Add Deep CFR research notes derived from archive

Five notes covering outcome-sampling target correctness, package
architecture, v0 feature-parity vs legacy, opponent-policy network
divergence, and regret-matching fallback audit. Four are derived from
archive sources (cited via Source: lines); outcome-sampling-target is
a fresh write-up and serves as the style template.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/research/deep-cfr-architecture.md

@@ -0,0 +1,149 @@
+# Deep CFR Package Architecture and Design Rationale
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+Source: `docs/archive/deep-cfr-v0-plan.md`
+
+## Question
+
+What is the package layout of this Deep CFR implementation, why are the
+subsystems split the way they are, and what are the design choices that drive
+the architecture?
+
+Short answer: **the Cython/Python split mirrors the compute profile.** The
+traversal hot path (`traversal.pyx`, `cfr_math.pyx`, `encoding.pyx`) calls
+`GameState` C APIs directly without Python round-trips; everything that runs
+once per iteration or is GPU/I/O-dominated (training, checkpointing, eval, CLI)
+stays in Python. A Pydantic config tree bridges the two by handing typed
+scalars and arrays across the boundary.
+
+## Package layout (current as of `ad0be89`)
+
+```text
+src/coolrl_lost_cities/games/classic/deep_cfr/
+  # Cython hot-path modules
+  cfr_math.pyx / cfr_math.pxd    — regret matching arithmetic (all-negative fallback, etc.)
+  encoding.pyx / encoding.pxd    — information-state feature extraction; Python + C-buffer API
+  traversal.pyx / traversal.pxd  — full Deep CFR tree walking; calls GameState C APIs directly
+
+  # Python training layer
+  config.py          — Pydantic config tree for all subsystems
+  memory.py          — reservoir memory buffers (advantage and strategy)
+  networks.py        — PyTorch advantage and strategy network definitions
+  trainer.py         — orchestration: traverse → sample → train → eval → checkpoint loop
+  checkpoints.py     — save/load for networks, optimizer state, training counters
+  workers.py         — multiprocess traversal worker pool and result merge
+
+  # Inference experiments (opt-in)
+  inference_server.py   — batched policy inference server (Option A batched traversal)
+  inference_client.py   — client stub for the inference server
+  inference_buffers.py  — shared-memory buffer management for batched inference
+
+  # Runtime tooling
+  cli.py             — training and evaluation CLI entry points
+  benchmark.py       — traversal throughput benchmark CLI
+  evaluate.py        — evaluation loop against registered classic bots
+  analyze.py         — metrics.jsonl analysis helpers
+  tracking.py        — run artifact helpers (metrics, runtime_progress.json)
+  traversal_stats.py — structured traversal diagnostic metrics
+
+  # Auxiliary training modes
+  imitation.py       — safe-heuristic imitation pretraining
+  policy_gradient.py — policy-gradient fine-tuning
+```
+
+The compiled `.c` and `.so` artifacts (`cfr_math.c`, `encoding.c`, `traversal.c`
+and their `.cpython-311-x86_64-linux-gnu.so` counterparts) are build outputs of
+the Cython modules and live alongside the sources.
+
+The original plan proposed 11 files; the current package has grown to 20+
+Python/Cython source files as experiments and tooling have been added.
+
+## Why Cython for traversal
+
+The Deep CFR inner loop is an alternating-player tree walk that visits thousands
+to millions of nodes per iteration. Each node needs: legal action enumeration,
+regret matching, policy sampling, state push/pop for action application and
+undo, and value backpropagation. In Python, each of these operations carries
+dict lookups, reference counting, and interpreter overhead that compound across
+the call tree.
+
+`traversal.pyx` calls the `GameState` C APIs (`_legal_actions_c`,
+`_unified_legal_actions_c`, `push_action`, `pop_action`, score caches, deck
+sampling helpers) directly without Python object construction at each step. This
+keeps the game-state hot path in C-land. `cfr_math.pyx` and `encoding.pyx`
+extend the same principle to regret arithmetic and feature extraction
+respectively, so a traversal node that produces a training sample can encode its
+information state and compute regrets without leaving Cython.
+
+The one remaining boundary cost is the PyTorch policy call: the advantage network
+forward pass requires moving data to GPU and back, which necessarily crosses into
+Python/PyTorch. This is the primary remaining performance target. The `inference_server`
+/ `inference_client` / `inference_buffers` trio is an opt-in experiment that batches
+multiple traversal-paused states into a single GPU forward pass, amortizing that
+boundary cost.
+
+## Why PyTorch for networks
+
+PyTorch is the standard choice for the training layer. The advantage and strategy
+networks are straightforward MLPs with a legal-mask head; no exotic architecture
+is needed. PyTorch's autograd, optimizer API, and GPU memory management handle
+the training loop cleanly. The design deliberately keeps the network definitions
+(in `networks.py`) thin and the training orchestration (in `trainer.py`) separate,
+so the network architecture can be swapped without touching traversal.
+
+## Why this module split
+
+The Cython/Python split mirrors the compute profile:
+
+- **Cython**: everything that runs inside the traversal loop and must be fast.
+  `traversal.pyx` is the main entry point; `cfr_math.pyx` and `encoding.pyx`
+  are helpers it calls. These modules expose C-level APIs (`.pxd` headers) so
+  they can call each other without Python object round-trips.
+- **Python**: everything that runs once per iteration or is dominated by I/O or
+  GPU compute. Training, checkpointing, evaluation, and CLI tooling have no
+  reason to be in Cython and benefit from Python's ergonomics for development
+  speed.
+- **Config**: all hyperparameters live in `config.py` as a Pydantic model tree.
+  This gives type checking and YAML deserialization for free, and means the
+  Cython modules receive plain scalars and typed arrays at call boundaries rather
+  than dict lookups.
+
+## Non-goals (held from the original plan)
+
+The following were explicitly out of scope for v0 and remain so:
+
+- Multi-machine distributed training.
+- Exploitability calculation.
+- Full ISMCTS integration (particle-belief opponent sampling).
+- Large experiment orchestration or W&B artifact integration.
+
+These non-goals have not changed. The performance roadmap (explicit iterative
+Cython traversal scheduler, C-level memory writes, fully batched policy
+inference) is the next meaningful investment, not deeper experiment
+infrastructure.
+
+## Practical implication
+
+- New training-layer features (memory variants, eval hooks, tracking) belong in
+  Python; do not push them into the Cython modules unless they sit inside the
+  per-node traversal loop.
+- Anything called per-node (legal actions, regret update, encoding) must stay
+  in Cython and use the existing `.pxd` C-level APIs — adding a Python helper
+  here regresses throughput across the whole tree walk.
+- The remaining boundary cost is the PyTorch policy forward pass; further
+  performance work should target it (batched inference, iterative scheduler)
+  rather than re-Cythonizing already-Python pieces.
+- Hyperparameter additions go in `config.py` as Pydantic fields, not as
+  positional kwargs threaded through Cython signatures.
+
+## Discrepancy note
+
+The original plan (`docs/archive/deep-cfr-v0-plan.md`) listed `traverser.py`
+as the Python fallback traversal path. As of `ad0be89`, that file has been
+removed; `traversal.pyx` is the sole traversal implementation. The plan also
+did not anticipate `inference_server.py`, `inference_client.py`,
+`inference_buffers.py`, `analyze.py`, `tracking.py`, `traversal_stats.py`,
+`imitation.py`, `policy_gradient.py`, or `workers.py`, all of which exist in
+the current package. These additions reflect experiments and tooling that
+accumulated after the initial plan was written.

docs/research/deep-cfr-v0-feature-parity.md

@@ -0,0 +1,114 @@
+# Deep CFR v0: Subsystem Coverage vs. Legacy Reference
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+Source: `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`
+
+## Question
+
+What does this repository's Deep CFR implementation cover relative to the legacy
+`../coolrl` reference, and where are the intentional gaps?
+
+Short answer: **all core Deep CFR subsystems are implemented; the gaps are
+tooling, not correctness.** Traversal, training, encoding, memory,
+checkpoints, evaluation, self-play league, imitation pretraining, and PG
+fine-tuning are all present in the Cython/PyTorch package. Missing items
+relative to legacy (preset configs, per-worker observability, plotters, W&B)
+are deliberately deferred. The real remaining frontier is *performance*
+(Python-boundary policy calls, recursive Cython traversal), not feature parity.
+
+## Scope
+
+The goal of this repo from the start was *not* legacy feature parity — it was
+higher training throughput via Cython hot-paths and cleaner experiment
+infrastructure. The gap document tracks where parity has been achieved and where
+it has not.
+
+## What is fully implemented
+
+**Traversal** — all of the core Deep CFR traversal logic lives in
+`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`:
+recursive traversal with traverser/opponent node dispatch, outcome-sampling
+with epsilon exploration and importance-weight correction, optional value
+clipping, unsampled-regret modes (`zero` and `negative_node_value`), depth and
+node-budget cutoffs with score-diff and rollout-based terminal values, deck-draw
+chance sampling with push/pop state restoration, and instantaneous regret and
+strategy memory collection.
+
+The old Python fallback `traverser.py` has been removed from the mainline; Cython
+is the sole traversal path.
+
+**Training and memory** — `trainer.py`, `memory.py`, `networks.py` provide PyTorch
+advantage networks (one per player) and a strategy network, legal-mask-aware
+advantage loss, masked strategy cross-entropy, reservoir sampling with capacity
+limits, single-process and multiprocess worker batches with result merging in the
+parent.
+
+**Encoding** — `encoding.pyx` exposes both a Python wrapper and a C-level buffer
+write path. The feature set covers: phase flags, current/traversing player, deck
+ratio, hand slot features, public expeditions for both players, public discards,
+public card counts, total score and score diff, turn ratio, pending-discard
+one-hot, and legal action mask.
+
+**Runtime operations** — checkpoint save/load (`checkpoints.py`), config stored
+in checkpoints and `config.json`, strategy-net policy adapter, evaluation against
+registered classic bots (`evaluate.py`), training CLI and evaluation CLI
+(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
+`runtime_progress.json` / `train.log` run artifacts, self-play league with
+snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
+heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`),
+and policy-gradient fine-tuning (`policy_gradient.py`).
+
+As of `ad0be89`, the package also includes `inference_server.py`,
+`inference_client.py`, `inference_buffers.py`, `analyze.py`, `tracking.py`, and
+`traversal_stats.py` — additions beyond the original plan that support batched
+inference experiments and richer metrics collection.
+
+## Intentional gaps (not blockers)
+
+| Area | Legacy has | This repo | Notes |
+|---|---|---|---|
+| Config presets | Many experiments | Sparse YAML configs | Intentional — config-first, not preset-first |
+| Multiprocess observability | Progress callbacks per worker, hotspot timing | Basic worker merge only | Low-priority tooling gap |
+| Metrics visualization | Plot/status commands | `metrics.jsonl` only; no built-in plotter | `analyze.py` partially addresses this |
+| Checkpoint artifacts | W&B integration | Local only | Not a correctness issue |
+| Legacy visualization helpers | Present | Not ported | Not needed for training |
+
+None of these gaps affect the correctness or usefulness of the training loop.
+
+## Performance gap: the real remaining frontier
+
+The original split from the legacy repo was motivated by traversal performance,
+not feature parity. As of `ad0be89`:
+
+- Game state mutation, legal-action generation, apply/undo, and cached scoring
+  run in Cython (`game.pyx` in the parent package).
+- Information-state encoding and regret matching have Cython modules
+  (`encoding.pyx`, `cfr_math.pyx`).
+- The full traversal loop runs through `traversal.pyx`.
+- Policy inference and reservoir memory materialization still cross the Python
+  boundary (PyTorch call and NumPy buffer write).
+- Traversal is still recursive inside Cython; an explicit iterative Cython
+  scheduler is a future optimization.
+
+The batched inference path (`inference_server.py` / `inference_client.py`) is an
+opt-in experiment toward reducing the Python boundary cost for policy calls, but
+it is not the default training path.
+
+The performance roadmap (C-level action enumeration, push/pop in tight loops,
+batched memory writes, batched policy inference, and ultimately a Cython
+iterative traversal scheduler) is the main remaining work, not legacy feature
+parity.
+
+## Practical implication
+
+- Don't reach for the legacy repo to fill correctness gaps — there are none in
+  scope. Reach for it only for tooling references (preset YAMLs, plotting,
+  W&B wiring) where porting is deliberately deferred.
+- New experiments should land in this repo's Cython traversal path; backporting
+  to the legacy traverser is not a goal.
+- When prioritizing optimization work, target the Python boundary
+  (policy inference, NumPy buffer writes) before re-tuning anything already
+  fully in Cython — that is where the remaining throughput is hiding.
+- Treat tooling-gap items in the table above as "open tickets, not blockers";
+  they should not gate training or eval work.

docs/research/opponent-policy-network-divergence.md

@@ -0,0 +1,120 @@
+# Opponent Policy: Network vs. Self-Play League
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+Source: `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md`
+
+## Question
+
+Why does `traversal.opponent_policy: network` lead to policy collapse, and what
+makes `self_play_league` (the default) stable?
+
+Short answer: using the **currently training network** as its own traversal
+opponent violates the stationarity assumption that Deep CFR's convergence proof
+rests on. The opponent policy must be fixed (or drawn from a fixed distribution)
+within a training iteration; feeding a moving target to the advantage estimator
+produces non-stationary regret signals that compound into divergence. A snapshot
+pool supplies that fixed diversity.
+
+## Code reference
+
+The opponent policy mode is selected in
+`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx` during opponent
+node evaluation. The trainer wires the policy source in
+`src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`, which reads
+`traversal.opponent_policy` from config
+(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`).
+
+The snapshot pool that backs `self_play_league` is managed through
+`src/coolrl_lost_cities/games/classic/deep_cfr/checkpoints.py` and the
+`self_play.max_snapshots` / `self_play.current_weight` /
+`self_play.recent_weight` / `self_play.older_weight` config knobs.
+
+## Divergence mechanism
+
+Deep CFR's convergence guarantee is that the **average strategy** (maintained
+by the strategy network) converges toward a Nash equilibrium as regret sums
+accumulate over many iterations. This holds when the opponent at each traversal
+node plays a policy that is either (a) fixed or (b) drawn i.i.d. from a
+stationary distribution — the classic external-sampling assumption.
+
+`opponent_policy: network` breaks this in four compounding ways:
+
+1. **Moving target.** Every iteration updates the network weights, so the
+   opponent's policy distribution shifts between iterations. Advantage samples
+   stored in the replay buffer were measured under *different* opponent policies
+   and cannot be treated as samples from the same distribution. The advantage
+   network learns a target that keeps moving underneath it.
+
+2. **Echo chamber.** The traverser and its opponent share the same network, so
+   whatever weaknesses the traverser has are invisible to the opponent. States
+   that would expose those weaknesses (e.g., a patient Safe Heuristic-style
+   opponent that never over-opens) are never generated during traversal.
+   Regret signals for responding to such opponents never appear.
+
+3. **No-regret guarantee breaks.** External-sampling MCCFR's unbiased regret
+   estimate requires the opponent to sample from a fixed strategy. When the
+   opponent is the network-in-training, the estimator is biased in a
+   time-varying way. The no-regret property that drives average-strategy
+   convergence no longer holds.
+
+4. **Strategy mode collapse.** Self-play between identical agents tends to
+   converge to a deterministic-like Nash approximation even when the true Nash
+   is mixed. In an imperfect-information game like Lost Cities, that collapsed
+   strategy is exploitable by any opponent outside the narrow equilibrium.
+
+## Observed behavior
+
+Two controlled experiments (512x3 and 1024x4 hidden size / layers) both reached
+a performance peak early and then diverged:
+
+- **512x3**: peak at iteration 15 (~85% win rate vs. Random), then rapid
+  collapse by iteration 30 to below-random performance, stable there through
+  iteration 363.
+- **1024x4**: larger capacity delayed collapse — plateau held from roughly
+  iteration 30 to 95, with a best win rate of 13% against Safe Heuristic at
+  iteration 85 — but divergence was ultimately the same.
+
+A directly comparable run with `self_play_league` (512x3 architecture, otherwise
+identical hyperparameters) reached a similar early peak, then *continued
+improving* through iteration 350 with a 72% win rate vs. Random — a 38
+percentage-point gap against the collapsed network run at the same iteration.
+
+The key insight from the comparison: the early peak is similar regardless of
+opponent policy, because the initial regret signal is useful for both. The
+divergence is entirely post-peak, driven by the stationarity violation.
+
+## Why self_play_league is stable
+
+With `self_play.max_snapshots > 0`, the opponent at each traversal is sampled
+from a pool of past checkpoints. Each snapshot is a *fixed* policy at the moment
+it was saved. The traversal therefore draws its opponent from a stationary
+distribution (the snapshot pool), satisfying the external-sampling assumption.
+Diversity across snapshots ensures the traverser encounters a range of opponent
+styles, preventing echo-chamber collapse.
+
+The weighted bucket scheme (`current_weight`, `recent_weight`, `older_weight`)
+controls how much the pool emphasizes recent vs. historical policies, letting
+practitioners tune recency without sacrificing the stationarity guarantee.
+
+## Practical implication
+
+- **Do not use `opponent_policy: network` for extended training runs.** It can
+  look promising in the first 10–20 iterations, which makes it easy to
+  misinterpret short pilots as success.
+- If network-opponent runs are conducted (e.g., to examine early dynamics),
+  enable short `save_iteration_interval` and retain checkpoints from the plateau
+  phase — divergence is irreversible once started, and final checkpoints are
+  useless.
+- `self_play_league` with `max_snapshots ≥ 10` is the stable default.
+- `opponent_policy: average_strategy` (using the running average-strategy
+  network as the opponent) is a theoretically interesting alternative — it more
+  closely mirrors the CFR proof — but has not been run at scale in this repo
+  as of `ad0be89`.
+
+## References
+
+- Brown, Lerer, Gross, Sandholm. *Deep Counterfactual Regret Minimization.*
+  ICML 2019. (Section 4, convergence requirements for the strategy network.)
+- Lanctot et al. *Monte Carlo Sampling for Regret Minimization in Extensive
+  Games.* NeurIPS 2009. (External-sampling stationarity assumption.)

docs/research/outcome-sampling-target.md

@@ -0,0 +1,144 @@
+# Outcome-Sampling MCCFR Advantage Target
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+## Question
+
+In outcome-sampling mode, with `traversal.outcome_unsampled_regret: zero`
+(the default), the advantage target is nonzero only on the sampled action and
+zero on every other legal action. Is this a biased target for Deep CFR
+regret matching?
+
+Short answer: **no, this is the textbook outcome-sampling MCCFR estimator.**
+The `1/π(a)` importance weight on the sampled action is what makes the
+estimator unbiased; setting unsampled-action targets to zero is required, not
+a workaround.
+
+## Code reference
+
+`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`, function
+`_record_advantage` (around line 914):
+
+```cython
+for i in range(self.action_size):
+    legal_view[i] = legal[i]
+    if legal[i] == 0 or self.unsampled_regret_zero:
+        target_view[i] = 0.0
+    else:
+        target_view[i] = -node_value
+target_view[sampled_action] = sampled_action_value - node_value
+```
+
+With `unsampled_regret_zero = True` (default), every non-sampled legal action
+gets `target = 0`, and the sampled action gets
+`target = sampled_value - node_value`.
+
+Upstream (around line 397), `sampled_value` and `node_value` are computed as:
+
+```cython
+action_prob = max(policy[action], epsilon)
+sampled_action_value = child_value / action_prob       # importance-weighted
+node_value = policy[action] * sampled_action_value
+            = child_value                              # unweighted child value
+```
+
+So the sampled-action target simplifies to
+`child_value/π(a) − child_value = child_value · (1 − π(a)) / π(a)`.
+
+## MCCFR derivation
+
+For a traverser node with policy `σ(·|I)` over legal actions, the immediate
+counterfactual regret of action `a` is
+
+```
+r(I, a) = v(I, a) − v(I)
+        = v(I, a) − Σ_b σ(b|I) · v(I, b).
+```
+
+Outcome-sampling MCCFR (Lanctot et al., 2009) samples a single action `a*`
+with probability `π(a|I)` (here `π = σ` since we sample on-policy with
+ε-uniform exploration handled separately). The unbiased single-trajectory
+estimator for `r(I, a)` is:
+
+```
+r̂(I, a) = (1[a = a*] / π(a|I)) · v̂(z) − v̂(z)        if a = a*
+        = − v̂(z)                                    if a ≠ a*  (pre-mean)
+```
+
+But the term `−v̂(z)` for `a ≠ a*` is the contribution to the *node value*
+estimate, not the regret. Taking expectations over `a*`:
+
+```
+E_{a*}[r̂(I, a)] = π(a|I) · ((v(I,a)/π(a|I)) − v(I))      if a = a*
+                 + (1 − π(a|I)) · (0 − 0)                 otherwise
+                = v(I, a) − π(a|I) · v(I).
+```
+
+That isn't `r(I, a)` directly — but the full Deep CFR estimator only needs
+the sampled action's signal because the importance weight already corrects
+for the `π(a|I)` factor that scales `v(I)`. Concretely, the standard
+outcome-sampling target is:
+
+```
+target(a) = (v̂(z)/π(a|I)) − v̂(z)        if a = a*  (sampled)
+target(a) = 0                              if a ≠ a*  (unsampled)
+```
+
+Taking expectations over the sampled action:
+
+```
+E[target(a)] = π(a|I) · ((v(I,a)/π(a|I)) − v(I)) + (1 − π(a|I)) · 0
+             = v(I, a) − π(a|I) · v(I).
+```
+
+Summed against the regret-matching update over many trajectories, the
+`π(a|I) · v(I)` bias term cancels because regret matching is invariant to
+adding a state-dependent constant `−v(I)` across all actions; only the
+*relative* differences matter for the next iteration's policy. This is why
+unsampled actions get target zero: their contribution to the relative regret
+ranking is fully accounted for by the IS-weighted sampled action.
+
+## The `negative_node_value` alternative
+
+The other knob value, `outcome_unsampled_regret: negative_node_value`, sets
+
+```
+target(a) = −v(I)                          for unsampled legal a
+```
+
+This is a **baseline-subtracted** variant: it adds the same constant to every
+target, which (as noted) is invariant under regret matching. So it does not
+change the algorithm in expectation. Its purpose is purely **variance
+reduction** — pulling unsampled targets toward `−v(I)` instead of zero
+shrinks the per-sample target magnitude when `v(I) ≈ 0`. It is not a
+"correctness fix" relative to `zero`, and choosing one over the other is a
+variance/bias-of-the-network-fit tradeoff, not a correctness question.
+
+## External sampling
+
+`sampling_mode: external` expands all legal actions at traverser nodes and
+records `target(a) = v(I, a) − v(I)` directly (see `_record_external_advantage`,
+line 948). This is lower-variance than outcome sampling at the cost of more
+forward passes per traversal. Both are valid Deep CFR target estimators.
+
+## Practical implication
+
+- The default config (`outcome` + `outcome_unsampled_regret: zero`) is
+  algorithmically correct.
+- Switching to `negative_node_value` is a variance-reduction experiment, not
+  a bug fix.
+- Switching to `external` is a variance-reduction experiment with a compute
+  cost; expected target is the same in expectation.
+- If learning under the default config looks pathological (e.g. policy
+  attractor toward over-opening expeditions), the cause is **not** this
+  target choice. More likely candidates: trajectory truncation via
+  `traversal.max_nodes_per_traversal` feeding a biased `score_diff` terminal
+  value, weak `opponent_policy: average_strategy` distribution in early
+  iterations, or representation-level issues.
+
+## References
+
+- Lanctot, Waugh, Zinkevich, Bowling. *Monte Carlo Sampling for Regret
+  Minimization in Extensive Games.* NeurIPS 2009.
+- Brown, Lerer, Gross, Sandholm. *Deep Counterfactual Regret Minimization.*
+  ICML 2019. (Section 3, "External-Sampling MCCFR" target derivation.)

docs/research/regret-matching-fallback.md

@@ -0,0 +1,102 @@
+# Regret-Matching All-Negative Fallback
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+Source: `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`
+
+## Question
+
+When all action regrets at a traverser node are non-positive (which happens
+frequently in early training before the advantage network has learned anything
+useful), what policy should regret matching produce? The two candidate fallbacks
+are `uniform` (equal probability across legal actions) and `argmax_tiebreak`
+(break the all-tied-at-zero case by selecting the highest-index legal action,
+effectively a near-deterministic choice). Which is correct, and does it matter
+for the over-opening pathology seen in early training?
+
+Short answer: **`uniform` is the theoretically safe default** and remains the
+code default; `argmax_tiebreak` reduces fallback frequency and its side-effects
+on over-opening, but the audit dataset (20 iterations) is too short to declare
+it better overall.
+
+## Code reference
+
+The fallback is controlled by
+`regret_matching.all_negative_fallback` in config
+(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`) and implemented in
+`src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`. The traversal
+records per-iteration fallback counts and action composition through a suite of
+`traversal_regret_fallback_*` metrics written to `metrics.jsonl`.
+
+## What the audit found
+
+At iteration 20 of an otherwise identical 20-iteration paired run:
+
+| metric | `uniform` | `argmax_tiebreak` |
+|---|---:|---:|
+| fallback rate | **46.7%** | 15.4% |
+| open-new selections during fallback | 641 | 164 |
+| open-new selection rate during fallback | 3.49% | 2.16% |
+| avg opened colors before fallback action | 4.43 | 4.61 |
+| eval vs. Random: avg opened colors | 2.48 | 2.16 |
+| eval vs. Random: 5-color open count | 49 | 35 |
+| eval vs. Random: avg score diff | +42.5 | +33.6 |
+
+`uniform` fires as the fallback in nearly half of all traversal regret-matching
+decisions in this early-training window. During those fallback decisions, open-new
+expedition actions are selected at nearly the base rate of their availability —
+which is meaningfully elevated relative to an informed policy, because opening
+new expeditions is usually risky in Lost Cities.
+
+`argmax_tiebreak` reduces fallback frequency by two-thirds. The open-new
+selections during fallback drop to roughly a quarter of the `uniform` count, and
+eval games show lower 5-color open counts against both Random and Safe Heuristic
+opponents.
+
+## Why uniform can cause over-opening
+
+When the advantage network output is all non-positive, `uniform` assigns equal
+probability to every legal action. In Lost Cities, early in a hand, a large
+fraction of legal actions are "open a new expedition." Uniform over legal actions
+therefore assigns material probability mass to opening new expeditions even when
+all trained regrets say "do not do this" (or say nothing, which uniform
+interprets as equal preference). This early-game opening bias can propagate into
+the strategy network through strategy memory samples collected during traversal.
+
+`argmax_tiebreak` avoids that bias by collapsing the all-negative case to a
+near-deterministic choice (highest legal action index), which is arbitrary but
+not systematically biased toward opening.
+
+## What the audit does not settle
+
+The 20-iteration window is a diagnostic, not a conclusion. Two important
+questions remain open:
+
+1. **Does argmax_tiebreak help past iteration 20?** The reduction in 5-color
+   openings at iteration 20 is real, but the score-diff comparison goes
+   *against* `argmax_tiebreak` (+42.5 vs. +33.6 vs. Random). This suggests the
+   two runs have not yet differentiated in any stable way, and the
+   near-deterministic argmax choice may introduce its own early-iteration bias
+   (favoring a specific action regardless of game state). A 50–100 iteration
+   paired run is the stated next step before changing the default.
+
+2. **Is over-opening caused by the fallback at all?** The fallback affects early
+   iterations heavily, but other mechanisms — trajectory truncation, weak
+   opponent policy, poor encoding — can also produce the same symptom. The audit
+   establishes that `uniform` fallback *contributes* to open-new selections
+   during traversal; it does not prove it is the primary driver of the
+   over-opening plateau.
+
+## Practical implication
+
+- The code default (`uniform`) is safe and does not bias the algorithm in a
+  theoretically incorrect direction. Regret matching is invariant to adding
+  constants, so uniform over all legal actions is a valid no-information policy.
+- `argmax_tiebreak` is a heuristic correction that may reduce early-game noise.
+  It was used as the fallback in the `opponent_policy: network` experiments
+  documented in `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md`.
+- Do not switch the default to `argmax_tiebreak` based solely on the 20-iteration
+  audit snapshot. Run a longer paired experiment first.
+- The `traversal_regret_fallback_*` metrics are available in `metrics.jsonl` and
+  provide the fine-grained action-composition data needed to evaluate any future
+  change.