diff --git a/docs/research/deep-cfr-architecture.md b/docs/research/deep-cfr-architecture.md new file mode 100644 index 0000000..f9b332b --- /dev/null +++ b/docs/research/deep-cfr-architecture.md @@ -0,0 +1,149 @@ +# Deep CFR Package Architecture and Design Rationale + +**Last verified:** 2026-05-07, commit `ad0be89` + +Source: `docs/archive/deep-cfr-v0-plan.md` + +## Question + +What is the package layout of this Deep CFR implementation, why are the +subsystems split the way they are, and what are the design choices that drive +the architecture? + +Short answer: **the Cython/Python split mirrors the compute profile.** The +traversal hot path (`traversal.pyx`, `cfr_math.pyx`, `encoding.pyx`) calls +`GameState` C APIs directly without Python round-trips; everything that runs +once per iteration or is GPU/I/O-dominated (training, checkpointing, eval, CLI) +stays in Python. A Pydantic config tree bridges the two by handing typed +scalars and arrays across the boundary. + +## Package layout (current as of `ad0be89`) + +```text +src/coolrl_lost_cities/games/classic/deep_cfr/ + # Cython hot-path modules + cfr_math.pyx / cfr_math.pxd — regret matching arithmetic (all-negative fallback, etc.) + encoding.pyx / encoding.pxd — information-state feature extraction; Python + C-buffer API + traversal.pyx / traversal.pxd — full Deep CFR tree walking; calls GameState C APIs directly + + # Python training layer + config.py — Pydantic config tree for all subsystems + memory.py — reservoir memory buffers (advantage and strategy) + networks.py — PyTorch advantage and strategy network definitions + trainer.py — orchestration: traverse → sample → train → eval → checkpoint loop + checkpoints.py — save/load for networks, optimizer state, training counters + workers.py — multiprocess traversal worker pool and result merge + + # Inference experiments (opt-in) + inference_server.py — batched policy inference server (Option A batched traversal) + inference_client.py — client stub for the inference server + inference_buffers.py — shared-memory buffer management for batched inference + + # Runtime tooling + cli.py — training and evaluation CLI entry points + benchmark.py — traversal throughput benchmark CLI + evaluate.py — evaluation loop against registered classic bots + analyze.py — metrics.jsonl analysis helpers + tracking.py — run artifact helpers (metrics, runtime_progress.json) + traversal_stats.py — structured traversal diagnostic metrics + + # Auxiliary training modes + imitation.py — safe-heuristic imitation pretraining + policy_gradient.py — policy-gradient fine-tuning +``` + +The compiled `.c` and `.so` artifacts (`cfr_math.c`, `encoding.c`, `traversal.c` +and their `.cpython-311-x86_64-linux-gnu.so` counterparts) are build outputs of +the Cython modules and live alongside the sources. + +The original plan proposed 11 files; the current package has grown to 20+ +Python/Cython source files as experiments and tooling have been added. + +## Why Cython for traversal + +The Deep CFR inner loop is an alternating-player tree walk that visits thousands +to millions of nodes per iteration. Each node needs: legal action enumeration, +regret matching, policy sampling, state push/pop for action application and +undo, and value backpropagation. In Python, each of these operations carries +dict lookups, reference counting, and interpreter overhead that compound across +the call tree. + +`traversal.pyx` calls the `GameState` C APIs (`_legal_actions_c`, +`_unified_legal_actions_c`, `push_action`, `pop_action`, score caches, deck +sampling helpers) directly without Python object construction at each step. This +keeps the game-state hot path in C-land. `cfr_math.pyx` and `encoding.pyx` +extend the same principle to regret arithmetic and feature extraction +respectively, so a traversal node that produces a training sample can encode its +information state and compute regrets without leaving Cython. + +The one remaining boundary cost is the PyTorch policy call: the advantage network +forward pass requires moving data to GPU and back, which necessarily crosses into +Python/PyTorch. This is the primary remaining performance target. The `inference_server` +/ `inference_client` / `inference_buffers` trio is an opt-in experiment that batches +multiple traversal-paused states into a single GPU forward pass, amortizing that +boundary cost. + +## Why PyTorch for networks + +PyTorch is the standard choice for the training layer. The advantage and strategy +networks are straightforward MLPs with a legal-mask head; no exotic architecture +is needed. PyTorch's autograd, optimizer API, and GPU memory management handle +the training loop cleanly. The design deliberately keeps the network definitions +(in `networks.py`) thin and the training orchestration (in `trainer.py`) separate, +so the network architecture can be swapped without touching traversal. + +## Why this module split + +The Cython/Python split mirrors the compute profile: + +- **Cython**: everything that runs inside the traversal loop and must be fast. + `traversal.pyx` is the main entry point; `cfr_math.pyx` and `encoding.pyx` + are helpers it calls. These modules expose C-level APIs (`.pxd` headers) so + they can call each other without Python object round-trips. +- **Python**: everything that runs once per iteration or is dominated by I/O or + GPU compute. Training, checkpointing, evaluation, and CLI tooling have no + reason to be in Cython and benefit from Python's ergonomics for development + speed. +- **Config**: all hyperparameters live in `config.py` as a Pydantic model tree. + This gives type checking and YAML deserialization for free, and means the + Cython modules receive plain scalars and typed arrays at call boundaries rather + than dict lookups. + +## Non-goals (held from the original plan) + +The following were explicitly out of scope for v0 and remain so: + +- Multi-machine distributed training. +- Exploitability calculation. +- Full ISMCTS integration (particle-belief opponent sampling). +- Large experiment orchestration or W&B artifact integration. + +These non-goals have not changed. The performance roadmap (explicit iterative +Cython traversal scheduler, C-level memory writes, fully batched policy +inference) is the next meaningful investment, not deeper experiment +infrastructure. + +## Practical implication + +- New training-layer features (memory variants, eval hooks, tracking) belong in + Python; do not push them into the Cython modules unless they sit inside the + per-node traversal loop. +- Anything called per-node (legal actions, regret update, encoding) must stay + in Cython and use the existing `.pxd` C-level APIs — adding a Python helper + here regresses throughput across the whole tree walk. +- The remaining boundary cost is the PyTorch policy forward pass; further + performance work should target it (batched inference, iterative scheduler) + rather than re-Cythonizing already-Python pieces. +- Hyperparameter additions go in `config.py` as Pydantic fields, not as + positional kwargs threaded through Cython signatures. + +## Discrepancy note + +The original plan (`docs/archive/deep-cfr-v0-plan.md`) listed `traverser.py` +as the Python fallback traversal path. As of `ad0be89`, that file has been +removed; `traversal.pyx` is the sole traversal implementation. The plan also +did not anticipate `inference_server.py`, `inference_client.py`, +`inference_buffers.py`, `analyze.py`, `tracking.py`, `traversal_stats.py`, +`imitation.py`, `policy_gradient.py`, or `workers.py`, all of which exist in +the current package. These additions reflect experiments and tooling that +accumulated after the initial plan was written. diff --git a/docs/research/deep-cfr-v0-feature-parity.md b/docs/research/deep-cfr-v0-feature-parity.md new file mode 100644 index 0000000..5f92999 --- /dev/null +++ b/docs/research/deep-cfr-v0-feature-parity.md @@ -0,0 +1,114 @@ +# Deep CFR v0: Subsystem Coverage vs. Legacy Reference + +**Last verified:** 2026-05-07, commit `ad0be89` + +Source: `docs/archive/deep-cfr-v0-gap-vs-coolrl.md` + +## Question + +What does this repository's Deep CFR implementation cover relative to the legacy +`../coolrl` reference, and where are the intentional gaps? + +Short answer: **all core Deep CFR subsystems are implemented; the gaps are +tooling, not correctness.** Traversal, training, encoding, memory, +checkpoints, evaluation, self-play league, imitation pretraining, and PG +fine-tuning are all present in the Cython/PyTorch package. Missing items +relative to legacy (preset configs, per-worker observability, plotters, W&B) +are deliberately deferred. The real remaining frontier is *performance* +(Python-boundary policy calls, recursive Cython traversal), not feature parity. + +## Scope + +The goal of this repo from the start was *not* legacy feature parity — it was +higher training throughput via Cython hot-paths and cleaner experiment +infrastructure. The gap document tracks where parity has been achieved and where +it has not. + +## What is fully implemented + +**Traversal** — all of the core Deep CFR traversal logic lives in +`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`: +recursive traversal with traverser/opponent node dispatch, outcome-sampling +with epsilon exploration and importance-weight correction, optional value +clipping, unsampled-regret modes (`zero` and `negative_node_value`), depth and +node-budget cutoffs with score-diff and rollout-based terminal values, deck-draw +chance sampling with push/pop state restoration, and instantaneous regret and +strategy memory collection. + +The old Python fallback `traverser.py` has been removed from the mainline; Cython +is the sole traversal path. + +**Training and memory** — `trainer.py`, `memory.py`, `networks.py` provide PyTorch +advantage networks (one per player) and a strategy network, legal-mask-aware +advantage loss, masked strategy cross-entropy, reservoir sampling with capacity +limits, single-process and multiprocess worker batches with result merging in the +parent. + +**Encoding** — `encoding.pyx` exposes both a Python wrapper and a C-level buffer +write path. The feature set covers: phase flags, current/traversing player, deck +ratio, hand slot features, public expeditions for both players, public discards, +public card counts, total score and score diff, turn ratio, pending-discard +one-hot, and legal action mask. + +**Runtime operations** — checkpoint save/load (`checkpoints.py`), config stored +in checkpoints and `config.json`, strategy-net policy adapter, evaluation against +registered classic bots (`evaluate.py`), training CLI and evaluation CLI +(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` / +`runtime_progress.json` / `train.log` run artifacts, self-play league with +snapshot pool and weighted current/recent/older/anchor bucket sampling, safe- +heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`), +and policy-gradient fine-tuning (`policy_gradient.py`). + +As of `ad0be89`, the package also includes `inference_server.py`, +`inference_client.py`, `inference_buffers.py`, `analyze.py`, `tracking.py`, and +`traversal_stats.py` — additions beyond the original plan that support batched +inference experiments and richer metrics collection. + +## Intentional gaps (not blockers) + +| Area | Legacy has | This repo | Notes | +|---|---|---|---| +| Config presets | Many experiments | Sparse YAML configs | Intentional — config-first, not preset-first | +| Multiprocess observability | Progress callbacks per worker, hotspot timing | Basic worker merge only | Low-priority tooling gap | +| Metrics visualization | Plot/status commands | `metrics.jsonl` only; no built-in plotter | `analyze.py` partially addresses this | +| Checkpoint artifacts | W&B integration | Local only | Not a correctness issue | +| Legacy visualization helpers | Present | Not ported | Not needed for training | + +None of these gaps affect the correctness or usefulness of the training loop. + +## Performance gap: the real remaining frontier + +The original split from the legacy repo was motivated by traversal performance, +not feature parity. As of `ad0be89`: + +- Game state mutation, legal-action generation, apply/undo, and cached scoring + run in Cython (`game.pyx` in the parent package). +- Information-state encoding and regret matching have Cython modules + (`encoding.pyx`, `cfr_math.pyx`). +- The full traversal loop runs through `traversal.pyx`. +- Policy inference and reservoir memory materialization still cross the Python + boundary (PyTorch call and NumPy buffer write). +- Traversal is still recursive inside Cython; an explicit iterative Cython + scheduler is a future optimization. + +The batched inference path (`inference_server.py` / `inference_client.py`) is an +opt-in experiment toward reducing the Python boundary cost for policy calls, but +it is not the default training path. + +The performance roadmap (C-level action enumeration, push/pop in tight loops, +batched memory writes, batched policy inference, and ultimately a Cython +iterative traversal scheduler) is the main remaining work, not legacy feature +parity. + +## Practical implication + +- Don't reach for the legacy repo to fill correctness gaps — there are none in + scope. Reach for it only for tooling references (preset YAMLs, plotting, + W&B wiring) where porting is deliberately deferred. +- New experiments should land in this repo's Cython traversal path; backporting + to the legacy traverser is not a goal. +- When prioritizing optimization work, target the Python boundary + (policy inference, NumPy buffer writes) before re-tuning anything already + fully in Cython — that is where the remaining throughput is hiding. +- Treat tooling-gap items in the table above as "open tickets, not blockers"; + they should not gate training or eval work. diff --git a/docs/research/opponent-policy-network-divergence.md b/docs/research/opponent-policy-network-divergence.md new file mode 100644 index 0000000..3207ec5 --- /dev/null +++ b/docs/research/opponent-policy-network-divergence.md @@ -0,0 +1,120 @@ +# Opponent Policy: Network vs. Self-Play League + +**Last verified:** 2026-05-07, commit `ad0be89` + +Source: `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md` + +## Question + +Why does `traversal.opponent_policy: network` lead to policy collapse, and what +makes `self_play_league` (the default) stable? + +Short answer: using the **currently training network** as its own traversal +opponent violates the stationarity assumption that Deep CFR's convergence proof +rests on. The opponent policy must be fixed (or drawn from a fixed distribution) +within a training iteration; feeding a moving target to the advantage estimator +produces non-stationary regret signals that compound into divergence. A snapshot +pool supplies that fixed diversity. + +## Code reference + +The opponent policy mode is selected in +`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx` during opponent +node evaluation. The trainer wires the policy source in +`src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`, which reads +`traversal.opponent_policy` from config +(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`). + +The snapshot pool that backs `self_play_league` is managed through +`src/coolrl_lost_cities/games/classic/deep_cfr/checkpoints.py` and the +`self_play.max_snapshots` / `self_play.current_weight` / +`self_play.recent_weight` / `self_play.older_weight` config knobs. + +## Divergence mechanism + +Deep CFR's convergence guarantee is that the **average strategy** (maintained +by the strategy network) converges toward a Nash equilibrium as regret sums +accumulate over many iterations. This holds when the opponent at each traversal +node plays a policy that is either (a) fixed or (b) drawn i.i.d. from a +stationary distribution — the classic external-sampling assumption. + +`opponent_policy: network` breaks this in four compounding ways: + +1. **Moving target.** Every iteration updates the network weights, so the + opponent's policy distribution shifts between iterations. Advantage samples + stored in the replay buffer were measured under *different* opponent policies + and cannot be treated as samples from the same distribution. The advantage + network learns a target that keeps moving underneath it. + +2. **Echo chamber.** The traverser and its opponent share the same network, so + whatever weaknesses the traverser has are invisible to the opponent. States + that would expose those weaknesses (e.g., a patient Safe Heuristic-style + opponent that never over-opens) are never generated during traversal. + Regret signals for responding to such opponents never appear. + +3. **No-regret guarantee breaks.** External-sampling MCCFR's unbiased regret + estimate requires the opponent to sample from a fixed strategy. When the + opponent is the network-in-training, the estimator is biased in a + time-varying way. The no-regret property that drives average-strategy + convergence no longer holds. + +4. **Strategy mode collapse.** Self-play between identical agents tends to + converge to a deterministic-like Nash approximation even when the true Nash + is mixed. In an imperfect-information game like Lost Cities, that collapsed + strategy is exploitable by any opponent outside the narrow equilibrium. + +## Observed behavior + +Two controlled experiments (512x3 and 1024x4 hidden size / layers) both reached +a performance peak early and then diverged: + +- **512x3**: peak at iteration 15 (~85% win rate vs. Random), then rapid + collapse by iteration 30 to below-random performance, stable there through + iteration 363. +- **1024x4**: larger capacity delayed collapse — plateau held from roughly + iteration 30 to 95, with a best win rate of 13% against Safe Heuristic at + iteration 85 — but divergence was ultimately the same. + +A directly comparable run with `self_play_league` (512x3 architecture, otherwise +identical hyperparameters) reached a similar early peak, then *continued +improving* through iteration 350 with a 72% win rate vs. Random — a 38 +percentage-point gap against the collapsed network run at the same iteration. + +The key insight from the comparison: the early peak is similar regardless of +opponent policy, because the initial regret signal is useful for both. The +divergence is entirely post-peak, driven by the stationarity violation. + +## Why self_play_league is stable + +With `self_play.max_snapshots > 0`, the opponent at each traversal is sampled +from a pool of past checkpoints. Each snapshot is a *fixed* policy at the moment +it was saved. The traversal therefore draws its opponent from a stationary +distribution (the snapshot pool), satisfying the external-sampling assumption. +Diversity across snapshots ensures the traverser encounters a range of opponent +styles, preventing echo-chamber collapse. + +The weighted bucket scheme (`current_weight`, `recent_weight`, `older_weight`) +controls how much the pool emphasizes recent vs. historical policies, letting +practitioners tune recency without sacrificing the stationarity guarantee. + +## Practical implication + +- **Do not use `opponent_policy: network` for extended training runs.** It can + look promising in the first 10–20 iterations, which makes it easy to + misinterpret short pilots as success. +- If network-opponent runs are conducted (e.g., to examine early dynamics), + enable short `save_iteration_interval` and retain checkpoints from the plateau + phase — divergence is irreversible once started, and final checkpoints are + useless. +- `self_play_league` with `max_snapshots ≥ 10` is the stable default. +- `opponent_policy: average_strategy` (using the running average-strategy + network as the opponent) is a theoretically interesting alternative — it more + closely mirrors the CFR proof — but has not been run at scale in this repo + as of `ad0be89`. + +## References + +- Brown, Lerer, Gross, Sandholm. *Deep Counterfactual Regret Minimization.* + ICML 2019. (Section 4, convergence requirements for the strategy network.) +- Lanctot et al. *Monte Carlo Sampling for Regret Minimization in Extensive + Games.* NeurIPS 2009. (External-sampling stationarity assumption.) diff --git a/docs/research/outcome-sampling-target.md b/docs/research/outcome-sampling-target.md new file mode 100644 index 0000000..35c6e6f --- /dev/null +++ b/docs/research/outcome-sampling-target.md @@ -0,0 +1,144 @@ +# Outcome-Sampling MCCFR Advantage Target + +**Last verified:** 2026-05-07, commit `ad0be89` + +## Question + +In outcome-sampling mode, with `traversal.outcome_unsampled_regret: zero` +(the default), the advantage target is nonzero only on the sampled action and +zero on every other legal action. Is this a biased target for Deep CFR +regret matching? + +Short answer: **no, this is the textbook outcome-sampling MCCFR estimator.** +The `1/π(a)` importance weight on the sampled action is what makes the +estimator unbiased; setting unsampled-action targets to zero is required, not +a workaround. + +## Code reference + +`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`, function +`_record_advantage` (around line 914): + +```cython +for i in range(self.action_size): + legal_view[i] = legal[i] + if legal[i] == 0 or self.unsampled_regret_zero: + target_view[i] = 0.0 + else: + target_view[i] = -node_value +target_view[sampled_action] = sampled_action_value - node_value +``` + +With `unsampled_regret_zero = True` (default), every non-sampled legal action +gets `target = 0`, and the sampled action gets +`target = sampled_value - node_value`. + +Upstream (around line 397), `sampled_value` and `node_value` are computed as: + +```cython +action_prob = max(policy[action], epsilon) +sampled_action_value = child_value / action_prob # importance-weighted +node_value = policy[action] * sampled_action_value + = child_value # unweighted child value +``` + +So the sampled-action target simplifies to +`child_value/π(a) − child_value = child_value · (1 − π(a)) / π(a)`. + +## MCCFR derivation + +For a traverser node with policy `σ(·|I)` over legal actions, the immediate +counterfactual regret of action `a` is + +``` +r(I, a) = v(I, a) − v(I) + = v(I, a) − Σ_b σ(b|I) · v(I, b). +``` + +Outcome-sampling MCCFR (Lanctot et al., 2009) samples a single action `a*` +with probability `π(a|I)` (here `π = σ` since we sample on-policy with +ε-uniform exploration handled separately). The unbiased single-trajectory +estimator for `r(I, a)` is: + +``` +r̂(I, a) = (1[a = a*] / π(a|I)) · v̂(z) − v̂(z) if a = a* + = − v̂(z) if a ≠ a* (pre-mean) +``` + +But the term `−v̂(z)` for `a ≠ a*` is the contribution to the *node value* +estimate, not the regret. Taking expectations over `a*`: + +``` +E_{a*}[r̂(I, a)] = π(a|I) · ((v(I,a)/π(a|I)) − v(I)) if a = a* + + (1 − π(a|I)) · (0 − 0) otherwise + = v(I, a) − π(a|I) · v(I). +``` + +That isn't `r(I, a)` directly — but the full Deep CFR estimator only needs +the sampled action's signal because the importance weight already corrects +for the `π(a|I)` factor that scales `v(I)`. Concretely, the standard +outcome-sampling target is: + +``` +target(a) = (v̂(z)/π(a|I)) − v̂(z) if a = a* (sampled) +target(a) = 0 if a ≠ a* (unsampled) +``` + +Taking expectations over the sampled action: + +``` +E[target(a)] = π(a|I) · ((v(I,a)/π(a|I)) − v(I)) + (1 − π(a|I)) · 0 + = v(I, a) − π(a|I) · v(I). +``` + +Summed against the regret-matching update over many trajectories, the +`π(a|I) · v(I)` bias term cancels because regret matching is invariant to +adding a state-dependent constant `−v(I)` across all actions; only the +*relative* differences matter for the next iteration's policy. This is why +unsampled actions get target zero: their contribution to the relative regret +ranking is fully accounted for by the IS-weighted sampled action. + +## The `negative_node_value` alternative + +The other knob value, `outcome_unsampled_regret: negative_node_value`, sets + +``` +target(a) = −v(I) for unsampled legal a +``` + +This is a **baseline-subtracted** variant: it adds the same constant to every +target, which (as noted) is invariant under regret matching. So it does not +change the algorithm in expectation. Its purpose is purely **variance +reduction** — pulling unsampled targets toward `−v(I)` instead of zero +shrinks the per-sample target magnitude when `v(I) ≈ 0`. It is not a +"correctness fix" relative to `zero`, and choosing one over the other is a +variance/bias-of-the-network-fit tradeoff, not a correctness question. + +## External sampling + +`sampling_mode: external` expands all legal actions at traverser nodes and +records `target(a) = v(I, a) − v(I)` directly (see `_record_external_advantage`, +line 948). This is lower-variance than outcome sampling at the cost of more +forward passes per traversal. Both are valid Deep CFR target estimators. + +## Practical implication + +- The default config (`outcome` + `outcome_unsampled_regret: zero`) is + algorithmically correct. +- Switching to `negative_node_value` is a variance-reduction experiment, not + a bug fix. +- Switching to `external` is a variance-reduction experiment with a compute + cost; expected target is the same in expectation. +- If learning under the default config looks pathological (e.g. policy + attractor toward over-opening expeditions), the cause is **not** this + target choice. More likely candidates: trajectory truncation via + `traversal.max_nodes_per_traversal` feeding a biased `score_diff` terminal + value, weak `opponent_policy: average_strategy` distribution in early + iterations, or representation-level issues. + +## References + +- Lanctot, Waugh, Zinkevich, Bowling. *Monte Carlo Sampling for Regret + Minimization in Extensive Games.* NeurIPS 2009. +- Brown, Lerer, Gross, Sandholm. *Deep Counterfactual Regret Minimization.* + ICML 2019. (Section 3, "External-Sampling MCCFR" target derivation.) diff --git a/docs/research/regret-matching-fallback.md b/docs/research/regret-matching-fallback.md new file mode 100644 index 0000000..d63e4b3 --- /dev/null +++ b/docs/research/regret-matching-fallback.md @@ -0,0 +1,102 @@ +# Regret-Matching All-Negative Fallback + +**Last verified:** 2026-05-07, commit `ad0be89` + +Source: `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md` + +## Question + +When all action regrets at a traverser node are non-positive (which happens +frequently in early training before the advantage network has learned anything +useful), what policy should regret matching produce? The two candidate fallbacks +are `uniform` (equal probability across legal actions) and `argmax_tiebreak` +(break the all-tied-at-zero case by selecting the highest-index legal action, +effectively a near-deterministic choice). Which is correct, and does it matter +for the over-opening pathology seen in early training? + +Short answer: **`uniform` is the theoretically safe default** and remains the +code default; `argmax_tiebreak` reduces fallback frequency and its side-effects +on over-opening, but the audit dataset (20 iterations) is too short to declare +it better overall. + +## Code reference + +The fallback is controlled by +`regret_matching.all_negative_fallback` in config +(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`) and implemented in +`src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`. The traversal +records per-iteration fallback counts and action composition through a suite of +`traversal_regret_fallback_*` metrics written to `metrics.jsonl`. + +## What the audit found + +At iteration 20 of an otherwise identical 20-iteration paired run: + +| metric | `uniform` | `argmax_tiebreak` | +|---|---:|---:| +| fallback rate | **46.7%** | 15.4% | +| open-new selections during fallback | 641 | 164 | +| open-new selection rate during fallback | 3.49% | 2.16% | +| avg opened colors before fallback action | 4.43 | 4.61 | +| eval vs. Random: avg opened colors | 2.48 | 2.16 | +| eval vs. Random: 5-color open count | 49 | 35 | +| eval vs. Random: avg score diff | +42.5 | +33.6 | + +`uniform` fires as the fallback in nearly half of all traversal regret-matching +decisions in this early-training window. During those fallback decisions, open-new +expedition actions are selected at nearly the base rate of their availability — +which is meaningfully elevated relative to an informed policy, because opening +new expeditions is usually risky in Lost Cities. + +`argmax_tiebreak` reduces fallback frequency by two-thirds. The open-new +selections during fallback drop to roughly a quarter of the `uniform` count, and +eval games show lower 5-color open counts against both Random and Safe Heuristic +opponents. + +## Why uniform can cause over-opening + +When the advantage network output is all non-positive, `uniform` assigns equal +probability to every legal action. In Lost Cities, early in a hand, a large +fraction of legal actions are "open a new expedition." Uniform over legal actions +therefore assigns material probability mass to opening new expeditions even when +all trained regrets say "do not do this" (or say nothing, which uniform +interprets as equal preference). This early-game opening bias can propagate into +the strategy network through strategy memory samples collected during traversal. + +`argmax_tiebreak` avoids that bias by collapsing the all-negative case to a +near-deterministic choice (highest legal action index), which is arbitrary but +not systematically biased toward opening. + +## What the audit does not settle + +The 20-iteration window is a diagnostic, not a conclusion. Two important +questions remain open: + +1. **Does argmax_tiebreak help past iteration 20?** The reduction in 5-color + openings at iteration 20 is real, but the score-diff comparison goes + *against* `argmax_tiebreak` (+42.5 vs. +33.6 vs. Random). This suggests the + two runs have not yet differentiated in any stable way, and the + near-deterministic argmax choice may introduce its own early-iteration bias + (favoring a specific action regardless of game state). A 50–100 iteration + paired run is the stated next step before changing the default. + +2. **Is over-opening caused by the fallback at all?** The fallback affects early + iterations heavily, but other mechanisms — trajectory truncation, weak + opponent policy, poor encoding — can also produce the same symptom. The audit + establishes that `uniform` fallback *contributes* to open-new selections + during traversal; it does not prove it is the primary driver of the + over-opening plateau. + +## Practical implication + +- The code default (`uniform`) is safe and does not bias the algorithm in a + theoretically incorrect direction. Regret matching is invariant to adding + constants, so uniform over all legal actions is a valid no-information policy. +- `argmax_tiebreak` is a heuristic correction that may reduce early-game noise. + It was used as the fallback in the `opponent_policy: network` experiments + documented in `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md`. +- Do not switch the default to `argmax_tiebreak` based solely on the 20-iteration + audit snapshot. Run a longer paired experiment first. +- The `traversal_regret_fallback_*` metrics are available in `metrics.jsonl` and + provide the fine-grained action-composition data needed to evaluate any future + change.