# Regret-Matching All-Negative Fallback

**Last verified:** 2026-05-07, commit `ad0be89`

Source: `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`

## Question

When all action regrets at a traverser node are non-positive (which happens
frequently in early training before the advantage network has learned anything
useful), what policy should regret matching produce? The two candidate fallbacks
are `uniform` (equal probability across legal actions) and `argmax_tiebreak`
(break the all-tied-at-zero case by selecting the highest-index legal action,
effectively a near-deterministic choice). Which is correct, and does it matter
for the over-opening pathology seen in early training?

Short answer: **`uniform` is the theoretically safe default** and remains the
code default; `argmax_tiebreak` reduces fallback frequency and its side-effects
on over-opening, but the audit dataset (20 iterations) is too short to declare
it better overall.

## Code reference

The fallback is controlled by
`regret_matching.all_negative_fallback` in config
(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`) and implemented in
`src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`. The traversal
records per-iteration fallback counts and action composition through a suite of
`traversal_regret_fallback_*` metrics written to `metrics.jsonl`.

## What the audit found

At iteration 20 of an otherwise identical 20-iteration paired run:

| metric | `uniform` | `argmax_tiebreak` |
|---|---:|---:|
| fallback rate | **46.7%** | 15.4% |
| open-new selections during fallback | 641 | 164 |
| open-new selection rate during fallback | 3.49% | 2.16% |
| avg opened colors before fallback action | 4.43 | 4.61 |
| eval vs. Random: avg opened colors | 2.48 | 2.16 |
| eval vs. Random: 5-color open count | 49 | 35 |
| eval vs. Random: avg score diff | +42.5 | +33.6 |

`uniform` fires as the fallback in nearly half of all traversal regret-matching
decisions in this early-training window. During those fallback decisions, open-new
expedition actions are selected at nearly the base rate of their availability —
which is meaningfully elevated relative to an informed policy, because opening
new expeditions is usually risky in Lost Cities.

`argmax_tiebreak` reduces fallback frequency by two-thirds. The open-new
selections during fallback drop to roughly a quarter of the `uniform` count, and
eval games show lower 5-color open counts against both Random and Safe Heuristic
opponents.

## Why uniform can cause over-opening

When the advantage network output is all non-positive, `uniform` assigns equal
probability to every legal action. In Lost Cities, early in a hand, a large
fraction of legal actions are "open a new expedition." Uniform over legal actions
therefore assigns material probability mass to opening new expeditions even when
all trained regrets say "do not do this" (or say nothing, which uniform
interprets as equal preference). This early-game opening bias can propagate into
the strategy network through strategy memory samples collected during traversal.

`argmax_tiebreak` avoids that bias by collapsing the all-negative case to a
near-deterministic choice (highest legal action index), which is arbitrary but
not systematically biased toward opening.

## What the audit does not settle

The 20-iteration window is a diagnostic, not a conclusion. Two important
questions remain open:

1. **Does argmax_tiebreak help past iteration 20?** The reduction in 5-color
   openings at iteration 20 is real, but the score-diff comparison goes
   *against* `argmax_tiebreak` (+42.5 vs. +33.6 vs. Random). This suggests the
   two runs have not yet differentiated in any stable way, and the
   near-deterministic argmax choice may introduce its own early-iteration bias
   (favoring a specific action regardless of game state). A 50–100 iteration
   paired run is the stated next step before changing the default.

2. **Is over-opening caused by the fallback at all?** The fallback affects early
   iterations heavily, but other mechanisms — trajectory truncation, weak
   opponent policy, poor encoding — can also produce the same symptom. The audit
   establishes that `uniform` fallback *contributes* to open-new selections
   during traversal; it does not prove it is the primary driver of the
   over-opening plateau.

## Practical implication

- The code default (`uniform`) is safe and does not bias the algorithm in a
  theoretically incorrect direction. Regret matching is invariant to adding
  constants, so uniform over all legal actions is a valid no-information policy.
- `argmax_tiebreak` is a heuristic correction that may reduce early-game noise.
  It was used as the fallback in the `opponent_policy: network` experiments
  documented in `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md`.
- Do not switch the default to `argmax_tiebreak` based solely on the 20-iteration
  audit snapshot. Run a longer paired experiment first.
- The `traversal_regret_fallback_*` metrics are available in `metrics.jsonl` and
  provide the fine-grained action-composition data needed to evaluate any future
  change.