coolguyandClaude Opus 4.7
edad3b47da
Five notes covering outcome-sampling target correctness, package architecture, v0 feature-parity vs legacy, opponent-policy network divergence, and regret-matching fallback audit. Four are derived from archive sources (cited via Source: lines); outcome-sampling-target is a fresh write-up and serves as the style template. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
103 lines
5.0 KiB
Markdown
103 lines
5.0 KiB
Markdown
# Regret-Matching All-Negative Fallback
|
||
|
||
**Last verified:** 2026-05-07, commit `ad0be89`
|
||
|
||
Source: `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`
|
||
|
||
## Question
|
||
|
||
When all action regrets at a traverser node are non-positive (which happens
|
||
frequently in early training before the advantage network has learned anything
|
||
useful), what policy should regret matching produce? The two candidate fallbacks
|
||
are `uniform` (equal probability across legal actions) and `argmax_tiebreak`
|
||
(break the all-tied-at-zero case by selecting the highest-index legal action,
|
||
effectively a near-deterministic choice). Which is correct, and does it matter
|
||
for the over-opening pathology seen in early training?
|
||
|
||
Short answer: **`uniform` is the theoretically safe default** and remains the
|
||
code default; `argmax_tiebreak` reduces fallback frequency and its side-effects
|
||
on over-opening, but the audit dataset (20 iterations) is too short to declare
|
||
it better overall.
|
||
|
||
## Code reference
|
||
|
||
The fallback is controlled by
|
||
`regret_matching.all_negative_fallback` in config
|
||
(`src/coolrl_lost_cities/games/classic/deep_cfr/config.py`) and implemented in
|
||
`src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`. The traversal
|
||
records per-iteration fallback counts and action composition through a suite of
|
||
`traversal_regret_fallback_*` metrics written to `metrics.jsonl`.
|
||
|
||
## What the audit found
|
||
|
||
At iteration 20 of an otherwise identical 20-iteration paired run:
|
||
|
||
| metric | `uniform` | `argmax_tiebreak` |
|
||
|---|---:|---:|
|
||
| fallback rate | **46.7%** | 15.4% |
|
||
| open-new selections during fallback | 641 | 164 |
|
||
| open-new selection rate during fallback | 3.49% | 2.16% |
|
||
| avg opened colors before fallback action | 4.43 | 4.61 |
|
||
| eval vs. Random: avg opened colors | 2.48 | 2.16 |
|
||
| eval vs. Random: 5-color open count | 49 | 35 |
|
||
| eval vs. Random: avg score diff | +42.5 | +33.6 |
|
||
|
||
`uniform` fires as the fallback in nearly half of all traversal regret-matching
|
||
decisions in this early-training window. During those fallback decisions, open-new
|
||
expedition actions are selected at nearly the base rate of their availability —
|
||
which is meaningfully elevated relative to an informed policy, because opening
|
||
new expeditions is usually risky in Lost Cities.
|
||
|
||
`argmax_tiebreak` reduces fallback frequency by two-thirds. The open-new
|
||
selections during fallback drop to roughly a quarter of the `uniform` count, and
|
||
eval games show lower 5-color open counts against both Random and Safe Heuristic
|
||
opponents.
|
||
|
||
## Why uniform can cause over-opening
|
||
|
||
When the advantage network output is all non-positive, `uniform` assigns equal
|
||
probability to every legal action. In Lost Cities, early in a hand, a large
|
||
fraction of legal actions are "open a new expedition." Uniform over legal actions
|
||
therefore assigns material probability mass to opening new expeditions even when
|
||
all trained regrets say "do not do this" (or say nothing, which uniform
|
||
interprets as equal preference). This early-game opening bias can propagate into
|
||
the strategy network through strategy memory samples collected during traversal.
|
||
|
||
`argmax_tiebreak` avoids that bias by collapsing the all-negative case to a
|
||
near-deterministic choice (highest legal action index), which is arbitrary but
|
||
not systematically biased toward opening.
|
||
|
||
## What the audit does not settle
|
||
|
||
The 20-iteration window is a diagnostic, not a conclusion. Two important
|
||
questions remain open:
|
||
|
||
1. **Does argmax_tiebreak help past iteration 20?** The reduction in 5-color
|
||
openings at iteration 20 is real, but the score-diff comparison goes
|
||
*against* `argmax_tiebreak` (+42.5 vs. +33.6 vs. Random). This suggests the
|
||
two runs have not yet differentiated in any stable way, and the
|
||
near-deterministic argmax choice may introduce its own early-iteration bias
|
||
(favoring a specific action regardless of game state). A 50–100 iteration
|
||
paired run is the stated next step before changing the default.
|
||
|
||
2. **Is over-opening caused by the fallback at all?** The fallback affects early
|
||
iterations heavily, but other mechanisms — trajectory truncation, weak
|
||
opponent policy, poor encoding — can also produce the same symptom. The audit
|
||
establishes that `uniform` fallback *contributes* to open-new selections
|
||
during traversal; it does not prove it is the primary driver of the
|
||
over-opening plateau.
|
||
|
||
## Practical implication
|
||
|
||
- The code default (`uniform`) is safe and does not bias the algorithm in a
|
||
theoretically incorrect direction. Regret matching is invariant to adding
|
||
constants, so uniform over all legal actions is a valid no-information policy.
|
||
- `argmax_tiebreak` is a heuristic correction that may reduce early-game noise.
|
||
It was used as the fallback in the `opponent_policy: network` experiments
|
||
documented in `docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md`.
|
||
- Do not switch the default to `argmax_tiebreak` based solely on the 20-iteration
|
||
audit snapshot. Run a longer paired experiment first.
|
||
- The `traversal_regret_fallback_*` metrics are available in `metrics.jsonl` and
|
||
provide the fine-grained action-composition data needed to evaluate any future
|
||
change.
|