coolguyandClaude Opus 4.7
f63c4b8059
Audit of the Deep CFR information-state encoding identified two tiers of
non-pure features and removed both:
- Tier 3 (judgment): is_bad_open_candidate, open_risk_score,
is_safe_continuation. Same heuristic family used to label bad_open in
evaluation, embedded as model input.
- Tier 2 (projection): recoverable_score_no_bonus,
recoverable_margin_no_bonus, min_needed_to_break_even,
cards_needed_for_bonus, has_bonus_path. Mechanical but assumption-laden
("commit and play all currently-playable cards"). The no_bonus form is
asymmetric: it amplifies the immediate -20 penalty while truncating the
+20 bonus upside, biasing the model toward the same "don't open" basin
the diagnostics already flagged.
Input dim 365 -> 297. DERIVED_PLAYABILITY_PER_COLOR 19 -> 15;
SLOT_AWARE_PLAYABILITY_PER_SLOT 12 -> 6. Test shape assertions updated.
Also adds selectivity diagnostic infrastructure used to reach this point:
- traversal.outcome_unsampled_first_open_prior_alpha config field with
signed-prior overlay on unsampled first-open advantage targets (A1).
- analyze_first_open_counterfactual.py --post-policy to swap the
policy_player rollout policy and isolate selection bias (D1).
- analyze_first_open_followup.py to inspect post-forced-open behavior
(E2): same-color play vs discard counts, other-open rate, terminal
hand composition.
Findings recorded in docs/plans/deep-cfr-selectivity.md sections 3-6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
598 lines
26 KiB
Markdown
598 lines
26 KiB
Markdown
# Deep CFR Selectivity Investigation
|
||
|
||
Last updated: 2026-05-10
|
||
|
||
## Current conclusion
|
||
|
||
The current Deep CFR baseline is not mainly blocked by model size or LCFR
|
||
time-weighting. The strongest signal so far is a selectivity failure: the model
|
||
does not reliably learn to distinguish good first opens from bad first opens.
|
||
|
||
The `outcome_sampling_epsilon=0.05` setting produced a short-run improvement
|
||
around 200 iterations, but the improvement did not hold through 500 iterations.
|
||
Weighting ablations did not recover the 200-iteration peak, so further LCFR
|
||
alpha tuning is lower priority than directly inspecting or changing the
|
||
first-open advantage target.
|
||
|
||
## Baseline symptoms
|
||
|
||
The 512x3 dense-eval baseline showed improving training losses, but the main
|
||
game-quality metrics against `safe_heuristic_strict` did not improve enough to
|
||
indicate a useful policy.
|
||
|
||
Observed pattern:
|
||
|
||
- Advantage loss can move in the expected direction while eval quality remains
|
||
poor.
|
||
- Bad first-open behavior stays high.
|
||
- Open quality, measured by score per opened color, stays negative.
|
||
- Current-policy evaluation did not rescue the result, so the issue is not only
|
||
average-policy lag.
|
||
|
||
## Closed hypotheses
|
||
|
||
### More outcome sampling helps, but only up to a point
|
||
|
||
Short 200-iteration ablations with `outcome_unsampled_regret=zero`:
|
||
|
||
| setting | safe strict score diff | win rate | bad open rate | score/opened color |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| `epsilon=0.20` | -57.87 | 0.07 | 0.901 | -8.40 |
|
||
| `epsilon=0.10` | -58.66 | 0.05 | 0.913 | -8.63 |
|
||
| `epsilon=0.05` | -40.01 | 0.12 | 0.893 | -6.25 |
|
||
| `epsilon=0.02` | -53.55 | - | - | - |
|
||
|
||
Conclusion: `epsilon=0.05` was the best short-run candidate. Lowering to
|
||
`0.02` was worse, and larger values were also worse.
|
||
|
||
### Negative unsampled regret was not sufficient
|
||
|
||
The `epsilon=0.05` plus `outcome_unsampled_regret=negative_node_value` variant
|
||
ended around safe strict score diff `-44.53` at 200 iterations, worse than
|
||
`epsilon=0.05` with `zero`.
|
||
|
||
Conclusion: the negative target suppresses opening, but it does not selectively
|
||
preserve good opens.
|
||
|
||
### The 200-iteration epsilon=0.05 peak did not hold
|
||
|
||
Confirmation run:
|
||
|
||
- Run: `runs/2026-05-09_010747_confirm-eps-005-zero-512x3-det-500`
|
||
- W&B group: `eps005-confirmation-512x3-v1`
|
||
- Config delta:
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=500`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
|
||
Safe heuristic strict metrics:
|
||
|
||
| iteration | win rate | score diff | bad open rate | score/opened color |
|
||
| ---: | ---: | ---: | ---: | ---: |
|
||
| 200 | 0.12 | -40.01 | 0.893 | -6.25 |
|
||
| 250 | 0.08 | -46.91 | 0.900 | -7.29 |
|
||
| 300 | 0.08 | -45.93 | 0.895 | -6.30 |
|
||
| 350 | 0.07 | -54.02 | 0.899 | -7.83 |
|
||
| 400 | 0.04 | -49.64 | 0.912 | -7.83 |
|
||
| 450 | 0.06 | -49.31 | 0.917 | -6.71 |
|
||
| 500 | 0.06 | -61.24 | 0.903 | -8.30 |
|
||
|
||
Conclusion: `epsilon=0.05` creates a real short-run peak, but the behavior is
|
||
not stable through 500 iterations.
|
||
|
||
### LCFR time-weighting is not the sole cause of the degradation
|
||
|
||
Weighting ablation group:
|
||
|
||
- W&B group: `eps005-weighting-ablation-512x3-v1`
|
||
- Common config:
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=300`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
|
||
Compared runs:
|
||
|
||
| run | iter 200 diff | iter 250 diff | iter 300 diff | iter 300 win | iter 300 bad open |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| LCFR alpha=1.0 confirmation | -40.01 | -46.91 | -45.93 | 0.08 | 0.895 |
|
||
| `training_weighting.mode=none` | -50.98 | -48.46 | -56.27 | 0.04 | 0.912 |
|
||
| `training_weighting.lcfr_alpha=0.5` | -66.81 | -58.97 | -50.77 | 0.10 | 0.891 |
|
||
|
||
Open-quality comparison:
|
||
|
||
| run | iter 200 score/opened color | iter 300 score/opened color |
|
||
| --- | ---: | ---: |
|
||
| LCFR alpha=1.0 confirmation | -6.25 | -6.30 |
|
||
| `training_weighting.mode=none` | -7.71 | -7.70 |
|
||
| `training_weighting.lcfr_alpha=0.5` | -9.94 | -7.85 |
|
||
|
||
Conclusion: neither removing time-weighting nor softening LCFR to `alpha=0.5`
|
||
beat the original `alpha=1.0` confirmation by the primary score-diff metric at
|
||
300 iterations. LCFR may affect stability, but it is not the main lever.
|
||
|
||
## First-open diagnostic
|
||
|
||
Diagnostic output:
|
||
|
||
- `runs/tmp/first_open_advantage_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Observed values:
|
||
|
||
| checkpoint | bad selection | good selection | bad advantage | good advantage | sampled turns |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | 0.0625 | 0.0582 | -12.82 | -16.60 | 8560 |
|
||
| iter 500 | 0.0067 | 0.0067 | -35.42 | -44.32 | 40942 |
|
||
|
||
Interpretation: by 500 iterations the model strongly suppresses opening
|
||
overall. It suppresses good opens along with bad opens, which is the core
|
||
selectivity failure.
|
||
|
||
## First-open target audit
|
||
|
||
Script:
|
||
|
||
- `scripts/analyze_first_open_targets.py`
|
||
|
||
Output:
|
||
|
||
- `runs/tmp/first_open_target_audit_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Method: regenerate short interleaved traversal batches from existing
|
||
checkpoints and bucket first-open advantage targets by action quality. Because
|
||
`outcome_unsampled_regret=zero` sets unsampled legal actions to zero, the most
|
||
informative statistic is the sampled-action target distribution, not the full
|
||
legal-candidate target distribution.
|
||
|
||
Sampled target summary:
|
||
|
||
| checkpoint | bucket | candidates | policy prob | sampled rate | sampled target mean | sampled target positive |
|
||
| --- | --- | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | good open | 796 | 0.078 | 0.078 | -40.71 | 0.419 |
|
||
| iter 200 | bad open | 7142 | 0.081 | 0.081 | -25.65 | 0.424 |
|
||
| iter 500 | good open | 800 | 0.064 | 0.058 | 8.39 | 0.435 |
|
||
| iter 500 | bad open | 8322 | 0.069 | 0.071 | 12.61 | 0.433 |
|
||
|
||
Interpretation: the regenerated traversal targets do not rank good opens above
|
||
bad opens. At both inspected checkpoints, bad-open candidates receive slightly
|
||
higher policy probability and sampled rate than good-open candidates. The
|
||
sampled target mean is also better for bad opens than good opens. This points
|
||
to a target or metric-alignment problem before model capacity or LCFR tuning.
|
||
|
||
## First-open counterfactual audit
|
||
|
||
Script:
|
||
|
||
- `scripts/analyze_first_open_counterfactual.py`
|
||
|
||
Output:
|
||
|
||
- `runs/tmp/first_open_counterfactual_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Method: collect first-open candidate states from existing checkpoints, force
|
||
each first-open candidate once, and compare the resulting continuation value
|
||
against the current policy's best non-open action from the same state.
|
||
|
||
`delta_open = value(force open) - value(best non-open)`.
|
||
|
||
Counterfactual summary against `safe_heuristic_strict`:
|
||
|
||
| checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate |
|
||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | good open | 40 | -26.57 | -26.5 | 0.050 | 0.000 | 0.000 |
|
||
| iter 200 | bad open | 460 | -23.15 | -21.0 | 0.130 | 0.036 | 0.037 |
|
||
| iter 500 | good open | 30 | -23.43 | -17.5 | 0.167 | 0.067 | 0.067 |
|
||
| iter 500 | bad open | 470 | -11.18 | -8.0 | 0.226 | 0.020 | 0.019 |
|
||
|
||
Interpretation: the heuristic `open_bad` label is not entirely misaligned with
|
||
continuation value. Forced bad opens are usually worse than the best non-open
|
||
alternative. However, heuristic `open_good` also often loses to best non-open in
|
||
these sampled states, so "recoverable eventually" is not the same as "open now."
|
||
|
||
Combined with the target audit, this points toward target/objective alignment:
|
||
the traversal target is not making the bad-open-vs-non-open mistake clearly
|
||
negative, even when the counterfactual continuation usually is negative.
|
||
|
||
## Open questions
|
||
|
||
1. Does the traversal target itself provide separable labels for good first
|
||
opens versus bad first opens?
|
||
2. Is the model receiving too sparse or too noisy a signal at the first-open
|
||
decision point?
|
||
3. Would target shaping around first-open decisions improve score/opened color
|
||
without increasing bad-open rate?
|
||
4. Is evaluation showing a policy-selection problem, or is the advantage model
|
||
already misranking good and bad opens before strategy extraction?
|
||
|
||
## Recommended next experiments
|
||
|
||
### 1. Deeper first-open target audit
|
||
|
||
Before another long training run, inspect sampled first-open decision records
|
||
more directly:
|
||
|
||
- Group candidate first-open actions into good-open and bad-open buckets.
|
||
- Compare target values before model prediction, not only final learned
|
||
advantages.
|
||
- Report distributions, not just means.
|
||
- Check whether the target ranks good opens above bad opens in the same
|
||
information-state context family.
|
||
|
||
Success criterion: the target distribution should show a usable separation
|
||
between good and bad opens. If it does not, the training target is the blocker.
|
||
|
||
### 2. First-open replay reweighting
|
||
|
||
Implemented option:
|
||
|
||
- `optimization.advantage_first_open_fraction`
|
||
|
||
Meaning: reserve this fraction of each advantage minibatch for samples where a
|
||
first-open action was legally available at the traverser decision. The target is
|
||
not changed; only replay sampling frequency changes. This keeps the experiment
|
||
on the pure self-play side more than hand-written target shaping.
|
||
|
||
Initial planned run:
|
||
|
||
- `run.experiment_name=first-open-reweight-50-512x3-det-500`
|
||
- `optimization.advantage_first_open_fraction=0.5`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=500`
|
||
|
||
Primary comparison is the `eps=0.05` confirmation run and the pure external
|
||
sampling run. Success requires bad-open rate and score/opened color to improve
|
||
together without a large score-diff regression.
|
||
|
||
Result:
|
||
|
||
- Run: `runs/2026-05-09_235808_first-open-reweight-50-512x3-det-500-indexed`
|
||
- W&B group: `first-open-replay-v1`
|
||
- Commit: `8217cec` (`Speed up first-open memory sampling`)
|
||
|
||
The scan-based first implementation was stopped after 12 iterations because
|
||
first-open sampling scanned the full replay memory and pushed iteration time
|
||
above 60 seconds. The indexed-memory version kept first-open sampling near
|
||
0.25 seconds per player at 4M advantage samples and completed 500 iterations.
|
||
|
||
Final `safe_heuristic_strict` comparison:
|
||
|
||
| Run | Iter | Score diff | Win rate | Bad open | Score/opened |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| baseline `confirm-eps-005-zero-512x3-det-500` | 500 | -61.24 | 0.06 | 0.903 | -8.30 |
|
||
| pure external `pure-external-512x3-det-500` | 500 | -50.38 | 0.06 | 0.925 | -9.79 |
|
||
| first-open reweight 50% indexed | 500 | -52.20 | 0.07 | 0.919 | -9.39 |
|
||
|
||
Conclusion: first-open replay reweighting alone did not solve selectivity. It
|
||
improved score diff versus the baseline final checkpoint, but it did not reduce
|
||
bad-open rate and made score/opened color worse. The run briefly looked better
|
||
around 120-200 iterations, then regressed by 300-500 iterations. This suggests
|
||
the replay emphasis is not enough if the underlying target does not separate
|
||
good first opens from bad first opens.
|
||
|
||
### 3. First-open target prior (A1) — running 2026-05-10
|
||
|
||
Hypothesis: target audit shows the regenerated traversal target does not
|
||
separate good first opens from bad first opens. With
|
||
`outcome_unsampled_regret=zero`, every unsampled first-open candidate is
|
||
labeled with target `0`, regardless of whether it is heuristically good
|
||
or bad. A small signed prior on unsampled first-open play actions should
|
||
break the symmetry without overriding the high-variance sampled-action
|
||
target.
|
||
|
||
Implementation:
|
||
|
||
- Config field: `traversal.outcome_unsampled_first_open_prior_alpha` (float,
|
||
default `0.0`).
|
||
- Plumbed through `InterleavedTraversalConfig` and
|
||
`run_interleaved_traversal_batch`. Cython traversal path unchanged
|
||
(default scheduler is interleaved; pyx path is a follow-up if needed).
|
||
- Effect site: `_after_child` in `interleaved_traversal.py`. When
|
||
`is_first_open` and `alpha != 0`, unsampled legal play actions whose color
|
||
has an empty expedition are overwritten in `target` with `+alpha` if the
|
||
visible `recoverable_score` for that color is `>= 0`, else `-alpha`. The
|
||
sampled action's cell stays as `sampled_action_value - node_value`.
|
||
- `recoverable_score` mirrors `evaluate._visible_recoverable_summary` for
|
||
the empty-expedition case (no future-card lookahead, hand-only signal).
|
||
- Tests: `test_first_open_prior_overrides_unsampled_play_targets_with_signed_alpha`,
|
||
`test_first_open_prior_zero_alpha_is_noop`.
|
||
|
||
Note on "pure self-play": this prior introduces a hand-written heuristic
|
||
into the advantage target. The opponent and rollout policy remain
|
||
self-play (no external bot). This is treated as a diagnostic experiment;
|
||
if effective, the permanent solution is a counterfactual-based prior
|
||
(A2) that recovers full pure self-play.
|
||
|
||
Initial planned run:
|
||
|
||
- `run.experiment_name=first-open-prior-alpha5-512x3-det-200`
|
||
- `traversal.outcome_unsampled_first_open_prior_alpha=5.0`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=200`
|
||
- W&B group: `first-open-prior-v1`
|
||
|
||
Primary comparisons:
|
||
|
||
- `confirm-eps-005-zero-512x3-det-500` (baseline; iter 200 reference)
|
||
- `first-open-reweight-50-512x3-det-500-indexed` (replay-side intervention)
|
||
|
||
Success criterion: `bad_open_rate` and `score_per_opened_color` improve
|
||
*together* at iter 200 without large regression in `avg_score_diff0`. If
|
||
positive, follow up with a 500-iter run to test stability. If null/regress,
|
||
try alpha sweep (e.g. 2.0, 10.0) before abandoning the direction.
|
||
|
||
Result (2026-05-10):
|
||
|
||
- Run: `runs/2026-05-10_072718_first-open-prior-alpha5-512x3-det-200`
|
||
- W&B: synced online to group `first-open-prior-v1` (run `teuh915r`)
|
||
- Commit: see HEAD at run start
|
||
|
||
`safe_heuristic_strict` at iter 200:
|
||
|
||
| Run | Score diff | Win rate | Bad open | Score/opened |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 |
|
||
| **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** |
|
||
|
||
Other opponents at iter 200: random +32.51 / 0.86, noisy_safe -71.52 / 0.07,
|
||
safe_heuristic -81.81 / 0.02, safe_heuristic_loose -81.24 / 0.05.
|
||
|
||
Conclusion: **mixed result, net regression.** The prior did shift behavior
|
||
in the intended direction on one axis — `bad_open_rate` dropped from 0.893
|
||
to 0.796 (~10% absolute reduction). This is the only metric where the
|
||
hypothesis "the prior breaks ranking symmetry" looks supported.
|
||
|
||
But the overall game-quality metrics regressed: score diff worsened
|
||
(-40 → -67), win rate collapsed (0.12 → 0.02), and `score_per_opened_color`
|
||
got worse (-6.25 → -8.85). The model fails the success criterion (which
|
||
required `bad_open_rate` AND `score_per_opened_color` to improve together).
|
||
|
||
Two interpretations:
|
||
|
||
1. **α = 5.0 too strong.** The prior is overpowering the sampled-action
|
||
target rather than acting as a weak symmetry-breaker. The good-open prior
|
||
pushes the model to open *more often*, but those forced opens are bad
|
||
*given the actual game state*, just labeled good by the visible-only
|
||
heuristic. The counterfactual audit already flagged this: heuristic
|
||
`open_good` candidates often lose to best non-open in real continuation.
|
||
2. **Heuristic itself misaligned.** Even the right α won't help if the
|
||
sign assigned to candidates is wrong relative to true game value.
|
||
|
||
Next steps (in order):
|
||
|
||
- (A1.b) α sweep at 200 iter: α ∈ {1.0, 2.0} to test "weaker prior" hypothesis.
|
||
If α=2.0 still regresses score diff while reducing bad_open, the prior shape
|
||
itself is wrong, not just strength.
|
||
- (A2) Counterfactual prior: replace heuristic sign with sign of
|
||
`value(force open) - value(best non-open)` from a small in-traversal
|
||
rollout. Cleaner signal, recovers pure self-play, but more expensive.
|
||
- If both fail to improve score diff, the issue is upstream of unsampled
|
||
regret labeling (likely traversal sample distribution or sampled-action
|
||
target variance), and the next experiment family should target those.
|
||
|
||
### 4. D1 diagnostic: counterfactual with strong post-policy (2026-05-10)
|
||
|
||
Question: are forced-open continuation values low because the openings
|
||
themselves are bad, or because the self-play rollout policy poisons the
|
||
post-action play (selection bias)?
|
||
|
||
Method: re-ran `analyze_first_open_counterfactual.py` on the
|
||
`confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter
|
||
500), but with `--post-policy safe_heuristic_strict`. The opponent and
|
||
state-collection policy stayed the same; only the policy_player's actions
|
||
*after* the forced first action used the strong fixed bot.
|
||
|
||
Output: `runs/tmp/first_open_counterfactual_d1_strong_post_policy_200_vs_500.jsonl`
|
||
|
||
`delta_open = value(force open) - value(best non-open)` comparison:
|
||
|
||
| iter | bucket | n | self-play post | strong post | shift |
|
||
| ---: | --- | ---: | ---: | ---: | ---: |
|
||
| 200 | open_good | 40 | -26.57 | **-3.35** | +23.22 |
|
||
| 200 | open_bad | 460 | -23.15 | **-5.52** | +17.63 |
|
||
| 500 | open_good | 30 | -23.43 | **-10.90** | +12.53 |
|
||
| 500 | open_bad | 470 | -11.18 | **-8.99** | +2.19 |
|
||
|
||
`delta_positive_rate`:
|
||
|
||
| iter | bucket | self-play post | strong post |
|
||
| ---: | --- | ---: | ---: |
|
||
| 200 | open_good | 0.050 | 0.225 |
|
||
| 200 | open_bad | 0.130 | 0.317 |
|
||
| 500 | open_good | 0.167 | 0.300 |
|
||
| 500 | open_bad | 0.226 | 0.230 |
|
||
|
||
Verdict — two findings, both important:
|
||
|
||
**1. Selection bias is real and significant.** Strong post-policy
|
||
improves forced-open continuation values by 12–23 points on average. The
|
||
self-play policy is meaningfully poisoning rollouts: forced opens look
|
||
much less bad once a competent player handles the followup.
|
||
|
||
**2. Heuristic labels do not separate cleanly even under strong post-policy.**
|
||
At iter 200, `open_good` delta_mean is only ~2 points better than
|
||
`open_bad` (-3.35 vs -5.52). At iter 500, ranking is essentially flat
|
||
(`open_good` -10.9 vs `open_bad` -8.99 — slightly *worse*). Median deltas
|
||
agree. Sample size for `open_good` is small (30–40) so noise contributes,
|
||
but there is no clean signal that the heuristic `recoverable_score`
|
||
classifier matches actual continuation value.
|
||
|
||
Implications:
|
||
|
||
- A1 prior was destined to fail — the heuristic sign is at best weakly
|
||
aligned with continuation value, even with optimistic post-policy.
|
||
- A2 with self-play rollouts would inherit selection bias and likely
|
||
reproduce the same misranking. A2 with a strong post-policy would
|
||
give clean signs but breaks pure self-play.
|
||
- The deeper bottleneck is post-open play quality. Until the trained
|
||
policy plays competently *after* opening, training signals about
|
||
*whether to open* will be biased toward "don't open."
|
||
|
||
Candidate next directions (decision pending):
|
||
|
||
- (E1) Train with `cutoff_rollout_policy=safe_heuristic` instead of `random`.
|
||
Already a config option; gives leaf nodes stronger value estimates
|
||
during traversal. Trades some pure-self-play purity for a stronger
|
||
bootstrap signal. Cheap to test.
|
||
- (E2) Investigate post-open behavior directly: forced-open + observe
|
||
next-2-3 turns. Diagnoses *why* the model can't follow up (e.g. always
|
||
discards followup cards, switches color, etc.).
|
||
- (E3) Curriculum / staged training that exposes the network to good
|
||
post-open trajectories before forcing it to make first-open decisions.
|
||
- A2 deferred unless we adopt a strong post-policy in rollouts.
|
||
|
||
### 5. E2 diagnostic: post-forced-open behavior (2026-05-10)
|
||
|
||
Question: D1 showed selection bias is real — *what is the model actually
|
||
doing after a forced first-open* that poisons rollouts?
|
||
|
||
Method: forced each first-open candidate at sampled states, then observed
|
||
the policy_player's next 3 decisions (window). Used baseline checkpoints
|
||
(`confirm-eps-005-zero` iter 200, iter 500). Categorized each followup
|
||
decision relative to the forced-open color.
|
||
|
||
Output: `runs/tmp/first_open_followup_baseline_200_vs_500.jsonl`
|
||
Script: `scripts/analyze_first_open_followup.py`
|
||
|
||
Per-window mean counts (3 policy_player decisions = ~1.5 game turns;
|
||
each game turn includes one play/discard plus one draw):
|
||
|
||
| iter | bucket | n | held@force | plays(same) | discards(same) | open_other | other_discard | held@end |
|
||
| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| 200 | good | 33 | 4.82 | 0.15 | 0.15 | **0.49** | 0.21 | 2.70 |
|
||
| 200 | bad | 367 | 1.97 | 0.03 | 0.12 | 0.10 | 0.75 | 1.08 |
|
||
| 500 | good | 22 | 4.45 | 0.14 | **0.46** | 0.14 | 0.27 | 1.23 |
|
||
| 500 | bad | 378 | 2.56 | 0.01 | 0.23 | 0.05 | 0.71 | 1.29 |
|
||
|
||
Findings:
|
||
|
||
**1. Model under-plays followup cards even when it holds many.** At iter
|
||
200, good-open candidates start with ~4.8 same-color cards in hand. In the
|
||
next 3 decisions, the model plays only 0.15 of them on average (3% of
|
||
its window). 2.7 cards remain in hand at terminal — nearly 3 cards of
|
||
the just-opened color never reach the expedition.
|
||
|
||
**2. Model opens additional new colors after a forced open.** At iter 200
|
||
good bucket, `open_other` = 0.49 in a 3-decision window — the model is
|
||
~3× more likely to open *another* new color than to follow up the one it
|
||
was forced into. This compounds the recoverable-score drag.
|
||
|
||
**3. By iter 500 the model actively dumps the forced color.** Good bucket
|
||
at iter 500: `same-color discards` 0.46 vs `same-color plays` 0.14. The
|
||
model discards followup cards more than 3× as often as it plays them,
|
||
despite holding 4.45 of them at force time. This is the strongest single
|
||
piece of evidence so far that the post-open value head is poisoned.
|
||
|
||
**4. Bad-open bucket shows even sharper avoidance.** Hand has ~2
|
||
same-color cards, plays effectively zero. Almost all play-phase decisions
|
||
are "discard other" (0.71–0.75). The model treats forced opens as bad
|
||
news to liquidate rather than commit to.
|
||
|
||
Interpretation: the trained advantage network *learned* that opens of
|
||
these colors are net-negative, so once forced into one, it hedges by
|
||
opening other colors and dumping followup cards. That hedging is rational
|
||
under the policy's own value estimates but is the exact mechanism that
|
||
keeps forced-open continuation values low and keeps the training target
|
||
labeling opens as bad. Closed loop.
|
||
|
||
This combined with D1 means:
|
||
|
||
- A1/A2 priors operate on first-open *labels*. They cannot fix a model
|
||
that, even after committing to an open, refuses to follow up.
|
||
- Any fix needs to either (a) produce stronger leaf values during training
|
||
so the value head learns post-open play matters, or (b) explicitly
|
||
curriculum or prior-shape the *post-open* decision (not the open
|
||
decision).
|
||
|
||
Updated next-step priorities:
|
||
|
||
- **(E1) `cutoff_rollout_policy=safe_heuristic` training ablation.**
|
||
Strongest single lever: gives traversal leaves stronger value estimates
|
||
during training, which should ripple back to "open + follow up" signal.
|
||
Pure-self-play purity dented, but only at cutoff leaves.
|
||
- (E1.b) Sanity check: at iter 200 baseline, the same-color discard rate
|
||
is already 0.12 in good bucket — early. So this is not a late-training
|
||
collapse; it's baked in from early iterations. Curriculum-style fix
|
||
would have to start very early.
|
||
- A2 effectively dead unless paired with strong post-policy in rollouts.
|
||
|
||
### 6. Input feature audit and cleanup (2026-05-10)
|
||
|
||
After D1 + E2 confirmed the bottleneck is *training signal* and not *input
|
||
poverty*, audited every feature the model receives. Goal: align the input
|
||
representation with a strict pure-self-play definition by removing any
|
||
feature that embeds a hand-coded judgment or strategy assumption.
|
||
|
||
Three tiers found:
|
||
|
||
- **Tier 1**: pure game state and single-step game rules (hand cards,
|
||
expedition state, discard piles, scores, public histogram, legal-action
|
||
mask, `playable_*`/`dead_*` rule checks, `unknown_remaining_count`,
|
||
etc.). No judgment. Kept.
|
||
- **Tier 2**: projection-based features that compute "if I commit and
|
||
play all currently-playable cards, what is the score?" — mechanical
|
||
but assumption-laden. Includes `recoverable_score_no_bonus`,
|
||
`recoverable_margin_no_bonus`, `min_needed_to_break_even`,
|
||
`cards_needed_for_bonus`, `has_bonus_path`. Removed.
|
||
- **Tier 3**: explicit hand-coded judgments — `is_bad_open_candidate`,
|
||
`open_risk_score`, `is_safe_continuation`. Same heuristic family used
|
||
to label `bad_open` in evaluation. Removed.
|
||
|
||
Notable additional finding: the `no_bonus` projection family is
|
||
asymmetric. It includes wager multipliers but excludes the +20 bonus,
|
||
which systematically *under*-estimates the upside of commitable
|
||
expeditions by roughly 20 × P(complete bonus). This bias points in the
|
||
same direction as the phase 1 trap ("opens look like loss"), so removing
|
||
the projection family is consistent with diagnosing-not-baking-in the
|
||
trap.
|
||
|
||
Implementation:
|
||
|
||
- `encoding.pyx`: `DERIVED_PLAYABILITY_PER_COLOR` 19 → 15;
|
||
`SLOT_AWARE_PLAYABILITY_PER_SLOT` 12 → 6 (across two cleanup passes).
|
||
- Test shape assertions updated.
|
||
- Cython module rebuilt.
|
||
|
||
Input dimension change: 365 → 341 (Tier 3 removal) → **297** (Tier 2
|
||
removal). Net 18.6% reduction.
|
||
|
||
Result is not yet measured. The hypothesis is that with the heuristic
|
||
crutches removed, training behavior is a cleaner measurement of what
|
||
Deep CFR can do in this game from raw input. The model may stay in the
|
||
phase 1 trap longer or fail more visibly, both of which are useful
|
||
information.
|
||
|
||
### 7. Short open-selectivity ablation
|
||
|
||
Run a 200-300 iteration ablation only after the target audit identifies a
|
||
specific change. Candidate changes include:
|
||
|
||
- first-open target shaping,
|
||
- modified unsampled-open penalty,
|
||
- outcome sampling focused on open-relevant branches,
|
||
- or strategy/eval selection that separates current and average policy at the
|
||
open decision.
|
||
|
||
Primary metrics:
|
||
|
||
- `eval/safe_heuristic_strict/avg_score_diff0`
|
||
- `eval/safe_heuristic_strict/win_rate0`
|
||
- `eval/safe_heuristic_strict/bad_open_rate`
|
||
- `eval/safe_heuristic_strict/score_per_opened_color`
|
||
|
||
Do not promote to 500+ iterations unless bad-open rate and score/opened color
|
||
both improve without degrading score diff.
|
||
|
||
## Operational notes
|
||
|
||
- Keep one GPU training run active at a time.
|
||
- Use `tmux` plus `.compute.lock` for training.
|
||
- Mirror real experiments to W&B.
|
||
- Use one W&B group per hypothesis family.
|
||
- Do not start a long 2000-iteration run until a short diagnostic run shows
|
||
stable selectivity improvement.
|