coolguy
79f3ca0a58
이전 레포(../coolrl) commit 5a98855, da0d4cd에서 시도된 regret_matching_epsilon 튜닝(eps1e3 → eps1e4)이 zero-pit 대응책이었고 1e-4가 best로 채택되어 현재 default 0.0001로 남아있음을 doc에 기록. 이 lever는 이미 당겨진 상태이며 R1의 새 zero-pit은 다른 원인 (opponent=discard_only가 만든 game-theoretic 균형)이라는 점 명시.
845 lines
38 KiB
Markdown
845 lines
38 KiB
Markdown
# Deep CFR Selectivity Investigation
|
||
|
||
Last updated: 2026-05-10
|
||
|
||
## Current conclusion
|
||
|
||
The current Deep CFR baseline is not mainly blocked by model size or LCFR
|
||
time-weighting. The strongest signal so far is a selectivity failure: the model
|
||
does not reliably learn to distinguish good first opens from bad first opens.
|
||
|
||
The `outcome_sampling_epsilon=0.05` setting produced a short-run improvement
|
||
around 200 iterations, but the improvement did not hold through 500 iterations.
|
||
Weighting ablations did not recover the 200-iteration peak, so further LCFR
|
||
alpha tuning is lower priority than directly inspecting or changing the
|
||
first-open advantage target.
|
||
|
||
## Baseline symptoms
|
||
|
||
The 512x3 dense-eval baseline showed improving training losses, but the main
|
||
game-quality metrics against `heuristic_cautious` did not improve enough to
|
||
indicate a useful policy.
|
||
|
||
Observed pattern:
|
||
|
||
- Advantage loss can move in the expected direction while eval quality remains
|
||
poor.
|
||
- Bad first-open behavior stays high.
|
||
- Open quality, measured by score per opened color, stays negative.
|
||
- Current-policy evaluation did not rescue the result, so the issue is not only
|
||
average-policy lag.
|
||
|
||
## Closed hypotheses
|
||
|
||
### More outcome sampling helps, but only up to a point
|
||
|
||
Short 200-iteration ablations with `outcome_unsampled_regret=zero`:
|
||
|
||
| setting | safe strict score diff | win rate | bad open rate | score/opened color |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| `epsilon=0.20` | -57.87 | 0.07 | 0.901 | -8.40 |
|
||
| `epsilon=0.10` | -58.66 | 0.05 | 0.913 | -8.63 |
|
||
| `epsilon=0.05` | -40.01 | 0.12 | 0.893 | -6.25 |
|
||
| `epsilon=0.02` | -53.55 | - | - | - |
|
||
|
||
Conclusion: `epsilon=0.05` was the best short-run candidate. Lowering to
|
||
`0.02` was worse, and larger values were also worse.
|
||
|
||
### regret_matching_epsilon = 1e-4 was the zero-pit countermeasure (prior repo)
|
||
|
||
이전 레포(`../coolrl`)에서 zero-pit 발생 당시 시도한 epsilon 튜닝 기록
|
||
(commit `5a98855`, `da0d4cd`, 2026-05-06). 이건 `outcome_sampling_epsilon`이
|
||
아니라 `regret_matching_epsilon`이라는 별도 파라미터.
|
||
|
||
- **eps1e3** (`regret_matching_epsilon=1.0e-3`): zero-pit timeout은 줄였지만
|
||
safe 계열 avg_diff 개선 X. selectivity floor를 과하게 만들 가능성 의심.
|
||
- **eps1e4** (`regret_matching_epsilon=1.0e-4`): floor 더 낮춰서 219 iter
|
||
수동 중단. random 승률 0.86 / score_diff +32, safe 승률 0.03~0.05 /
|
||
score_diff -57~-74. zero-pit timeout은 거의 안 나오지만 selectivity
|
||
실패는 그대로.
|
||
|
||
`1.0e-4`가 그때 best로 채택되어 현재 `default.yaml`의 기본값으로 남아 있음
|
||
(`regret_matching_epsilon: 0.0001`). R0, R1 양쪽 모두 이 설정 사용 중.
|
||
|
||
**이 lever는 이미 당겨져 있다.** R1에서 zero-pit이 다시 나타난 것은
|
||
floor가 풀려서가 아니라, opponent=discard_only가 "do nothing" 균형을
|
||
*수학적으로* 안전하게 만든 별개 원인. epsilon 재조정으로 풀리는 구조가 아님.
|
||
|
||
### Negative unsampled regret was not sufficient
|
||
|
||
The `epsilon=0.05` plus `outcome_unsampled_regret=negative_node_value` variant
|
||
ended around safe strict score diff `-44.53` at 200 iterations, worse than
|
||
`epsilon=0.05` with `zero`.
|
||
|
||
Conclusion: the negative target suppresses opening, but it does not selectively
|
||
preserve good opens.
|
||
|
||
### The 200-iteration epsilon=0.05 peak did not hold
|
||
|
||
Confirmation run:
|
||
|
||
- Run: `runs/2026-05-09_010747_confirm-eps-005-zero-512x3-det-500`
|
||
- W&B group: `eps005-confirmation-512x3-v1`
|
||
- Config delta:
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=500`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
|
||
Safe heuristic strict metrics:
|
||
|
||
| iteration | win rate | score diff | bad open rate | score/opened color |
|
||
| ---: | ---: | ---: | ---: | ---: |
|
||
| 200 | 0.12 | -40.01 | 0.893 | -6.25 |
|
||
| 250 | 0.08 | -46.91 | 0.900 | -7.29 |
|
||
| 300 | 0.08 | -45.93 | 0.895 | -6.30 |
|
||
| 350 | 0.07 | -54.02 | 0.899 | -7.83 |
|
||
| 400 | 0.04 | -49.64 | 0.912 | -7.83 |
|
||
| 450 | 0.06 | -49.31 | 0.917 | -6.71 |
|
||
| 500 | 0.06 | -61.24 | 0.903 | -8.30 |
|
||
|
||
Conclusion: `epsilon=0.05` creates a real short-run peak, but the behavior is
|
||
not stable through 500 iterations.
|
||
|
||
### LCFR time-weighting is not the sole cause of the degradation
|
||
|
||
Weighting ablation group:
|
||
|
||
- W&B group: `eps005-weighting-ablation-512x3-v1`
|
||
- Common config:
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=300`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
|
||
Compared runs:
|
||
|
||
| run | iter 200 diff | iter 250 diff | iter 300 diff | iter 300 win | iter 300 bad open |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| LCFR alpha=1.0 confirmation | -40.01 | -46.91 | -45.93 | 0.08 | 0.895 |
|
||
| `training_weighting.mode=none` | -50.98 | -48.46 | -56.27 | 0.04 | 0.912 |
|
||
| `training_weighting.lcfr_alpha=0.5` | -66.81 | -58.97 | -50.77 | 0.10 | 0.891 |
|
||
|
||
Open-quality comparison:
|
||
|
||
| run | iter 200 score/opened color | iter 300 score/opened color |
|
||
| --- | ---: | ---: |
|
||
| LCFR alpha=1.0 confirmation | -6.25 | -6.30 |
|
||
| `training_weighting.mode=none` | -7.71 | -7.70 |
|
||
| `training_weighting.lcfr_alpha=0.5` | -9.94 | -7.85 |
|
||
|
||
Conclusion: neither removing time-weighting nor softening LCFR to `alpha=0.5`
|
||
beat the original `alpha=1.0` confirmation by the primary score-diff metric at
|
||
300 iterations. LCFR may affect stability, but it is not the main lever.
|
||
|
||
## First-open diagnostic
|
||
|
||
Diagnostic output:
|
||
|
||
- `runs/tmp/first_open_advantage_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Observed values:
|
||
|
||
| checkpoint | bad selection | good selection | bad advantage | good advantage | sampled turns |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | 0.0625 | 0.0582 | -12.82 | -16.60 | 8560 |
|
||
| iter 500 | 0.0067 | 0.0067 | -35.42 | -44.32 | 40942 |
|
||
|
||
Interpretation: by 500 iterations the model strongly suppresses opening
|
||
overall. It suppresses good opens along with bad opens, which is the core
|
||
selectivity failure.
|
||
|
||
## First-open target audit
|
||
|
||
Script:
|
||
|
||
- `scripts/analyze_first_open_targets.py`
|
||
|
||
Output:
|
||
|
||
- `runs/tmp/first_open_target_audit_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Method: regenerate short interleaved traversal batches from existing
|
||
checkpoints and bucket first-open advantage targets by action quality. Because
|
||
`outcome_unsampled_regret=zero` sets unsampled legal actions to zero, the most
|
||
informative statistic is the sampled-action target distribution, not the full
|
||
legal-candidate target distribution.
|
||
|
||
Sampled target summary:
|
||
|
||
| checkpoint | bucket | candidates | policy prob | sampled rate | sampled target mean | sampled target positive |
|
||
| --- | --- | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | good open | 796 | 0.078 | 0.078 | -40.71 | 0.419 |
|
||
| iter 200 | bad open | 7142 | 0.081 | 0.081 | -25.65 | 0.424 |
|
||
| iter 500 | good open | 800 | 0.064 | 0.058 | 8.39 | 0.435 |
|
||
| iter 500 | bad open | 8322 | 0.069 | 0.071 | 12.61 | 0.433 |
|
||
|
||
Interpretation: the regenerated traversal targets do not rank good opens above
|
||
bad opens. At both inspected checkpoints, bad-open candidates receive slightly
|
||
higher policy probability and sampled rate than good-open candidates. The
|
||
sampled target mean is also better for bad opens than good opens. This points
|
||
to a target or metric-alignment problem before model capacity or LCFR tuning.
|
||
|
||
## First-open counterfactual audit
|
||
|
||
Script:
|
||
|
||
- `scripts/analyze_first_open_counterfactual.py`
|
||
|
||
Output:
|
||
|
||
- `runs/tmp/first_open_counterfactual_confirm_eps005_200_vs_500.jsonl`
|
||
|
||
Method: collect first-open candidate states from existing checkpoints, force
|
||
each first-open candidate once, and compare the resulting continuation value
|
||
against the current policy's best non-open action from the same state.
|
||
|
||
`delta_open = value(force open) - value(best non-open)`.
|
||
|
||
Counterfactual summary against `heuristic_cautious`:
|
||
|
||
| checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate |
|
||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| iter 200 | good open | 40 | -26.57 | -26.5 | 0.050 | 0.000 | 0.000 |
|
||
| iter 200 | bad open | 460 | -23.15 | -21.0 | 0.130 | 0.036 | 0.037 |
|
||
| iter 500 | good open | 30 | -23.43 | -17.5 | 0.167 | 0.067 | 0.067 |
|
||
| iter 500 | bad open | 470 | -11.18 | -8.0 | 0.226 | 0.020 | 0.019 |
|
||
|
||
Interpretation: the heuristic `open_bad` label is not entirely misaligned with
|
||
continuation value. Forced bad opens are usually worse than the best non-open
|
||
alternative. However, heuristic `open_good` also often loses to best non-open in
|
||
these sampled states, so "recoverable eventually" is not the same as "open now."
|
||
|
||
Combined with the target audit, this points toward target/objective alignment:
|
||
the traversal target is not making the bad-open-vs-non-open mistake clearly
|
||
negative, even when the counterfactual continuation usually is negative.
|
||
|
||
## Open questions
|
||
|
||
1. Does the traversal target itself provide separable labels for good first
|
||
opens versus bad first opens?
|
||
2. Is the model receiving too sparse or too noisy a signal at the first-open
|
||
decision point?
|
||
3. Would target shaping around first-open decisions improve score/opened color
|
||
without increasing bad-open rate?
|
||
4. Is evaluation showing a policy-selection problem, or is the advantage model
|
||
already misranking good and bad opens before strategy extraction?
|
||
|
||
## Recommended next experiments
|
||
|
||
### 1. Deeper first-open target audit
|
||
|
||
Before another long training run, inspect sampled first-open decision records
|
||
more directly:
|
||
|
||
- Group candidate first-open actions into good-open and bad-open buckets.
|
||
- Compare target values before model prediction, not only final learned
|
||
advantages.
|
||
- Report distributions, not just means.
|
||
- Check whether the target ranks good opens above bad opens in the same
|
||
information-state context family.
|
||
|
||
Success criterion: the target distribution should show a usable separation
|
||
between good and bad opens. If it does not, the training target is the blocker.
|
||
|
||
### 2. First-open replay reweighting
|
||
|
||
Implemented option:
|
||
|
||
- `optimization.advantage_first_open_fraction`
|
||
|
||
Meaning: reserve this fraction of each advantage minibatch for samples where a
|
||
first-open action was legally available at the traverser decision. The target is
|
||
not changed; only replay sampling frequency changes. This keeps the experiment
|
||
on the pure self-play side more than hand-written target shaping.
|
||
|
||
Initial planned run:
|
||
|
||
- `run.experiment_name=first-open-reweight-50-512x3-det-500`
|
||
- `optimization.advantage_first_open_fraction=0.5`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=500`
|
||
|
||
Primary comparison is the `eps=0.05` confirmation run and the pure external
|
||
sampling run. Success requires bad-open rate and score/opened color to improve
|
||
together without a large score-diff regression.
|
||
|
||
Result:
|
||
|
||
- Run: `runs/2026-05-09_235808_first-open-reweight-50-512x3-det-500-indexed`
|
||
- W&B group: `first-open-replay-v1`
|
||
- Commit: `8217cec` (`Speed up first-open memory sampling`)
|
||
|
||
The scan-based first implementation was stopped after 12 iterations because
|
||
first-open sampling scanned the full replay memory and pushed iteration time
|
||
above 60 seconds. The indexed-memory version kept first-open sampling near
|
||
0.25 seconds per player at 4M advantage samples and completed 500 iterations.
|
||
|
||
Final `heuristic_cautious` comparison:
|
||
|
||
| Run | Iter | Score diff | Win rate | Bad open | Score/opened |
|
||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||
| baseline `confirm-eps-005-zero-512x3-det-500` | 500 | -61.24 | 0.06 | 0.903 | -8.30 |
|
||
| pure external `pure-external-512x3-det-500` | 500 | -50.38 | 0.06 | 0.925 | -9.79 |
|
||
| first-open reweight 50% indexed | 500 | -52.20 | 0.07 | 0.919 | -9.39 |
|
||
|
||
Conclusion: first-open replay reweighting alone did not solve selectivity. It
|
||
improved score diff versus the baseline final checkpoint, but it did not reduce
|
||
bad-open rate and made score/opened color worse. The run briefly looked better
|
||
around 120-200 iterations, then regressed by 300-500 iterations. This suggests
|
||
the replay emphasis is not enough if the underlying target does not separate
|
||
good first opens from bad first opens.
|
||
|
||
### 3. First-open target prior (A1) — running 2026-05-10
|
||
|
||
Hypothesis: target audit shows the regenerated traversal target does not
|
||
separate good first opens from bad first opens. With
|
||
`outcome_unsampled_regret=zero`, every unsampled first-open candidate is
|
||
labeled with target `0`, regardless of whether it is heuristically good
|
||
or bad. A small signed prior on unsampled first-open play actions should
|
||
break the symmetry without overriding the high-variance sampled-action
|
||
target.
|
||
|
||
Implementation:
|
||
|
||
- Config field: `traversal.outcome_unsampled_first_open_prior_alpha` (float,
|
||
default `0.0`).
|
||
- Plumbed through `InterleavedTraversalConfig` and
|
||
`run_interleaved_traversal_batch`. Cython traversal path unchanged
|
||
(default scheduler is interleaved; pyx path is a follow-up if needed).
|
||
- Effect site: `_after_child` in `interleaved_traversal.py`. When
|
||
`is_first_open` and `alpha != 0`, unsampled legal play actions whose color
|
||
has an empty expedition are overwritten in `target` with `+alpha` if the
|
||
visible `recoverable_score` for that color is `>= 0`, else `-alpha`. The
|
||
sampled action's cell stays as `sampled_action_value - node_value`.
|
||
- `recoverable_score` mirrors `evaluate._visible_recoverable_summary` for
|
||
the empty-expedition case (no future-card lookahead, hand-only signal).
|
||
- Tests: `test_first_open_prior_overrides_unsampled_play_targets_with_signed_alpha`,
|
||
`test_first_open_prior_zero_alpha_is_noop`.
|
||
|
||
Note on "pure self-play": this prior introduces a hand-written heuristic
|
||
into the advantage target. The opponent and rollout policy remain
|
||
self-play (no external bot). This is treated as a diagnostic experiment;
|
||
if effective, the permanent solution is a counterfactual-based prior
|
||
(A2) that recovers full pure self-play.
|
||
|
||
Initial planned run:
|
||
|
||
- `run.experiment_name=first-open-prior-alpha5-512x3-det-200`
|
||
- `traversal.outcome_unsampled_first_open_prior_alpha=5.0`
|
||
- `traversal.outcome_unsampled_regret=zero`
|
||
- `traversal.outcome_sampling_epsilon=0.05`
|
||
- `run.deterministic=true`
|
||
- `run.max_iterations=200`
|
||
- W&B group: `first-open-prior-v1`
|
||
|
||
Primary comparisons:
|
||
|
||
- `confirm-eps-005-zero-512x3-det-500` (baseline; iter 200 reference)
|
||
- `first-open-reweight-50-512x3-det-500-indexed` (replay-side intervention)
|
||
|
||
Success criterion: `bad_open_rate` and `score_per_opened_color` improve
|
||
*together* at iter 200 without large regression in `avg_score_diff0`. If
|
||
positive, follow up with a 500-iter run to test stability. If null/regress,
|
||
try alpha sweep (e.g. 2.0, 10.0) before abandoning the direction.
|
||
|
||
Result (2026-05-10):
|
||
|
||
- Run: `runs/2026-05-10_072718_first-open-prior-alpha5-512x3-det-200`
|
||
- W&B: synced online to group `first-open-prior-v1` (run `teuh915r`)
|
||
- Commit: see HEAD at run start
|
||
|
||
`heuristic_cautious` at iter 200:
|
||
|
||
| Run | Score diff | Win rate | Bad open | Score/opened |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 |
|
||
| **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** |
|
||
|
||
Other opponents at iter 200: random +32.51 / 0.86, heuristic_noisy -71.52 / 0.07,
|
||
heuristic_balanced -81.81 / 0.02, heuristic_aggressive -81.24 / 0.05.
|
||
|
||
Conclusion: **mixed result, net regression.** The prior did shift behavior
|
||
in the intended direction on one axis — `bad_open_rate` dropped from 0.893
|
||
to 0.796 (~10% absolute reduction). This is the only metric where the
|
||
hypothesis "the prior breaks ranking symmetry" looks supported.
|
||
|
||
But the overall game-quality metrics regressed: score diff worsened
|
||
(-40 → -67), win rate collapsed (0.12 → 0.02), and `score_per_opened_color`
|
||
got worse (-6.25 → -8.85). The model fails the success criterion (which
|
||
required `bad_open_rate` AND `score_per_opened_color` to improve together).
|
||
|
||
Two interpretations:
|
||
|
||
1. **α = 5.0 too strong.** The prior is overpowering the sampled-action
|
||
target rather than acting as a weak symmetry-breaker. The good-open prior
|
||
pushes the model to open *more often*, but those forced opens are bad
|
||
*given the actual game state*, just labeled good by the visible-only
|
||
heuristic. The counterfactual audit already flagged this: heuristic
|
||
`open_good` candidates often lose to best non-open in real continuation.
|
||
2. **Heuristic itself misaligned.** Even the right α won't help if the
|
||
sign assigned to candidates is wrong relative to true game value.
|
||
|
||
Next steps (in order):
|
||
|
||
- (A1.b) α sweep at 200 iter: α ∈ {1.0, 2.0} to test "weaker prior" hypothesis.
|
||
If α=2.0 still regresses score diff while reducing bad_open, the prior shape
|
||
itself is wrong, not just strength.
|
||
- (A2) Counterfactual prior: replace heuristic sign with sign of
|
||
`value(force open) - value(best non-open)` from a small in-traversal
|
||
rollout. Cleaner signal, recovers pure self-play, but more expensive.
|
||
- If both fail to improve score diff, the issue is upstream of unsampled
|
||
regret labeling (likely traversal sample distribution or sampled-action
|
||
target variance), and the next experiment family should target those.
|
||
|
||
### 4. D1 diagnostic: counterfactual with strong post-policy (2026-05-10)
|
||
|
||
Question: are forced-open continuation values low because the openings
|
||
themselves are bad, or because the self-play rollout policy poisons the
|
||
post-action play (selection bias)?
|
||
|
||
Method: re-ran `analyze_first_open_counterfactual.py` on the
|
||
`confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter
|
||
500), but with `--post-policy heuristic_cautious`. The opponent and
|
||
state-collection policy stayed the same; only the policy_player's actions
|
||
*after* the forced first action used the strong fixed bot.
|
||
|
||
Output: `runs/tmp/first_open_counterfactual_d1_strong_post_policy_200_vs_500.jsonl`
|
||
|
||
`delta_open = value(force open) - value(best non-open)` comparison:
|
||
|
||
| iter | bucket | n | self-play post | strong post | shift |
|
||
| ---: | --- | ---: | ---: | ---: | ---: |
|
||
| 200 | open_good | 40 | -26.57 | **-3.35** | +23.22 |
|
||
| 200 | open_bad | 460 | -23.15 | **-5.52** | +17.63 |
|
||
| 500 | open_good | 30 | -23.43 | **-10.90** | +12.53 |
|
||
| 500 | open_bad | 470 | -11.18 | **-8.99** | +2.19 |
|
||
|
||
`delta_positive_rate`:
|
||
|
||
| iter | bucket | self-play post | strong post |
|
||
| ---: | --- | ---: | ---: |
|
||
| 200 | open_good | 0.050 | 0.225 |
|
||
| 200 | open_bad | 0.130 | 0.317 |
|
||
| 500 | open_good | 0.167 | 0.300 |
|
||
| 500 | open_bad | 0.226 | 0.230 |
|
||
|
||
Verdict — two findings, both important:
|
||
|
||
**1. Selection bias is real and significant.** Strong post-policy
|
||
improves forced-open continuation values by 12–23 points on average. The
|
||
self-play policy is meaningfully poisoning rollouts: forced opens look
|
||
much less bad once a competent player handles the followup.
|
||
|
||
**2. Heuristic labels do not separate cleanly even under strong post-policy.**
|
||
At iter 200, `open_good` delta_mean is only ~2 points better than
|
||
`open_bad` (-3.35 vs -5.52). At iter 500, ranking is essentially flat
|
||
(`open_good` -10.9 vs `open_bad` -8.99 — slightly *worse*). Median deltas
|
||
agree. Sample size for `open_good` is small (30–40) so noise contributes,
|
||
but there is no clean signal that the heuristic `recoverable_score`
|
||
classifier matches actual continuation value.
|
||
|
||
Implications:
|
||
|
||
- A1 prior was destined to fail — the heuristic sign is at best weakly
|
||
aligned with continuation value, even with optimistic post-policy.
|
||
- A2 with self-play rollouts would inherit selection bias and likely
|
||
reproduce the same misranking. A2 with a strong post-policy would
|
||
give clean signs but breaks pure self-play.
|
||
- The deeper bottleneck is post-open play quality. Until the trained
|
||
policy plays competently *after* opening, training signals about
|
||
*whether to open* will be biased toward "don't open."
|
||
|
||
Candidate next directions (decision pending):
|
||
|
||
- (E1) Train with `cutoff_rollout_policy=heuristic_balanced` instead of `random`.
|
||
Already a config option; gives leaf nodes stronger value estimates
|
||
during traversal. Trades some pure-self-play purity for a stronger
|
||
bootstrap signal. Cheap to test.
|
||
- (E2) Investigate post-open behavior directly: forced-open + observe
|
||
next-2-3 turns. Diagnoses *why* the model can't follow up (e.g. always
|
||
discards followup cards, switches color, etc.).
|
||
- (E3) Curriculum / staged training that exposes the network to good
|
||
post-open trajectories before forcing it to make first-open decisions.
|
||
- A2 deferred unless we adopt a strong post-policy in rollouts.
|
||
|
||
### 5. E2 diagnostic: post-forced-open behavior (2026-05-10)
|
||
|
||
Question: D1 showed selection bias is real — *what is the model actually
|
||
doing after a forced first-open* that poisons rollouts?
|
||
|
||
Method: forced each first-open candidate at sampled states, then observed
|
||
the policy_player's next 3 decisions (window). Used baseline checkpoints
|
||
(`confirm-eps-005-zero` iter 200, iter 500). Categorized each followup
|
||
decision relative to the forced-open color.
|
||
|
||
Output: `runs/tmp/first_open_followup_baseline_200_vs_500.jsonl`
|
||
Script: `scripts/analyze_first_open_followup.py`
|
||
|
||
Per-window mean counts (3 policy_player decisions = ~1.5 game turns;
|
||
each game turn includes one play/discard plus one draw):
|
||
|
||
| iter | bucket | n | held@force | plays(same) | discards(same) | open_other | other_discard | held@end |
|
||
| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| 200 | good | 33 | 4.82 | 0.15 | 0.15 | **0.49** | 0.21 | 2.70 |
|
||
| 200 | bad | 367 | 1.97 | 0.03 | 0.12 | 0.10 | 0.75 | 1.08 |
|
||
| 500 | good | 22 | 4.45 | 0.14 | **0.46** | 0.14 | 0.27 | 1.23 |
|
||
| 500 | bad | 378 | 2.56 | 0.01 | 0.23 | 0.05 | 0.71 | 1.29 |
|
||
|
||
Findings:
|
||
|
||
**1. Model under-plays followup cards even when it holds many.** At iter
|
||
200, good-open candidates start with ~4.8 same-color cards in hand. In the
|
||
next 3 decisions, the model plays only 0.15 of them on average (3% of
|
||
its window). 2.7 cards remain in hand at terminal — nearly 3 cards of
|
||
the just-opened color never reach the expedition.
|
||
|
||
**2. Model opens additional new colors after a forced open.** At iter 200
|
||
good bucket, `open_other` = 0.49 in a 3-decision window — the model is
|
||
~3× more likely to open *another* new color than to follow up the one it
|
||
was forced into. This compounds the recoverable-score drag.
|
||
|
||
**3. By iter 500 the model actively dumps the forced color.** Good bucket
|
||
at iter 500: `same-color discards` 0.46 vs `same-color plays` 0.14. The
|
||
model discards followup cards more than 3× as often as it plays them,
|
||
despite holding 4.45 of them at force time. This is the strongest single
|
||
piece of evidence so far that the post-open value head is poisoned.
|
||
|
||
**4. Bad-open bucket shows even sharper avoidance.** Hand has ~2
|
||
same-color cards, plays effectively zero. Almost all play-phase decisions
|
||
are "discard other" (0.71–0.75). The model treats forced opens as bad
|
||
news to liquidate rather than commit to.
|
||
|
||
Interpretation: the trained advantage network *learned* that opens of
|
||
these colors are net-negative, so once forced into one, it hedges by
|
||
opening other colors and dumping followup cards. That hedging is rational
|
||
under the policy's own value estimates but is the exact mechanism that
|
||
keeps forced-open continuation values low and keeps the training target
|
||
labeling opens as bad. Closed loop.
|
||
|
||
This combined with D1 means:
|
||
|
||
- A1/A2 priors operate on first-open *labels*. They cannot fix a model
|
||
that, even after committing to an open, refuses to follow up.
|
||
- Any fix needs to either (a) produce stronger leaf values during training
|
||
so the value head learns post-open play matters, or (b) explicitly
|
||
curriculum or prior-shape the *post-open* decision (not the open
|
||
decision).
|
||
|
||
Updated next-step priorities:
|
||
|
||
- **(E1) `cutoff_rollout_policy=heuristic_balanced` training ablation.**
|
||
Strongest single lever: gives traversal leaves stronger value estimates
|
||
during training, which should ripple back to "open + follow up" signal.
|
||
Pure-self-play purity dented, but only at cutoff leaves.
|
||
- (E1.b) Sanity check: at iter 200 baseline, the same-color discard rate
|
||
is already 0.12 in good bucket — early. So this is not a late-training
|
||
collapse; it's baked in from early iterations. Curriculum-style fix
|
||
would have to start very early.
|
||
- A2 effectively dead unless paired with strong post-policy in rollouts.
|
||
|
||
### 6. Input feature audit and cleanup (2026-05-10)
|
||
|
||
After D1 + E2 confirmed the bottleneck is *training signal* and not *input
|
||
poverty*, audited every feature the model receives. Goal: align the input
|
||
representation with a strict pure-self-play definition by removing any
|
||
feature that embeds a hand-coded judgment or strategy assumption.
|
||
|
||
Three tiers found:
|
||
|
||
- **Tier 1**: pure game state and single-step game rules (hand cards,
|
||
expedition state, discard piles, scores, public histogram, legal-action
|
||
mask, `playable_*`/`dead_*` rule checks, `unknown_remaining_count`,
|
||
etc.). No judgment. Kept.
|
||
- **Tier 2**: projection-based features that compute "if I commit and
|
||
play all currently-playable cards, what is the score?" — mechanical
|
||
but assumption-laden. Includes `recoverable_score_no_bonus`,
|
||
`recoverable_margin_no_bonus`, `min_needed_to_break_even`,
|
||
`cards_needed_for_bonus`, `has_bonus_path`. Removed.
|
||
- **Tier 3**: explicit hand-coded judgments — `is_bad_open_candidate`,
|
||
`open_risk_score`, `is_safe_continuation`. Same heuristic family used
|
||
to label `bad_open` in evaluation. Removed.
|
||
|
||
Notable additional finding: the `no_bonus` projection family is
|
||
asymmetric. It includes wager multipliers but excludes the +20 bonus,
|
||
which systematically *under*-estimates the upside of commitable
|
||
expeditions by roughly 20 × P(complete bonus). This bias points in the
|
||
same direction as the phase 1 trap ("opens look like loss"), so removing
|
||
the projection family is consistent with diagnosing-not-baking-in the
|
||
trap.
|
||
|
||
Implementation:
|
||
|
||
- `encoding.pyx`: `DERIVED_PLAYABILITY_PER_COLOR` 19 → 15;
|
||
`SLOT_AWARE_PLAYABILITY_PER_SLOT` 12 → 6 (across two cleanup passes).
|
||
- Test shape assertions updated.
|
||
- Cython module rebuilt.
|
||
|
||
Input dimension change: 365 → 341 (Tier 3 removal) → **297** (Tier 2
|
||
removal). Net 18.6% reduction.
|
||
|
||
Result is not yet measured. The hypothesis is that with the heuristic
|
||
crutches removed, training behavior is a cleaner measurement of what
|
||
Deep CFR can do in this game from raw input. The model may stay in the
|
||
phase 1 trap longer or fail more visibly, both of which are useful
|
||
information.
|
||
|
||
### 7. ColorSharedNetwork chunked-layout bug (2026-05-10)
|
||
|
||
While re-examining the archived `2026-05-07_092137_color_shared_attention_1000iter`
|
||
run (killed at iter 41), discovered the `ColorSharedNetwork` implementation
|
||
in `networks.py` was not actually color-aware. The forward pass split the
|
||
input vector into `input_dim // n_colors` contiguous slices and ran them
|
||
through a shared encoder. The slice boundaries do not align with the actual
|
||
encoding layout — adjacent slices contain phase flags, hand slots,
|
||
expedition state, scores, etc. mixed together. The "shared color encoder"
|
||
was therefore sharing weights across semantically unrelated chunks, not
|
||
across per-color blocks.
|
||
|
||
This means the prior conclusion that "color_shared / attention archive run
|
||
was inconclusive" was charitable. The architecture being measured was a
|
||
chunked-input network mislabeled `color_shared`, not a real per-color
|
||
shared architecture. We have *no* signal on whether a properly per-color
|
||
architecture would help.
|
||
|
||
Fix landed in this same session:
|
||
|
||
- Added `compute_lost_cities_color_layout(input_dim)` in `networks.py`. For
|
||
the standard Lost Cities schema (n_colors=5, hand_size=8, n_ranks=9), it
|
||
recognises the four valid `input_dim` values (171, 219, 249, 297 across
|
||
derived/slot-aware flag combinations) and returns per-color and common
|
||
index lists derived from the actual encoding layout.
|
||
- Per-color block (39 dims with `derived_playability` on): both players'
|
||
expedition state for that color, discard top metadata, public-histogram
|
||
row, pending-discard one-hot bit, legal-action draw-pile bit, and the
|
||
`derived_playability` per-color block. Slot-aware features stay in
|
||
common because they are slot-major, not color-major.
|
||
- `ColorSharedNetwork.forward` now indexes per-color blocks via the layout
|
||
when `input_dim` matches a known schema. For other input dims (unit
|
||
tests, non-Lost Cities use), it falls back to the old chunked slicing
|
||
with a `UserWarning`, preserving backward compatibility for tests but
|
||
making the legacy behaviour visible.
|
||
- New unit tests cover both branches and the layout helper.
|
||
|
||
This is purely an implementation correctness fix; no fair test of the
|
||
architecture has been run yet. Fair test deferred — the diagnosis from
|
||
sections 4–5 (selection bias, post-open behaviour) suggests that even a
|
||
correct color-aware encoder would not break the closed loop on its own.
|
||
|
||
### 8. Interleaved scheduler honours all_negative_fallback (2026-05-10)
|
||
|
||
The interleaved traversal scheduler's `_regret_matching` was hard-coded to
|
||
spread fallback policy uniformly across legal actions, regardless of the
|
||
configured `regret_matching.all_negative_fallback`. The default config has
|
||
shipped with `all_negative_fallback: argmax_tiebreak` since
|
||
`618d5f8 Promote avg-strategy 1000iter to default.yaml` based on prior
|
||
20-iteration audit + 1000-iteration empirical evidence (see
|
||
`docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`), but the
|
||
default scheduler was switched to interleaved in `09bbe7c Make interleaved
|
||
traversal the default`, after which the configured fallback mode silently
|
||
no-op'd in interleaved code paths.
|
||
|
||
Fix:
|
||
|
||
- `_regret_matching(advantages, legal_mask, epsilon, fallback_mode="uniform")`
|
||
in `interleaved_traversal.py` now honours `argmax_tiebreak` by
|
||
concentrating policy mass on the lowest-index tied action (deterministic
|
||
tiebreak; the Cython recursive traverser randomises ties using its
|
||
per-traverser RNG, which the batched policy does not have).
|
||
- `BatchedPolicy` accepts `fallback_mode` and threads it through.
|
||
- `InterleavedTraversalConfig` carries `all_negative_fallback`.
|
||
- `run_interleaved_traversal_batch`, `trainer.py`, `workers.py`, and the
|
||
`analyze_first_open_targets.py` callers pass the field through.
|
||
|
||
Also bumped `traversal.outcome_sampling_epsilon` in `default.yaml` from
|
||
0.2 to 0.05. The 200-iteration sweep (section 1) showed 0.05 produced the
|
||
best short-run heuristic_cautious score diff (-40.01 vs -57.87 for
|
||
0.20). All recent experimental runs already used 0.05; the default now
|
||
matches actual experimental practice.
|
||
|
||
These changes do not target the diagnosed selection-bias bottleneck. They
|
||
align config intent with actual scheduler behaviour and make the default
|
||
config reproduce known-best knob settings out of the box.
|
||
|
||
### 9. Naming, plot curation, and tiered eval cadence (2026-05-10)
|
||
|
||
Hygiene changes — none target the diagnosed selection-bias bottleneck,
|
||
but they make the codebase honestly reflect the pure-self-play stance
|
||
and reduce dashboard noise.
|
||
|
||
Bot family rename (drop the unhelpful `safe_` prefix; suffixes now
|
||
describe behaviour):
|
||
|
||
| Old | New |
|
||
| --- | --- |
|
||
| `safe_heuristic_loose` | `heuristic_aggressive` |
|
||
| `safe_heuristic` | `heuristic_balanced` |
|
||
| `safe_heuristic_strict` | `heuristic_cautious` |
|
||
| `noisy_safe` | `heuristic_noisy` |
|
||
| `passive_discard` | `discard_only` |
|
||
|
||
Class renames in `bots/`: `SafeHeuristicBot` → `HeuristicBot`,
|
||
`SafeHeuristicParams` → `HeuristicParams`, `PassiveDiscardBot` →
|
||
`DiscardOnlyBot`, plus the loose/strict parameter constants. Backwards
|
||
compatibility was dropped intentionally — no aliases. Active configs,
|
||
docs, scripts, tests updated; archive files (read-only by policy)
|
||
left intact and may still reference old names.
|
||
|
||
Analyze plot curation (`deep_cfr/analyze.py`):
|
||
|
||
- New `analysis_00_core.png` dashboard with 10 heuristic-free metrics
|
||
(loss/{advantage, strategy}; vs `heuristic_cautious`:
|
||
`avg_score_diff0`, `win_rate0`, `avg_opened_colors`,
|
||
`positive_expedition_rate`, `bonus_expedition_rate`,
|
||
`score_per_opened_color`, `policy_entropy`; vs `random`: `win_rate0`).
|
||
- Removed `analysis_05_open_quality.png` (bad/weak/good open rates,
|
||
recoverable score) and `analysis_07_calibration.png` (calibration gap,
|
||
recoverable mean) — both relied on the heuristic `recoverable_score`
|
||
classifier we already dropped from inputs.
|
||
- Removed `SELECTIVITY_PLOTS` and `plot_selectivity` (heuristic-laden).
|
||
- `SUMMARY_EVAL_METRICS` no longer includes `bad_open_rate` or
|
||
`calibration_gap`.
|
||
|
||
`PlotSpec` gained an optional `opponents` allowlist so the new core
|
||
section can pin a specific opponent per panel without restructuring the
|
||
existing `plot_section` plumbing.
|
||
|
||
Tiered evaluation cadence (`EvaluationConfig`):
|
||
|
||
- Added `extended_opponents: tuple[str, ...]` and `extended_eval_every:
|
||
int = 0`.
|
||
- Method `opponents_for_iteration(iteration)` returns the core list at
|
||
every `eval_every`, and appends `extended_opponents` (de-duplicated)
|
||
when `iteration` is also a multiple of `extended_eval_every`.
|
||
- `default.yaml` now uses 3 core opponents
|
||
(`random`, `discard_only`, `heuristic_cautious`) every 5 iterations
|
||
and 3 extended opponents (`heuristic_balanced`, `heuristic_aggressive`,
|
||
`heuristic_noisy`) every 50 iterations.
|
||
- `random` is the floor sanity. `discard_only` is the zero-pit
|
||
detector / absolute-score reference (its score is always 0, so
|
||
`eval/discard_only/avg_score_diff0` directly equals our model's
|
||
raw average score). `heuristic_cautious` is the ceiling and the
|
||
archive-comparable benchmark used in sections 1–6.
|
||
|
||
Net effect on ongoing eval cost: ~50% reduction (3 opponents × every
|
||
5 iter, plus 6 opponents × every 50 iter, vs the prior 6 opponents
|
||
× every 5).
|
||
|
||
### 10. Short open-selectivity ablation
|
||
|
||
Run a 200-300 iteration ablation only after the target audit identifies a
|
||
specific change. Candidate changes include:
|
||
|
||
- first-open target shaping,
|
||
- modified unsampled-open penalty,
|
||
- outcome sampling focused on open-relevant branches,
|
||
- or strategy/eval selection that separates current and average policy at the
|
||
open decision.
|
||
|
||
Primary metrics:
|
||
|
||
- `eval/heuristic_cautious/avg_score_diff0`
|
||
- `eval/heuristic_cautious/win_rate0`
|
||
- `eval/heuristic_cautious/bad_open_rate`
|
||
- `eval/heuristic_cautious/score_per_opened_color`
|
||
|
||
Do not promote to 500+ iterations unless bad-open rate and score/opened color
|
||
both improve without degrading score diff.
|
||
|
||
## Operational notes
|
||
|
||
- Keep one GPU training run active at a time.
|
||
- Use `tmux` plus `.compute.lock` for training.
|
||
- Mirror real experiments to W&B.
|
||
- Use one W&B group per hypothesis family.
|
||
- Do not start a long 2000-iteration run until a short diagnostic run shows
|
||
stable selectivity improvement.
|
||
|
||
## §10. R1 — discard_only opponent diagnostic (2026-05-10)
|
||
|
||
### Hypothesis
|
||
|
||
Trap의 본질이 **자기참조 회로(self-similar opponent)**인지 **모델 자체
|
||
credit assignment 실패**인지 분리하기 위해, opponent_policy를
|
||
`discard_only` (항상 카드만 버리는 fixed bot)로 고정. opponent
|
||
dynamics를 거의 0으로 만들고 self-reference를 끊은 환경에서 모델이
|
||
trap을 벗어나는지 본다.
|
||
|
||
### Run
|
||
|
||
- Run dir: `runs/2026-05-10_173912_r1-discard-only-opp-512x3-det-300`
|
||
- W&B group: `r1-discard-only-opp-v1`
|
||
- Iters: 300 (R0 끊긴 시점과 같은 길이로 비교)
|
||
- Single change vs R0: `traversal.opponent_policy: discard_only`
|
||
- All else (capacity 2M, eval games 100, eps=0.05) identical
|
||
|
||
### Result (iter 300, vs R0 clean-baseline iter 300)
|
||
|
||
| metric | R0 (over-open) | R1 (zero-pit) | Δ |
|
||
| --- | ---: | ---: | ---: |
|
||
| score_diff vs cautious | -47.99 | **-69.56** | -21.57 worse |
|
||
| opened_colors vs cautious | 4.93 | **2.54** | -2.39 |
|
||
| play_action_rate vs cautious | 0.02 | **0.00** | -0.02 |
|
||
| pos_expeditions/game cautious | 0.76 | **0.08** | -0.68 |
|
||
| neg_expeditions/game cautious | 4.06 | 2.44 | -1.62 |
|
||
| avg_game_length cautious | 1053 | **5282** | +4229 (timeout) |
|
||
| score_diff vs discard_only | -27.19 | **0.00** | +27.19 |
|
||
| opened_colors vs discard_only | 4.98 | **0.00** | -4.98 |
|
||
| bad_open_rate vs discard_only | 0.91 | **0.00** | -0.91 |
|
||
| score_diff vs random | +49.61 | **-9.30** | -58.91 |
|
||
| opened_colors vs random | 4.97 | 1.45 | -3.52 |
|
||
|
||
### Interpretation: trap shifted, did not break
|
||
|
||
Trap 본질 진단 결과 **"자기참조 회로"는 충분조건이 아니다.**
|
||
|
||
- vs `discard_only` opponent: 모델이 **정확히 0개 색을 열고 0점으로 비김**
|
||
(zero-pit). 즉 closed-loop self-reference 깬 후에도 trap이 다른
|
||
형태(zero-pit)로 재출현.
|
||
- vs `random`: R0에서 +49 (제압)하던 상대에게 R1은 -9 (패배). random은
|
||
카드를 두기라도 하니까 음수 expedition으로 끝나도 양수 게임 일부 만들어서 점수
|
||
내지만, R1 모델은 0점 (전혀 안 둠).
|
||
- vs `heuristic_cautious`: opened_colors 절반(4.93→2.54)으로 줄었지만
|
||
play_action_rate 0% — 연 expedition을 채우지 못함. avg_game_length
|
||
5x 증가 → max_steps timeout 다발. score_diff 22점 더 악화.
|
||
- Action 분포 (vs cautious): **discard 50% / draw_pile 50% / play 0% /
|
||
draw_deck 0%**. 즉 "버리고 상대 버린 거 줍기"의 단일 패턴.
|
||
- Loss 트렌드: advantage loss 1500 → 3500 (꾸준히 상승), strategy loss
|
||
0.7 → 1.4 (꾸준히 상승). 모델이 점점 더 noisy한 target을 못 fit함.
|
||
|
||
### What this proves
|
||
|
||
1. **Trap의 본질은 모델 자체 value function 실패 (credit assignment)**, not
|
||
self-reference. self-reference는 trap 강화 요인이지만 제거해도 다른 형태로
|
||
재출현.
|
||
2. **모델이 일관되게 못하는 일은 "expedition에 카드 두기" 행동**. R0에서
|
||
over-open 무차별 5색 + 채우지 못함. R1에서 아예 안 열되 채우지도 않음.
|
||
양 극단 어디든 play_action_rate ≈ 0%.
|
||
3. **Outcome sampling + 본인 정책 rollout만으로 long-horizon credit chain을
|
||
못 닫는다.** "open이 좋은가?"의 답은 "이후 카드를 잘 두는가"에 종속이고,
|
||
모델이 카드 두기를 못 배우는 한 open 가치는 노이즈로만 추정됨.
|
||
|
||
### Recommended next direction
|
||
|
||
이전에 hygiene 개선(R1 = 4M capacity + 500 eval)을 첫 후보로 두었으나, R1의
|
||
명확한 결과로 hygiene으로는 trap 본질 못 풀 것이 거의 확정. **R2는 trap에 직접
|
||
작용하는 변경이어야 함.**
|
||
|
||
후보 (강도 약→강):
|
||
|
||
1. **Curriculum (작은 게임 → 큰 게임)**: 3색 5랭크 mini Lost Cities로 시작 →
|
||
credit chain 짧아짐 → expedition 사이클이 게임 8턴 안에 닫힘. 모델이
|
||
"카드 두기" 학습 가능성 확보. 풀 게임에 weight transfer.
|
||
2. **Opening / playing 네트워크 분리**: closed loop 회로의 한 회선 물리적
|
||
절단. opening 네트워크는 follow-up이 noise여도 자체 신호로 학습 가능.
|
||
아키텍처 작업 큼.
|
||
3. **Cutoff rollout heuristic + external opponent (조합)**: opening 가치
|
||
추정에 외부 합리적 정책 끼움. pure self-play 가설 명확히 폐기.
|
||
|
||
저자 추천: **R2 = curriculum**. 가장 적은 아키텍처 변경으로 credit chain
|
||
직접 단축. 작은 게임에서 follow-up 학습 성공이 풀 게임 학습의 핵심 prior가
|
||
됨.
|
||
|