diff --git a/docs/plans/deep-cfr-selectivity.md b/docs/plans/deep-cfr-selectivity.md new file mode 100644 index 0000000..6d42581 --- /dev/null +++ b/docs/plans/deep-cfr-selectivity.md @@ -0,0 +1,188 @@ +# Deep CFR Selectivity Investigation + +Last updated: 2026-05-09 + +## Current conclusion + +The current Deep CFR baseline is not mainly blocked by model size or LCFR +time-weighting. The strongest signal so far is a selectivity failure: the model +does not reliably learn to distinguish good first opens from bad first opens. + +The `outcome_sampling_epsilon=0.05` setting produced a short-run improvement +around 200 iterations, but the improvement did not hold through 500 iterations. +Weighting ablations did not recover the 200-iteration peak, so further LCFR +alpha tuning is lower priority than directly inspecting or changing the +first-open advantage target. + +## Baseline symptoms + +The 512x3 dense-eval baseline showed improving training losses, but the main +game-quality metrics against `safe_heuristic_strict` did not improve enough to +indicate a useful policy. + +Observed pattern: + +- Advantage loss can move in the expected direction while eval quality remains + poor. +- Bad first-open behavior stays high. +- Open quality, measured by score per opened color, stays negative. +- Current-policy evaluation did not rescue the result, so the issue is not only + average-policy lag. + +## Closed hypotheses + +### More outcome sampling helps, but only up to a point + +Short 200-iteration ablations with `outcome_unsampled_regret=zero`: + +| setting | safe strict score diff | win rate | bad open rate | score/opened color | +| --- | ---: | ---: | ---: | ---: | +| `epsilon=0.20` | -57.87 | 0.07 | 0.901 | -8.40 | +| `epsilon=0.10` | -58.66 | 0.05 | 0.913 | -8.63 | +| `epsilon=0.05` | -40.01 | 0.12 | 0.893 | -6.25 | +| `epsilon=0.02` | -53.55 | - | - | - | + +Conclusion: `epsilon=0.05` was the best short-run candidate. Lowering to +`0.02` was worse, and larger values were also worse. + +### Negative unsampled regret was not sufficient + +The `epsilon=0.05` plus `outcome_unsampled_regret=negative_node_value` variant +ended around safe strict score diff `-44.53` at 200 iterations, worse than +`epsilon=0.05` with `zero`. + +Conclusion: the negative target suppresses opening, but it does not selectively +preserve good opens. + +### The 200-iteration epsilon=0.05 peak did not hold + +Confirmation run: + +- Run: `runs/2026-05-09_010747_confirm-eps-005-zero-512x3-det-500` +- W&B group: `eps005-confirmation-512x3-v1` +- Config delta: + - `run.deterministic=true` + - `run.max_iterations=500` + - `traversal.outcome_sampling_epsilon=0.05` + - `traversal.outcome_unsampled_regret=zero` + +Safe heuristic strict metrics: + +| iteration | win rate | score diff | bad open rate | score/opened color | +| ---: | ---: | ---: | ---: | ---: | +| 200 | 0.12 | -40.01 | 0.893 | -6.25 | +| 250 | 0.08 | -46.91 | 0.900 | -7.29 | +| 300 | 0.08 | -45.93 | 0.895 | -6.30 | +| 350 | 0.07 | -54.02 | 0.899 | -7.83 | +| 400 | 0.04 | -49.64 | 0.912 | -7.83 | +| 450 | 0.06 | -49.31 | 0.917 | -6.71 | +| 500 | 0.06 | -61.24 | 0.903 | -8.30 | + +Conclusion: `epsilon=0.05` creates a real short-run peak, but the behavior is +not stable through 500 iterations. + +### LCFR time-weighting is not the sole cause of the degradation + +Weighting ablation group: + +- W&B group: `eps005-weighting-ablation-512x3-v1` +- Common config: + - `run.deterministic=true` + - `run.max_iterations=300` + - `traversal.outcome_sampling_epsilon=0.05` + - `traversal.outcome_unsampled_regret=zero` + +Compared runs: + +| run | iter 200 diff | iter 250 diff | iter 300 diff | iter 300 win | iter 300 bad open | +| --- | ---: | ---: | ---: | ---: | ---: | +| LCFR alpha=1.0 confirmation | -40.01 | -46.91 | -45.93 | 0.08 | 0.895 | +| `training_weighting.mode=none` | -50.98 | -48.46 | -56.27 | 0.04 | 0.912 | +| `training_weighting.lcfr_alpha=0.5` | -66.81 | -58.97 | -50.77 | 0.10 | 0.891 | + +Open-quality comparison: + +| run | iter 200 score/opened color | iter 300 score/opened color | +| --- | ---: | ---: | +| LCFR alpha=1.0 confirmation | -6.25 | -6.30 | +| `training_weighting.mode=none` | -7.71 | -7.70 | +| `training_weighting.lcfr_alpha=0.5` | -9.94 | -7.85 | + +Conclusion: neither removing time-weighting nor softening LCFR to `alpha=0.5` +beat the original `alpha=1.0` confirmation by the primary score-diff metric at +300 iterations. LCFR may affect stability, but it is not the main lever. + +## First-open diagnostic + +Diagnostic output: + +- `runs/tmp/first_open_advantage_confirm_eps005_200_vs_500.jsonl` + +Observed values: + +| checkpoint | bad selection | good selection | bad advantage | good advantage | sampled turns | +| --- | ---: | ---: | ---: | ---: | ---: | +| iter 200 | 0.0625 | 0.0582 | -12.82 | -16.60 | 8560 | +| iter 500 | 0.0067 | 0.0067 | -35.42 | -44.32 | 40942 | + +Interpretation: by 500 iterations the model strongly suppresses opening +overall. It suppresses good opens along with bad opens, which is the core +selectivity failure. + +## Open questions + +1. Does the traversal target itself provide separable labels for good first + opens versus bad first opens? +2. Is the model receiving too sparse or too noisy a signal at the first-open + decision point? +3. Would target shaping around first-open decisions improve score/opened color + without increasing bad-open rate? +4. Is evaluation showing a policy-selection problem, or is the advantage model + already misranking good and bad opens before strategy extraction? + +## Recommended next experiments + +### 1. Deeper first-open target audit + +Before another long training run, inspect sampled first-open decision records +more directly: + +- Group candidate first-open actions into good-open and bad-open buckets. +- Compare target values before model prediction, not only final learned + advantages. +- Report distributions, not just means. +- Check whether the target ranks good opens above bad opens in the same + information-state context family. + +Success criterion: the target distribution should show a usable separation +between good and bad opens. If it does not, the training target is the blocker. + +### 2. Short open-selectivity ablation + +Run a 200-300 iteration ablation only after the target audit identifies a +specific change. Candidate changes include: + +- first-open target shaping, +- modified unsampled-open penalty, +- outcome sampling focused on open-relevant branches, +- or strategy/eval selection that separates current and average policy at the + open decision. + +Primary metrics: + +- `eval/safe_heuristic_strict/avg_score_diff0` +- `eval/safe_heuristic_strict/win_rate0` +- `eval/safe_heuristic_strict/bad_open_rate` +- `eval/safe_heuristic_strict/score_per_opened_color` + +Do not promote to 500+ iterations unless bad-open rate and score/opened color +both improve without degrading score diff. + +## Operational notes + +- Keep one GPU training run active at a time. +- Use `tmux` plus `.compute.lock` for training. +- Mirror real experiments to W&B. +- Use one W&B group per hypothesis family. +- Do not start a long 2000-iteration run until a short diagnostic run shows + stable selectivity improvement.