Rename bot family, curate analyze plots, tier evaluation cadence
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -17,7 +17,7 @@ first-open advantage target.
|
||||
## Baseline symptoms
|
||||
|
||||
The 512x3 dense-eval baseline showed improving training losses, but the main
|
||||
game-quality metrics against `safe_heuristic_strict` did not improve enough to
|
||||
game-quality metrics against `heuristic_cautious` did not improve enough to
|
||||
indicate a useful policy.
|
||||
|
||||
Observed pattern:
|
||||
@@ -176,7 +176,7 @@ against the current policy's best non-open action from the same state.
|
||||
|
||||
`delta_open = value(force open) - value(best non-open)`.
|
||||
|
||||
Counterfactual summary against `safe_heuristic_strict`:
|
||||
Counterfactual summary against `heuristic_cautious`:
|
||||
|
||||
| checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -257,7 +257,7 @@ first-open sampling scanned the full replay memory and pushed iteration time
|
||||
above 60 seconds. The indexed-memory version kept first-open sampling near
|
||||
0.25 seconds per player at 4M advantage samples and completed 500 iterations.
|
||||
|
||||
Final `safe_heuristic_strict` comparison:
|
||||
Final `heuristic_cautious` comparison:
|
||||
|
||||
| Run | Iter | Score diff | Win rate | Bad open | Score/opened |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -331,15 +331,15 @@ Result (2026-05-10):
|
||||
- W&B: synced online to group `first-open-prior-v1` (run `teuh915r`)
|
||||
- Commit: see HEAD at run start
|
||||
|
||||
`safe_heuristic_strict` at iter 200:
|
||||
`heuristic_cautious` at iter 200:
|
||||
|
||||
| Run | Score diff | Win rate | Bad open | Score/opened |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 |
|
||||
| **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** |
|
||||
|
||||
Other opponents at iter 200: random +32.51 / 0.86, noisy_safe -71.52 / 0.07,
|
||||
safe_heuristic -81.81 / 0.02, safe_heuristic_loose -81.24 / 0.05.
|
||||
Other opponents at iter 200: random +32.51 / 0.86, heuristic_noisy -71.52 / 0.07,
|
||||
heuristic_balanced -81.81 / 0.02, heuristic_aggressive -81.24 / 0.05.
|
||||
|
||||
Conclusion: **mixed result, net regression.** The prior did shift behavior
|
||||
in the intended direction on one axis — `bad_open_rate` dropped from 0.893
|
||||
@@ -382,7 +382,7 @@ post-action play (selection bias)?
|
||||
|
||||
Method: re-ran `analyze_first_open_counterfactual.py` on the
|
||||
`confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter
|
||||
500), but with `--post-policy safe_heuristic_strict`. The opponent and
|
||||
500), but with `--post-policy heuristic_cautious`. The opponent and
|
||||
state-collection policy stayed the same; only the policy_player's actions
|
||||
*after* the forced first action used the strong fixed bot.
|
||||
|
||||
@@ -434,7 +434,7 @@ Implications:
|
||||
|
||||
Candidate next directions (decision pending):
|
||||
|
||||
- (E1) Train with `cutoff_rollout_policy=safe_heuristic` instead of `random`.
|
||||
- (E1) Train with `cutoff_rollout_policy=heuristic_balanced` instead of `random`.
|
||||
Already a config option; gives leaf nodes stronger value estimates
|
||||
during traversal. Trades some pure-self-play purity for a stronger
|
||||
bootstrap signal. Cheap to test.
|
||||
@@ -510,7 +510,7 @@ This combined with D1 means:
|
||||
|
||||
Updated next-step priorities:
|
||||
|
||||
- **(E1) `cutoff_rollout_policy=safe_heuristic` training ablation.**
|
||||
- **(E1) `cutoff_rollout_policy=heuristic_balanced` training ablation.**
|
||||
Strongest single lever: gives traversal leaves stronger value estimates
|
||||
during training, which should ripple back to "open + follow up" signal.
|
||||
Pure-self-play purity dented, but only at cutoff leaves.
|
||||
@@ -635,7 +635,7 @@ Fix:
|
||||
|
||||
Also bumped `traversal.outcome_sampling_epsilon` in `default.yaml` from
|
||||
0.2 to 0.05. The 200-iteration sweep (section 1) showed 0.05 produced the
|
||||
best short-run safe_heuristic_strict score diff (-40.01 vs -57.87 for
|
||||
best short-run heuristic_cautious score diff (-40.01 vs -57.87 for
|
||||
0.20). All recent experimental runs already used 0.05; the default now
|
||||
matches actual experimental practice.
|
||||
|
||||
@@ -643,7 +643,71 @@ These changes do not target the diagnosed selection-bias bottleneck. They
|
||||
align config intent with actual scheduler behaviour and make the default
|
||||
config reproduce known-best knob settings out of the box.
|
||||
|
||||
### 9. Short open-selectivity ablation
|
||||
### 9. Naming, plot curation, and tiered eval cadence (2026-05-10)
|
||||
|
||||
Hygiene changes — none target the diagnosed selection-bias bottleneck,
|
||||
but they make the codebase honestly reflect the pure-self-play stance
|
||||
and reduce dashboard noise.
|
||||
|
||||
Bot family rename (drop the unhelpful `safe_` prefix; suffixes now
|
||||
describe behaviour):
|
||||
|
||||
| Old | New |
|
||||
| --- | --- |
|
||||
| `safe_heuristic_loose` | `heuristic_aggressive` |
|
||||
| `safe_heuristic` | `heuristic_balanced` |
|
||||
| `safe_heuristic_strict` | `heuristic_cautious` |
|
||||
| `noisy_safe` | `heuristic_noisy` |
|
||||
| `passive_discard` | `discard_only` |
|
||||
|
||||
Class renames in `bots/`: `SafeHeuristicBot` → `HeuristicBot`,
|
||||
`SafeHeuristicParams` → `HeuristicParams`, `PassiveDiscardBot` →
|
||||
`DiscardOnlyBot`, plus the loose/strict parameter constants. Backwards
|
||||
compatibility was dropped intentionally — no aliases. Active configs,
|
||||
docs, scripts, tests updated; archive files (read-only by policy)
|
||||
left intact and may still reference old names.
|
||||
|
||||
Analyze plot curation (`deep_cfr/analyze.py`):
|
||||
|
||||
- New `analysis_00_core.png` dashboard with 10 heuristic-free metrics
|
||||
(loss/{advantage, strategy}; vs `heuristic_cautious`:
|
||||
`avg_score_diff0`, `win_rate0`, `avg_opened_colors`,
|
||||
`positive_expedition_rate`, `bonus_expedition_rate`,
|
||||
`score_per_opened_color`, `policy_entropy`; vs `random`: `win_rate0`).
|
||||
- Removed `analysis_05_open_quality.png` (bad/weak/good open rates,
|
||||
recoverable score) and `analysis_07_calibration.png` (calibration gap,
|
||||
recoverable mean) — both relied on the heuristic `recoverable_score`
|
||||
classifier we already dropped from inputs.
|
||||
- Removed `SELECTIVITY_PLOTS` and `plot_selectivity` (heuristic-laden).
|
||||
- `SUMMARY_EVAL_METRICS` no longer includes `bad_open_rate` or
|
||||
`calibration_gap`.
|
||||
|
||||
`PlotSpec` gained an optional `opponents` allowlist so the new core
|
||||
section can pin a specific opponent per panel without restructuring the
|
||||
existing `plot_section` plumbing.
|
||||
|
||||
Tiered evaluation cadence (`EvaluationConfig`):
|
||||
|
||||
- Added `extended_opponents: tuple[str, ...]` and `extended_eval_every:
|
||||
int = 0`.
|
||||
- Method `opponents_for_iteration(iteration)` returns the core list at
|
||||
every `eval_every`, and appends `extended_opponents` (de-duplicated)
|
||||
when `iteration` is also a multiple of `extended_eval_every`.
|
||||
- `default.yaml` now uses 3 core opponents
|
||||
(`random`, `discard_only`, `heuristic_cautious`) every 5 iterations
|
||||
and 3 extended opponents (`heuristic_balanced`, `heuristic_aggressive`,
|
||||
`heuristic_noisy`) every 50 iterations.
|
||||
- `random` is the floor sanity. `discard_only` is the zero-pit
|
||||
detector / absolute-score reference (its score is always 0, so
|
||||
`eval/discard_only/avg_score_diff0` directly equals our model's
|
||||
raw average score). `heuristic_cautious` is the ceiling and the
|
||||
archive-comparable benchmark used in sections 1–6.
|
||||
|
||||
Net effect on ongoing eval cost: ~50% reduction (3 opponents × every
|
||||
5 iter, plus 6 opponents × every 50 iter, vs the prior 6 opponents
|
||||
× every 5).
|
||||
|
||||
### 10. Short open-selectivity ablation
|
||||
|
||||
Run a 200-300 iteration ablation only after the target audit identifies a
|
||||
specific change. Candidate changes include:
|
||||
@@ -656,10 +720,10 @@ specific change. Candidate changes include:
|
||||
|
||||
Primary metrics:
|
||||
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0`
|
||||
- `eval/safe_heuristic_strict/win_rate0`
|
||||
- `eval/safe_heuristic_strict/bad_open_rate`
|
||||
- `eval/safe_heuristic_strict/score_per_opened_color`
|
||||
- `eval/heuristic_cautious/avg_score_diff0`
|
||||
- `eval/heuristic_cautious/win_rate0`
|
||||
- `eval/heuristic_cautious/bad_open_rate`
|
||||
- `eval/heuristic_cautious/score_per_opened_color`
|
||||
|
||||
Do not promote to 500+ iterations unless bad-open rate and score/opened color
|
||||
both improve without degrading score diff.
|
||||
|
||||
@@ -39,7 +39,7 @@ The experiment must produce either a recommended new `network` config or a docum
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. At least one tested config produces win-rate trajectories vs `safe_heuristic_strict` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
|
||||
1. At least one tested config produces win-rate trajectories vs `heuristic_cautious` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
|
||||
2. `iteration_seconds`, `traversal_seconds`, `advantage_train_seconds`, `strategy_train_seconds`, and `policy_network_seconds` (eval) are captured for each tested size and written to `docs/performance.md`.
|
||||
3. A recommended `network` config emerges from the data, OR the experiment documents why the current size should be kept, with specific rationale.
|
||||
|
||||
@@ -72,7 +72,7 @@ configs/deep_cfr/model-size-1536x8.yaml
|
||||
- `traversal_seconds` — traversal phase.
|
||||
- `advantage_train_seconds` — advantage network optimization.
|
||||
- `strategy_train_seconds` — strategy network optimization.
|
||||
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `safe_heuristic_strict`.
|
||||
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `heuristic_cautious`.
|
||||
- At eval iterations: `eval/<opponent>/policy_network_seconds` — needed for the AMP/TRT prerequisite check.
|
||||
- **Memory monitoring:** watch GPU VRAM during the 1024x6 and 1536x8 runs. If a run OOMs or VRAM > 20 GB, reduce `optimization.advantage_batch_size` and `optimization.strategy_batch_size` by half (1024 → 512) and note the change in the results table. Do not adjust traversal settings.
|
||||
|
||||
@@ -259,8 +259,8 @@ if non_eval:
|
||||
eval_rows = {r['iteration']: r for r in rows if r.get('evaluation_seconds')}
|
||||
for it in [50, 100, 150, 200]:
|
||||
if it in eval_rows:
|
||||
wr = eval_rows[it].get('eval/safe_heuristic_strict/win_rate', 'n/a')
|
||||
print(f' iter={it} safe_heuristic_strict win_rate={wr}')
|
||||
wr = eval_rows[it].get('eval/heuristic_cautious/win_rate', 'n/a')
|
||||
print(f' iter={it} heuristic_cautious win_rate={wr}')
|
||||
"
|
||||
done
|
||||
```
|
||||
@@ -269,7 +269,7 @@ done
|
||||
|
||||
After the grid completes, append a date-stamped experiment subsection to `docs/performance.md` under the "Experiments" heading. The subsection must include:
|
||||
|
||||
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `safe_heuristic_strict` at {50, 100, 150, 200} for each config.
|
||||
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `heuristic_cautious` at {50, 100, 150, 200} for each config.
|
||||
- A `policy_network_seconds` column from eval rows — this is the key data for the AMP/compile/TRT prerequisite check.
|
||||
- The recommendation that follows from the decision tree below.
|
||||
|
||||
@@ -302,7 +302,7 @@ Apply this logic after the grid completes:
|
||||
|
||||
### Branch A — a size unlocks the curve AND iter time is acceptable
|
||||
|
||||
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `safe_heuristic_strict` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
|
||||
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `heuristic_cautious` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
|
||||
|
||||
**Action:**
|
||||
1. Recommend that config as the new `network` default.
|
||||
|
||||
@@ -152,7 +152,7 @@ Cython production rewrite. It intentionally uses per-context RNG so interleaved
|
||||
execution order does not change the random stream for another context. That
|
||||
lets the prototype assert value/stat/sample parity against a recursive prototype
|
||||
while measuring realized batch size. Production Cython parity is a later Phase 2
|
||||
gate because the real path also has safe-heuristic opponents, average-strategy
|
||||
gate because the real path also has heuristic-balanced opponents, average-strategy
|
||||
opponents, self-play league snapshots, deck-draw chance sampling, external
|
||||
sampling, and cutoff rollouts.
|
||||
|
||||
@@ -312,7 +312,7 @@ Required feature expansion:
|
||||
|
||||
- Support `opponent_policy: average_strategy`, matching the default config's
|
||||
opponent branch.
|
||||
- Keep unsupported branches guarded (`self_play_league`, `safe_heuristic`,
|
||||
- Keep unsupported branches guarded (`self_play_league`, `heuristic_balanced`,
|
||||
random rollout cutoffs, external sampling).
|
||||
- Add parity tests for the average-strategy fixed-opponent branch.
|
||||
- Verify a default-policy interleaved run starts and emits batch metrics.
|
||||
|
||||
@@ -26,7 +26,7 @@ Re-enable `torch.compile` on the Deep CFR trainer's networks at a model size whe
|
||||
|
||||
1. **Model-size precondition met.** The active `default.yaml` (or the targeted variant) has `network.hidden_size ≥ 1024` *or* `network.num_layers ≥ 6`, *or* an architecture (e.g. `color_shared` with non-trivial `color_attention_layers`) whose per-call forward time exceeds ~150 μs at the trainer's training batch size on the target GPU. If neither condition holds, this plan is **not merged**; the branch is parked.
|
||||
2. **Iter-time improvement.** With `compile.trainer.enabled: true` on the chosen larger model and the same seed, the 1000-iter projection improves by at least **5%** vs the no-compile baseline on the same machine (measured on `home`). Eval and checkpointing should be disabled for the bench window, matching the protocol used in the 2026-05-07 experiment.
|
||||
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `safe_heuristic` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
|
||||
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `heuristic_balanced` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
|
||||
4. **No checkpoint-format break.** Checkpoints saved with compile enabled must load cleanly when compile is disabled, and vice versa. (Handled via `_clean_state_dict()`; see Risks.)
|
||||
5. **No multiprocessing-worker break.** Whether `inference_backend` is `local` or `server`, traversal workers must continue to receive uncompiled `state_dict`s without `_orig_mod.` prefixes.
|
||||
6. **(Secondary) Inference-server forward.** If step B below is taken, the server's `policy_network_seconds` decreases by at least 20% at the chosen model size, with no traversal-path correctness regression. If step B does not produce a measurable win, it is left disabled and the plan still ships with step A only.
|
||||
|
||||
Reference in New Issue
Block a user