coolguyandClaude Opus 4.7
004b913a7b
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
165 lines
6.4 KiB
Markdown
165 lines
6.4 KiB
Markdown
# Deep CFR Reproducibility
|
|
|
|
**Last verified:** 2026-05-08, commit `0f85fa8`
|
|
|
|
## Summary
|
|
|
|
Deep CFR training is deterministic for short single-worker checks, but the
|
|
default multi-worker traversal path is not bitwise reproducible across repeated
|
|
runs with the same seed.
|
|
|
|
The likely source is multiprocessing result ordering, not evaluation. With
|
|
`traversal.num_workers=1`, repeated runs matched exactly for core training and
|
|
evaluation metrics across iterations 1-3. With `traversal.num_workers=1`,
|
|
inserting evaluation every iteration did not change the training trajectory.
|
|
|
|
## Evidence
|
|
|
|
### Multi-worker runs diverged despite matching seed and config
|
|
|
|
Two 512x3 runs used the same resolved training config except for
|
|
`run.experiment_name`, `run.max_iterations`, and `evaluation.eval_every`:
|
|
|
|
- `runs/2026-05-08_022808_model-size-512x3`
|
|
- `runs/2026-05-08_051124_baseline-512x3-2000-dense-eval`
|
|
|
|
They matched on iteration 1 traversal size and memory size, then diverged from
|
|
iteration 2:
|
|
|
|
| Iteration | Metric | model-size-512x3 | baseline-512x3-2000 |
|
|
| --- | --- | ---: | ---: |
|
|
| 1 | `traversal/nodes` | 169976 | 169976 |
|
|
| 1 | `memory/advantage` | 84664 | 84664 |
|
|
| 2 | `traversal/nodes` | 157664 | 173489 |
|
|
| 2 | `memory/advantage` | 163292 | 171205 |
|
|
|
|
This divergence happens before either run reaches its first evaluation point in
|
|
the 200-iteration model-size run, so evaluation frequency does not explain the
|
|
initial split.
|
|
|
|
### Single-worker repeated runs matched
|
|
|
|
Two temporary runs used:
|
|
|
|
- `traversal.num_workers=1`
|
|
- `run.max_iterations=3`
|
|
- `evaluation.eval_every=1`
|
|
- `evaluation.games=20`
|
|
- `evaluation.opponents=[random,heuristic_cautious]`
|
|
- W&B disabled
|
|
|
|
Runs:
|
|
|
|
- `runs/tmp/2026-05-08_152357_repro-single-worker-a`
|
|
- `runs/tmp/2026-05-08_152448_repro-single-worker-b`
|
|
|
|
Core metrics matched exactly:
|
|
|
|
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` |
|
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
|
|
| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
|
|
| 3 | 142128 | 258823 | 1089.841465830803 | 0.50 | 0.00 |
|
|
|
|
Timing counters differed, as expected.
|
|
|
|
### Evaluation did not perturb single-worker training
|
|
|
|
Two temporary runs compared eval disabled vs. eval every iteration:
|
|
|
|
- `runs/tmp/2026-05-08_152813_repro-single-worker-no-eval`
|
|
- `runs/tmp/2026-05-08_152852_repro-single-worker-with-eval`
|
|
|
|
With `traversal.num_workers=1`, common non-timing, non-eval fields matched
|
|
exactly across iterations 1-3. Evaluation added eval metrics and wall-clock
|
|
cost, but did not change:
|
|
|
|
- `traversal/nodes`
|
|
- `memory/advantage`
|
|
- `memory/strategy`
|
|
- `samples/advantage`
|
|
- `samples/strategy`
|
|
- `loss/advantage`
|
|
- `loss/strategy`
|
|
|
|
## Likely Cause
|
|
|
|
The parallel traversal path processes worker results in completion order.
|
|
`DeepCFRTrainer._run_traversals_parallel` waits for `FIRST_COMPLETED` futures,
|
|
then immediately adds the completed batch's samples into reservoir memory
|
|
(`src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py:548`).
|
|
|
|
Reservoir insertion is order-sensitive because each sample increments `seen`,
|
|
and capacity replacement draws from the trainer RNG
|
|
(`src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:26`). Even before
|
|
capacity is reached, list order affects later sampled batches because memory
|
|
sampling draws indices from the same RNG
|
|
(`src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:57`).
|
|
|
|
Therefore two runs can share the same seeds and configs but diverge if worker
|
|
completion order differs due to OS scheduling, process timing, or device timing.
|
|
|
|
## Implications
|
|
|
|
- Same seed does not guarantee bitwise reproducibility for default
|
|
multi-worker Deep CFR training.
|
|
- Short deterministic checks should use `traversal.num_workers=1`.
|
|
- Multi-worker experiment comparisons should be interpreted as stochastic
|
|
repeated runs, even when `run.seed` is identical.
|
|
- Eval frequency is not currently implicated in training trajectory divergence,
|
|
based on the single-worker eval/no-eval check above.
|
|
|
|
## Proposed Fix
|
|
|
|
The current implementation target is stronger than same-worker-count stability:
|
|
worker count should be a performance setting, not an algorithm setting. On the
|
|
same code/config/seed and same GPU/CUDA stack, `traversal.num_workers=1`, `4`,
|
|
and `8` should produce the same non-timing metrics.
|
|
|
|
Implement that by making the logical traversal stream independent of process
|
|
scheduling:
|
|
|
|
1. Assign a stable traversal ID to every traversal, such as
|
|
`(iteration, player, traversal_index)`.
|
|
2. Derive all traversal-local RNG streams from that traversal ID and a purpose
|
|
token, not from worker ID, batch ID, or completion order.
|
|
3. Keep a canonical logical traversal list for each iteration. `num_workers`
|
|
should only decide how that list is partitioned for execution.
|
|
4. Return samples and stats with their traversal IDs.
|
|
5. Buffer completed futures for an iteration.
|
|
6. Insert `advantage_samples` and `strategy_samples` into memory in sorted
|
|
traversal ID order.
|
|
7. Accumulate stats in the same sorted traversal ID order.
|
|
8. Stabilize interleaved scheduler request/context ordering so ready queue and
|
|
policy request processing do not depend on set/dict iteration or worker
|
|
timing.
|
|
9. Enable PyTorch deterministic settings for reproducibility/debug runs:
|
|
`torch.use_deterministic_algorithms(True)`,
|
|
`torch.backends.cudnn.benchmark = False`,
|
|
`torch.backends.cuda.matmul.allow_tf32 = False`, and
|
|
`torch.backends.cudnn.allow_tf32 = False`.
|
|
10. Re-run the repeated-seed check across `traversal.num_workers=1`, `4`, and
|
|
`8`.
|
|
11. If differences remain, inspect remaining CPU-side ordering and PyTorch
|
|
operator-level nondeterminism.
|
|
|
|
This is a larger change than sorting completed worker batches, but it is the
|
|
right target if same GPU/CUDA runs should remain stable across different CPU
|
|
machines and different worker counts.
|
|
|
|
## Batch Size Scope
|
|
|
|
The deterministic target treats traversal and evaluation inference batch sizes
|
|
as execution details:
|
|
|
|
- `traversal.interleave_max_batch`
|
|
- `evaluation.batch_size`
|
|
|
|
Changing these should not change non-timing metrics once traversal IDs, request
|
|
ordering, and result merge order are stable.
|
|
|
|
Training mini-batch sizes are different. Changing
|
|
`optimization.advantage_batch_size` or `optimization.strategy_batch_size`
|
|
changes the gradient estimate and optimizer trajectory, so those values remain
|
|
ordinary experimental variables. They are not expected to match across runs.
|