Document Deep CFR reproducibility policy

docs/research/deep-cfr-reproducibility-policy.md

@@ -0,0 +1,136 @@
+# Deep CFR Reproducibility Policy
+
+**Last verified:** 2026-05-08, commit `0f85fa8`
+**Source:** `docs/research/deep-cfr-reproducibility.md`
+
+## Policy
+
+Deep CFR experiments should target two different kinds of reproducibility:
+
+1. **Debug reproducibility:** short runs should be bitwise reproducible on the
+   same machine when using deterministic settings.
+2. **Research reproducibility:** reported conclusions should survive repeated
+   seeds and reasonable hardware differences, even when exact metric rows do
+   not match bit-for-bit.
+
+Do not treat a single seeded multi-worker GPU run as a final research result.
+Use it as an exploratory signal unless it is confirmed with matched seeds.
+
+## Hardware Expectations
+
+Exact same metrics are not guaranteed across machines, even with the same CUDA
+version. GPU model, driver, PyTorch build, CPU scheduling, multiprocessing
+timing, and floating-point reduction order can all affect exact trajectories.
+
+Expected reliability by setting:
+
+| Setting | Expected result |
+| --- | --- |
+| Same machine, same GPU, same code, `traversal.num_workers=1` | Best current option for bitwise debug checks |
+| Same machine, same GPU, same code, multi-worker traversal | Current implementation target: bitwise stable across repeated runs |
+| Same GPU/CUDA but different CPU | Current implementation target: same seed should follow the same logical trajectory |
+| Different GPU or PyTorch/CUDA build | Exact equality is not expected; use multi-seed conclusions |
+
+## Required Experiment Practice
+
+For exploratory experiments:
+
+- Single-seed runs are acceptable.
+- Record the resolved config, run directory, W&B URL when used, and final eval
+  metrics.
+- Label conclusions as provisional.
+
+For claims worth keeping:
+
+- Use matched seeds across baseline and treatment.
+- Use at least 3 seeds; prefer 5 when runtime allows.
+- Report mean and standard deviation for the key metrics.
+- Keep `run.seed`, commit SHA, GPU, CUDA/PyTorch versions, and
+  `traversal.num_workers` visible in the run record.
+
+Example matched-seed comparison:
+
+```text
+baseline:  seeds 79, 80, 81
+treatment: seeds 79, 80, 81
+```
+
+Compare the treatment against the baseline seed-by-seed, then report aggregate
+statistics.
+
+## Batch Size Interpretation
+
+Not every batch size is a reproducibility-neutral setting.
+
+Batching that should be algorithm-neutral under the deterministic target:
+
+- traversal/interleaved policy inference batch size, such as
+  `traversal.interleave_max_batch`
+- evaluation inference batch size, such as `evaluation.batch_size`
+
+These settings should affect throughput, not the logical trajectory. Same code,
+seed, hardware stack, and training config should produce the same non-timing
+metrics when only these batching knobs change.
+
+Batching that is an algorithmic training setting:
+
+- `optimization.advantage_batch_size`
+- `optimization.strategy_batch_size`
+- update counts or optimizer schedule parameters
+
+These settings change gradient estimates or optimizer updates. They are
+experiment variables, not reproducibility-neutral execution settings. A run with
+training mini-batch size 3 is not expected to match one with training mini-batch
+size 7.
+
+## Debug Mode
+
+Use debug mode when checking exact reproducibility or suspected regressions:
+
+- `traversal.num_workers=1`
+- `evaluation.num_workers=1`
+- PyTorch deterministic settings enabled:
+  - `torch.use_deterministic_algorithms(True)`
+  - `torch.backends.cudnn.benchmark = False`
+  - `torch.backends.cuda.matmul.allow_tf32 = False`
+  - `torch.backends.cudnn.allow_tf32 = False`
+- short `run.max_iterations`, usually 3
+- checkpoint writes disabled unless specifically needed
+- W&B disabled
+
+The expected check is to run the same command twice and compare non-timing
+metrics in `metrics.jsonl`, including:
+
+- `traversal/nodes`
+- `memory/advantage`
+- `memory/strategy`
+- `loss/advantage`
+- `loss/strategy`
+- eval scores, if evaluation is enabled
+
+Timing fields are not expected to match exactly.
+
+## Implementation Priorities
+
+Current implementation target:
+
+1. Add a small reproducibility smoke script that runs two short debug jobs and
+   compares non-timing metrics.
+2. Make worker count a performance setting, not an algorithm setting:
+   `traversal.num_workers=1`, `4`, and `8` should produce the same non-timing
+   metrics on the same code/config/seed.
+3. Assign stable traversal IDs, derive RNG streams from traversal IDs, and merge
+   samples by traversal order rather than worker, batch, or completion order.
+4. Stabilize interleaved scheduler request/context ordering so CPU scheduling
+   differences do not change the logical traversal path.
+5. Enable PyTorch deterministic settings for reproducibility/debug runs. Keep
+   the speed impact explicit when comparing runtime.
+6. Re-test same-seed runs across `traversal.num_workers=1`, `4`, and `8`.
+7. Re-test on the same GPU/CUDA stack with a different CPU when available.
+
+## Current Interpretation Rule
+
+Until traversal-ID based deterministic scheduling and merge are implemented,
+default multi-worker runs with the same seed may diverge. Interpret exact curves
+cautiously. For research conclusions, prefer matched multi-seed comparisons over
+bitwise row matching.

docs/research/deep-cfr-reproducibility.md

@@ -0,0 +1,164 @@
+# Deep CFR Reproducibility
+
+**Last verified:** 2026-05-08, commit `0f85fa8`
+
+## Summary
+
+Deep CFR training is deterministic for short single-worker checks, but the
+default multi-worker traversal path is not bitwise reproducible across repeated
+runs with the same seed.
+
+The likely source is multiprocessing result ordering, not evaluation. With
+`traversal.num_workers=1`, repeated runs matched exactly for core training and
+evaluation metrics across iterations 1-3. With `traversal.num_workers=1`,
+inserting evaluation every iteration did not change the training trajectory.
+
+## Evidence
+
+### Multi-worker runs diverged despite matching seed and config
+
+Two 512x3 runs used the same resolved training config except for
+`run.experiment_name`, `run.max_iterations`, and `evaluation.eval_every`:
+
+- `runs/2026-05-08_022808_model-size-512x3`
+- `runs/2026-05-08_051124_baseline-512x3-2000-dense-eval`
+
+They matched on iteration 1 traversal size and memory size, then diverged from
+iteration 2:
+
+| Iteration | Metric | model-size-512x3 | baseline-512x3-2000 |
+| --- | --- | ---: | ---: |
+| 1 | `traversal/nodes` | 169976 | 169976 |
+| 1 | `memory/advantage` | 84664 | 84664 |
+| 2 | `traversal/nodes` | 157664 | 173489 |
+| 2 | `memory/advantage` | 163292 | 171205 |
+
+This divergence happens before either run reaches its first evaluation point in
+the 200-iteration model-size run, so evaluation frequency does not explain the
+initial split.
+
+### Single-worker repeated runs matched
+
+Two temporary runs used:
+
+- `traversal.num_workers=1`
+- `run.max_iterations=3`
+- `evaluation.eval_every=1`
+- `evaluation.games=20`
+- `evaluation.opponents=[random,safe_heuristic_strict]`
+- W&B disabled
+
+Runs:
+
+- `runs/tmp/2026-05-08_152357_repro-single-worker-a`
+- `runs/tmp/2026-05-08_152448_repro-single-worker-b`
+
+Core metrics matched exactly:
+
+| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
+| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
+| 3 | 142128 | 258823 | 1089.841465830803 | 0.50 | 0.00 |
+
+Timing counters differed, as expected.
+
+### Evaluation did not perturb single-worker training
+
+Two temporary runs compared eval disabled vs. eval every iteration:
+
+- `runs/tmp/2026-05-08_152813_repro-single-worker-no-eval`
+- `runs/tmp/2026-05-08_152852_repro-single-worker-with-eval`
+
+With `traversal.num_workers=1`, common non-timing, non-eval fields matched
+exactly across iterations 1-3. Evaluation added eval metrics and wall-clock
+cost, but did not change:
+
+- `traversal/nodes`
+- `memory/advantage`
+- `memory/strategy`
+- `samples/advantage`
+- `samples/strategy`
+- `loss/advantage`
+- `loss/strategy`
+
+## Likely Cause
+
+The parallel traversal path processes worker results in completion order.
+`DeepCFRTrainer._run_traversals_parallel` waits for `FIRST_COMPLETED` futures,
+then immediately adds the completed batch's samples into reservoir memory
+(`src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py:548`).
+
+Reservoir insertion is order-sensitive because each sample increments `seen`,
+and capacity replacement draws from the trainer RNG
+(`src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:26`). Even before
+capacity is reached, list order affects later sampled batches because memory
+sampling draws indices from the same RNG
+(`src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:57`).
+
+Therefore two runs can share the same seeds and configs but diverge if worker
+completion order differs due to OS scheduling, process timing, or device timing.
+
+## Implications
+
+- Same seed does not guarantee bitwise reproducibility for default
+  multi-worker Deep CFR training.
+- Short deterministic checks should use `traversal.num_workers=1`.
+- Multi-worker experiment comparisons should be interpreted as stochastic
+  repeated runs, even when `run.seed` is identical.
+- Eval frequency is not currently implicated in training trajectory divergence,
+  based on the single-worker eval/no-eval check above.
+
+## Proposed Fix
+
+The current implementation target is stronger than same-worker-count stability:
+worker count should be a performance setting, not an algorithm setting. On the
+same code/config/seed and same GPU/CUDA stack, `traversal.num_workers=1`, `4`,
+and `8` should produce the same non-timing metrics.
+
+Implement that by making the logical traversal stream independent of process
+scheduling:
+
+1. Assign a stable traversal ID to every traversal, such as
+   `(iteration, player, traversal_index)`.
+2. Derive all traversal-local RNG streams from that traversal ID and a purpose
+   token, not from worker ID, batch ID, or completion order.
+3. Keep a canonical logical traversal list for each iteration. `num_workers`
+   should only decide how that list is partitioned for execution.
+4. Return samples and stats with their traversal IDs.
+5. Buffer completed futures for an iteration.
+6. Insert `advantage_samples` and `strategy_samples` into memory in sorted
+   traversal ID order.
+7. Accumulate stats in the same sorted traversal ID order.
+8. Stabilize interleaved scheduler request/context ordering so ready queue and
+   policy request processing do not depend on set/dict iteration or worker
+   timing.
+9. Enable PyTorch deterministic settings for reproducibility/debug runs:
+   `torch.use_deterministic_algorithms(True)`,
+   `torch.backends.cudnn.benchmark = False`,
+   `torch.backends.cuda.matmul.allow_tf32 = False`, and
+   `torch.backends.cudnn.allow_tf32 = False`.
+10. Re-run the repeated-seed check across `traversal.num_workers=1`, `4`, and
+   `8`.
+11. If differences remain, inspect remaining CPU-side ordering and PyTorch
+    operator-level nondeterminism.
+
+This is a larger change than sorting completed worker batches, but it is the
+right target if same GPU/CUDA runs should remain stable across different CPU
+machines and different worker counts.
+
+## Batch Size Scope
+
+The deterministic target treats traversal and evaluation inference batch sizes
+as execution details:
+
+- `traversal.interleave_max_batch`
+- `evaluation.batch_size`
+
+Changing these should not change non-timing metrics once traversal IDs, request
+ordering, and result merge order are stable.
+
+Training mini-batch sizes are different. Changing
+`optimization.advantage_batch_size` or `optimization.strategy_batch_size`
+changes the gradient estimate and optimizer trajectory, so those values remain
+ordinary experimental variables. They are not expected to match across runs.