coorl-lost-cities/docs/research/deep-cfr-reproducibility.md

Deep CFR Reproducibility

Last verified: 2026-05-08, commit 0f85fa8

Summary

Deep CFR training is deterministic for short single-worker checks, but the default multi-worker traversal path is not bitwise reproducible across repeated runs with the same seed.

The likely source is multiprocessing result ordering, not evaluation. With traversal.num_workers=1, repeated runs matched exactly for core training and evaluation metrics across iterations 1-3. With traversal.num_workers=1, inserting evaluation every iteration did not change the training trajectory.

Evidence

Multi-worker runs diverged despite matching seed and config

Two 512x3 runs used the same resolved training config except for run.experiment_name, run.max_iterations, and evaluation.eval_every:

• runs/2026-05-08_022808_model-size-512x3
• runs/2026-05-08_051124_baseline-512x3-2000-dense-eval

They matched on iteration 1 traversal size and memory size, then diverged from iteration 2:

Iteration	Metric	model-size-512x3	baseline-512x3-2000
1	traversal/nodes	169976	169976
1	memory/advantage	84664	84664
2	traversal/nodes	157664	173489
2	memory/advantage	163292	171205

This divergence happens before either run reaches its first evaluation point in the 200-iteration model-size run, so evaluation frequency does not explain the initial split.

Single-worker repeated runs matched

Two temporary runs used:

• traversal.num_workers=1
• run.max_iterations=3
• evaluation.eval_every=1
• evaluation.games=20
• evaluation.opponents=[random,heuristic_cautious]
• W&B disabled

Runs:

• runs/tmp/2026-05-08_152357_repro-single-worker-a
• runs/tmp/2026-05-08_152448_repro-single-worker-b

Core metrics matched exactly:

Iteration	traversal/nodes	memory/advantage	loss/advantage	eval/random/win_rate0	eval/heuristic_cautious/win_rate0
1	170822	85096	803.9475702643394	0.75	0.10
2	206225	187987	821.3355012834072	0.50	0.05
3	142128	258823	1089.841465830803	0.50	0.00

Timing counters differed, as expected.

Evaluation did not perturb single-worker training

Two temporary runs compared eval disabled vs. eval every iteration:

• runs/tmp/2026-05-08_152813_repro-single-worker-no-eval
• runs/tmp/2026-05-08_152852_repro-single-worker-with-eval

With traversal.num_workers=1, common non-timing, non-eval fields matched exactly across iterations 1-3. Evaluation added eval metrics and wall-clock cost, but did not change:

• traversal/nodes
• memory/advantage
• memory/strategy
• samples/advantage
• samples/strategy
• loss/advantage
• loss/strategy

Likely Cause

The parallel traversal path processes worker results in completion order. DeepCFRTrainer._run_traversals_parallel waits for FIRST_COMPLETED futures, then immediately adds the completed batch's samples into reservoir memory (src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py:548).

Reservoir insertion is order-sensitive because each sample increments seen, and capacity replacement draws from the trainer RNG (src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:26). Even before capacity is reached, list order affects later sampled batches because memory sampling draws indices from the same RNG (src/coolrl_lost_cities/games/classic/deep_cfr/memory.py:57).

Therefore two runs can share the same seeds and configs but diverge if worker completion order differs due to OS scheduling, process timing, or device timing.

Implications

• Same seed does not guarantee bitwise reproducibility for default multi-worker Deep CFR training.
• Short deterministic checks should use traversal.num_workers=1.
• Multi-worker experiment comparisons should be interpreted as stochastic repeated runs, even when run.seed is identical.
• Eval frequency is not currently implicated in training trajectory divergence, based on the single-worker eval/no-eval check above.

Proposed Fix

The current implementation target is stronger than same-worker-count stability: worker count should be a performance setting, not an algorithm setting. On the same code/config/seed and same GPU/CUDA stack, traversal.num_workers=1, 4, and 8 should produce the same non-timing metrics.

Implement that by making the logical traversal stream independent of process scheduling:

1. Assign a stable traversal ID to every traversal, such as (iteration, player, traversal_index).
2. Derive all traversal-local RNG streams from that traversal ID and a purpose token, not from worker ID, batch ID, or completion order.
3. Keep a canonical logical traversal list for each iteration. num_workers should only decide how that list is partitioned for execution.
4. Return samples and stats with their traversal IDs.
5. Buffer completed futures for an iteration.
6. Insert advantage_samples and strategy_samples into memory in sorted traversal ID order.
7. Accumulate stats in the same sorted traversal ID order.
8. Stabilize interleaved scheduler request/context ordering so ready queue and policy request processing do not depend on set/dict iteration or worker timing.
9. Enable PyTorch deterministic settings for reproducibility/debug runs: torch.use_deterministic_algorithms(True), torch.backends.cudnn.benchmark = False, torch.backends.cuda.matmul.allow_tf32 = False, and torch.backends.cudnn.allow_tf32 = False.
10. Re-run the repeated-seed check across traversal.num_workers=1, 4, and 8.
11. If differences remain, inspect remaining CPU-side ordering and PyTorch operator-level nondeterminism.

This is a larger change than sorting completed worker batches, but it is the right target if same GPU/CUDA runs should remain stable across different CPU machines and different worker counts.

Batch Size Scope

The deterministic target treats traversal and evaluation inference batch sizes as execution details:

• traversal.interleave_max_batch
• evaluation.batch_size

Changing these should not change non-timing metrics once traversal IDs, request ordering, and result merge order are stable.

Training mini-batch sizes are different. Changing optimization.advantage_batch_size or optimization.strategy_batch_size changes the gradient estimate and optimizer trajectory, so those values remain ordinary experimental variables. They are not expected to match across runs.