Adds torch.autocast(fp16) + GradScaler around _train_advantage and
_train_strategy when run.use_amp=true and device=cuda. CPU/non-CUDA
falls back to fp32 no-op. Mitigations:
- scaler.unscale_(optimizer) before grad_clip.
- nonfinite-loss guard skips overflowing batches and counts them.
- diff.float().square() in advantage loss to avoid fp16 overflow.
- strategy mask/log_softmax kept in fp32.
New metrics: amp/grad_scale, amp/nonfinite_loss_count.
Tests: AMP CUDA smoke + CPU fallback in test_deep_cfr_trainer.py.
Bench: scripts/bench_amp_trainer.py micro-benches train phases under
synthetic replay memory. smoke.yaml result is fp32 3.22ms / AMP 3.92ms
(0.82×, regression). 100-iter A/B on default.yaml deliberately
skipped: smoke regression mirrors the 2026-05-07 torch.compile
regression dynamic (dispatch overhead > kernel benefit at this model
size) and re-confirming on the same size adds no information.
Default stays run.use_amp: false. Re-enable trigger documented in
docs/performance.md: hidden_size >= 1024 or num_layers >= 6, then run
the bench script + 100-iter A/B before flipping default.
The experiment finds a model size where (a) learning-curve gains
justify compute, and (b) forward time is large enough to amortize
AMP/compile/TensorRT overhead. Outcome gates re-enabling those four
deferred optimizations and Option A.
Tested grid: hidden={512,768,1024,1536} × layers={3,4,6,8} subset.
200 iterations per config on home (RTX 3090), single seed initially,
second seed for boundary configs. Eval cadence held at default.
Decision tree included for: success → recommend new default and
trigger downstream plans; null → document and stay; expensive-but-
better → defer until AMP/compile/TRT land.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
performance.md additions:
- Batched Traversal Inference design decision (A vs B vs C with
rationale).
- Option A bench result and structural ceiling (realized batch ~7.2,
IPC overhead exceeds GPU gain at small model size).
- Post-A optimization calculus: why compile/TensorRT remain
iter-neutral today and become meaningful only after model growth
and/or denser eval. Sequencing matters; do not retest these on the
current small model.
- Free-threaded Python (3.13t/3.14t) note: cleanest endpoint in
principle, but PyTorch maturity + Cython nogil audit cost block
near-term adoption.
docs/plans/ (4 plan documents for Codex execution):
- batched_traversal_inference_server.md (executed; deferred).
- amp_trainer.md.
- torch_compile.md.
- cython_safe_heuristic_bots.md (executed; first-pass landed).
docs/reports/ (3 cost reports):
- cost_pytorch_free_threaded_2026-05-07.md: WAIT 3-6 months;
PyTorch wheels exist but our Cython is the gating cost.
- cost_cython_nogil_audit_2026-05-07.md: medium effort, traversal.pyx
carries 90% of blockers; Steps 1-3 (cfr_math/encoding nogil
keywords, TraversalStats cdef class) are safe and cheap, Steps
4-6 wait for triggers.
- cost_pytorch_cuda_multithread_2026-05-07.md: risky;
optimizer.step / load_state_dict race silently with concurrent
forward; per-thread default streams unset means naive threading
serializes on default stream anyway.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the central inference server pattern: a dedicated GPU
process owns advantage/strategy/league networks, batches policy
requests across traversal workers via shared-memory tensor pool, and
returns logits. Workers route forward calls through InferenceClient /
NetworkProxy when traversal.inference_backend == "server".
Default remains traversal.inference_backend: local. The server
backend regresses iter time ~3.8× on the inspected default config
(small-model dispatch + sync-blocking traversal capping realized
batch at ~num_workers=8 instead of the bs=64-256 needed to amortize
IPC overhead). Keeping the implementation behind the flag lets us
re-enable when (a) model size grows, (b) per-worker interleaved
traversal lands, or (c) eval becomes dominant — see
docs/performance.md "Option A Bench Result and Structural Ceiling"
for the full diagnosis.
Plumbing included:
- inference_buffers.py: shared-memory tensor pool with slot
management.
- inference_client.py: per-worker client + NetworkProxy adapter for
the existing traversal.pyx call sites.
- inference_server.py: spawn-context server process with
batch-window aggregation, weight sync, shutdown sentinel.
- bench_inference_backend.py: A/B between local and server backends
with eval/checkpoint disabled.
- test_inference_server.py: round-trip and integration tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython implementation in heuristic_cy.pyx achieves ~2.55× speedup on
opponent_act_seconds (200-game eval: 59.20s → 23.24s). Original Python
implementation preserved verbatim in heuristic_py.py as the equivalence
reference. Action-sequence equivalence is verified by
test_safe_heuristic_equivalence.py against seeded game corpora.
Key implementation notes:
- File-local wraparound=True override required for negative discard
indexing; Cython global wraparound=False would segfault.
- annotation_typing=False preserves verbatim Python semantics.
- _CachedState materializes hands/expeditions/discards/deck once per
act() call — this is the dominant performance win.
Further C-array optimization of _card_value_for_me /
_card_value_for_opponent / _color_commitment / _bonus_potential is
deferred. The current 2.55× delivers most of the dense-eval future
benefit; further work is gated on actually adopting denser eval
schedules (eval_every=5, games=1000).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Measured DeepCFRMLP forward at bs={1,4,16,64,256,1024} on RTX 3090.
Per-state cost drops 232× from bs=1 (80 µs) to bs=256 (0.34 µs) while
per-call latency stays near 90 µs through bs=256. Policy-call supply
from a real run is ~368 states per traversal and ~200k per iteration,
well above the bs=64–256 plateau, so batched inference is not
supply-limited. GPU forward is not the limiter once batching exists.
Verdict: Optimization Priorities #5 (batched traversal inference) is
worth pursuing. End-to-end gain will still be bounded by encoding and
worker-GPU coordination overhead.
- scripts/profile_gpu_forward.py: standalone profiling script
- docs/performance.md: new "GPU forward profiling for batched traversal"
experiment section with table, supply estimate, and verdict
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the link from the failed torch.compile experiment to Optimization
Priorities #5 explicit, so a future revisit happens at the right time
(once compile is on the dominant phase, not just trainer optimization
steps).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrapping trainer networks with torch.compile produced a 4.8% regression
in iteration time on default.yaml (17.93s → 18.79s). Two causes: (1)
the dominant phase is CPU traversal which bypasses the compiled
wrapper, (2) DeepCFRMLP is too small for compile dispatch overhead to
pay back. Implementation kept on experiments/torch-compile for future
revisits when the trainer model or inference path changes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopt a 5-namespace scheme so wandb groups related metrics in the
sidebar and capture-group regex (eval/(random|safe_heuristic)/win_rate0
vs eval/(?:random|safe_heuristic)/win_rate0) controls panel splitting:
- loss/{advantage,strategy}
- samples/{advantage,strategy,advantage_player_N}
- memory/{advantage,strategy,advantage_player_N}
- time/{iteration_seconds,traversal_seconds,advantage_train_seconds,
strategy_train_seconds,evaluation_seconds,memory_add_seconds,
checkpoint_seconds,batch_tensor_seconds,nodes_per_second,
advantage_player_N_sample_seconds,strategy_sample_seconds}
- traversal/{nodes,terminals,depth_cutoffs,node_limit_cutoffs,
max_depth_reached,endpoints,avg_endpoint_depth,
endpoint_depth_bucket_*,regret_fallback_*,sampled_actions}
- eval/<opponent>/<metric> (3-level so opponent can be the capture group)
`iteration` keeps no namespace (it's the wandb step axis). Internal
TraversalStats.to_dict() and benchmark.py's standalone result dict
keep their flat names — only the trainer's emitted metrics are
remapped, with the traversal_*→traversal/* translation done at
insertion into runtime_metrics.
analyze.py updated to read the new keys (PlotSpec metrics, color map,
opponent_names parser, _first_existing_eval lookup). Tests updated for
the new eval_metrics dict keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the timer wrapped every call to _evaluate(), but on iterations
that skip eval (iteration % eval_every != 0) the function returns
immediately and the recorded value was just function-call overhead
(~3 µs), which made W&B show a wildly bimodal "evaluation_seconds"
metric. Now only set the key when eval_metrics is non-empty so
non-eval iterations have no data point.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append a rough iters_per_hour estimate (3600/iteration_seconds) to the
console summary so users running long jobs can eyeball ETA without doing
the math.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make traversal progress and iteration-complete summary lines easier to
visually scan during long runs by leading with [i=N]. Drop the redundant
"iteration=N" kv from the body to keep lines short.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The avg-strategy 1000iter file (with the recent +traversals/+LR/+LCFR
changes) is the canonical "best-known" config. Renamed it to
default.yaml so users start from a single, obvious entry point and
override one field per ablation via --set. Other 12 configs moved to
configs/archive/ — kept for historical reproduction, not for active use.
- configs/deep_cfr/{default.yaml, smoke.yaml} are the only active configs
- experiment_name shortened to "deep-cfr-default" (was a long mouthful)
- AGENTS.md examples and Project Layout section rewritten around
default.yaml; ablation example shows the override-one-field pattern
- Tests pointed at the archived slot-playability config for the legacy
reproduction assertions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If eval_every is positive but max_iterations falls before the next scheduled
eval iteration (including resume cases where current_iteration is already
past the last eval boundary), log a one-time warning at run start so the
user notices the misconfiguration. We deliberately do not force an
end-of-run eval, which would distort time budgets and reproducibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Default is one baseline + one treatment, sequential, same seed, with a
shared --wandb-tag hypothesis label for W&B Compare Runs filtering.
Multi-seed only on explicit request; never run two trainings on the same
GPU.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brief guidance on roles (notes = purpose, tags = filter categories)
plus three anti-patterns to avoid. Otherwise free-form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Exposes wandb.init's notes field through the CLI so each run can carry a
short description of its purpose, visible on the W&B run page alongside
tags and config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- AGENTS.md: add Weights & Biases section covering install (extra),
online/offline modes, per-run wandb/ layout, sync command, and the
source-of-truth note (metrics.jsonl, not W&B).
- CLAUDE.md: replace soft "before making changes" wording with a
mandatory session-start instruction to read AGENTS.md in full.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Append eval_seconds and its share of iteration_seconds to the per-iteration
console summary when evaluation actually ran, so users watching the log can
see how much wall time eval is consuming without parsing JSON metrics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop checkpoint.directory from config — config defines what an experiment
is, not where its outputs go. The CLI now computes the run directory from
run.experiment_name plus a timestamp, defaulting to runs/tmp/ for
throwaway runs and runs/ when --keep is passed.
- Remove CheckpointConfig.directory and DeepCFRConfig.checkpoint_path
- DeepCFRTrainer takes run_dir: Path explicitly
- CLI: add --keep boolean; --resume requires an explicit path (no shortcut)
- Auto path: runs/[tmp/]<YYYY-MM-DD_HHMMSS>_<experiment_name-kebab>/
- Rename 13 configs to kebab-case; strip directory: lines; kebab their
experiment_name values
- Rewrite AGENTS.md training/run sections; document
archive/tmp/<flat> layout, --keep, kebab-case scope
- Update tests for new run_dir flow and dropped --resume shortcut
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove legacy aliases, rename max_hours to max_minutes, collapse the
four checkpoint save flags into save_every + save_latest, and change
defaults to safer values (opponent_policy=self_play_league,
device=auto, eval_every=50, max_depth=null). Migrate all archived
yaml configs and tests to the new schema.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror Deep CFR training metrics to W&B via a new WandbRunTracker
wired through CompositeRunTracker; wandb is an optional extra so
default installs and runs stay unchanged. Train CLI gains
--wandb/--wandb-project/--wandb-mode/--wandb-name/--wandb-tag, and
train() now closes the tracker in a finally block so runs finalize
even on early exit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strategy network (학습 중인 average policy)를 traversal opponent로 사용하는
새 옵션 추가. Deep CFR 이론적 수렴이 average strategy에 대한 보장이라는
점에 착안 — opponent_policy=network의 발산 문제를 완화할 수 있는지 실증.
구현:
- config: opponent_policy validator에 average_strategy 추가
- traversal.pyx: opponent_policy_id=3, strategy_network 인자, softmax 기반
policy 도출 (_policy_from_strategy_network)
- workers.py: TraversalWorkerBatch에 strategy_network state_dict 추가
- trainer.py: 직렬/병렬 traversal call에 strategy_network 전달
- 1000-iter 실험 config 추가 (opponent_policy=network와 동일 hyperparam)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
같은 hyperparameter (512x3, 2x updates, argmax_tiebreak)에 opponent_policy
만 self_play_league로 다른 run과의 iteration별 비교 표 추가. iter 350
시점에서 Random WR 38%p 격차 (network 34% vs league 72%) 확인 — 발산
가설을 실증.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
opponent_policy=network 설정으로 1000-iter 실험 두 개 (512x3, 1024x4)를
돌린 결과 두 실험 모두 policy collapse가 발생함을 확인. 큰 capacity는
plateau를 늘리지만 발산 자체를 막지 못함. 향후 학습은 self_play_league
기본값을 유지할 것을 권고.
- 실험에 사용한 config 두 개 추가
- 발견 분석 문서 추가 (원인, 비교, 권고)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add deep_cfr_color_shared_512x3.yaml: color_shared without attention
- Add deep_cfr_color_shared_attention_512x3.yaml: color_shared with 2-layer attention
- Both use 512 hidden size and 3 MLP layers
- Configs are ready for experimental training runs
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add network.kind field to config (mlp/color_shared)
- Implement ColorSharedNetwork that:
- Splits input into 5 equal color blocks
- Encodes each block with shared weights
- Pools with mean/max aggregation
- Concatenates pooled embeddings with remainder
- Outputs same action logits as MLP
- Add ColorAttention for optional self-attention over color embeddings
- Add network.color_attention_layers and color_attention_heads config
- Maintain full backward compatibility (default kind=mlp)
- Add 28 comprehensive tests covering all architectures
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
coolguy
