coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Refactor docs/performance.md per librarian routing rule

Browse Source 
docs/performance.md grew to 914 lines because dated experiments and
design analyses kept getting appended instead of routed to
docs/archive/ and docs/research/ as AGENTS.md prescribes. The
oversize check from librarian Stage 1 surfaced the file; this commit
acts on that finding by extracting the parts that belong elsewhere
and trimming the source to a focused current-state reference.

Extracts (verbatim from the original prose, with cross-link headers
and a brief routing note added at top):

- docs/archive/deep-cfr-performance-experiments-2026-05-07.md
  bundles torch.compile (regression), AMP (regression), GPU forward
  profiling (decision support), and Option B interleaved traversal
  (pass) — same date, same theme.
- docs/research/batched-traversal-inference-decision.md captures the
  durable A vs B vs C rationale with a closing "Outcome" pointer to
  the post-bench archive doc.
- docs/archive/post-a-optimization-calculus-2026-05-07.md preserves
  the forward-looking sequencing recorded pre-bench.
- docs/archive/option-a-bench-result-2026-05-07.md preserves the
  regression diagnosis and re-enable criteria.

docs/performance.md is now 345 lines, holds sections 1–9 (current
runtime / bottleneck / device / AMP status / batching / eval /
TensorRT / priorities), and ends with a "See Also" linking the four
extracts.

Also reworded the AGENTS.md soft-cap rule from a bare "~500-line
soft cap" to clarify the intent: the cap is a *routing trigger* (is
content piling up that should live in archive/research?), not a
split mandate. Reduces the risk of future agents shredding a useful
doc just to satisfy a number.

scripts/librarian.sh now exits 0 against the working tree.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
coolguy
2026-05-07 23:57:39 +09:00
co-authored by Claude Opus 4.7
parent 6ecb233bdd
commit 1cd9950bd3
7 changed files with 703 additions and 595 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+4 -1
View File
@@ -378,7 +378,10 @@ contention slows both unevenly and breaks the comparison.
- archive 본문 복붙 금지 — 대신 `Source:` 링크 + distill. - archive 본문 복붙 금지 — 대신 `Source:` 링크 + distill.
- `file.py:NN` 인용은 작성 시점에 `rg`로 검증. - `file.py:NN` 인용은 작성 시점에 `rg`로 검증.
- 한 주제 한 파일 — `foo-v2.md` 만들지 말 것. - 한 주제 한 파일 — `foo-v2.md` 만들지 말 것.
- 파일당 ~500줄 soft cap. - 파일당 ~500줄 soft cap — 강제 분할이 아니라 "다른 위치(archive/research)로
가야 할 내용이 누적됐는지" routing 점검 트리거. 초과 시 dated 실험은
`docs/archive/`로, 항구적 분석은 `docs/research/`로 보내고 본 파일은
현재 상태 reference만 남긴다.
## Notes For Future Agents ## Notes For Future Agents
docs/archive/deep-cfr-performance-experiments-2026-05-07.md
+245
View File
@@ -0,0 +1,245 @@
# Deep CFR Performance Experiments (2026-05-07)
**Source:** Extracted from `docs/performance.md` § "Experiments" on
2026-05-08 to honor the "dated experiment records → docs/archive/"
routing rule from AGENTS.md.
Bundles four trainer- and traversal-side optimization experiments
performed 2026-05-07 on the small `default.yaml` model
(3-layer, 512-hidden). Three of them (`torch.compile`, AMP, GPU forward
profiling) surfaced the same underlying truth — the model is too small
for kernel-fusion / lower-precision wins to amortize their dispatch
overhead. The fourth (Option B interleaved traversal) passed and became
the new default.
## `torch.compile` on trainer networks (regression)
Wrapped both advantage networks and the strategy network with
`torch.compile()` at trainer construction time. Implementation also
required a `_clean_state_dict()` helper to strip the `_orig_mod.` prefix
that compiled modules add to `state_dict()`, plus a `_orig_mod`-routed
path for `load_state_dict()` so multiprocessing traversal workers and
checkpoint restoration could keep using the uncompiled `DeepCFRMLP`
class.
Measurement (8 iterations on `default.yaml`, eval and checkpoint
disabled, iteration 1 dropped as compile warm-up):
| | iter mean | 1000-iter projection |
| --- | ---: | ---: |
| Baseline (no compile) | 17.93s | 4.98h |
| `torch.compile` on trainer nets | 18.79s | 5.22h |
| Effect | +0.86s (+4.8%) | +14 min |
Net result: regression. Two reasons:
- Traversal is ~60% of iteration time and runs in CPU multiprocessing
workers that reconstruct networks from cleaned `state_dict`s, so they
bypass the compiled wrapper entirely.
- `DeepCFRMLP` (512-hidden, 3-layer) is small enough that the compiled
call dispatch overhead exceeds the kernel-fusion benefit.
Implementation preserved on branch `experiments/torch-compile` for
revisiting if the trainer model grows substantially or after the
batched-traversal-inference work in Optimization Priorities #5 lands —
that is the change that would put compile on the dominant phase, not
just on the trainer's optimization steps. Not enabled on `main`.
## AMP on trainer networks (regression)
Wrapped the trainer optimization phases with `torch.autocast(fp16)` and
`torch.amp.GradScaler`: `_train_advantage` and `_train_strategy` now run
their network forward/backward/optimizer step through the AMP path when
`run.use_amp=true` and the trainer device is CUDA.
Safety mitigations included in the implementation:
- `GradScaler.unscale_(optimizer)` is called before `clip_grad_norm_`.
- Non-finite loss guard increments `amp/nonfinite_loss_count` and skips the
bad step instead of applying it.
- Advantage squared loss computes `diff.float().square()` so the loss
reduction is fp32 even when the forward path is autocast to fp16.
- Strategy logits are cast back to fp32 before `masked_fill` and
`log_softmax`.
- Metrics now expose `amp/grad_scale` and `amp/nonfinite_loss_count`.
Measurement used the small `smoke.yaml` config with synthetic replay-memory
samples via:
```bash
uv run python scripts/bench_amp_trainer.py \
--config configs/deep_cfr/smoke.yaml \
--runs 3 \
--warmup 1 \
--device cuda
```
| | mean ms/call | speedup vs fp32 |
| --- | ---: | ---: |
| fp32 | 3.22 | 1.00× |
| AMP (fp16) | 3.92 | 0.82× |
Net result: regression. This matches the same dispatch-overhead-vs-kernel
benefit dynamic as the `torch.compile` regression above: the current trainer
model and smoke workload are too small for AMP's lower-precision kernels to
pay back autocast and scaler bookkeeping overhead.
The full `default.yaml` 100-iteration A/B was intentionally skipped. Given
the small-model regression and the matching `torch.compile` precedent on the
same model family, there is no current evidence that spending GPU time on the
longer A/B would produce a different decision. The infrastructure is kept
merged but default-off: `run.use_amp=false` remains the default, and
re-enabling is a one-field config flip.
Re-measure AMP only after the model grows to at least `hidden_size >= 1024`
or `num_layers >= 6`. At that point run both the fast
`scripts/bench_amp_trainer.py` micro-bench and the formal 100-iteration
fp32-vs-AMP A/B. If AMP still provides less than 5% speedup at that larger
model size, keep it default-off and raise the next re-measure trigger to an
even larger model.
## GPU forward profiling for batched traversal (decision support)
To decide whether Optimization Priorities #5 (batched traversal inference) is
worth implementing, profiled `DeepCFRMLP` from `default.yaml`
(input_dim=365, output_dim=22, hidden=512, 3 layers, ReLU) on an RTX 3090 in
`eval()` + `inference_mode`, with 10-iter warm-up and 1000-iter measurement
per batch size. Script: `scripts/profile_gpu_forward.py`.
| Batch size | μs/call | μs/state | Speedup vs bs=1 |
| ---: | ---: | ---: | ---: |
| 1 | 80.07 | 80.074 | 1.00× |
| 4 | 81.20 | 20.299 | 3.94× |
| 16 | 91.30 | 5.706 | 14.03× |
| 64 | 93.61 | 1.463 | 54.75× |
| 256 | 88.34 | 0.345 | 232.03× |
| 1024 | 161.95 | 0.158 | 506.30× |
Policy-call supply from
`runs/tmp/2026-05-07_181155_deep-cfr-default/metrics.jsonl`: mean
`traversal/nodes` ≈ 205,810 over 280 traversals/player → ~368 policy calls per
traversal (rough upper bound on batchable states), ~200k per iteration across
560 traversals.
Verdict: **Priority #5 is worth pursuing.** Per-state cost drops from 80 μs at
bs=1 to 0.34 μs at bs=256 (>230×). The available supply of ~368 states per
traversal sits comfortably in the bs=64–256 range where μs/call plateaus near
90 μs. End-to-end gain will be bounded by encoding and worker-GPU coordination
overhead, but the GPU forward is not the limiter once batching is in place.
## Option B interleaved traversal (pass)
Implemented a non-default interleaved traversal scheduler behind:
```yaml
traversal:
scheduler: interleaved
```
The recursive Cython path remains the default. The current interleaved path is
a Python explicit-stack production prototype, guarded to the narrow case
`sampling_mode=outcome`, `opponent_policy=network`,
`cutoff_value_mode=score_diff`, `cutoff_rollouts=0`, and
`inference_backend=local`.
Measurement used `configs/deep_cfr/default.yaml` with evaluation and
checkpointing disabled, 10 iterations, first 2 iterations dropped as warm-up.
Because interleaved currently supports only `opponent_policy=network`, the
recursive baseline used the same opponent-policy override.
Commands:
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-recursive-network-10i \
--set traversal.opponent_policy=network \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-interleaved-workers-10i \
--set traversal.scheduler=interleaved \
--set traversal.opponent_policy=network \
--set traversal.num_workers=8 \
--set traversal.worker_chunk_size=64 \
--set traversal.interleave_width=64 \
--set traversal.interleave_max_batch=128 \
--set traversal.progress_every_traversals=0 \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-interleaved-cuda-single-10i \
--set traversal.scheduler=interleaved \
--set traversal.opponent_policy=network \
--set traversal.num_workers=0 \
--set traversal.interleave_width=64 \
--set traversal.interleave_max_batch=128 \
--set traversal.progress_every_traversals=0 \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
```
Result paths:
- `runs/2026-05-07_225044_option-b-recursive-network-10i`
- `runs/2026-05-07_225342_option-b-interleaved-workers-10i`
- `runs/2026-05-07_225542_option-b-interleaved-cuda-single-10i`
Warm-up-excluded means:
| Mode | iter s | traversal s | traversal speedup | iter speedup | nodes/s | batch mean | batch max |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| recursive, 8 workers, chunk 8 | 16.49 | 10.22 | 1.00× | 1.00× | 17.6k | 1.0 | 1 |
| interleaved, 8 workers, chunk 64 | 10.81 | 4.92 | 2.08× | 1.53× | 31.3k | 35.2 | 64 |
| interleaved, single CUDA process | 11.51 | 5.35 | 1.91× | 1.43× | 24.7k | 36.7 | 64 |
Net result: **PASS for the Phase 3 traversal-speed gate.** The best candidate
is the 8-worker interleaved path with larger worker chunks. It reaches real
batches near the target regime (`max_batch_size=64`) and more than doubles
traversal wall-clock versus the recursive network-opponent baseline.
The single-process CUDA path confirms that GPU forward is no longer the
dominant cost once batching works (`interleaved/forward_seconds` averaged
0.64s versus 4.53s for CPU worker forward), but it gives up multiprocessing
game-state throughput and is slower end-to-end than 8 interleaved CPU workers.
Important caveat: multi-traversal interleaving uses per-context RNG streams, so
exact recursive-batch RNG ordering is intentionally not preserved. The Phase 2
single-traversal parity test matches recursive stats and sample target
checksums under identical RNG seed. This means the default now favors the
measured traversal-speed win over byte-identical sample ordering. If future
learning curves show unexplained drift, first compare against the recursive
fallback:
```bash
--set traversal.scheduler=recursive \
--set traversal.worker_chunk_size=8 \
--set traversal.progress_every_traversals=10
```
Follow-up: `average_strategy` support was added after the initial Phase 3
network-opponent A/B so the interleaved path can run the actual default opponent
policy. A 10-iteration throughput check with default opponent policy,
evaluation/checkpoint disabled, and the same 8-worker chunk-64 interleaving
settings produced warm-up-excluded means:
| Mode | iter s | traversal s | batch mean | batch max |
| --- | ---: | ---: | ---: | ---: |
| interleaved, default `average_strategy` | 10.61 | 4.85 | 28.3 | 64 |
Run: `runs/2026-05-07_230419_option-b-interleaved-average-strategy-10i`.
This follow-up unblocked making interleaved traversal the default. The default
switch was made with the caveat above rather than waiting for a long-run A/B.
docs/archive/option-a-bench-result-2026-05-07.md
+187
View File
@@ -0,0 +1,187 @@
# Option A Bench Result and Structural Ceiling (2026-05-07)
**Source:** Extracted from `docs/performance.md` § "Option A Bench
Result and Structural Ceiling" on 2026-05-08.
Records the post-implementation Option A benchmark (regression: 0.21×
traversal), the diagnosis of the sync-blocking policy boundary as the
real ceiling, and the criteria under which to revisit Option A.
**Related:**
- Design rationale: `docs/research/batched-traversal-inference-decision.md`
- Forward-looking sequencing (pre-bench): `docs/archive/post-a-optimization-calculus-2026-05-07.md`
- Trainer-side experiments from the same date: `docs/archive/deep-cfr-performance-experiments-2026-05-07.md`
---
Option A (`traversal.inference_backend: server`) was implemented and
benchmarked. **Result: regression. A is deferred. `default.yaml` stays on
`local`. The implementation is preserved behind the flag for future revisit.**
## Bench numbers
`scripts/bench_inference_backend.py --device cuda --iterations 5 --warmup 1`,
RTX 3090, after a per-call IPC fix (replaced `multiprocessing.Manager()`
queues/events with spawn-context primitives, slot reuse per worker batch,
shared memory confirmed in use for state/response payloads).
| Backend | iter | traversal | adv_train | strat_train | mem_add | batch_tensor |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `local` | 16.75s | 10.81s | 3.91s | 2.02s | 0.95s | 3.56s |
| `server` | 57.61s | 51.61s | 3.96s | 2.02s | 0.50s | 3.57s |
| Speedup | 0.29× | 0.21× | 0.99× | 1.00× | 1.89× | 1.00× |
Raw: `runs/bench/2026-05-07_193335_inference_backend/results.json`.
Training and eval phases are unchanged (as expected — A only touches the
traversal forward path). The regression is contained in `traversal_seconds`,
which is ~5× worse.
## Diagnosis
The server emits per-flush batch stats. **Mean batch size: ~7.2–7.9, max 8.**
This is the structural ceiling, not a tunable misconfiguration:
- Traversal recursion is **sync-blocking** at the policy call site. Each
worker has at most one in-flight policy request at a time.
- In-flight requests at the server ≤ `num_workers` = 8.
- The server further splits each batch by `(network_kind, network_index)`,
so the actual GPU forward group size is roughly half of that — about 4
rows per group.
Per-state cost at this realized batch size, from the GPU profile table:
| Realized batch | μs/state |
| ---: | ---: |
| 1 | 80.07 |
| 4 | 20.30 |
| 8 (extrapolated) | ~12 |
| 64 | 1.46 |
| 256 | 0.34 |
So the GPU is doing ~12μs per state instead of the projected ~1.5μs at
bs=64. The IPC round-trip per call (queue post + server scheduler + event
wakeup, even with shared-memory payload) is on the order of hundreds of μs
per call, which exceeds both the local CPU forward (~80–200μs at bs=1 on
this small MLP) and the marginal GPU gain. Net: per-call cost roughly
doubles or triples, compounded across ~205k calls/iter, gives the observed
5× traversal regression.
`batch_window_us` and `max_batch` tuning cannot escape this ceiling —
there are simply not 64 concurrent in-flight requests to coalesce when only
8 workers are blocking-sync.
## What this means for the headline GPU profile (`scripts/profile_gpu_forward.py`)
The earlier "230× speedup at bs=256" is a **per-state GPU forward**
microbenchmark, not an end-to-end traversal speedup. Realizing that gain
requires *actually feeding the GPU* with bs=64+ batches. Sync-blocking
multi-worker traversal cannot do that. Reaching the bs=64 regime needs
either:
- Per-worker traversal interleaving (worker advances `worker_chunk_size`
traversals concurrently, suspending at each policy call — Option B
shape), which requires turning Cython traversal recursion into a
resumable state machine. Same scope as a partial Option C, localized to
worker scope.
- Option C proper (single-process vectorized traversal).
Both require restructuring traversal. Option A's "additive, no traversal
changes" property turned out to also mean "cannot drive the batch size up."
## Clarifying the traversal bottleneck: sync policy boundary, not SIMD
The tempting shorthand is "Python/GIL prevents traversal from using SIMD or
threads." The more precise diagnosis is narrower:
- Lost Cities game mechanics are already mostly Cython C-level operations.
`legal_actions`, action push/pop, and cached scoring are not Python list
walks on the hot path.
- The traversal recursion is Cython, but it synchronously crosses back into
Python/PyTorch at every policy-needed state: encode a single info state,
run one-row PyTorch forward, copy logits back to CPU/Numpy, then continue
recursion.
- This boundary makes every traversal worker **sync-blocking**. With
`num_workers=8`, the inference server can see at most eight in-flight
requests before per-network splitting, no matter how large `max_batch` is.
- GIL-free threading would help only after the same path is made
`nogil`-clean or after traversal is restructured so policy calls can be
batched. Simply "using SIMD" does not address the one-row policy boundary.
So the actionable bottleneck is **policy-call scheduling shape**, not scalar
game-rule arithmetic. The highest-leverage experiment is Option B:
per-worker interleaved traversal, where one worker advances many traversals,
suspends each at a policy request, batches those requests, and resumes the
corresponding continuations.
Microbench evidence (2026-05-07, `configs/deep_cfr/default.yaml`,
`experiments/traversal_policy_boundary/bench_policy_boundary.py`):
| Device | Component | median μs/call | p95 μs/call |
| --- | --- | ---: | ---: |
| CPU | encode + legal | 3.10 | 3.81 |
| CPU | push + pop | 0.15 | 0.22 |
| CPU | policy boundary bs=1 | 111.50 | 125.46 |
| CPU | torch forward bs=64 | 12.84 | 13.14 |
| CUDA | policy boundary bs=1 | 181.30 | 194.77 |
| CUDA | torch forward bs=64 | 2.55 | 2.75 |
This confirms the bottleneck is not Cython game-rule scalar work. The
single-request policy boundary is ~36× larger than encode+legal on CPU, while
CUDA bs=64 forward is ~71× cheaper than the current CUDA bs=1 boundary.
Expected upside is bounded by the fraction of traversal currently spent at
policy calls. Moving realized GPU forward from the current ~4-8 row regime
(~12-20μs/state) to bs=64 (~1.46μs/state) is an ~8-14× improvement on the
forward component, but not on game recursion, sample creation, or replay
writes. For the observed `local` traversal around 10-13s/iter, a realistic
first target is roughly **1.5-3× traversal speedup** if Option B reaches the
bs=64 regime without adding comparable scheduler overhead. Larger claims need
a prototype because traversal has substantial non-forward work.
## Why deferring A (not deleting) is the right call
- The plumbing (server process, shared-memory client, weight sync, config
flag) is complete and tested. Re-enabling is a config flip.
- The fundamental issue at this model size is that **GPU forward time is
too small to amortize IPC overhead** at any realistic batch size we can
drive without restructuring traversal. Bigger model changes that
arithmetic; the same plumbing then becomes useful.
- The `mem_add_seconds` row showed a real 1.89× win, suggesting the
shared-memory replay-write path adopted along the way is worth keeping
even with `local` backend. (Confirm separately; this is a side effect.)
## Re-enable A when one of these holds
1. **Model grows** to ~1024 hidden / ~6 layers (compile/TRT discussion in
`docs/archive/post-a-optimization-calculus-2026-05-07.md`). Forward
time scales with FLOPs while IPC overhead is fixed; at some point IPC
becomes a small fraction.
2. **Per-worker interleaved traversal** ships (Option B-shape refactor).
Drives realized batch toward 64 and reclaims the profile table's gains.
3. **Eval becomes the dominant phase** (`eval_every: 5`,
`evaluation.games: 1000`). Eval is not sync-blocking traversal; it is
already batch_size=64 in eval code. The same inference server can
serve eval directly without the worker-side ceiling.
## Free-threaded Python (3.13t) note
Free-threaded Python + Cython `nogil` would let many threads (well above
core count) run game logic concurrently in one process, with shared memory
and no IPC. With ~64 threads sync-blocking on policy calls, the server
would naturally see bs=64. **In principle this is the cleanest endpoint.**
In practice as of early 2026:
- Free-threaded Python is an opt-in build (`python3.13t`), still
experimental, with measurable single-thread overhead.
- PyTorch's free-threaded compatibility is partial.
- Cython `nogil`-cleanliness audit on the existing game engine is still
required and was the original reason `nogil` threading was deferred in
the design decision.
- No project-level adoption pressure on `python3.13t` today.
So free-threaded Python does change the architectural answer, but it does
not unblock A *now*. Track the ecosystem; revisit when (a) `python3.13t`
becomes mainstream or (b) the Cython engine is `nogil`-cleaned for other
reasons.
docs/archive/post-a-optimization-calculus-2026-05-07.md
+98
View File
@@ -0,0 +1,98 @@
# Post-A Optimization Calculus (2026-05-07)
**Source:** Extracted from `docs/performance.md` § "Post-A Optimization
Calculus (forward-looking, 2026-05-07)" on 2026-05-08.
Forward-looking sequencing recorded *before* Option A was benched. Has
not been measured. See
`docs/archive/option-a-bench-result-2026-05-07.md` for the actual bench
which deferred Option A — some assumptions below ("once Option A
lands") have to be re-evaluated in light of that result.
---
Once Option A lands, the bottleneck shape changes. This section records the
expected sequencing for follow-up work. It is forward-looking and has not been
measured yet — verify against bench numbers after A is benchmarked.
## Why compile / TensorRT are negligible *today* but become meaningful later
Today (small model: 3-layer, 512 hidden):
- `torch.compile` on the trainer's networks already regressed (see the
2026-05-07 experiment in
`docs/archive/deep-cfr-performance-experiments-2026-05-07.md`).
The model is too small for kernel fusion to beat compile dispatch
overhead.
- `torch.compile` / TensorRT on the inference-server forward (post-A) would
shave ~30–50% off ~90μs/call → ~50–70μs/call. With forward share of an iter
reduced to <1% by A's batching, the iter-level multiplier is ~1.00–1.01×.
Negligible.
Two compounding shifts can flip this:
1. **Larger model.** Going from 512 hidden / 3 layers to ~1024 hidden /
~6 layers pushes the forward call out of dispatch-bound territory into
kernel-bound territory. Compile fusion and TensorRT both deliver real
1.5–2× on the forward call itself once the kernel is large enough to
amortize launch overhead. Forward share of iter time also rebalances upward
because per-call time scales with FLOPs while batching gain is fixed.
2. **Denser, larger evaluation.** Moving toward `eval_every: 5` and
`evaluation.games: 1000` makes evaluation about half of iteration wall-clock
(see the amortized eval table in `docs/performance.md`). Eval is pure
inference, so TensorRT on the inference-server's forward path applies
directly.
When both shifts happen together, an illustrative future iter (rough order of
magnitude only):
| Configuration | Iter time (rough) |
| --- | ---: |
| Today (small model, eval_every=25) | 17.85s |
| + A (batched traversal inference) | ~14s |
| + larger model (≈4× FLOPs), no compile/TRT | ~50s |
| + dense eval (eval_every=5, games=1000) | ~70s |
| + compile (trainer) + TensorRT (inference) | ~45s |
That last row is where compile/TensorRT contributes ~1.5× iter — the same
tooling that is iter-neutral today. The numbers above are illustrative; real
ratios depend on model size, kernel autotune outcomes, and the eval-vs-train
balance.
## Tooling split
- **TensorRT**: applies only to inference (no backward). Targets:
- inference-server forward in traversal,
- inference-server forward in evaluation.
Both are served by the same A-era server, so a single TensorRT integration
covers both.
- **`torch.compile`**: applies to trainer's advantage/strategy training
(forward+backward+optimizer). The 2026-05-07 regression on a small model
does **not** generalize — it must be re-measured on whatever larger model
config we settle on. Do not conclude "compile is bad" from the small-model
data point.
## Recommended sequencing
Do this in order. Skipping ahead is the failure mode that creates misleading
"compile/TRT didn't help" data.
1. **Now**: benchmark A (`scripts/bench_inference_backend.py`) and confirm the
`local` vs `server` multipliers on `home` and `remote`. Validate the iter
1.2–1.3× / traversal 1.5–2× working estimate.
2. **Next**: experiment with a larger network config. Measure compute vs
learning-curve trade-off with the existing toolchain (no compile/TRT yet).
This step decides the model size that future optimizations target.
It is also the prerequisite for revisiting AMP, `torch.compile`, and
TensorRT: all three are dispatch-overhead-bound on the current small model.
3. **Then**: re-measure `torch.compile` on the trainer at the chosen model
size. The earlier regression was size-bound; expect a different result.
4. **Then**: integrate TensorRT into the inference server (covers traversal
and eval forward simultaneously). Bound the gain by the post-step-2
`policy_network_seconds` share, not the headline TensorRT speedup.
5. **In parallel with 2–4**: if denser eval is operationally useful, raise
`evaluation.games` and lower `evaluation.eval_every`. This step does not
require code changes but sharply increases the value of step 4.
Out of scope until A bench numbers are in: Option C, `nogil` threading, async
inference client, compiled encoding.
docs/performance.md
+20 -589
View File
@@ -323,592 +323,23 @@ Based on the current metrics, the more plausible performance work is:
post-processing before assuming TensorRT is the main lever. post-processing before assuming TensorRT is the main lever.
The inspected eval row shows those costs dominate the slowest opponents. The inspected eval row shows those costs dominate the slowest opponents.
## Experiments ## See Also
### `torch.compile` on trainer networks (2026-05-07, regression) The following dated records and design notes were extracted from this
document on 2026-05-08 to keep `performance.md` focused on current-state
Wrapped both advantage networks and the strategy network with reference material. The split followed the AGENTS.md "Docs & Experiment
`torch.compile()` at trainer construction time. Implementation also Workflow" routing rules:
required a `_clean_state_dict()` helper to strip the `_orig_mod.` prefix
that compiled modules add to `state_dict()`, plus a `_orig_mod`-routed - **Performance experiments (2026-05-07)** — `torch.compile` regression,
path for `load_state_dict()` so multiprocessing traversal workers and AMP regression, GPU forward profiling, Option B interleaved traversal
checkpoint restoration could keep using the uncompiled `DeepCFRMLP` pass: `docs/archive/deep-cfr-performance-experiments-2026-05-07.md`.
class. - **Batched traversal inference design rationale (A vs B vs C)** —
durable architecture note:
Measurement (8 iterations on `default.yaml`, eval and checkpoint `docs/research/batched-traversal-inference-decision.md`.
disabled, iteration 1 dropped as compile warm-up): - **Post-A optimization calculus** — forward-looking sequencing
recorded before Option A was benched:
| | iter mean | 1000-iter projection | `docs/archive/post-a-optimization-calculus-2026-05-07.md`.
| --- | ---: | ---: | - **Option A bench result + structural-ceiling diagnosis** — why
| Baseline (no compile) | 17.93s | 4.98h | Option A regressed (sync-blocking policy boundary, not IPC plumbing)
| `torch.compile` on trainer nets | 18.79s | 5.22h | and the criteria for revisiting it:
| Effect | +0.86s (+4.8%) | +14 min | `docs/archive/option-a-bench-result-2026-05-07.md`.
Net result: regression. Two reasons:
- Traversal is ~60% of iteration time and runs in CPU multiprocessing
workers that reconstruct networks from cleaned `state_dict`s, so they
bypass the compiled wrapper entirely.
- `DeepCFRMLP` (512-hidden, 3-layer) is small enough that the compiled
call dispatch overhead exceeds the kernel-fusion benefit.
Implementation preserved on branch `experiments/torch-compile` for
revisiting if the trainer model grows substantially or after the
batched-traversal-inference work in Optimization Priorities #5 lands —
that is the change that would put compile on the dominant phase, not
just on the trainer's optimization steps. Not enabled on `main`.
### AMP on trainer networks (2026-05-07, regression)
Wrapped the trainer optimization phases with `torch.autocast(fp16)` and
`torch.amp.GradScaler`: `_train_advantage` and `_train_strategy` now run
their network forward/backward/optimizer step through the AMP path when
`run.use_amp=true` and the trainer device is CUDA.
Safety mitigations included in the implementation:
- `GradScaler.unscale_(optimizer)` is called before `clip_grad_norm_`.
- Non-finite loss guard increments `amp/nonfinite_loss_count` and skips the
bad step instead of applying it.
- Advantage squared loss computes `diff.float().square()` so the loss
reduction is fp32 even when the forward path is autocast to fp16.
- Strategy logits are cast back to fp32 before `masked_fill` and
`log_softmax`.
- Metrics now expose `amp/grad_scale` and `amp/nonfinite_loss_count`.
Measurement used the small `smoke.yaml` config with synthetic replay-memory
samples via:
```bash
uv run python scripts/bench_amp_trainer.py \
--config configs/deep_cfr/smoke.yaml \
--runs 3 \
--warmup 1 \
--device cuda
```
| | mean ms/call | speedup vs fp32 |
| --- | ---: | ---: |
| fp32 | 3.22 | 1.00× |
| AMP (fp16) | 3.92 | 0.82× |
Net result: regression. This matches the same dispatch-overhead-vs-kernel
benefit dynamic as the `torch.compile` regression above: the current trainer
model and smoke workload are too small for AMP's lower-precision kernels to
pay back autocast and scaler bookkeeping overhead.
The full `default.yaml` 100-iteration A/B was intentionally skipped. Given
the small-model regression and the matching `torch.compile` precedent on the
same model family, there is no current evidence that spending GPU time on the
longer A/B would produce a different decision. The infrastructure is kept
merged but default-off: `run.use_amp=false` remains the default, and
re-enabling is a one-field config flip.
Re-measure AMP only after the model grows to at least `hidden_size >= 1024`
or `num_layers >= 6`. At that point run both the fast
`scripts/bench_amp_trainer.py` micro-bench and the formal 100-iteration
fp32-vs-AMP A/B. If AMP still provides less than 5% speedup at that larger
model size, keep it default-off and raise the next re-measure trigger to an
even larger model.
### GPU forward profiling for batched traversal (2026-05-07, decision support)
To decide whether Optimization Priorities #5 (batched traversal inference) is
worth implementing, profiled `DeepCFRMLP` from `default.yaml`
(input_dim=365, output_dim=22, hidden=512, 3 layers, ReLU) on an RTX 3090 in
`eval()` + `inference_mode`, with 10-iter warm-up and 1000-iter measurement
per batch size. Script: `scripts/profile_gpu_forward.py`.
| Batch size | μs/call | μs/state | Speedup vs bs=1 |
| ---: | ---: | ---: | ---: |
| 1 | 80.07 | 80.074 | 1.00× |
| 4 | 81.20 | 20.299 | 3.94× |
| 16 | 91.30 | 5.706 | 14.03× |
| 64 | 93.61 | 1.463 | 54.75× |
| 256 | 88.34 | 0.345 | 232.03× |
| 1024 | 161.95 | 0.158 | 506.30× |
Policy-call supply from
`runs/tmp/2026-05-07_181155_deep-cfr-default/metrics.jsonl`: mean
`traversal/nodes` ≈ 205,810 over 280 traversals/player → ~368 policy calls per
traversal (rough upper bound on batchable states), ~200k per iteration across
560 traversals.
Verdict: **Priority #5 is worth pursuing.** Per-state cost drops from 80 μs at
bs=1 to 0.34 μs at bs=256 (>230×). The available supply of ~368 states per
traversal sits comfortably in the bs=64–256 range where μs/call plateaus near
90 μs. End-to-end gain will be bounded by encoding and worker-GPU coordination
overhead, but the GPU forward is not the limiter once batching is in place.
### Option B interleaved traversal (2026-05-07, pass)
Implemented a non-default interleaved traversal scheduler behind:
```yaml
traversal:
scheduler: interleaved
```
The recursive Cython path remains the default. The current interleaved path is
a Python explicit-stack production prototype, guarded to the narrow case
`sampling_mode=outcome`, `opponent_policy=network`,
`cutoff_value_mode=score_diff`, `cutoff_rollouts=0`, and
`inference_backend=local`.
Measurement used `configs/deep_cfr/default.yaml` with evaluation and
checkpointing disabled, 10 iterations, first 2 iterations dropped as warm-up.
Because interleaved currently supports only `opponent_policy=network`, the
recursive baseline used the same opponent-policy override.
Commands:
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-recursive-network-10i \
--set traversal.opponent_policy=network \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-interleaved-workers-10i \
--set traversal.scheduler=interleaved \
--set traversal.opponent_policy=network \
--set traversal.num_workers=8 \
--set traversal.worker_chunk_size=64 \
--set traversal.interleave_width=64 \
--set traversal.interleave_max_batch=128 \
--set traversal.progress_every_traversals=0 \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set run.max_iterations=10 \
--set run.experiment_name=option-b-interleaved-cuda-single-10i \
--set traversal.scheduler=interleaved \
--set traversal.opponent_policy=network \
--set traversal.num_workers=0 \
--set traversal.interleave_width=64 \
--set traversal.interleave_max_batch=128 \
--set traversal.progress_every_traversals=0 \
--set checkpoint.save_latest=false \
--set checkpoint.save_every=0 \
--set evaluation.eval_every=0
```
Result paths:
- `runs/2026-05-07_225044_option-b-recursive-network-10i`
- `runs/2026-05-07_225342_option-b-interleaved-workers-10i`
- `runs/2026-05-07_225542_option-b-interleaved-cuda-single-10i`
Warm-up-excluded means:
| Mode | iter s | traversal s | traversal speedup | iter speedup | nodes/s | batch mean | batch max |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| recursive, 8 workers, chunk 8 | 16.49 | 10.22 | 1.00× | 1.00× | 17.6k | 1.0 | 1 |
| interleaved, 8 workers, chunk 64 | 10.81 | 4.92 | 2.08× | 1.53× | 31.3k | 35.2 | 64 |
| interleaved, single CUDA process | 11.51 | 5.35 | 1.91× | 1.43× | 24.7k | 36.7 | 64 |
Net result: **PASS for the Phase 3 traversal-speed gate.** The best candidate
is the 8-worker interleaved path with larger worker chunks. It reaches real
batches near the target regime (`max_batch_size=64`) and more than doubles
traversal wall-clock versus the recursive network-opponent baseline.
The single-process CUDA path confirms that GPU forward is no longer the
dominant cost once batching works (`interleaved/forward_seconds` averaged
0.64s versus 4.53s for CPU worker forward), but it gives up multiprocessing
game-state throughput and is slower end-to-end than 8 interleaved CPU workers.
Important caveat: multi-traversal interleaving uses per-context RNG streams, so
exact recursive-batch RNG ordering is intentionally not preserved. The Phase 2
single-traversal parity test matches recursive stats and sample target
checksums under identical RNG seed. This means the default now favors the
measured traversal-speed win over byte-identical sample ordering. If future
learning curves show unexplained drift, first compare against the recursive
fallback:
```bash
--set traversal.scheduler=recursive \
--set traversal.worker_chunk_size=8 \
--set traversal.progress_every_traversals=10
```
Follow-up: `average_strategy` support was added after the initial Phase 3
network-opponent A/B so the interleaved path can run the actual default opponent
policy. A 10-iteration throughput check with default opponent policy,
evaluation/checkpoint disabled, and the same 8-worker chunk-64 interleaving
settings produced warm-up-excluded means:
| Mode | iter s | traversal s | batch mean | batch max |
| --- | ---: | ---: | ---: | ---: |
| interleaved, default `average_strategy` | 10.61 | 4.85 | 28.3 | 64 |
Run: `runs/2026-05-07_230419_option-b-interleaved-average-strategy-10i`.
This follow-up unblocked making interleaved traversal the default. The default
switch was made with the caveat above rather than waiting for a long-run A/B.
## Batched Traversal Inference: Design Decision (2026-05-07)
Three structural options were considered for Priority #5:
- **A. Central inference server.** Workers stay in multiprocessing and
reconstruct nothing on GPU. A separate server process owns the model, batches
incoming policy requests across workers, runs GPU forward, returns logits.
Worker traversal logic and the Cython recursion are untouched.
- **B. Per-worker batching.** Each worker interleaves multiple traversals
internally to form its own batches. GPU process count = worker count, so model
copies and GPU contention scale with workers. Batching efficiency is bounded
by per-worker in-flight count.
- **C. Single-process vectorized traversal.** Drop multiprocessing entirely.
Main process runs N traversals lockstep with explicit recursion stacks,
forming a natural batch dimension across traversal instances. Existing
recursive traversal can be kept and a new `traversal/batched.{py,pyx}` added
as a parallel backend gated by config; existing code is not modified.
### Decision: A
Reasons:
- **Hardware fit dominates.** A central inference server keeps multiprocessing,
so all available CPU cores stay productive on game logic. C is single-process,
so on a 32-core remote machine with a weak GPU it wastes 31 cores while the
weak GPU caps batching gains; A is strictly better there. On a 6-core / RTX
3090 box A and C are competitive but uncertain — C only wins when GPU forward
is the dominant share of traversal, and game logic in CFR traversal is not
negligible.
- **C is not the "ultimate" answer on multi-core machines.** A truly maximal
design would combine C's batched GPU forward with `nogil` threaded game
logic, which is strictly more complex than C alone. Plain C, by being
single-process, gives up CPU parallelism that the existing multiprocessing
path already exploits.
- **A is mostly additive.** New modules: `inference_server.py`,
`inference_client.py`, shared-memory tensor pool, weight-sync hook. Existing
touches are small: worker policy call site (one line), worker spawn/teardown
(server start/stop), trainer (periodic weight push). Cython traversal
recursion, game engine, replay/training paths are unchanged.
- **The hard part is IPC tuning, not code volume.** Latency budget vs GPU
forward, weight-staleness window, backpressure, and shared-memory tensor
layout. Code is small; the design surface is concentrated in one place.
### IPC: what crosses the process boundary
Only the encoded policy input and its response cross IPC:
- Forward request: encoded state vector, ~365 floats ≈ 1.5KB.
- Forward response: action logits, ~22 floats ≈ 88 bytes.
Game state, traversal recursion stack, event log, CFR regret/strategy
accumulators, and chance-node sampling history all stay inside the worker
process. The policy network consumes a flat encoded state (`input_dim=365`),
so the server needs no game-tree context to answer a request.
The replay-buffer write path (workers shipping collected regret/strategy
samples to the trainer) is separate, already exists today, and is reflected in
`memory_add_seconds` ≈ 1.25s/iter; A does not add to it.
### IPC mechanism: multiprocessing + shared memory
- **Big payload (state, logits)**: shared-memory tensors. Either
`torch.multiprocessing` with `tensor.share_memory_()` and a pre-allocated
buffer pool indexed by slot id, or `multiprocessing.shared_memory.SharedMemory`
with manual slot management. Pickle is bypassed for the data itself.
- **Control messages (slot index, request id)**: small `Queue`. Pickle still
happens here but only for ints/tuples, which is sub-microsecond and
negligible against ~90 μs GPU forward.
- The naive path (`multiprocessing.Queue(tensor)` with default pickle) is the
one that is slow and is what causes the "Python IPC is slow" reputation.
With shared memory, multiprocessing IPC is effectively on par with thread
shared-memory access for tensor traffic.
### Why not Cython `nogil` + threading instead of multiprocessing
Threading would avoid IPC entirely, but it requires the game-engine hot path
to be genuinely `nogil`-clean — no Python objects touched anywhere on the path.
Whether the existing Cython traversal qualifies is unknown and likely no:
auditing and migrating it to be fully `nogil`-clean is a substantial,
high-risk change to existing code, contradicting A's "mostly additive"
property. Additional drawbacks: a single segfault kills all threads;
multi-threaded CUDA usage has subtle context-sharing pitfalls; tooling and
prior art are weaker than for the multiprocessing pattern. Revisit only after
free-threaded Python (PEP 703) stabilizes or if a future profile shows the
shared-memory IPC is itself the limiter.
### Implementation plan
1. Prototype A on the 6-core / 3090 host with a single worker: validate
end-to-end correctness and measure IPC round-trip latency vs GPU forward.
2. Scale to multiple workers; tune `batch_window_us`, `max_batch`, and
`sync_every` (weight push frequency).
3. Deploy to the 32-core / weak-GPU remote and confirm CPU-side scaling holds
and the weak GPU is still the right place to keep the model.
4. Defer C. Re-evaluate only if A's measurements show GPU forward is no longer
on the critical path and game-logic CPU cost dominates — in that case the
right next step is C with `nogil` threading, not plain C.
## Post-A Optimization Calculus (forward-looking, 2026-05-07)
Once Option A lands, the bottleneck shape changes. This section records the
expected sequencing for follow-up work. It is forward-looking and has not been
measured yet — verify against bench numbers after A is benchmarked.
### Why compile / TensorRT are negligible *today* but become meaningful later
Today (small model: 3-layer, 512 hidden):
- `torch.compile` on the trainer's networks already regressed (see the
2026-05-07 experiment above). The model is too small for kernel fusion to
beat compile dispatch overhead.
- `torch.compile` / TensorRT on the inference-server forward (post-A) would
shave ~30–50% off ~90μs/call → ~50–70μs/call. With forward share of an iter
reduced to <1% by A's batching, the iter-level multiplier is ~1.00–1.01×.
Negligible.
Two compounding shifts can flip this:
1. **Larger model.** Going from 512 hidden / 3 layers to ~1024 hidden /
~6 layers pushes the forward call out of dispatch-bound territory into
kernel-bound territory. Compile fusion and TensorRT both deliver real
1.5–2× on the forward call itself once the kernel is large enough to
amortize launch overhead. Forward share of iter time also rebalances upward
because per-call time scales with FLOPs while batching gain is fixed.
2. **Denser, larger evaluation.** Moving toward `eval_every: 5` and
`evaluation.games: 1000` makes evaluation about half of iteration wall-clock
(see the amortized eval table earlier in this doc). Eval is pure inference,
so TensorRT on the inference-server's forward path applies directly.
When both shifts happen together, an illustrative future iter (rough order of
magnitude only):
| Configuration | Iter time (rough) |
| --- | ---: |
| Today (small model, eval_every=25) | 17.85s |
| + A (batched traversal inference) | ~14s |
| + larger model (≈4× FLOPs), no compile/TRT | ~50s |
| + dense eval (eval_every=5, games=1000) | ~70s |
| + compile (trainer) + TensorRT (inference) | ~45s |
That last row is where compile/TensorRT contributes ~1.5× iter — the same
tooling that is iter-neutral today. The numbers above are illustrative; real
ratios depend on model size, kernel autotune outcomes, and the eval-vs-train
balance.
### Tooling split
- **TensorRT**: applies only to inference (no backward). Targets:
- inference-server forward in traversal,
- inference-server forward in evaluation.
Both are served by the same A-era server, so a single TensorRT integration
covers both.
- **`torch.compile`**: applies to trainer's advantage/strategy training
(forward+backward+optimizer). The 2026-05-07 regression on a small model
does **not** generalize — it must be re-measured on whatever larger model
config we settle on. Do not conclude "compile is bad" from the small-model
data point.
### Recommended sequencing
Do this in order. Skipping ahead is the failure mode that creates misleading
"compile/TRT didn't help" data.
1. **Now**: benchmark A (`scripts/bench_inference_backend.py`) and confirm the
`local` vs `server` multipliers on `home` and `remote`. Validate the iter
1.2–1.3× / traversal 1.5–2× working estimate.
2. **Next**: experiment with a larger network config. Measure compute vs
learning-curve trade-off with the existing toolchain (no compile/TRT yet).
This step decides the model size that future optimizations target.
It is also the prerequisite for revisiting AMP, `torch.compile`, and
TensorRT: all three are dispatch-overhead-bound on the current small model.
3. **Then**: re-measure `torch.compile` on the trainer at the chosen model
size. The earlier regression was size-bound; expect a different result.
4. **Then**: integrate TensorRT into the inference server (covers traversal
and eval forward simultaneously). Bound the gain by the post-step-2
`policy_network_seconds` share, not the headline TensorRT speedup.
5. **In parallel with 2–4**: if denser eval is operationally useful, raise
`evaluation.games` and lower `evaluation.eval_every`. This step does not
require code changes but sharply increases the value of step 4.
Out of scope until A bench numbers are in: Option C, `nogil` threading, async
inference client, compiled encoding.
## Option A Bench Result and Structural Ceiling (2026-05-07)
Option A (`traversal.inference_backend: server`) was implemented and
benchmarked. **Result: regression. A is deferred. `default.yaml` stays on
`local`. The implementation is preserved behind the flag for future revisit.**
### Bench numbers
`scripts/bench_inference_backend.py --device cuda --iterations 5 --warmup 1`,
RTX 3090, after a per-call IPC fix (replaced `multiprocessing.Manager()`
queues/events with spawn-context primitives, slot reuse per worker batch,
shared memory confirmed in use for state/response payloads).
| Backend | iter | traversal | adv_train | strat_train | mem_add | batch_tensor |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `local` | 16.75s | 10.81s | 3.91s | 2.02s | 0.95s | 3.56s |
| `server` | 57.61s | 51.61s | 3.96s | 2.02s | 0.50s | 3.57s |
| Speedup | 0.29× | 0.21× | 0.99× | 1.00× | 1.89× | 1.00× |
Raw: `runs/bench/2026-05-07_193335_inference_backend/results.json`.
Training and eval phases are unchanged (as expected — A only touches the
traversal forward path). The regression is contained in `traversal_seconds`,
which is ~5× worse.
### Diagnosis
The server emits per-flush batch stats. **Mean batch size: ~7.2–7.9, max 8.**
This is the structural ceiling, not a tunable misconfiguration:
- Traversal recursion is **sync-blocking** at the policy call site. Each
worker has at most one in-flight policy request at a time.
- In-flight requests at the server ≤ `num_workers` = 8.
- The server further splits each batch by `(network_kind, network_index)`,
so the actual GPU forward group size is roughly half of that — about 4
rows per group.
Per-state cost at this realized batch size, from the GPU profile table:
| Realized batch | μs/state |
| ---: | ---: |
| 1 | 80.07 |
| 4 | 20.30 |
| 8 (extrapolated) | ~12 |
| 64 | 1.46 |
| 256 | 0.34 |
So the GPU is doing ~12μs per state instead of the projected ~1.5μs at
bs=64. The IPC round-trip per call (queue post + server scheduler + event
wakeup, even with shared-memory payload) is on the order of hundreds of μs
per call, which exceeds both the local CPU forward (~80–200μs at bs=1 on
this small MLP) and the marginal GPU gain. Net: per-call cost roughly
doubles or triples, compounded across ~205k calls/iter, gives the observed
5× traversal regression.
`batch_window_us` and `max_batch` tuning cannot escape this ceiling —
there are simply not 64 concurrent in-flight requests to coalesce when only
8 workers are blocking-sync.
### What this means for the headline GPU profile (`scripts/profile_gpu_forward.py`)
The earlier "230× speedup at bs=256" is a **per-state GPU forward**
microbenchmark, not an end-to-end traversal speedup. Realizing that gain
requires *actually feeding the GPU* with bs=64+ batches. Sync-blocking
multi-worker traversal cannot do that. Reaching the bs=64 regime needs
either:
- Per-worker traversal interleaving (worker advances `worker_chunk_size`
traversals concurrently, suspending at each policy call — Option B
shape), which requires turning Cython traversal recursion into a
resumable state machine. Same scope as a partial Option C, localized to
worker scope.
- Option C proper (single-process vectorized traversal).
Both require restructuring traversal. Option A's "additive, no traversal
changes" property turned out to also mean "cannot drive the batch size up."
### Clarifying the traversal bottleneck: sync policy boundary, not SIMD
The tempting shorthand is "Python/GIL prevents traversal from using SIMD or
threads." The more precise diagnosis is narrower:
- Lost Cities game mechanics are already mostly Cython C-level operations.
`legal_actions`, action push/pop, and cached scoring are not Python list
walks on the hot path.
- The traversal recursion is Cython, but it synchronously crosses back into
Python/PyTorch at every policy-needed state: encode a single info state,
run one-row PyTorch forward, copy logits back to CPU/Numpy, then continue
recursion.
- This boundary makes every traversal worker **sync-blocking**. With
`num_workers=8`, the inference server can see at most eight in-flight
requests before per-network splitting, no matter how large `max_batch` is.
- GIL-free threading would help only after the same path is made
`nogil`-clean or after traversal is restructured so policy calls can be
batched. Simply "using SIMD" does not address the one-row policy boundary.
So the actionable bottleneck is **policy-call scheduling shape**, not scalar
game-rule arithmetic. The highest-leverage experiment is Option B:
per-worker interleaved traversal, where one worker advances many traversals,
suspends each at a policy request, batches those requests, and resumes the
corresponding continuations.
Microbench evidence (2026-05-07, `configs/deep_cfr/default.yaml`,
`experiments/traversal_policy_boundary/bench_policy_boundary.py`):
| Device | Component | median μs/call | p95 μs/call |
| --- | --- | ---: | ---: |
| CPU | encode + legal | 3.10 | 3.81 |
| CPU | push + pop | 0.15 | 0.22 |
| CPU | policy boundary bs=1 | 111.50 | 125.46 |
| CPU | torch forward bs=64 | 12.84 | 13.14 |
| CUDA | policy boundary bs=1 | 181.30 | 194.77 |
| CUDA | torch forward bs=64 | 2.55 | 2.75 |
This confirms the bottleneck is not Cython game-rule scalar work. The
single-request policy boundary is ~36× larger than encode+legal on CPU, while
CUDA bs=64 forward is ~71× cheaper than the current CUDA bs=1 boundary.
Expected upside is bounded by the fraction of traversal currently spent at
policy calls. Moving realized GPU forward from the current ~4-8 row regime
(~12-20μs/state) to bs=64 (~1.46μs/state) is an ~8-14× improvement on the
forward component, but not on game recursion, sample creation, or replay
writes. For the observed `local` traversal around 10-13s/iter, a realistic
first target is roughly **1.5-3× traversal speedup** if Option B reaches the
bs=64 regime without adding comparable scheduler overhead. Larger claims need
a prototype because traversal has substantial non-forward work.
### Why deferring A (not deleting) is the right call
- The plumbing (server process, shared-memory client, weight sync, config
flag) is complete and tested. Re-enabling is a config flip.
- The fundamental issue at this model size is that **GPU forward time is
too small to amortize IPC overhead** at any realistic batch size we can
drive without restructuring traversal. Bigger model changes that
arithmetic; the same plumbing then becomes useful.
- The `mem_add_seconds` row showed a real 1.89× win, suggesting the
shared-memory replay-write path adopted along the way is worth keeping
even with `local` backend. (Confirm separately; this is a side effect.)
### Re-enable A when one of these holds
1. **Model grows** to ~1024 hidden / ~6 layers (compile/TRT discussion
above). Forward time scales with FLOPs while IPC overhead is fixed; at
some point IPC becomes a small fraction.
2. **Per-worker interleaved traversal** ships (Option B-shape refactor).
Drives realized batch toward 64 and reclaims the profile table's gains.
3. **Eval becomes the dominant phase** (`eval_every: 5`,
`evaluation.games: 1000`). Eval is not sync-blocking traversal; it is
already batch_size=64 in eval code. The same inference server can
serve eval directly without the worker-side ceiling.
### Free-threaded Python (3.13t) note
Free-threaded Python + Cython `nogil` would let many threads (well above
core count) run game logic concurrently in one process, with shared memory
and no IPC. With ~64 threads sync-blocking on policy calls, the server
would naturally see bs=64. **In principle this is the cleanest endpoint.**
In practice as of early 2026:
- Free-threaded Python is an opt-in build (`python3.13t`), still
experimental, with measurable single-thread overhead.
- PyTorch's free-threaded compatibility is partial.
- Cython `nogil`-cleanliness audit on the existing game engine is still
required and was the original reason `nogil` threading was deferred in
the design decision above.
- No project-level adoption pressure on `python3.13t` today.
So free-threaded Python does change the architectural answer, but it does
not unblock A *now*. Track the ecosystem; revisit when (a) `python3.13t`
becomes mainstream or (b) the Cython engine is `nogil`-cleaned for other
reasons.
docs/plans/librarian.md
+23 -5
View File
@@ -165,11 +165,29 @@ deterministic detection:
and derived research notes; LLM should decide whether two passages and derived research notes; LLM should decide whether two passages
are the *same idea* vs the *same evidence*. are the *same idea* vs the *same evidence*.
Open Stage 1 finding to address: `docs/performance.md` at 914 lines. ## Stage 1 finding closed
`docs/performance.md` 914-line oversize finding is resolved by
routing the dated experiments and design analysis out of the file:
- `docs/archive/deep-cfr-performance-experiments-2026-05-07.md` —
`torch.compile`, AMP, GPU-forward profiling, Option B (4 sub-experiments).
- `docs/research/batched-traversal-inference-decision.md` —
durable A/B/C design rationale.
- `docs/archive/post-a-optimization-calculus-2026-05-07.md` —
forward-looking sequencing recorded pre-bench.
- `docs/archive/option-a-bench-result-2026-05-07.md` —
bench regression + structural-ceiling diagnosis.
`docs/performance.md` trimmed to 345 lines and now points at the
extracts via a "See Also" section. AGENTS.md soft-cap rule reworded
to clarify it is a *routing trigger*, not a split mandate.
## Next Concrete Step ## Next Concrete Step
Address the open Stage 1 finding by splitting `docs/performance.md` Stage 2 — LLM judgment dispatcher. Read `librarian-<timestamp>.json`
into sub-topic notes under `docs/research/` (and dated archive + relevant doc bodies, route to `LIBRARIAN_LLM={claude|codex|gemini}`
entries where appropriate). Sketch the split as a 1-page sub-plan with `scripts/librarian-prompt.md` as system prompt, emit a unified
before doing the actual move so we don't shred a useful document. diff under `runs/tmp/librarian-<timestamp>.patch`. Initial use cases:
research-note drafts for promotable archive entries, MEMORY.md drift
fixups, duplicate-doc merge proposals.
docs/research/batched-traversal-inference-decision.md
+126
View File
@@ -0,0 +1,126 @@
# Batched Traversal Inference: Design Decision (A vs B vs C)
**Last verified:** 2026-05-07
**Source:** `docs/performance.md` § "Batched Traversal Inference: Design Decision"
(extracted to `docs/research/` on 2026-05-08).
**See also:** `docs/archive/option-a-bench-result-2026-05-07.md` for the
post-implementation bench, which deferred Option A. The design rationale
below is captured *as it stood at decision time* — Option A's eventual
regression sharpens but does not invalidate the framework: the bench
later showed that the sync-blocking policy boundary, not the IPC
plumbing, was the actual ceiling.
## Problem
Traversal currently calls the policy network one row at a time. The GPU
profile (`scripts/profile_gpu_forward.py`) shows >230× per-state
speedup at bs=256 vs bs=1, with `~368` policy-needed states per
traversal as the available supply. We need a structural change to
reach that batch regime.
## Three structural options
- **A. Central inference server.** Workers stay in multiprocessing and
reconstruct nothing on GPU. A separate server process owns the model, batches
incoming policy requests across workers, runs GPU forward, returns logits.
Worker traversal logic and the Cython recursion are untouched.
- **B. Per-worker batching.** Each worker interleaves multiple traversals
internally to form its own batches. GPU process count = worker count, so model
copies and GPU contention scale with workers. Batching efficiency is bounded
by per-worker in-flight count.
- **C. Single-process vectorized traversal.** Drop multiprocessing entirely.
Main process runs N traversals lockstep with explicit recursion stacks,
forming a natural batch dimension across traversal instances. Existing
recursive traversal can be kept and a new `traversal/batched.{py,pyx}` added
as a parallel backend gated by config; existing code is not modified.
## Decision: A
Reasons:
- **Hardware fit dominates.** A central inference server keeps multiprocessing,
so all available CPU cores stay productive on game logic. C is single-process,
so on a 32-core remote machine with a weak GPU it wastes 31 cores while the
weak GPU caps batching gains; A is strictly better there. On a 6-core / RTX
3090 box A and C are competitive but uncertain — C only wins when GPU forward
is the dominant share of traversal, and game logic in CFR traversal is not
negligible.
- **C is not the "ultimate" answer on multi-core machines.** A truly maximal
design would combine C's batched GPU forward with `nogil` threaded game
logic, which is strictly more complex than C alone. Plain C, by being
single-process, gives up CPU parallelism that the existing multiprocessing
path already exploits.
- **A is mostly additive.** New modules: `inference_server.py`,
`inference_client.py`, shared-memory tensor pool, weight-sync hook. Existing
touches are small: worker policy call site (one line), worker spawn/teardown
(server start/stop), trainer (periodic weight push). Cython traversal
recursion, game engine, replay/training paths are unchanged.
- **The hard part is IPC tuning, not code volume.** Latency budget vs GPU
forward, weight-staleness window, backpressure, and shared-memory tensor
layout. Code is small; the design surface is concentrated in one place.
## IPC: what crosses the process boundary
Only the encoded policy input and its response cross IPC:
- Forward request: encoded state vector, ~365 floats ≈ 1.5KB.
- Forward response: action logits, ~22 floats ≈ 88 bytes.
Game state, traversal recursion stack, event log, CFR regret/strategy
accumulators, and chance-node sampling history all stay inside the worker
process. The policy network consumes a flat encoded state (`input_dim=365`),
so the server needs no game-tree context to answer a request.
The replay-buffer write path (workers shipping collected regret/strategy
samples to the trainer) is separate, already exists today, and is reflected in
`memory_add_seconds` ≈ 1.25s/iter; A does not add to it.
## IPC mechanism: multiprocessing + shared memory
- **Big payload (state, logits)**: shared-memory tensors. Either
`torch.multiprocessing` with `tensor.share_memory_()` and a pre-allocated
buffer pool indexed by slot id, or `multiprocessing.shared_memory.SharedMemory`
with manual slot management. Pickle is bypassed for the data itself.
- **Control messages (slot index, request id)**: small `Queue`. Pickle still
happens here but only for ints/tuples, which is sub-microsecond and
negligible against ~90 μs GPU forward.
- The naive path (`multiprocessing.Queue(tensor)` with default pickle) is the
one that is slow and is what causes the "Python IPC is slow" reputation.
With shared memory, multiprocessing IPC is effectively on par with thread
shared-memory access for tensor traffic.
## Why not Cython `nogil` + threading instead of multiprocessing
Threading would avoid IPC entirely, but it requires the game-engine hot path
to be genuinely `nogil`-clean — no Python objects touched anywhere on the path.
Whether the existing Cython traversal qualifies is unknown and likely no:
auditing and migrating it to be fully `nogil`-clean is a substantial,
high-risk change to existing code, contradicting A's "mostly additive"
property. Additional drawbacks: a single segfault kills all threads;
multi-threaded CUDA usage has subtle context-sharing pitfalls; tooling and
prior art are weaker than for the multiprocessing pattern. Revisit only after
free-threaded Python (PEP 703) stabilizes or if a future profile shows the
shared-memory IPC is itself the limiter.
## Implementation plan (as of decision time)
1. Prototype A on the 6-core / 3090 host with a single worker: validate
end-to-end correctness and measure IPC round-trip latency vs GPU forward.
2. Scale to multiple workers; tune `batch_window_us`, `max_batch`, and
`sync_every` (weight push frequency).
3. Deploy to the 32-core / weak-GPU remote and confirm CPU-side scaling holds
and the weak GPU is still the right place to keep the model.
4. Defer C. Re-evaluate only if A's measurements show GPU forward is no longer
on the critical path and game-logic CPU cost dominates — in that case the
right next step is C with `nogil` threading, not plain C.
## Outcome (post-bench, see archive)
The plan was executed and Option A was benchmarked. Result: 0.21×
traversal regression because the *sync-blocking* policy boundary capped
realized batch size at ~7 (not the IPC plumbing). The structural
ceiling diagnosis and the criteria for revisiting A live in
`docs/archive/option-a-bench-result-2026-05-07.md`. Option B
(per-worker interleaved traversal) became the production path because
it actually drives realized batch size up by suspending traversals at
each policy call.