coorl-lost-cities/docs/archive/option-a-bench-result-2026-05-07.md

# Option A Bench Result and Structural Ceiling (2026-05-07)

**Source:** Extracted from `docs/performance.md` § "Option A Bench
Result and Structural Ceiling" on 2026-05-08.

Records the post-implementation Option A benchmark (regression: 0.21×
traversal), the diagnosis of the sync-blocking policy boundary as the
real ceiling, and the criteria under which to revisit Option A.

**Related:**
- Design rationale: `docs/research/batched-traversal-inference-decision.md`
- Forward-looking sequencing (pre-bench): `docs/archive/post-a-optimization-calculus-2026-05-07.md`
- Trainer-side experiments from the same date: `docs/archive/deep-cfr-performance-experiments-2026-05-07.md`

---

Option A (`traversal.inference_backend: server`) was implemented and
benchmarked. **Result: regression. A is deferred. `default.yaml` stays on
`local`. The implementation is preserved behind the flag for future revisit.**

## Bench numbers

`scripts/bench_inference_backend.py --device cuda --iterations 5 --warmup 1`,
RTX 3090, after a per-call IPC fix (replaced `multiprocessing.Manager()`
queues/events with spawn-context primitives, slot reuse per worker batch,
shared memory confirmed in use for state/response payloads).

| Backend | iter | traversal | adv_train | strat_train | mem_add | batch_tensor |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `local` | 16.75s | 10.81s | 3.91s | 2.02s | 0.95s | 3.56s |
| `server` | 57.61s | 51.61s | 3.96s | 2.02s | 0.50s | 3.57s |
| Speedup | 0.29× | 0.21× | 0.99× | 1.00× | 1.89× | 1.00× |

Raw: `runs/bench/2026-05-07_193335_inference_backend/results.json`.

Training and eval phases are unchanged (as expected — A only touches the
traversal forward path). The regression is contained in `traversal_seconds`,
which is ~5× worse.

## Diagnosis

The server emits per-flush batch stats. **Mean batch size: ~7.2–7.9, max 8.**
This is the structural ceiling, not a tunable misconfiguration:

- Traversal recursion is **sync-blocking** at the policy call site. Each
  worker has at most one in-flight policy request at a time.
- In-flight requests at the server ≤ `num_workers` = 8.
- The server further splits each batch by `(network_kind, network_index)`,
  so the actual GPU forward group size is roughly half of that — about 4
  rows per group.

Per-state cost at this realized batch size, from the GPU profile table:

| Realized batch | μs/state |
| ---: | ---: |
| 1 | 80.07 |
| 4 | 20.30 |
| 8 (extrapolated) | ~12 |
| 64 | 1.46 |
| 256 | 0.34 |

So the GPU is doing ~12μs per state instead of the projected ~1.5μs at
bs=64. The IPC round-trip per call (queue post + server scheduler + event
wakeup, even with shared-memory payload) is on the order of hundreds of μs
per call, which exceeds both the local CPU forward (~80–200μs at bs=1 on
this small MLP) and the marginal GPU gain. Net: per-call cost roughly
doubles or triples, compounded across ~205k calls/iter, gives the observed
5× traversal regression.

`batch_window_us` and `max_batch` tuning cannot escape this ceiling —
there are simply not 64 concurrent in-flight requests to coalesce when only
8 workers are blocking-sync.

## What this means for the headline GPU profile (`scripts/profile_gpu_forward.py`)

The earlier "230× speedup at bs=256" is a **per-state GPU forward**
microbenchmark, not an end-to-end traversal speedup. Realizing that gain
requires *actually feeding the GPU* with bs=64+ batches. Sync-blocking
multi-worker traversal cannot do that. Reaching the bs=64 regime needs
either:

- Per-worker traversal interleaving (worker advances `worker_chunk_size`
  traversals concurrently, suspending at each policy call — Option B
  shape), which requires turning Cython traversal recursion into a
  resumable state machine. Same scope as a partial Option C, localized to
  worker scope.
- Option C proper (single-process vectorized traversal).

Both require restructuring traversal. Option A's "additive, no traversal
changes" property turned out to also mean "cannot drive the batch size up."

## Clarifying the traversal bottleneck: sync policy boundary, not SIMD

The tempting shorthand is "Python/GIL prevents traversal from using SIMD or
threads." The more precise diagnosis is narrower:

- Lost Cities game mechanics are already mostly Cython C-level operations.
  `legal_actions`, action push/pop, and cached scoring are not Python list
  walks on the hot path.
- The traversal recursion is Cython, but it synchronously crosses back into
  Python/PyTorch at every policy-needed state: encode a single info state,
  run one-row PyTorch forward, copy logits back to CPU/Numpy, then continue
  recursion.
- This boundary makes every traversal worker **sync-blocking**. With
  `num_workers=8`, the inference server can see at most eight in-flight
  requests before per-network splitting, no matter how large `max_batch` is.
- GIL-free threading would help only after the same path is made
  `nogil`-clean or after traversal is restructured so policy calls can be
  batched. Simply "using SIMD" does not address the one-row policy boundary.

So the actionable bottleneck is **policy-call scheduling shape**, not scalar
game-rule arithmetic. The highest-leverage experiment is Option B:
per-worker interleaved traversal, where one worker advances many traversals,
suspends each at a policy request, batches those requests, and resumes the
corresponding continuations.

Microbench evidence (2026-05-07, `configs/deep_cfr/default.yaml`,
`experiments/traversal_policy_boundary/bench_policy_boundary.py`):

| Device | Component | median μs/call | p95 μs/call |
| --- | --- | ---: | ---: |
| CPU | encode + legal | 3.10 | 3.81 |
| CPU | push + pop | 0.15 | 0.22 |
| CPU | policy boundary bs=1 | 111.50 | 125.46 |
| CPU | torch forward bs=64 | 12.84 | 13.14 |
| CUDA | policy boundary bs=1 | 181.30 | 194.77 |
| CUDA | torch forward bs=64 | 2.55 | 2.75 |

This confirms the bottleneck is not Cython game-rule scalar work. The
single-request policy boundary is ~36× larger than encode+legal on CPU, while
CUDA bs=64 forward is ~71× cheaper than the current CUDA bs=1 boundary.

Expected upside is bounded by the fraction of traversal currently spent at
policy calls. Moving realized GPU forward from the current ~4-8 row regime
(~12-20μs/state) to bs=64 (~1.46μs/state) is an ~8-14× improvement on the
forward component, but not on game recursion, sample creation, or replay
writes. For the observed `local` traversal around 10-13s/iter, a realistic
first target is roughly **1.5-3× traversal speedup** if Option B reaches the
bs=64 regime without adding comparable scheduler overhead. Larger claims need
a prototype because traversal has substantial non-forward work.

## Why deferring A (not deleting) is the right call

- The plumbing (server process, shared-memory client, weight sync, config
  flag) is complete and tested. Re-enabling is a config flip.
- The fundamental issue at this model size is that **GPU forward time is
  too small to amortize IPC overhead** at any realistic batch size we can
  drive without restructuring traversal. Bigger model changes that
  arithmetic; the same plumbing then becomes useful.
- The `mem_add_seconds` row showed a real 1.89× win, suggesting the
  shared-memory replay-write path adopted along the way is worth keeping
  even with `local` backend. (Confirm separately; this is a side effect.)

## Re-enable A when one of these holds

1. **Model grows** to ~1024 hidden / ~6 layers (compile/TRT discussion in
   `docs/archive/post-a-optimization-calculus-2026-05-07.md`). Forward
   time scales with FLOPs while IPC overhead is fixed; at some point IPC
   becomes a small fraction.
2. **Per-worker interleaved traversal** ships (Option B-shape refactor).
   Drives realized batch toward 64 and reclaims the profile table's gains.
3. **Eval becomes the dominant phase** (`eval_every: 5`,
   `evaluation.games: 1000`). Eval is not sync-blocking traversal; it is
   already batch_size=64 in eval code. The same inference server can
   serve eval directly without the worker-side ceiling.

## Free-threaded Python (3.13t) note

Free-threaded Python + Cython `nogil` would let many threads (well above
core count) run game logic concurrently in one process, with shared memory
and no IPC. With ~64 threads sync-blocking on policy calls, the server
would naturally see bs=64. **In principle this is the cleanest endpoint.**

In practice as of early 2026:

- Free-threaded Python is an opt-in build (`python3.13t`), still
  experimental, with measurable single-thread overhead.
- PyTorch's free-threaded compatibility is partial.
- Cython `nogil`-cleanliness audit on the existing game engine is still
  required and was the original reason `nogil` threading was deferred in
  the design decision.
- No project-level adoption pressure on `python3.13t` today.

So free-threaded Python does change the architectural answer, but it does
not unblock A *now*. Track the ecosystem; revisit when (a) `python3.13t`
becomes mainstream or (b) the Cython engine is `nogil`-cleaned for other
reasons.