coorl-lost-cities/docs/research/batched-traversal-inference-decision.md

Batched Traversal Inference: Design Decision (A vs B vs C)

Last verified: 2026-05-07 Source: docs/performance.md § "Batched Traversal Inference: Design Decision" (extracted to docs/research/ on 2026-05-08). See also: docs/archive/option-a-bench-result-2026-05-07.md for the post-implementation bench, which deferred Option A. The design rationale below is captured as it stood at decision time — Option A's eventual regression sharpens but does not invalidate the framework: the bench later showed that the sync-blocking policy boundary, not the IPC plumbing, was the actual ceiling.

Problem

Traversal currently calls the policy network one row at a time. The GPU profile (scripts/profile_gpu_forward.py) shows >230× per-state speedup at bs=256 vs bs=1, with ~368 policy-needed states per traversal as the available supply. We need a structural change to reach that batch regime.

Three structural options

• A. Central inference server. Workers stay in multiprocessing and reconstruct nothing on GPU. A separate server process owns the model, batches incoming policy requests across workers, runs GPU forward, returns logits. Worker traversal logic and the Cython recursion are untouched.
• B. Per-worker batching. Each worker interleaves multiple traversals internally to form its own batches. GPU process count = worker count, so model copies and GPU contention scale with workers. Batching efficiency is bounded by per-worker in-flight count.
• C. Single-process vectorized traversal. Drop multiprocessing entirely. Main process runs N traversals lockstep with explicit recursion stacks, forming a natural batch dimension across traversal instances. Existing recursive traversal can be kept and a new traversal/batched.{py,pyx} added as a parallel backend gated by config; existing code is not modified.

Decision: A

Reasons:

• Hardware fit dominates. A central inference server keeps multiprocessing, so all available CPU cores stay productive on game logic. C is single-process, so on a 32-core remote machine with a weak GPU it wastes 31 cores while the weak GPU caps batching gains; A is strictly better there. On a 6-core / RTX 3090 box A and C are competitive but uncertain — C only wins when GPU forward is the dominant share of traversal, and game logic in CFR traversal is not negligible.
• C is not the "ultimate" answer on multi-core machines. A truly maximal design would combine C's batched GPU forward with nogil threaded game logic, which is strictly more complex than C alone. Plain C, by being single-process, gives up CPU parallelism that the existing multiprocessing path already exploits.
• A is mostly additive. New modules: inference_server.py, inference_client.py, shared-memory tensor pool, weight-sync hook. Existing touches are small: worker policy call site (one line), worker spawn/teardown (server start/stop), trainer (periodic weight push). Cython traversal recursion, game engine, replay/training paths are unchanged.
• The hard part is IPC tuning, not code volume. Latency budget vs GPU forward, weight-staleness window, backpressure, and shared-memory tensor layout. Code is small; the design surface is concentrated in one place.

IPC: what crosses the process boundary

Only the encoded policy input and its response cross IPC:

• Forward request: encoded state vector, ~365 floats ≈ 1.5KB.
• Forward response: action logits, ~22 floats ≈ 88 bytes.

Game state, traversal recursion stack, event log, CFR regret/strategy accumulators, and chance-node sampling history all stay inside the worker process. The policy network consumes a flat encoded state (input_dim=365), so the server needs no game-tree context to answer a request.

The replay-buffer write path (workers shipping collected regret/strategy samples to the trainer) is separate, already exists today, and is reflected in memory_add_seconds ≈ 1.25s/iter; A does not add to it.

IPC mechanism: multiprocessing + shared memory

• Big payload (state, logits): shared-memory tensors. Either torch.multiprocessing with tensor.share_memory_() and a pre-allocated buffer pool indexed by slot id, or multiprocessing.shared_memory.SharedMemory with manual slot management. Pickle is bypassed for the data itself.
• Control messages (slot index, request id): small Queue. Pickle still happens here but only for ints/tuples, which is sub-microsecond and negligible against ~90 μs GPU forward.
• The naive path (multiprocessing.Queue(tensor) with default pickle) is the one that is slow and is what causes the "Python IPC is slow" reputation. With shared memory, multiprocessing IPC is effectively on par with thread shared-memory access for tensor traffic.

Why not Cython nogil + threading instead of multiprocessing

Threading would avoid IPC entirely, but it requires the game-engine hot path to be genuinely nogil-clean — no Python objects touched anywhere on the path. Whether the existing Cython traversal qualifies is unknown and likely no: auditing and migrating it to be fully nogil-clean is a substantial, high-risk change to existing code, contradicting A's "mostly additive" property. Additional drawbacks: a single segfault kills all threads; multi-threaded CUDA usage has subtle context-sharing pitfalls; tooling and prior art are weaker than for the multiprocessing pattern. Revisit only after free-threaded Python (PEP 703) stabilizes or if a future profile shows the shared-memory IPC is itself the limiter.

Implementation plan (as of decision time)

1. Prototype A on the 6-core / 3090 host with a single worker: validate end-to-end correctness and measure IPC round-trip latency vs GPU forward.
2. Scale to multiple workers; tune batch_window_us, max_batch, and sync_every (weight push frequency).
3. Deploy to the 32-core / weak-GPU remote and confirm CPU-side scaling holds and the weak GPU is still the right place to keep the model.
4. Defer C. Re-evaluate only if A's measurements show GPU forward is no longer on the critical path and game-logic CPU cost dominates — in that case the right next step is C with nogil threading, not plain C.

Outcome (post-bench, see archive)

The plan was executed and Option A was benchmarked. Result: 0.21× traversal regression because the sync-blocking policy boundary capped realized batch size at ~7 (not the IPC plumbing). The structural ceiling diagnosis and the criteria for revisiting A live in docs/archive/option-a-bench-result-2026-05-07.md. Option B (per-worker interleaved traversal) became the production path because it actually drives realized batch size up by suspending traversals at each policy call.