Add batched traversal inference server (Option A) behind opt-in flag
Implements the central inference server pattern: a dedicated GPU process owns advantage/strategy/league networks, batches policy requests across traversal workers via shared-memory tensor pool, and returns logits. Workers route forward calls through InferenceClient / NetworkProxy when traversal.inference_backend == "server". Default remains traversal.inference_backend: local. The server backend regresses iter time ~3.8× on the inspected default config (small-model dispatch + sync-blocking traversal capping realized batch at ~num_workers=8 instead of the bs=64-256 needed to amortize IPC overhead). Keeping the implementation behind the flag lets us re-enable when (a) model size grows, (b) per-worker interleaved traversal lands, or (c) eval becomes dominant — see docs/performance.md "Option A Bench Result and Structural Ceiling" for the full diagnosis. Plumbing included: - inference_buffers.py: shared-memory tensor pool with slot management. - inference_client.py: per-worker client + NetworkProxy adapter for the existing traversal.pyx call sites. - inference_server.py: spawn-context server process with batch-window aggregation, weight sync, shutdown sentinel. - bench_inference_backend.py: A/B between local and server backends with eval/checkpoint disabled. - test_inference_server.py: round-trip and integration tests. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -46,6 +46,7 @@ traversal:
|
||||
progress_every_traversals: 10
|
||||
endpoint_depth_bucket_width: 100
|
||||
endpoint_depth_bucket_max: 1000
|
||||
inference_backend: local
|
||||
|
||||
regret_matching:
|
||||
all_negative_fallback: argmax_tiebreak
|
||||
@@ -97,3 +98,11 @@ evaluation:
|
||||
batch_size: 64
|
||||
device: trainer
|
||||
num_workers: 4
|
||||
|
||||
inference_server:
|
||||
device: cuda
|
||||
num_slots: null
|
||||
max_batch: 256
|
||||
batch_window_us: 200
|
||||
weight_sync_every: 1
|
||||
use_amp: false
|
||||
|
||||
Reference in New Issue
Block a user