Add Deep CFR research notes derived from archive

Five notes covering outcome-sampling target correctness, package
architecture, v0 feature-parity vs legacy, opponent-policy network
divergence, and regret-matching fallback audit. Four are derived from
archive sources (cited via Source: lines); outcome-sampling-target is
a fresh write-up and serves as the style template.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

docs/research/deep-cfr-architecture.md

@@ -0,0 +1,149 @@
+# Deep CFR Package Architecture and Design Rationale
+
+**Last verified:** 2026-05-07, commit `ad0be89`
+
+Source: `docs/archive/deep-cfr-v0-plan.md`
+
+## Question
+
+What is the package layout of this Deep CFR implementation, why are the
+subsystems split the way they are, and what are the design choices that drive
+the architecture?
+
+Short answer: **the Cython/Python split mirrors the compute profile.** The
+traversal hot path (`traversal.pyx`, `cfr_math.pyx`, `encoding.pyx`) calls
+`GameState` C APIs directly without Python round-trips; everything that runs
+once per iteration or is GPU/I/O-dominated (training, checkpointing, eval, CLI)
+stays in Python. A Pydantic config tree bridges the two by handing typed
+scalars and arrays across the boundary.
+
+## Package layout (current as of `ad0be89`)
+
+```text
+src/coolrl_lost_cities/games/classic/deep_cfr/
+  # Cython hot-path modules
+  cfr_math.pyx / cfr_math.pxd    — regret matching arithmetic (all-negative fallback, etc.)
+  encoding.pyx / encoding.pxd    — information-state feature extraction; Python + C-buffer API
+  traversal.pyx / traversal.pxd  — full Deep CFR tree walking; calls GameState C APIs directly
+
+  # Python training layer
+  config.py          — Pydantic config tree for all subsystems
+  memory.py          — reservoir memory buffers (advantage and strategy)
+  networks.py        — PyTorch advantage and strategy network definitions
+  trainer.py         — orchestration: traverse → sample → train → eval → checkpoint loop
+  checkpoints.py     — save/load for networks, optimizer state, training counters
+  workers.py         — multiprocess traversal worker pool and result merge
+
+  # Inference experiments (opt-in)
+  inference_server.py   — batched policy inference server (Option A batched traversal)
+  inference_client.py   — client stub for the inference server
+  inference_buffers.py  — shared-memory buffer management for batched inference
+
+  # Runtime tooling
+  cli.py             — training and evaluation CLI entry points
+  benchmark.py       — traversal throughput benchmark CLI
+  evaluate.py        — evaluation loop against registered classic bots
+  analyze.py         — metrics.jsonl analysis helpers
+  tracking.py        — run artifact helpers (metrics, runtime_progress.json)
+  traversal_stats.py — structured traversal diagnostic metrics
+
+  # Auxiliary training modes
+  imitation.py       — safe-heuristic imitation pretraining
+  policy_gradient.py — policy-gradient fine-tuning
+```
+
+The compiled `.c` and `.so` artifacts (`cfr_math.c`, `encoding.c`, `traversal.c`
+and their `.cpython-311-x86_64-linux-gnu.so` counterparts) are build outputs of
+the Cython modules and live alongside the sources.
+
+The original plan proposed 11 files; the current package has grown to 20+
+Python/Cython source files as experiments and tooling have been added.
+
+## Why Cython for traversal
+
+The Deep CFR inner loop is an alternating-player tree walk that visits thousands
+to millions of nodes per iteration. Each node needs: legal action enumeration,
+regret matching, policy sampling, state push/pop for action application and
+undo, and value backpropagation. In Python, each of these operations carries
+dict lookups, reference counting, and interpreter overhead that compound across
+the call tree.
+
+`traversal.pyx` calls the `GameState` C APIs (`_legal_actions_c`,
+`_unified_legal_actions_c`, `push_action`, `pop_action`, score caches, deck
+sampling helpers) directly without Python object construction at each step. This
+keeps the game-state hot path in C-land. `cfr_math.pyx` and `encoding.pyx`
+extend the same principle to regret arithmetic and feature extraction
+respectively, so a traversal node that produces a training sample can encode its
+information state and compute regrets without leaving Cython.
+
+The one remaining boundary cost is the PyTorch policy call: the advantage network
+forward pass requires moving data to GPU and back, which necessarily crosses into
+Python/PyTorch. This is the primary remaining performance target. The `inference_server`
+/ `inference_client` / `inference_buffers` trio is an opt-in experiment that batches
+multiple traversal-paused states into a single GPU forward pass, amortizing that
+boundary cost.
+
+## Why PyTorch for networks
+
+PyTorch is the standard choice for the training layer. The advantage and strategy
+networks are straightforward MLPs with a legal-mask head; no exotic architecture
+is needed. PyTorch's autograd, optimizer API, and GPU memory management handle
+the training loop cleanly. The design deliberately keeps the network definitions
+(in `networks.py`) thin and the training orchestration (in `trainer.py`) separate,
+so the network architecture can be swapped without touching traversal.
+
+## Why this module split
+
+The Cython/Python split mirrors the compute profile:
+
+- **Cython**: everything that runs inside the traversal loop and must be fast.
+  `traversal.pyx` is the main entry point; `cfr_math.pyx` and `encoding.pyx`
+  are helpers it calls. These modules expose C-level APIs (`.pxd` headers) so
+  they can call each other without Python object round-trips.
+- **Python**: everything that runs once per iteration or is dominated by I/O or
+  GPU compute. Training, checkpointing, evaluation, and CLI tooling have no
+  reason to be in Cython and benefit from Python's ergonomics for development
+  speed.
+- **Config**: all hyperparameters live in `config.py` as a Pydantic model tree.
+  This gives type checking and YAML deserialization for free, and means the
+  Cython modules receive plain scalars and typed arrays at call boundaries rather
+  than dict lookups.
+
+## Non-goals (held from the original plan)
+
+The following were explicitly out of scope for v0 and remain so:
+
+- Multi-machine distributed training.
+- Exploitability calculation.
+- Full ISMCTS integration (particle-belief opponent sampling).
+- Large experiment orchestration or W&B artifact integration.
+
+These non-goals have not changed. The performance roadmap (explicit iterative
+Cython traversal scheduler, C-level memory writes, fully batched policy
+inference) is the next meaningful investment, not deeper experiment
+infrastructure.
+
+## Practical implication
+
+- New training-layer features (memory variants, eval hooks, tracking) belong in
+  Python; do not push them into the Cython modules unless they sit inside the
+  per-node traversal loop.
+- Anything called per-node (legal actions, regret update, encoding) must stay
+  in Cython and use the existing `.pxd` C-level APIs — adding a Python helper
+  here regresses throughput across the whole tree walk.
+- The remaining boundary cost is the PyTorch policy forward pass; further
+  performance work should target it (batched inference, iterative scheduler)
+  rather than re-Cythonizing already-Python pieces.
+- Hyperparameter additions go in `config.py` as Pydantic fields, not as
+  positional kwargs threaded through Cython signatures.
+
+## Discrepancy note
+
+The original plan (`docs/archive/deep-cfr-v0-plan.md`) listed `traverser.py`
+as the Python fallback traversal path. As of `ad0be89`, that file has been
+removed; `traversal.pyx` is the sole traversal implementation. The plan also
+did not anticipate `inference_server.py`, `inference_client.py`,
+`inference_buffers.py`, `analyze.py`, `tracking.py`, `traversal_stats.py`,
+`imitation.py`, `policy_gradient.py`, or `workers.py`, all of which exist in
+the current package. These additions reflect experiments and tooling that
+accumulated after the initial plan was written.