Close librarian: full archive promote-survey + parallel dispatch
Second survey processed the remaining 12 archives via gemini after
the first batch of 3 was accepted. 12 drafts, 0 skips, 0 errors.
Every draft carries a deterministic Last-verified header
(2026-05-08, commit 5c221fb) thanks to the post-processing fix
landed in the previous commit. All 12 accepted into docs/research/
verbatim:
deep-cfr-evaluation-profile-plan
deep-cfr-legacy-experiment-reproduction
deep-cfr-legacy-runtime-comparison
deep-cfr-performance-experiments
deep-cfr-profile-advantage-memory-split
deep-cfr-profile
deep-cfr-regret-fallback-audit
deep-cfr-v0-gap-vs-coolrl
deep-cfr-v0-plan
fast-engine-next-optimizations
post-a-optimization-calculus
test-coverage-notes
docs/archive/ is now fully covered: every entry either has a
research counterpart by stem or by tail-match.
Also extracts _dispatch_one and adds --parallel N to
scripts/librarian_survey.py. ThreadPoolExecutor over the per-archive
work is safe because subprocess.run is network-bound (no GIL fight)
and each thread writes to its own output filename. Default stays
1 (sequential); --parallel 4 is the recommended speedup for large
surveys. The two surveys above ran sequentially; future runs can
opt in.
Plan declares librarian closed for new feature work. MEMORY drift
fixup and duplicate-merge modes stay deferred until a real input
surfaces. Stage 1 (5 deterministic checks) and Stage 2 (promote +
survey) remain operational.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
+45
-5
@@ -259,14 +259,54 @@ tests or cost control.
|
||||
|
||||
All three drafts accepted into `docs/research/`:
|
||||
`classic-port-notes.md`, `deep-cfr-batched-evaluation.md`,
|
||||
`deep-cfr-evaluation-profile.md`. 12 archive entries remain
|
||||
unprocessed for the next survey run.
|
||||
`deep-cfr-evaluation-profile.md`.
|
||||
|
||||
### Full-archive pass (2026-05-08, gemini, 12 archives)
|
||||
|
||||
Second survey processed the remaining 12 archives. 12 drafts, 0
|
||||
skips, 0 errors. Post-processing produced consistent
|
||||
`Last verified: 2026-05-08, commit 5c221fb` headers on every draft.
|
||||
All 12 accepted into `docs/research/` verbatim:
|
||||
|
||||
- deep-cfr-evaluation-profile-plan.md
|
||||
- deep-cfr-legacy-experiment-reproduction.md
|
||||
- deep-cfr-legacy-runtime-comparison.md
|
||||
- deep-cfr-performance-experiments.md
|
||||
- deep-cfr-profile-advantage-memory-split.md
|
||||
- deep-cfr-profile.md
|
||||
- deep-cfr-regret-fallback-audit.md
|
||||
- deep-cfr-v0-gap-vs-coolrl.md
|
||||
- deep-cfr-v0-plan.md
|
||||
- fast-engine-next-optimizations.md
|
||||
- post-a-optimization-calculus.md
|
||||
- test-coverage-notes.md
|
||||
|
||||
`docs/archive/` is now fully covered: every entry either has a
|
||||
research counterpart by stem or by tail-match. `librarian.sh`
|
||||
exits 0.
|
||||
|
||||
### Parallel dispatch added
|
||||
|
||||
`scripts/librarian_survey.py --parallel N` runs up to N concurrent
|
||||
LLM calls via `ThreadPoolExecutor`. `subprocess.run` is mostly
|
||||
network-bound, so threads are enough — no GIL fight. Default
|
||||
remains 1 (sequential) for backward compatibility; explicit opt-in
|
||||
to widen. The two surveys above ran sequentially; future runs can
|
||||
collapse wall-clock significantly with `--parallel 4`.
|
||||
|
||||
## Stage 2 remaining
|
||||
|
||||
- MEMORY.md drift fixup mode (read drift report, propose one-line
|
||||
diffs). Currently no drift to act on, so deferred.
|
||||
- Duplicate-doc merge proposal mode.
|
||||
- MEMORY.md drift fixup mode — currently no drift to act on
|
||||
(Stage 1 reports clean), so deferred until a real drift surfaces.
|
||||
- Duplicate-doc merge proposal mode — same shape; defer until a
|
||||
duplicate is detected.
|
||||
|
||||
## Status: closed for new feature work
|
||||
|
||||
Librarian is in maintenance mode. New archive entries will surface
|
||||
via `librarian_survey.py` on the next run; new drift will surface
|
||||
via `librarian.sh`. Build mode resumes only when a real input
|
||||
appears that the deferred Stage 2 features would handle.
|
||||
|
||||
## Next Concrete Step
|
||||
|
||||
|
||||
@@ -0,0 +1,63 @@
|
||||
# Deep CFR Evaluation Profiling
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-evaluation-profile-plan.md`
|
||||
|
||||
## Question
|
||||
|
||||
How are evaluation runtime costs categorized in Deep CFR, and what do these metrics reveal about system bottlenecks?
|
||||
|
||||
Evaluation is a critical path for measuring agent progress, but its runtime can be unpredictable. To move beyond wall-clock guessing, the system instruments the evaluation loop with granular counters that distinguish between neural network inference, state encoding, and game engine overhead.
|
||||
|
||||
## Code reference
|
||||
|
||||
The primary instrumentation structure is the `EvalRuntimeCounters` dataclass in `src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py` (line 20). It tracks elapsed seconds across several distinct phases of a single game step:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class EvalRuntimeCounters:
|
||||
policy_turns: int = 0
|
||||
opponent_turns: int = 0
|
||||
policy_select_seconds: float = 0.0
|
||||
policy_legal_mask_seconds: float = 0.0
|
||||
policy_encoding_seconds: float = 0.0
|
||||
policy_network_seconds: float = 0.0
|
||||
policy_postprocess_seconds: float = 0.0
|
||||
opponent_act_seconds: float = 0.0
|
||||
apply_action_seconds: float = 0.0
|
||||
diagnostics_seconds: float = 0.0
|
||||
final_scoring_seconds: float = 0.0
|
||||
```
|
||||
|
||||
These metrics are updated in `select_actions` and `action_distribution` (lines 200-280), capturing the micro-timing of every policy request.
|
||||
|
||||
## Performance Analysis
|
||||
|
||||
The instrumentation allows for a tiered analysis of the evaluation bottleneck. By comparing these counters, one can pinpoint the specific layer responsible for performance degradation:
|
||||
|
||||
### 1. The Policy Path (`policy_select_seconds`)
|
||||
This is the total time spent by the agent under evaluation. It is further subdivided to identify efficiency gaps in the neural pipeline:
|
||||
- **`policy_network_seconds`**: Time spent inside the PyTorch `forward` pass. If this dominates, the bottleneck is model inference. For small models on CUDA, this often signals high kernel launch overhead for batch-size-1 requests.
|
||||
- **`policy_encoding_seconds`**: Time spent converting `GameState` objects into numerical info-state tensors. High values here suggest that the Python-based feature engineering is a bottleneck.
|
||||
- **`policy_legal_mask_seconds`**: Time spent calculating legal moves. In Lost Cities, this involves scanning the hand and board state.
|
||||
|
||||
### 2. Environment and Opponents
|
||||
- **`opponent_act_seconds`**: Time spent by the opponent bot. When evaluating against expensive bots (like heuristic-heavy search agents), this metric isolates their cost from the main agent's performance.
|
||||
- **`apply_action_seconds`**: The cost of the game engine itself (`GameState.apply_action`). High values indicate that the Cython game logic is the primary constraint.
|
||||
|
||||
## Interpretation
|
||||
|
||||
The relationship between these metrics dictates the optimization strategy. If `policy_network_seconds` is the primary driver, the system is "model-bound," and improvements should focus on batching evaluation games or using inference accelerators like TensorRT. Conversely, if `policy_encoding_seconds` dominates, the system is "feature-bound," and the feature extraction logic should be moved to Cython or vectorized.
|
||||
|
||||
When `opponent_act_seconds` dominates, any local optimizations to the strategy network or encoding will have negligible impact on total evaluation time, as the bottleneck resides in the external bot's implementation.
|
||||
|
||||
## Practical Implications
|
||||
|
||||
- **Optimization Priority**: Always check the ratio of `policy_network_seconds` to `policy_select_seconds` before attempting model optimizations.
|
||||
- **Device Selection**: Large `policy_network_seconds` on CUDA relative to CPU for small models is a known symptom of launch-latency saturation, justifying a move to CPU for serial evaluation.
|
||||
- **Regression Testing**: Evaluation metrics should be compared across iterations (e.g., comparing iteration 5 vs 10) to detect memory leaks or data structure bloat in the diagnostics path (`diagnostics_seconds`).
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/deep-cfr-evaluation-profile.md` (Analysis of CUDA vs CPU latency)
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py` (Implementation)
|
||||
@@ -0,0 +1,46 @@
|
||||
# Deep CFR Legacy Parity and Hyperparameter Mapping
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
|
||||
Source: `docs/archive/deep-cfr-legacy-experiment-reproduction.md`
|
||||
|
||||
## Question
|
||||
|
||||
How are legacy `coolrl` Deep CFR hyperparameters and feature semantics mapped to the current implementation to ensure parity and meaningful experiment reproduction?
|
||||
|
||||
## Code reference
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/config.py`, lines 63–64 (`EncodingConfig`) and 98–99 (`TraversalConfig`): Definition of parity-critical flags and traversal parameters.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx`, lines 156 and 209: Implementation of `_append_derived_playability_features_c` and `_append_slot_aware_playability_features_c`.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/networks.py`, lines 21–33: `_build_mlp` implementation that supports the legacy 3-layer, 256-hidden-unit architecture.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`, lines 408–411 and 606–607: Logic for outcome-sampling mixture and value clipping.
|
||||
|
||||
## Analysis
|
||||
|
||||
Reproduction of legacy Deep CFR experiments requires strict adherence to both hyperparameter values and specific feature engineering. The mapping between the legacy `coolrl` environment and this repository is achieved through the following core components:
|
||||
|
||||
### 1. Information-State Encoding
|
||||
The legacy experiment relied on specialized features beyond the raw board state. These are preserved through two critical flags in `EncodingConfig`:
|
||||
- **`derived_playability`**: Color-level features that calculate the utility or risk of playing into specific expeditions.
|
||||
- **`slot_aware_playability`**: Hand-slot local features that anchor actions to specific hand positions. This allows the model to distinguish between identical cards in different slots or empty slots, which is critical for high-tier play.
|
||||
|
||||
### 2. Network Architecture
|
||||
The current `DeepCFRMLP` supports configurable `hidden_size` and `num_layers`. To match legacy performance, the model must be configured with a 3-layer ReLU MLP (hidden size 256). The `_build_mlp` helper in `networks.py` ensures the layer stack is constructed identically to the legacy Torch implementation.
|
||||
|
||||
### 3. Traversal and Optimization
|
||||
Deep CFR performance is highly sensitive to the traversal mechanism. The current implementation matches legacy semantics via:
|
||||
- **Outcome Sampling Mixture**: On-policy sampling mixed with ε-uniform exploration (`outcome_sampling_epsilon: 0.2`).
|
||||
- **Value Clipping**: Restricting the range of sampled values (`outcome_sampling_value_clip: 500`) to prevent gradient instability.
|
||||
- **Batching**: Separate advantage and strategy batch sizes (legacy default: 1024) and update counts (legacy default: 256 per iteration).
|
||||
|
||||
## Practical implication
|
||||
|
||||
Maintaining this mapping allows for direct comparison between modern runs and legacy baselines. The **slot-aware encoding** is identified as the single most critical factor for parity in tier3 rulesets; without it, the information-state is insufficiently descriptive for the policy network to replicate legacy performance.
|
||||
|
||||
When evaluating current performance against legacy reports, ensure the `encoding.slot_aware_playability` flag is enabled and the `traversal` parameters match the 0.2/500 epsilon/clip baseline.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-legacy-experiment-reproduction.md` (Reproduction plan)
|
||||
- `configs/archive/deep-cfr-selfplay-full-depth-slot-playability.yaml` (Reference configuration)
|
||||
- `docs/research/deep-cfr-v0-feature-parity.md` (General subsystem coverage)
|
||||
@@ -0,0 +1,46 @@
|
||||
# Deep CFR Runtime: Legacy vs. Current Implementation
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-legacy-runtime-comparison-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
|
||||
How does the current Deep CFR implementation's performance compare to the legacy `../coolrl` codebase, and what architectural changes drove the observed speedups?
|
||||
|
||||
## Code reference
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py` (lines 337, 341, 347, 357): Core training loop timers for `time/traversal_seconds`, `time/advantage_train_seconds`, `time/strategy_train_seconds`, and `time/evaluation_seconds`.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py` (line 380): Batched action selection for evaluation games, allowing multiple environments to share a single GPU forward pass.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py` (line 891): `_evaluate_parallel` method implementing multi-process evaluation across different opponents.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`: Cython-optimized traversal logic providing the high-throughput foundation for MCCFR.
|
||||
|
||||
## Analysis
|
||||
|
||||
The current implementation demonstrates a significant performance leap over the legacy `../coolrl` system, with end-to-end wall time improvements ranging from 1.7x to 2.1x depending on the evaluation configuration. This speedup is the result of a deliberate shift toward batched GPU inference and parallelized environment execution.
|
||||
|
||||
### Training Iteration Throughput
|
||||
Normal training iterations (excluding evaluation) improved from approximately 11.5 seconds to 5.8 seconds (~1.97x speedup). This gain is primarily attributed to optimizations in the traversal and network update phases:
|
||||
|
||||
- **Traversal:** Improved from 7.16s to 3.16s (2.27x faster). This is driven by the Cythonized traversal loop and efficient management of worker chunking, which minimizes the overhead of Python-to-Cython transitions.
|
||||
- **Optimization:** Advantage and strategy training phases together improved from ~4.25s to ~2.65s (1.6x faster). The improvement here is largely due to more efficient tensor materialization from the replay buffers, reducing the time the GPU spends waiting for host-side data preparation.
|
||||
|
||||
### Evaluation Efficiency
|
||||
Evaluation was a major bottleneck in the legacy system, averaging 18.6 seconds per session. The current implementation offers two primary modes of improvement:
|
||||
|
||||
- **Batched Sequential:** By grouping evaluation games into chunks (default `batch_size: 64` in `evaluate.py:380`), the overhead of single-state GPU inference is mitigated. This reduces evaluation time to 14.8s (1.25x faster).
|
||||
- **Opponent-Parallel:** Parallelizing evaluation across multiple opponents (via `_evaluate_parallel` in `trainer.py:891`) further reduces wall-clock time to 6.4s. This represents a 2.9x speedup over the legacy evaluation average.
|
||||
|
||||
### End-to-End Comparison
|
||||
Using a simple cadence model of one evaluation every five iterations, the total wall time for a 5-iteration block dropped from ~76.2s in the legacy system to ~35.6s with parallel evaluation. This 2.14x overall speedup allows for more frequent checkpoints and faster hypothesis testing without increasing the total training budget.
|
||||
|
||||
## Practical implication
|
||||
|
||||
- **Strict Superiority:** The current implementation is significantly more efficient than the legacy code, establishing it as the definitive platform for all Lost Cities Deep CFR research.
|
||||
- **Bottleneck Distribution:** Despite these gains, traversal remains the largest phase (roughly 60% of iteration time). Future optimizations should prioritize batched traversal inference or "interleaved" execution to further leverage GPU compute during the traversal phase.
|
||||
- **Evaluation Scaling:** The 2.9x speedup in evaluation enables more frequent, high-fidelity monitoring (e.g., evaluating against a full suite of heuristic bots every 10 iterations) with minimal impact on total training time.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-legacy-runtime-comparison-2026-05-07.md`
|
||||
- `docs/performance.md`
|
||||
- `docs/research/deep-cfr-batched-evaluation.md`
|
||||
@@ -0,0 +1,58 @@
|
||||
# Deep CFR Performance Optimization and Scaling
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-performance-experiments-2026-05-07.md`
|
||||
|
||||
## Analysis of Trainer and Traversal Bottlenecks
|
||||
|
||||
Performance experiments on the baseline Deep CFR implementation (3-layer, 512-hidden MLP) reveal a clear trade-off between implementation complexity and hardware utilization. For models at this scale, the dominant overhead is not the raw floating-point operations on the GPU, but rather the Python-side dispatch and coordination logic.
|
||||
|
||||
Three specific optimization attempts—`torch.compile`, Automated Mixed Precision (AMP), and GPU forward profiling—converged on the same conclusion: the current model is too small to benefit from standard PyTorch optimization "magic" without architectural changes to how data is fed to the model.
|
||||
|
||||
### Dispatch Overhead vs. Kernel Fusion
|
||||
|
||||
Both `torch.compile` and AMP (fp16) resulted in small performance regressions (approx. 5-18%) on the `default.yaml` and `smoke.yaml` configurations.
|
||||
|
||||
- **`torch.compile` regression:** While compilation reduces kernel launch overhead and enables fusion, the `DeepCFRMLP` architecture (`src/coolrl_lost_cities/games/classic/deep_cfr/networks.py:35`) is shallow enough that the bookkeeping overhead of the compiled wrapper exceeds these gains. Furthermore, because traversal runs in separate CPU multiprocessing workers, they do not benefit from the trainer's compiled networks unless specifically re-compiled or shared in a serialized format.
|
||||
- **AMP regression:** Running the trainer optimization loops (`_train_advantage` at `trainer.py:1010` and `_train_strategy` at `trainer.py:1080`) in fp16 via `torch.autocast` proved counter-productive. The overhead of `GradScaler` bookkeeping and the frequent casting required for loss stability (e.g., computing squared loss in fp32 to maintain precision) outweighs the throughput win of lower-precision matrix multiplications at this model size.
|
||||
|
||||
**Rule of Thumb:** Re-evaluate these optimizations only when the model scale increases significantly (e.g., `hidden_size >= 1024` or `num_layers >= 6`).
|
||||
|
||||
### The Case for Batched Traversal
|
||||
|
||||
While the trainer is throughput-limited by dispatch, the traversal phase is the primary wall-clock bottleneck, accounting for approximately 60% of iteration time. Profiling the `DeepCFRMLP` forward pass (`scripts/profile_gpu_forward.py`) shows that the GPU is massively underutilized during standard recursive traversal (batch size = 1).
|
||||
|
||||
| Batch Size | μs per State | Efficiency vs. BS=1 |
|
||||
| :--- | :--- | :--- |
|
||||
| 1 | 80.07 | 1.00× |
|
||||
| 64 | 1.46 | 54.75× |
|
||||
| 256 | 0.34 | 232.03× |
|
||||
|
||||
A typical traversal visits ~360 nodes, providing a natural batching window. The transition from recursive traversal to an **interleaved scheduler** (`traversal.scheduler: interleaved`) leverages this by grouping policy requests across multiple concurrent traversals.
|
||||
|
||||
## Interleaved Traversal Architecture
|
||||
|
||||
The interleaved scheduler solves the "sequential bottleneck" of recursive MCCFR by decoupling state expansion from policy evaluation. Instead of waiting for a single forward pass per node, it maintains an explicit stack of active traversals and batches their network requests.
|
||||
|
||||
### Performance Gains
|
||||
|
||||
Comparing the recursive baseline against the interleaved scheduler (8 CPU workers, chunk size 64) on the `default.yaml` configuration:
|
||||
|
||||
- **Traversal wall-clock:** ~2.1× speedup.
|
||||
- **Total iteration time:** ~1.5× speedup.
|
||||
- **Node throughput:** Increased from 17.6k nodes/s to 31.3k nodes/s.
|
||||
|
||||
The 8-worker interleaved path is more effective than a single-process CUDA path because it preserves high CPU-side game-state throughput (encoding/decoding) while still benefiting from GPU batching.
|
||||
|
||||
## Practical Implications
|
||||
|
||||
- **Default Scheduler:** Interleaved traversal is the preferred default. The recursive path remains as a fallback (`traversal.scheduler: recursive`) for debugging or for verification of byte-identical RNG ordering.
|
||||
- **Training Stability:** Interleaved traversal uses per-context RNG streams. While it matches recursive statistics in expectation, it does not produce identical sample ordering.
|
||||
- **Future Scaling:** The next major throughput win lies in **Optimization Priority #5** (batched traversal inference), which will move the batched forward passes to a dedicated inference server, further reducing the worker-to-GPU coordination overhead.
|
||||
|
||||
## References
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`: Implementation of AMP and training loops.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/networks.py`: `DeepCFRMLP` architecture.
|
||||
- `configs/deep_cfr/default.yaml`: Configuration for interleaved scheduler.
|
||||
- `scripts/profile_gpu_forward.py`: GPU forward pass micro-benchmarks.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Advantage Memory Split Performance Optimization
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
|
||||
Why does splitting the advantage memory reservoir by player provide such a dramatic speedup in Deep CFR training, and what was the primary bottleneck in the unified memory implementation?
|
||||
|
||||
The speedup is primarily due to eliminating an $O(N)$ linear scan during the sampling phase of advantage network optimization. In the unified implementation, every sampling request for a specific player's advantages required filtering the entire reservoir, which becomes prohibitively expensive as memory capacity scales to millions of samples.
|
||||
|
||||
## Code reference
|
||||
|
||||
The split is implemented in `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py` by initializing two separate reservoir memories:
|
||||
|
||||
```python
|
||||
# Lines 239-241
|
||||
self.advantage_memories = [
|
||||
ReservoirMemory(self.config.memory.advantage_capacity) for _ in range(2)
|
||||
]
|
||||
```
|
||||
|
||||
During the traversal phase, samples are routed to the per-player memory in `_add_advantage_samples` (line 321):
|
||||
|
||||
```python
|
||||
self.advantage_memories[sample.player].add(sample, self.rng)
|
||||
```
|
||||
|
||||
The bottleneck in the previous unified implementation was located in `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py`, within the `sample` method (lines 62-67):
|
||||
|
||||
```python
|
||||
candidates = (
|
||||
self._samples
|
||||
if player is None
|
||||
else [sample for sample in self._samples if sample.player == player]
|
||||
)
|
||||
```
|
||||
|
||||
When training player $P$'s advantage network, the trainer must sample batches of advantages specifically for that player. If the reservoir is shared, `player=P` is passed to `sample()`, triggering the list comprehension. With a default capacity of 2,000,000 samples and 64 updates per iteration, this results in over 128 million object inspections per iteration for advantage training alone.
|
||||
|
||||
## Analysis
|
||||
|
||||
Deep CFR alternates between a traversal phase (where advantages are collected) and an optimization phase (where networks are updated). Because advantage networks are player-specific, we only ever need to sample advantages for one player at a time during the `_train_advantage` loop.
|
||||
|
||||
By splitting the memories at insertion time (during traversal), we move the $O(1)$ routing logic to a phase that already handles samples individually. This allows the sampling phase to treat its per-player reservoir as a pure pool of valid candidates, defaulting `player` to `None` and using `self._samples` directly. This transforms the sampling cost from $O(N \cdot K)$ to $O(B \cdot K)$, where $B$ is the batch size and $K$ is the number of updates.
|
||||
|
||||
Empirical results from the source profile show that sampling time for a single player dropped from approximately 1.65 seconds to 0.05 seconds per iteration—a ~30x improvement. Overall iteration time for non-evaluation steps dropped from 9.1 seconds to 5.8 seconds (~35% reduction).
|
||||
|
||||
## Practical implication
|
||||
|
||||
- **Scalability:** We can now scale `advantage_capacity` to the limits of system RAM without incurring a linear penalty in training time.
|
||||
- **Update Frequency:** The reduction in training overhead allows for more `advantage_updates_per_iteration` (K) if needed for better convergence, as the fixed cost of sampling is now negligible.
|
||||
- **Unified Strategy Memory:** Note that `strategy_memory` remains unified because the strategy network is shared (or trained on all-player data) in the current implementation. If the strategy network were also split per player, a similar optimization would apply.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md`
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py`
|
||||
@@ -0,0 +1,91 @@
|
||||
# Deep CFR Performance Profile: Advantage Training Bottlenecks
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-profile-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
Which components of the Deep CFR training loop dominate execution time, and how does performance scale as the training run progresses and memory buffers grow?
|
||||
|
||||
## Analysis
|
||||
Profile results from initial training runs indicate that **advantage network training** is the primary bottleneck in non-evaluation iterations, accounting for over 55% of total iteration time (e.g., 5.06s out of 9.13s).
|
||||
|
||||
The profile reveals a significant scaling issue: the time spent in advantage training grows roughly linearly with the iteration count. Specifically, the time spent sampling from advantage memory (`time/advantage_player_X_sample_seconds`) increased from ~0.5s at iteration 1 to ~8.2s at iteration 10. During this window, the total advantage memory size grew from approximately 43,000 to 205,000 samples.
|
||||
|
||||
### Bottleneck: Linear Scan in Memory Sampling
|
||||
The scaling bottleneck originates in the `ReservoirMemory.sample` implementation. When training on a shared advantage memory that requires player-specific filtering at sample time, the implementation performs a linear scan over the entire memory buffer to identify valid candidates.
|
||||
|
||||
With `advantage_updates_per_iteration` set to 256, the trainer performs 256 full scans of the growing memory buffer per player, per iteration. At iteration 10 (200k samples), this results in over 50 million object inspections per player per iteration, explaining the jump from sub-second sampling to nearly 10 seconds.
|
||||
|
||||
## Code Reference
|
||||
The training loop is orchestrated in `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`:
|
||||
- `run_iteration` (line 330): Coordinates traversal, advantage training, and strategy training.
|
||||
- `_train_advantage` (line 1014): Records sampling time and performs the update steps.
|
||||
|
||||
The sampling bottleneck occurs in `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py` (line 64):
|
||||
```python
|
||||
def sample(self, batch_size, rng, *, player=None):
|
||||
candidates = (
|
||||
self._samples
|
||||
if player is None
|
||||
else [sample for sample in self._samples if sample.player == player]
|
||||
)
|
||||
# ...
|
||||
```
|
||||
The list comprehension on line 67 triggers the full traversal of `self._samples` whenever `player` is specified.
|
||||
|
||||
## Practical Implication
|
||||
To maintain stable iteration times in long runs, **advantage memories must be split by player** at the trainer level. This ensures that `sample()` is called with `player=None` on a pre-filtered buffer, reducing the sampling operation to a constant-time index selection (plus the cost of object retrieval).
|
||||
|
||||
Post-split profiling (`docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md`) verified that this architectural change reduces iteration 10 sampling time from ~8.2s to ~0.12s, effectively decoupling training performance from total memory size.
|
||||
|
||||
## References
|
||||
- `docs/archive/deep-cfr-profile-2026-05-07.md` (Baseline Profile)
|
||||
- `docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md` (Optimization Result)
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py`
|
||||
|
||||
# Deep CFR Performance Profile: Advantage Training Bottlenecks
|
||||
|
||||
**Last verified:** 2026-05-08
|
||||
**Source:** `docs/archive/deep-cfr-profile-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
Which components of the Deep CFR training loop dominate execution time, and how does performance scale as the training run progresses and memory buffers grow?
|
||||
|
||||
## Analysis
|
||||
Profile results from initial training runs indicate that **advantage network training** is the primary bottleneck in non-evaluation iterations, accounting for over 55% of total iteration time (e.g., 5.06s out of 9.13s).
|
||||
|
||||
The profile reveals a significant scaling issue: the time spent in advantage training grows roughly linearly with the iteration count. Specifically, the time spent sampling from advantage memory (`time/advantage_player_X_sample_seconds`) increased from ~0.5s at iteration 1 to ~8.2s at iteration 10. During this window, the total advantage memory size grew from approximately 43,000 to 205,000 samples.
|
||||
|
||||
### Bottleneck: Linear Scan in Memory Sampling
|
||||
The scaling bottleneck originates in the `ReservoirMemory.sample` implementation. When training on a shared advantage memory that requires player-specific filtering at sample time, the implementation performs a linear scan over the entire memory buffer to identify valid candidates.
|
||||
|
||||
With `advantage_updates_per_iteration` set to 256, the trainer performs 256 full scans of the growing memory buffer per player, per iteration. At iteration 10 (200k samples), this results in over 50 million object inspections per player per iteration, explaining the jump from sub-second sampling to nearly 10 seconds.
|
||||
|
||||
## Code Reference
|
||||
The training loop is orchestrated in `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`:
|
||||
- `run_iteration` (line 330): Coordinates traversal, advantage training, and strategy training.
|
||||
- `_train_advantage` (line 1014): Records sampling time and performs the update steps.
|
||||
|
||||
The sampling bottleneck occurs in `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py` (line 64):
|
||||
```python
|
||||
def sample(self, batch_size, rng, *, player=None):
|
||||
candidates = (
|
||||
self._samples
|
||||
if player is None
|
||||
else [sample for sample in self._samples if sample.player == player]
|
||||
)
|
||||
# ...
|
||||
```
|
||||
The list comprehension on line 67 triggers the full traversal of `self._samples` whenever `player` is specified.
|
||||
|
||||
## Practical Implication
|
||||
To maintain stable iteration times in long runs, **advantage memories must be split by player** at the trainer level. This ensures that `sample()` is called with `player=None` on a pre-filtered buffer, reducing the sampling operation to a constant-time index selection (plus the cost of object retrieval).
|
||||
|
||||
Post-split profiling (`docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md`) verified that this architectural change reduces iteration 10 sampling time from ~8.2s to ~0.12s, effectively decoupling training performance from total memory size.
|
||||
|
||||
## References
|
||||
- `docs/archive/deep-cfr-profile-2026-05-07.md` (Baseline Profile)
|
||||
- `docs/archive/deep-cfr-profile-advantage-memory-split-2026-05-07.md` (Optimization Result)
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/memory.py`
|
||||
@@ -0,0 +1,61 @@
|
||||
# Deep CFR Regret Matching Fallback and Early Over-Opening
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
|
||||
In Deep CFR, when all legal actions at a traverser node have non-positive estimated regrets, the standard regret matching algorithm cannot produce a probability distribution by normalizing positive regrets. In such cases, a "fallback" policy must be used. Does the choice of fallback policy—specifically the default uniform distribution—contribute to the "over-opening" pathology (where the agent starts too many expeditions) observed in early Lost Cities training runs?
|
||||
|
||||
Short answer: **Yes.** Diagnostic audits show that in early iterations, nearly half of all regret-matching decisions fall back to the default policy because the network predicts negative regrets for all legal actions. In Lost Cities, the large number of legal "open new expedition" actions causes a uniform fallback to select an opening move more frequently than a more informed tie-breaking strategy would.
|
||||
|
||||
## Code reference
|
||||
|
||||
The regret matching fallback logic is implemented in `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx` within `_policy_from_networks` (around line 477):
|
||||
|
||||
```cython
|
||||
for i in range(self.action_size):
|
||||
if legal[i] != 0:
|
||||
positive = adv_view[i] if adv_view[i] > 0.0 else 0.0
|
||||
positive_sum += positive
|
||||
self.last_policy_regret_fallback = positive_sum <= self.epsilon
|
||||
```
|
||||
|
||||
If `positive_sum` is below the threshold, the policy is determined by the `all_negative_fallback` configuration. The default `uniform` mode uses `regret_matching_c` from `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx` (line 31), which assigns `1.0 / legal_count` to every legal action.
|
||||
|
||||
The alternative `argmax_tiebreak` mode (added in the May 2026 audit) instead identifies the legal action(s) with the highest (least negative) advantage:
|
||||
|
||||
```cython
|
||||
if selected < 0 or adv_view[i] > best:
|
||||
selected = i
|
||||
best = adv_view[i]
|
||||
tie_count = 1
|
||||
elif adv_view[i] == best:
|
||||
tie_count += 1
|
||||
if _next_u32(&self.rng) % <unsigned int>tie_count == 0:
|
||||
selected = i
|
||||
```
|
||||
|
||||
## Analysis
|
||||
|
||||
In Lost Cities, every move consists of playing or discarding a card, followed by drawing a card. Opening a new expedition incurs an immediate -20 point penalty. Early in training, before the advantage networks have learned the long-term value of expeditions, they frequently assign negative regrets to all "open new" actions.
|
||||
|
||||
If a traverser has 5 cards in hand that could each open a new expedition, and 3 cards that could be discarded or played on existing expeditions, a uniform fallback over these 8 legal actions will select an "open new" move with 62.5% probability. Because the network hasn't yet learned to distinguish between these "bad" (negative regret) moves, it defaults to a high-entropy selection that over-samples the most numerous action type: opening new expeditions.
|
||||
|
||||
The audit at iteration 20 revealed:
|
||||
- Under **uniform** fallback, the fallback rate was **46.6%**, and the agent opened an average of **4.43** colors per game.
|
||||
- Under **argmax_tiebreak**, the fallback rate dropped to **15.4%** (as the network only needs to find *one* action it "dislikes least"), and the agent opened **4.61** colors in the same iteration, though it showed lower 5-color opening counts against specific opponents.
|
||||
|
||||
The "over-opening" is not necessarily a failure of the network to learn, but a byproduct of how high-entropy fallbacks interact with the game's action space geometry.
|
||||
|
||||
## Practical implication
|
||||
|
||||
- **Uniform fallback is a source of exploration noise** that is biased by the action count of specific move types. In Lost Cities, this noise pushes the traverser toward starting expeditions it cannot afford.
|
||||
- **Argmax tie-breaking reduces fallback frequency** by forcing the agent to follow the network's relative preferences even when all absolute preferences are negative. This acts as a variance reduction technique for the policy.
|
||||
- While `argmax_tiebreak` reduces the frequency of "blind" openings, it should be used cautiously; if the network's relative rankings are also noise, it may converge to a different but equally pathological attractor.
|
||||
- For Lost Cities Deep CFR, `argmax_tiebreak` is recommended for further evaluation as a potential fix for the early-game opening plateau.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md`
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal_stats.py` (Implementation of fallback audit metrics).
|
||||
@@ -0,0 +1,42 @@
|
||||
# Deep CFR v0: Architectural Parity and Performance Gaps
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`
|
||||
|
||||
## Question
|
||||
|
||||
What is the implementation status of Deep CFR v0 relative to the legacy `coolrl` reference, and what are the primary architectural bottlenecks remaining for high-performance training?
|
||||
|
||||
## Code reference
|
||||
|
||||
The core algorithmic components have been ported to Cython to ensure C-level performance for the rules engine and the tree-walking loop:
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives.
|
||||
|
||||
## Analysis
|
||||
|
||||
As of May 2026, the implementation has achieved functional parity with the legacy reference. The migration to Cython has successfully eliminated the Python recursion limit as a primary constraint and significantly reduced the per-node overhead for game rules and state management. The "v0" implementation covers the full suite of required features: recursive traversal, terminal value handling (including rollouts and score-diff cutoffs), reservoir memory management, and PyTorch-based network training.
|
||||
|
||||
However, a "performance-critical gap" remains. While the traversal loop is in Cython, it remains **synchronous and recursive**. This architecture incurs significant costs at the Python/C boundary:
|
||||
1. **Synchronous Policy Inference:** Each node requiring a policy must wait for a PyTorch forward pass. Because these calls cross back into Python, they cannot be efficiently batched across different branches or traversal contexts, leading to poor GPU utilization.
|
||||
2. **Data Materialization:** Writing training samples from Cython-managed buffers into NumPy arrays for the reservoir memory involves frequent boundary crossings.
|
||||
3. **Lack of Concurrency:** The recursive depth-first search (DFS) pattern makes it difficult to interleave multiple traversal contexts, which is a prerequisite for effective inference batching.
|
||||
|
||||
## Practical implication
|
||||
|
||||
The implementation is algorithmically complete and suitable for verifying the correctness of the Deep CFR agent. However, for large-scale training, the current recursive DFS is a bottleneck. The recommended path forward is a **batched iterative traversal scheduler** implemented in Cython. Moving to an explicit stack-based or queue-based scheduler will allow the system to:
|
||||
- Interleave multiple traversal contexts.
|
||||
- Collect policy requests from many contexts into a single batch.
|
||||
- Execute a single large forward pass on the GPU, drastically reducing the impact of the Python/C boundary and maximizing throughput.
|
||||
|
||||
Until this transition is made, optimization efforts should focus on reducing the frequency and cost of policy calls rather than micro-optimizing the already efficient Cython rules engine.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`: Original status and gap analysis.
|
||||
- `docs/research/deep-cfr-v0-feature-parity.md`: Detailed subsystem coverage report.
|
||||
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Deep CFR Cython Traversal Architecture
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/deep-cfr-v0-plan.md`
|
||||
|
||||
## Question
|
||||
|
||||
How can Deep CFR traversal be implemented efficiently for the Lost Cities game-state hot path while correctly handling hidden information and chance sampling?
|
||||
|
||||
The core challenge in scaling Deep CFR for Lost Cities is the overhead of state manipulation and information-state encoding. A Python-heavy implementation would bottleneck on object instantiation and legal action masking. The architectural solution is a Cython-native traversal engine that operates directly on the C-level `GameState` API, using mutation and undo (push/pop) instead of state cloning.
|
||||
|
||||
## Code reference
|
||||
|
||||
The traversal engine is implemented in `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`. It relies on the `GameState` C API defined in `src/coolrl_lost_cities/games/classic/game.pxd`.
|
||||
|
||||
### Chance Sampling and Mutation
|
||||
|
||||
Instead of cloning the entire game state for each chance node or action, the traverser uses `_push_action_c` and `_pop_action_c` to navigate the tree. For chance sampling (deck order), it uses C-level deck swaps.
|
||||
|
||||
`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx` (around line 311):
|
||||
|
||||
```cython
|
||||
state._push_action_c(fixed_action)
|
||||
# ... traversal ...
|
||||
state._pop_action_c()
|
||||
if is_chance:
|
||||
state._swap_deck_cards_c(swapped_deck_index, state.deck_len - 1)
|
||||
```
|
||||
|
||||
This mutation-based approach significantly reduces memory allocation and garbage collection pressure compared to `state.clone()` calls in the inner loop of a Monte Carlo traversal.
|
||||
|
||||
### Information-State Encoding
|
||||
|
||||
The encoding layer translates the game state into a fixed-size vector for the advantage and strategy networks. It is critical that this encoding does not "leak" hidden information (such as the opponent's hand or future deck contents) which the current player cannot legally observe.
|
||||
|
||||
`src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx`, function `_encode_info_state_with_flags_c` (around line 291):
|
||||
|
||||
```cython
|
||||
out[idx] = 1.0 if state.phase_id == 0 else 0.0
|
||||
# ...
|
||||
out[idx] = <float>state.deck_len / <float>state.total_cards
|
||||
# ... encoding player hand, public expeditions, and discard piles ...
|
||||
```
|
||||
|
||||
The encoding includes:
|
||||
- Current game phase and active player.
|
||||
- The acting player's private hand.
|
||||
- Both players' public expeditions and the shared discard piles.
|
||||
- The remaining deck ratio and current scores.
|
||||
- Legal action masks for the current state.
|
||||
|
||||
## Analysis
|
||||
|
||||
### Performance Rationale
|
||||
|
||||
By keeping the entire traversal loop—from legal action generation to regret matching—within Cython, the system avoids the "Python tax" on every node transition. The `GameState` C structure provides direct pointer access to deck and hand buffers, allowing the traverser to perform thousands of rollouts per second on a single thread.
|
||||
|
||||
### Chance Handling via Mutation
|
||||
|
||||
In Deep CFR, modeling chance nodes by sampling compatible deck orders is standard. However, the efficiency of this sampling is often a bottleneck. The choice to use `_swap_deck_cards_c` followed by a `push_action` allows the traverser to simulate a specific chance outcome (e.g., drawing a specific card) without rebuilding the state. The `pop_action` call restores the deck and state counters, maintaining the integrity of the search tree.
|
||||
|
||||
### Non-leaking Encoding
|
||||
|
||||
A common failure mode in Deep CFR is "cheating" by including hidden information in the information-state vector. The v0 architecture strictly separates the `GameState` (which knows all) from the `encoding` (which only sees the current player's perspective). This ensures that the trained networks learn a true imperfect-information strategy rather than a policy that relies on privileged state knowledge.
|
||||
|
||||
## Practical implication
|
||||
|
||||
- **Developer Note:** When modifying the `GameState` or adding new card types, ensure that `_push_action_c` and `_pop_action_c` are updated to correctly restore any new state variables.
|
||||
- **Performance:** Any new features in the encoding (e.g., "derived playability" flags) should be implemented in `encoding.pyx` using C-level loops to avoid slowing down the traverser.
|
||||
- **Verification:** Always verify that `encoding.pyx` does not access `state.hand_cards[1 - player]` or other private indices belonging to the opponent.
|
||||
|
||||
## References
|
||||
|
||||
- Brown, Lerer, Gross, Sandholm. *Deep Counterfactual Regret Minimization.* ICML 2019.
|
||||
- Zinkevich, Johanson, Bowling, Piccione. *Regret Minimization in Games with Incomplete Information.* NeurIPS 2007. (CFR foundation.)
|
||||
- `docs/archive/deep-cfr-v0-plan.md`: Original design document for the Cython traversal pipeline.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Fast Engine Optimization Architecture
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
|
||||
Source: `docs/archive/fast-engine-next-optimizations.md`
|
||||
|
||||
## Question
|
||||
|
||||
How should the Cython-based game engine be optimized to support the high-throughput requirements of reinforcement learning algorithms like Deep CFR?
|
||||
|
||||
## Analysis
|
||||
|
||||
The core of the Lost Cities implementation (the "fast engine") is built as a Cython extension to provide a high-performance alternative to pure-Python game logic. While the current architecture provides the necessary speed for basic experimentation, it contains several intentional design choices—such as `cpdef` method wrappers and fragmented memory management—that become significant bottlenecks during large-scale self-play and traversal.
|
||||
|
||||
### Cython `pxd` API vs. Python Boundaries
|
||||
|
||||
The engine currently exposes its logic primarily through `cpdef` methods in `src/coolrl_lost_cities/games/classic/game.pyx`. While these methods are accessible from both Python and Cython, calling them from the Python interpreter incurs conversion overhead for every argument and return value. For algorithms like Deep CFR that visit millions of nodes per iteration, this overhead is prohibitive.
|
||||
|
||||
To achieve maximum performance, traversal code must bypass the Python interpreter entirely. This is possible by importing the game state's C-level API directly from `src/coolrl_lost_cities/games/classic/game.pxd`. High-performance modules, such as the Deep CFR trainer, should favor `cdef` methods like `_apply_action_unchecked_c` and `_legal_actions_c` which operate on raw C types and pointers.
|
||||
|
||||
### Contiguous Memory Allocation
|
||||
|
||||
In its current state, the `GameState` class manages its internal arrays using multiple discrete memory allocations. In `game.pyx` at lines 314–323, the `_configure` method performs approximately eleven separate `malloc` calls to set up the deck, hands, expeditions, and scoring buffers:
|
||||
|
||||
```cython
|
||||
self.deck_cards = <int*>malloc(self.total_cards * sizeof(int))
|
||||
self.hand_cards = <int*>malloc(2 * self.hand_size * sizeof(int))
|
||||
# ... and several others
|
||||
```
|
||||
|
||||
This fragmented allocation pattern increases memory management overhead and can lead to poor cache locality during state transitions. Consolidating these buffers into a single contiguous block of memory would not only improve cache performance but also simplify state cloning. A single `memcpy` could duplicate the entire game state, significantly speeding up the recursive branching required by CFR-based traversers.
|
||||
|
||||
### Zero-Copy Feature Extraction
|
||||
|
||||
A major cost in the current RL pipeline is the construction of observation vectors and action masks. Methods like `unified_legal_mask_np` (line 797 in `game.pyx`) currently build Python lists of booleans before converting them into NumPy arrays. This "build-and-convert" pattern generates excessive Python object churn in the inner loop of training.
|
||||
|
||||
The engine must transition toward a "zero-copy" paradigm where observation and mask construction logic writes directly into pre-allocated destination buffers. This pattern is already partially implemented in `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx` with the `encode_info_state_c` function, which accepts a `float*` pointer. Expanding this approach to all high-frequency data paths—including legal action masks—is essential for eliminating Python-level overhead during training.
|
||||
|
||||
## Practical Implication
|
||||
|
||||
Performance-critical components—specifically interleaved traversal and batched inference servers—should be designed to interface with the engine at the C level. Future refactoring of the `GameState` should prioritize a single-allocation memory model to enable rapid cloning and cache-efficient updates. All feature extraction logic must move toward buffer-writing APIs to ensure that node traversal speed is limited by algorithmic complexity rather than Python infrastructure.
|
||||
|
||||
## References
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/game.pxd`: Direct C-API definitions for the fast engine.
|
||||
- `src/coolrl_lost_cities/games/classic/game.pyx`: Core state management and `malloc`-based allocation logic.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx`: Reference implementation for buffer-based feature extraction.
|
||||
@@ -0,0 +1,54 @@
|
||||
# Post-A Optimization Calculus
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/post-a-optimization-calculus-2026-05-07.md`
|
||||
|
||||
## Question
|
||||
|
||||
Why do architectural optimizations like `torch.compile` and TensorRT integration yield negligible or even negative returns in the current Deep CFR implementation, and what specific triggers will shift them from "distractions" to "critical path" requirements?
|
||||
|
||||
## Code reference
|
||||
|
||||
The current baseline configuration is defined in `configs/deep_cfr/default.yaml`:
|
||||
|
||||
```yaml
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
```
|
||||
|
||||
Performance is measured via `scripts/bench_inference_backend.py`, which targets the traversal and evaluation inference paths. Current benchmarks (recorded in `docs/performance.md`) show that the wall-clock time is dominated by traversal (60%) and advantage/strategy training (39%), with evaluation occupying a small amortized share at the default `eval_every: 25`.
|
||||
|
||||
## Analysis
|
||||
|
||||
In the current development phase, kernel fusion and specialized inference engines are dispatch-bound rather than compute-bound. For a small model (3 layers, 512 hidden), the time spent executing the actual linear layers and activations is comparable to the overhead of the Python-to-C++ dispatch and CUDA kernel launch latency.
|
||||
|
||||
### The Small-Model Regression
|
||||
Empirical results in `docs/archive/deep-cfr-performance-experiments-2026-05-07.md` show that `torch.compile` on the trainer's networks regressed performance. This occurs because the compile-time overhead and the fusion of very small kernels do not amortize effectively; the overhead of the optimized dispatch path is greater than the execution time of the unoptimized kernels. Similarly, TensorRT on the inference-server forward pass would only shave ~20–40μs off a call that already takes ~90μs, providing an iteration-level gain of less than 1%.
|
||||
|
||||
### The Phase Shift
|
||||
The "calculus" shifts when two factors change the bottleneck profile:
|
||||
|
||||
1. **Model Scaling:** Increasing to ~1024 hidden units and ~6 layers pushes the model into kernel-bound territory. Per-call time scales with FLOPs, while batching gains and dispatch overhead remain relatively fixed. At this scale, the 1.5–2.0× speedup provided by TensorRT or `torch.compile` fusion begins to dominate the wall-clock time.
|
||||
2. **Evaluation Density:** Evaluation is pure inference. As we move toward denser evaluation (e.g., `eval_every: 5` and `evaluation.games: 1000`), evaluation can grow to occupy 50% or more of the total iteration time. Since the inference server handles both traversal and evaluation, any gain in the forward path (such as TensorRT) applies directly to this large time slice.
|
||||
|
||||
When combined, these shifts can make the same tools that are currently performance-neutral deliver a ~1.5× total iteration speedup.
|
||||
|
||||
## Recommended Sequencing
|
||||
|
||||
To avoid misleading benchmark data and wasted integration effort, optimizations must follow the growth of the model and evaluation load:
|
||||
|
||||
1. **Baseline Validation**: Confirm the "Option A" (batched traversal inference) multipliers using `scripts/bench_inference_backend.py`.
|
||||
2. **Model Selection**: Determine the target model size for production runs. This is the prerequisite for all subsequent optimization work.
|
||||
3. **Trainer Optimization**: Re-measure `torch.compile` on the trainer *only after* the model size is increased.
|
||||
4. **Inference Optimization**: Integrate TensorRT into the inference server once evaluation load or model size makes the forward pass a significant (>10%) share of wall-clock time.
|
||||
|
||||
## Practical Implication
|
||||
|
||||
Avoid premature optimization with `torch.compile` or TensorRT on the current small-model baseline. These tools should be treated as "Model Scale" features rather than "Algorithm" features; their value is unlocked by the compute intensity of the configuration, not the correctness of the implementation.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/archive/deep-cfr-performance-experiments-2026-05-07.md` (Small-model regression data)
|
||||
- `docs/archive/option-a-bench-result-2026-05-07.md` (Batched traversal benchmarks)
|
||||
- `docs/performance.md` (Current runtime bottleneck profile)
|
||||
@@ -0,0 +1,60 @@
|
||||
# Test Coverage Strategy for Python and Cython Modules
|
||||
|
||||
**Last verified:** 2026-05-08, commit `5c221fb`
|
||||
**Source:** `docs/archive/test-coverage-notes.md`
|
||||
|
||||
## Question
|
||||
How should the project manage test coverage reporting, particularly for performance-critical Cython extensions, without compromising development speed or contaminating standard build artifacts?
|
||||
|
||||
## Analysis
|
||||
The project utilizes a hybrid architecture where core game logic and algorithmic traversals are implemented in Cython (`.pyx`) for performance, while high-level coordination and configuration are in Python. Standard coverage tools (e.g., `coverage.py`) effectively track Python execution but require specific build-time instrumentation to observe line-level execution within Cython modules.
|
||||
|
||||
Enabling Cython tracing introduces several complications:
|
||||
1. **Performance Degradation**: Instrumenting tight loops in `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx` or `encoding.pyx` with `linetrace` can result in significant overhead, making large-scale tests or simulations prohibitively slow.
|
||||
2. **Artifact Contamination**: A `build_ext --inplace --force` command with tracing enabled overwrites the optimized `.so` files. If these artifacts are accidentally committed or used for benchmarking, they will report misleadingly slow performance.
|
||||
3. **Build Complexity**: It requires conditional logic in `setup.py` to toggle `compiler_directives` and `define_macros` based on environment variables.
|
||||
|
||||
## Practical Implication
|
||||
The project adopts a "Python-first, Cython-selective" coverage policy to balance visibility with performance.
|
||||
|
||||
### 1. Default Python-Only Coverage
|
||||
For routine development and CI, coverage is restricted to Python modules. This provides high-level assurance of test execution without impacting the speed of the Cython core. The standard reporting command is:
|
||||
|
||||
```bash
|
||||
uv run --with coverage coverage run --source=src/coolrl_lost_cities -m pytest tests/games/classic
|
||||
```
|
||||
|
||||
### 2. Isolated Cython Tracing
|
||||
When verification of Cython logic paths is required, it should be performed in an isolated environment (such as a separate `git worktree`) to prevent optimized build artifacts from being overwritten in the main development branch.
|
||||
|
||||
To enable tracing, `setup.py:12` would need to be modified (ideally via an environment variable like `CYTHON_COVERAGE=1`) to include:
|
||||
|
||||
```python
|
||||
# setup.py (proposed modification)
|
||||
extensions = cythonize(
|
||||
[...],
|
||||
compiler_directives={
|
||||
"linetrace": True,
|
||||
# ... other directives
|
||||
},
|
||||
define_macros=[("CYTHON_TRACE", "1")]
|
||||
)
|
||||
```
|
||||
|
||||
Additionally, a `.coveragerc` file must include the Cython plugin:
|
||||
|
||||
```ini
|
||||
[run]
|
||||
plugins = Cython.Coverage
|
||||
source = src/coolrl_lost_cities
|
||||
```
|
||||
|
||||
### 3. Recommendations
|
||||
* **Maintain Fast Defaults**: Keep the main tree "fast and boring." Avoid enabling Cython tracing by default.
|
||||
* **Targeted Audits**: Use Cython coverage only when introducing new complex logic in modules like `cfr_math.pyx` or `traversal.pyx` to ensure edge cases are exercised.
|
||||
* **External Trace**: Use a dedicated CI job or script for Cython coverage reporting rather than manual developer runs.
|
||||
|
||||
## References
|
||||
- `setup.py`: Extension definitions for `game.pyx`, `cfr_math.pyx`, `encoding.pyx`, `traversal.pyx`, and `heuristic_cy.pyx`.
|
||||
- `docs/archive/test-coverage-notes.md`: Initial profiling and snapshots.
|
||||
- [Cython Documentation: Debugging and profiling](https://cython.readthedocs.io/en/latest/src/userguide/debugging.html)
|
||||
+104
-32
@@ -14,13 +14,16 @@ Per archive output:
|
||||
the entry non-promotable.
|
||||
- `<stem>.ERROR.txt` — captured stderr, when the CLI itself failed.
|
||||
|
||||
Sequential; one LLM call per archive.
|
||||
Sequential by default; pass `--parallel N` to dispatch up to N concurrent
|
||||
LLM calls (subprocess.run is mostly waiting on the network, so threads
|
||||
are enough — no GIL fight).
|
||||
|
||||
Usage:
|
||||
uv run python scripts/librarian_survey.py
|
||||
LIBRARIAN_LLM=gemini uv run python scripts/librarian_survey.py
|
||||
uv run python scripts/librarian_survey.py --dry-run # list candidates only
|
||||
uv run python scripts/librarian_survey.py --max 3 # limit to first 3 candidates
|
||||
uv run python scripts/librarian_survey.py --max 3 # cap candidates
|
||||
uv run python scripts/librarian_survey.py --parallel 4 # 4 concurrent calls
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -30,6 +33,8 @@ import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
@@ -124,6 +129,42 @@ def _has_counterpart(archive_stem_no_date: str, research_stems: set[str]) -> boo
|
||||
return False
|
||||
|
||||
|
||||
def _dispatch_one(
|
||||
archive: Path,
|
||||
target: Path,
|
||||
*,
|
||||
root: Path,
|
||||
cmd: list[str],
|
||||
system_prompt: str,
|
||||
commit_sha: str,
|
||||
) -> tuple[Path, str, str, str]:
|
||||
"""Process a single archive end-to-end.
|
||||
|
||||
Returns ``(rel_archive, status, stem, payload)`` where ``status`` is
|
||||
one of ``"draft"``, ``"skip"``, or ``"error"``. ``payload`` is the
|
||||
text to write under ``out_dir / f"{stem}.<ext>"``. Pure function
|
||||
(apart from the subprocess call) so it is safe to call from a thread.
|
||||
"""
|
||||
rel_archive = archive.relative_to(root)
|
||||
rel_target = target.relative_to(root)
|
||||
archive_body = archive.read_text(encoding="utf-8")
|
||||
prompt = _assemble_prompt(system_prompt, rel_archive, archive_body, rel_target)
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
input=prompt,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
stem_out = DATE_SUFFIX.sub("", archive.stem)
|
||||
if result.returncode != 0:
|
||||
return (rel_archive, "error", stem_out, result.stderr)
|
||||
output = result.stdout.strip()
|
||||
if output.lstrip().upper().startswith("SKIP"):
|
||||
return (rel_archive, "skip", stem_out, output)
|
||||
return (rel_archive, "draft", stem_out, _post_process_draft(output, commit_sha))
|
||||
|
||||
|
||||
def _candidates(root: Path) -> list[tuple[Path, Path]]:
|
||||
"""Return (archive_path, suggested_research_target) for archives lacking a counterpart."""
|
||||
archive_dir = root / "docs" / "archive"
|
||||
@@ -152,7 +193,18 @@ def main() -> int:
|
||||
default=None,
|
||||
help="Process at most N archives (testing/cost guard).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--parallel",
|
||||
type=int,
|
||||
default=1,
|
||||
help=(
|
||||
"Number of concurrent LLM calls. Default 1 (sequential). "
|
||||
"Try 4 for a meaningful speedup on large surveys."
|
||||
),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
if args.parallel < 1:
|
||||
parser.error("--parallel must be >= 1")
|
||||
|
||||
root = _repo_root()
|
||||
candidates = _candidates(root)
|
||||
@@ -188,38 +240,58 @@ def main() -> int:
|
||||
drafts = 0
|
||||
skips = 0
|
||||
errors = 0
|
||||
for idx, (archive, target) in enumerate(candidates, 1):
|
||||
rel_archive = archive.relative_to(root)
|
||||
rel_target = target.relative_to(root)
|
||||
print(f"[{idx}/{len(candidates)}] {rel_archive} → {backend}", file=sys.stderr)
|
||||
total = len(candidates)
|
||||
print_lock = threading.Lock()
|
||||
|
||||
archive_body = archive.read_text(encoding="utf-8")
|
||||
prompt = _assemble_prompt(system_prompt, rel_archive, archive_body, rel_target)
|
||||
|
||||
result = subprocess.run(
|
||||
cmd,
|
||||
input=prompt,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
stem_out = DATE_SUFFIX.sub("", archive.stem)
|
||||
if result.returncode != 0:
|
||||
(out_dir / f"{stem_out}.ERROR.txt").write_text(result.stderr, encoding="utf-8")
|
||||
errors += 1
|
||||
print(f" ERROR (exit {result.returncode})", file=sys.stderr)
|
||||
continue
|
||||
|
||||
output = result.stdout.strip()
|
||||
if output.lstrip().upper().startswith("SKIP"):
|
||||
(out_dir / f"{stem_out}.SKIP.txt").write_text(output, encoding="utf-8")
|
||||
skips += 1
|
||||
print(f" SKIP ({output[:80]})", file=sys.stderr)
|
||||
def _record(idx: int, rel_archive: Path, status: str, stem: str, payload: str) -> str:
|
||||
if status == "error":
|
||||
(out_dir / f"{stem}.ERROR.txt").write_text(payload, encoding="utf-8")
|
||||
note = "ERROR"
|
||||
elif status == "skip":
|
||||
(out_dir / f"{stem}.SKIP.txt").write_text(payload, encoding="utf-8")
|
||||
note = f"SKIP ({payload[:80]})"
|
||||
else:
|
||||
normalized = _post_process_draft(output, commit_sha)
|
||||
(out_dir / f"{stem_out}.md").write_text(normalized, encoding="utf-8")
|
||||
drafts += 1
|
||||
print(f" draft ({len(normalized)} chars)", file=sys.stderr)
|
||||
(out_dir / f"{stem}.md").write_text(payload, encoding="utf-8")
|
||||
note = f"draft ({len(payload)} chars)"
|
||||
with print_lock:
|
||||
print(f"[{idx}/{total}] {rel_archive} → {backend}", file=sys.stderr)
|
||||
print(f" {note}", file=sys.stderr)
|
||||
return status
|
||||
|
||||
if args.parallel == 1:
|
||||
for idx, (archive, target) in enumerate(candidates, 1):
|
||||
rel_archive, status, stem_out, payload = _dispatch_one(
|
||||
archive,
|
||||
target,
|
||||
root=root,
|
||||
cmd=cmd,
|
||||
system_prompt=system_prompt,
|
||||
commit_sha=commit_sha,
|
||||
)
|
||||
kind = _record(idx, rel_archive, status, stem_out, payload)
|
||||
drafts += kind == "draft"
|
||||
skips += kind == "skip"
|
||||
errors += kind == "error"
|
||||
else:
|
||||
with ThreadPoolExecutor(max_workers=args.parallel) as pool:
|
||||
futures = {
|
||||
pool.submit(
|
||||
_dispatch_one,
|
||||
archive,
|
||||
target,
|
||||
root=root,
|
||||
cmd=cmd,
|
||||
system_prompt=system_prompt,
|
||||
commit_sha=commit_sha,
|
||||
): None
|
||||
for archive, target in candidates
|
||||
}
|
||||
for idx, fut in enumerate(as_completed(futures), 1):
|
||||
rel_archive, status, stem_out, payload = fut.result()
|
||||
kind = _record(idx, rel_archive, status, stem_out, payload)
|
||||
drafts += kind == "draft"
|
||||
skips += kind == "skip"
|
||||
errors += kind == "error"
|
||||
|
||||
print()
|
||||
print(f"Survey complete: {drafts} drafts, {skips} skips, {errors} errors.")
|
||||
|
||||
Reference in New Issue
Block a user