coorl-lost-cities/docs/archive/deep-cfr-v0-gap-vs-coolrl.md

# Deep CFR v0 Status vs Legacy coolrl

This document tracks the Lost Cities Deep CFR functionality in this repository
against the legacy implementation in `../coolrl`.

## Implemented In This Repository

### Traversal

Implemented:

1. Recursive `traverse(state, traverser, iteration, depth)` logic.
2. Terminal value handling.
3. Traverser vs opponent node behavior.
4. Advantage-network-driven policies during traversal.
5. Regret matching over legal actions.
6. Sampled action recursion with node-value calculation.
7. Instantaneous regret collection at traverser nodes.
8. Strategy-memory collection.
9. Depth and node-budget cutoffs.
10. Outcome-sampling epsilon.
11. Sampled action probability correction.
12. Optional sampled value clipping.
13. Unsampled regret modes:
    - `negative_node_value`
    - `zero`
14. Score-diff and rollout-based cutoff values.
15. Random and safe-heuristic cutoff rollout policies.
16. Deck-draw chance sampling with state restoration.

Important implementation note:

- The active training path now calls `deep_cfr/traversal.pyx`.
- The old Python recursive `deep_cfr/traverser.py` path has been removed from
  mainline code.
- The rules engine (`game.pyx`), encoding (`encoding.pyx`), regret-matching math
  (`cfr_math.pyx`), and Deep CFR tree-walking loop now have Cython
  implementations.

### Training And Memory

Implemented:

1. PyTorch advantage networks.
2. PyTorch strategy network.
3. Legal-mask-aware advantage loss.
4. Masked strategy cross-entropy loss.
5. Reservoir memory with capacity limits.
6. Batch sampling.
7. Legal masks stored with samples.
8. Single-process traversal.
9. Multiprocessing traversal worker batches.
10. Worker result merge in the parent trainer process.

### Encoding

Implemented information-state features:

1. Phase flags.
2. Current player.
3. Encoded player.
4. Deck ratio.
5. Player hand slot features.
6. Public expedition summaries for both players.
7. Public discard summaries.
8. Public card counts.
9. Total score and score diff features.
10. Turn ratio.
11. Pending-discard one-hot.
12. Legal action mask.

The encoding is still compact compared with the legacy feature set, but it now
contains the key public board, discard, score, and legal-action information.

### Runtime Operations

Implemented:

1. Checkpoint save/load.
2. Latest and per-iteration checkpoint files.
3. Config stored in checkpoints and `config.json`.
4. Strategy-net policy adapter.
5. Evaluation against registered classic bots.
6. Training CLI.
7. Evaluation CLI.
8. Traversal benchmark CLI.
9. Local run files:
   - `config.json`
   - `metrics.jsonl`
   - `runtime_progress.json`
   - `train.log`
10. Traversal benchmark metrics.
11. Self-play league snapshots.
12. Self-play league opponent selection from stored snapshots.
13. Safe-heuristic anchor opponent path.
14. Weighted current/recent/older/anchor self-play league buckets.
15. Safe-heuristic imitation pretraining.
16. Policy-gradient fine-tuning.
17. Single-vs-multiprocessing benchmark comparison.

## Remaining Differences From Legacy coolrl

The main Deep CFR v0 gaps listed earlier are now implemented. Remaining
differences are mostly experiment-system maturity and legacy-specific research
extras.

Still smaller than legacy:

1. Config is YAML-first and nested through Pydantic, but only one smoke preset
   exists under `configs/deep_cfr/`.
2. Multiprocessing exists, but it is intentionally simple:
   - no progress callback per worker batch
   - no hotspot timing profile
3. Metrics logging exists, but no plotting/status command exists yet.
4. Checkpoint artifacts are local only; W&B artifact integration is not added.
5. Legacy visualization helpers are not ported.

These remaining items are not blockers for running and iterating on Deep CFR v0.

## Performance-Critical Gap

This repository was split out to pursue much higher Lost Cities training
performance. From that perspective, the main remaining gap is not feature
parity with legacy `../coolrl`; it is the traversal backend.

Current state:

1. `GameState` mutation, legal-action generation, apply/undo, and cached scoring
   are implemented in Cython.
2. Information-state encoding and regret matching have Cython modules.
3. Full Deep CFR traversal now runs through `traversal.pyx`.
4. PyTorch policy inference and reservoir memory sample materialization still
   cross the Python boundary.
5. Traversal is still recursive inside Cython. The Python recursion-limit guard
   is no longer the main execution path, but an explicit iterative scheduler is
   still a future optimization.

Recommended performance roadmap:

1. Continue moving the traversal hot path away from Python object boundaries:
   - C-level legal action enumeration
   - C-level push/pop undo
   - terminal, depth cutoff, and node-budget cutoff
   - traverser/opponent node handling
   - outcome sampling
   - sampled action value correction
   - instantaneous regret calculation
   - strategy sample collection
   - traversal stats collection
2. Reduce Python boundary costs with batched memory writes.
3. Add batched network inference for policy calls.
4. Replace the recursive Cython DFS with an explicit Cython traversal scheduler.
5. Run multiple traversal contexts concurrently so policy-needed states can be
   encoded and evaluated in batches.

Python iterative traversal is not the preferred performance path. It would
remove Python recursion-limit risk, but it would keep most Python object and
callback overhead in the hot loop. For performance, the next serious step is a
Cython batched iterative traversal scheduler.

## Suggested Next Steps

1. Add batched memory writes from the Cython traversal engine.
2. Add benchmark output for recursive Cython traversal vs batched iterative
   traversal once the scheduler exists.
3. Add batched policy inference.
4. Add an explicit Cython iterative traversal scheduler.
5. Add a status/plot command that reads `metrics.jsonl`.
6. Add worker progress logging and hotspot timing profile.
7. Add W&B checkpoint artifacts after checkpoint quality is stable.