coorl-lost-cities/docs/deep-cfr-v0-plan.md

Deep CFR v0 Plan

The first Deep CFR target is an end-to-end training pipeline that runs on the classic Lost Cities game and exercises the Cython game-state hot path. It does not need to be a final research-grade implementation.

Goals

1. Add a real Deep CFR package under coolrl_lost_cities.games.classic.deep_cfr.
2. Keep traversal in Cython so it can call GameState C APIs directly: _legal_actions_c, _unified_legal_actions_c, push_action, pop_action, score caches, and deck-sampling helpers.
3. Train minimal PyTorch advantage and strategy networks from traversal output.
4. Save and load checkpoints for networks, optimizer state, config, and training counters.
5. Run a small smoke test that completes at least one training iteration on a tiny workload.
6. Add a small benchmark for Cython rollout/traversal steps per second.

Non-goals

• Multi-machine distributed training.
• Highly optimized replay-memory storage.
• Exploitability calculation.
• Perfect feature encoding.
• Full ISMCTS integration.
• Large experiment orchestration.

Proposed Package Shape

src/coolrl_lost_cities/games/classic/deep_cfr/
  cfr_math.pyx
  cfr_math.pxd
  encoding.pyx
  encoding.pxd
  traversal.pyx
  traversal.pxd
  config.py
  memory.py
  networks.py
  trainer.py
  checkpoints.py

cfr_math.pyx and encoding.pyx already exist as scaffolding. The next major piece is traversal.pyx.

Chance Handling

Lost Cities has hidden information from deck order and opponent hand contents. For v0, model chance by sampling compatible deck order through GameState mutation instead of cloning Python snapshots.

Initial implementation:

1. Use a deterministic RNG seed per traversal.
2. Shuffle the remaining deck region with C-level deck swaps.
3. Apply and undo actions with push_action and pop_action.
4. Do not expose opponent hand or exact unseen deck order in the information state encoding.

This is enough for smoke tests and speed work. More precise public-belief or particle sampling can come later.

Encoding v1

The current encoding is intentionally minimal. Deep CFR v0 should expand it without leaking hidden information.

Include:

• Current phase.
• Current player.
• Traversing player.
• Player hand.
• Both players' public expeditions.
• Public discard piles.
• Remaining deck ratio.
• Cached score or score diff.
• Legal action mask.

Do not include:

• Opponent hand contents.
• Exact hidden deck order.
• Any future card identity that the acting player cannot infer.

The encoding should keep a stable input_dim for a given config and expose both Python wrappers for tests and C-level buffer writes for traversal.

Traversal v0

The first traversal does not need every Deep CFR detail. It should prove that the Cython control flow is viable.

Steps:

1. Add traversal.pyx/.pxd.
2. Implement a deterministic random rollout helper that uses C-level legal actions and push_action/pop_action.
3. Add external-sampling Deep CFR traversal using regret matching.
4. Emit advantage-memory rows with (info_state, iteration, action_advantages).
5. Emit strategy-memory rows with (info_state, iteration, action_policy).
6. Add tests for determinism, push/pop restoration, terminal handling, and action legality.

The first version may return Python objects at module boundaries. Inner loops should stay Cython-native.

Python Training Layer

trainer.py should own orchestration:

1. Build or load networks.
2. Run traversal workers.
3. Append samples to advantage and strategy memories.
4. Train advantage networks per player.
5. Train the average strategy network.
6. Periodically evaluate using the existing policy/evaluation layer.
7. Periodically checkpoint.

memory.py can start simple with bounded Python/NumPy buffers. Replace it only after profiling.

Validation

Required v0 checks:

1. Cython build succeeds.
2. Unit tests for math, encoding, chance sampling, and traversal restoration.
3. A tiny one-iteration trainer smoke test.
4. A benchmark that reports rollout/traversal steps per second.

The first useful benchmark is not model quality. It is whether traversal is actually using the Cython hot path instead of rebuilding Python masks and snapshots.