139 lines
4.2 KiB
Markdown
139 lines
4.2 KiB
Markdown
# Deep CFR v0 Plan
|
|
|
|
The first Deep CFR target is an end-to-end training pipeline that runs on the
|
|
classic Lost Cities game and exercises the Cython game-state hot path. It does
|
|
not need to be a final research-grade implementation.
|
|
|
|
## Goals
|
|
|
|
1. Add a real Deep CFR package under
|
|
`coolrl_lost_cities.games.classic.deep_cfr`.
|
|
2. Keep traversal in Cython so it can call `GameState` C APIs directly:
|
|
`_legal_actions_c`, `_unified_legal_actions_c`, `push_action`,
|
|
`pop_action`, score caches, and deck-sampling helpers.
|
|
3. Train minimal PyTorch advantage and strategy networks from traversal output.
|
|
4. Save and load checkpoints for networks, optimizer state, config, and
|
|
training counters.
|
|
5. Run a small smoke test that completes at least one training iteration on a
|
|
tiny workload.
|
|
6. Add a small benchmark for Cython rollout/traversal steps per second.
|
|
|
|
## Non-goals
|
|
|
|
- Multi-machine distributed training.
|
|
- Highly optimized replay-memory storage.
|
|
- Exploitability calculation.
|
|
- Perfect feature encoding.
|
|
- Full ISMCTS integration.
|
|
- Large experiment orchestration.
|
|
|
|
## Proposed Package Shape
|
|
|
|
```text
|
|
src/coolrl_lost_cities/games/classic/deep_cfr/
|
|
cfr_math.pyx
|
|
cfr_math.pxd
|
|
encoding.pyx
|
|
encoding.pxd
|
|
traversal.pyx
|
|
traversal.pxd
|
|
config.py
|
|
memory.py
|
|
networks.py
|
|
trainer.py
|
|
checkpoints.py
|
|
```
|
|
|
|
`cfr_math.pyx` and `encoding.pyx` already exist as scaffolding. The next major
|
|
piece is `traversal.pyx`.
|
|
|
|
## Chance Handling
|
|
|
|
Lost Cities has hidden information from deck order and opponent hand contents.
|
|
For v0, model chance by sampling compatible deck order through `GameState`
|
|
mutation instead of cloning Python snapshots.
|
|
|
|
Initial implementation:
|
|
|
|
1. Use a deterministic RNG seed per traversal.
|
|
2. Shuffle the remaining deck region with C-level deck swaps.
|
|
3. Apply and undo actions with `push_action` and `pop_action`.
|
|
4. Do not expose opponent hand or exact unseen deck order in the information
|
|
state encoding.
|
|
|
|
This is enough for smoke tests and speed work. More precise public-belief or
|
|
particle sampling can come later.
|
|
|
|
## Encoding v1
|
|
|
|
The current encoding is intentionally minimal. Deep CFR v0 should expand it
|
|
without leaking hidden information.
|
|
|
|
Include:
|
|
|
|
- Current phase.
|
|
- Current player.
|
|
- Traversing player.
|
|
- Player hand.
|
|
- Both players' public expeditions.
|
|
- Public discard piles.
|
|
- Remaining deck ratio.
|
|
- Cached score or score diff.
|
|
- Legal action mask.
|
|
|
|
Do not include:
|
|
|
|
- Opponent hand contents.
|
|
- Exact hidden deck order.
|
|
- Any future card identity that the acting player cannot infer.
|
|
|
|
The encoding should keep a stable `input_dim` for a given config and expose both
|
|
Python wrappers for tests and C-level buffer writes for traversal.
|
|
|
|
## Traversal v0
|
|
|
|
The first traversal does not need every Deep CFR detail. It should prove that the
|
|
Cython control flow is viable.
|
|
|
|
Steps:
|
|
|
|
1. Add `traversal.pyx/.pxd`.
|
|
2. Implement a deterministic random rollout helper that uses C-level legal
|
|
actions and `push_action`/`pop_action`.
|
|
3. Add external-sampling Deep CFR traversal using regret matching.
|
|
4. Emit advantage-memory rows with `(info_state, iteration, action_advantages)`.
|
|
5. Emit strategy-memory rows with `(info_state, iteration, action_policy)`.
|
|
6. Add tests for determinism, push/pop restoration, terminal handling, and
|
|
action legality.
|
|
|
|
The first version may return Python objects at module boundaries. Inner loops
|
|
should stay Cython-native.
|
|
|
|
## Python Training Layer
|
|
|
|
`trainer.py` should own orchestration:
|
|
|
|
1. Build or load networks.
|
|
2. Run traversal workers.
|
|
3. Append samples to advantage and strategy memories.
|
|
4. Train advantage networks per player.
|
|
5. Train the average strategy network.
|
|
6. Periodically evaluate using the existing policy/evaluation layer.
|
|
7. Periodically checkpoint.
|
|
|
|
`memory.py` can start simple with bounded Python/NumPy buffers. Replace it only
|
|
after profiling.
|
|
|
|
## Validation
|
|
|
|
Required v0 checks:
|
|
|
|
1. Cython build succeeds.
|
|
2. Unit tests for math, encoding, chance sampling, and traversal restoration.
|
|
3. A tiny one-iteration trainer smoke test.
|
|
4. A benchmark that reports rollout/traversal steps per second.
|
|
|
|
The first useful benchmark is not model quality. It is whether traversal is
|
|
actually using the Cython hot path instead of rebuilding Python masks and
|
|
snapshots.
|