285 lines
6.8 KiB
Markdown
285 lines
6.8 KiB
Markdown
# Deep CFR v0 Gap vs Legacy coolrl
|
|
|
|
This document compares the current `coolrl-lost-cities` Deep CFR v0 smoke
|
|
pipeline with the legacy Lost Cities Deep CFR implementation in `../coolrl`.
|
|
|
|
The current implementation proves that Cython traversal primitives, PyTorch
|
|
networks, memory collection, and a one-iteration smoke run can work together. It
|
|
is not yet equivalent to the legacy implementation.
|
|
|
|
## Current v0
|
|
|
|
Implemented:
|
|
|
|
1. Cython traversal primitives:
|
|
- `random_rollout_value`
|
|
- `root_action_values`
|
|
- direct `GameState` C API use for legal actions and push/pop restoration
|
|
2. Minimal Cython information-state encoding.
|
|
3. Small PyTorch MLP.
|
|
4. Simple in-memory sample storage.
|
|
5. Minimal trainer that:
|
|
- samples root action values with random rollouts
|
|
- builds advantage-like targets
|
|
- trains advantage networks and strategy network once
|
|
6. Smoke tests for traversal restoration and trainer execution.
|
|
|
|
This is a scaffold, not a complete Deep CFR algorithm.
|
|
|
|
## Major Algorithm Gaps
|
|
|
|
### Recursive Deep CFR Traversal
|
|
|
|
Legacy `coolrl` has recursive outcome-sampling traversal. Current v0 does not.
|
|
|
|
Missing:
|
|
|
|
1. Recursive `traverse(state, traverser, iteration, depth)` logic.
|
|
2. Terminal value handling at every node.
|
|
3. Traverser node vs opponent node behavior.
|
|
4. Node value calculation from sampled actions.
|
|
5. Instantaneous regret calculation at traverser nodes.
|
|
6. Strategy-memory collection at traverser and/or opponent nodes.
|
|
7. Depth cutoff and node-budget cutoff.
|
|
|
|
This is the highest-priority gap.
|
|
|
|
### Network-driven Policies During Traversal
|
|
|
|
Current v0 does not use the advantage networks inside traversal. It estimates
|
|
root action values with random rollouts.
|
|
|
|
Legacy flow:
|
|
|
|
```text
|
|
encode information state
|
|
advantage network forward pass
|
|
regret matching over legal actions
|
|
sample action from policy
|
|
recurse
|
|
store regrets / strategy
|
|
```
|
|
|
|
Current flow:
|
|
|
|
```text
|
|
enumerate root actions
|
|
random rollout from each child
|
|
train on resulting root targets
|
|
```
|
|
|
|
### Outcome Sampling Controls
|
|
|
|
Legacy `coolrl` supports:
|
|
|
|
1. `outcome_sampling_epsilon`
|
|
2. sampled action probability correction
|
|
3. optional sampled value clipping
|
|
4. unsampled regret modes:
|
|
- `negative_node_value`
|
|
- `zero`
|
|
|
|
Current v0 has none of these.
|
|
|
|
### Cutoff Values
|
|
|
|
Legacy traversal supports:
|
|
|
|
1. score-diff cutoff values
|
|
2. random rollout cutoff values
|
|
3. safe-heuristic rollout cutoff values
|
|
4. rollout max-step timeouts
|
|
5. cutoff stats
|
|
|
|
Current v0 only uses random rollouts at the root-action helper level.
|
|
|
|
## Training and Memory Gaps
|
|
|
|
### Reservoir Memory
|
|
|
|
Legacy implementation has separate `AdvantageMemory` and `StrategyMemory` with:
|
|
|
|
1. capacity limits
|
|
2. reservoir sampling
|
|
3. legal masks stored with each sample
|
|
4. batch sampling
|
|
5. sample merging from traversal workers
|
|
|
|
Current v0 stores simple `TrainingSample` objects in a list-like memory.
|
|
|
|
### Legal-mask-aware Losses
|
|
|
|
Legacy advantage loss only trains legal action outputs:
|
|
|
|
```text
|
|
masked MSE over legal actions
|
|
```
|
|
|
|
Legacy strategy loss uses masked policy learning:
|
|
|
|
```text
|
|
masked logits -> log_softmax -> cross entropy against stored policy
|
|
```
|
|
|
|
Current v0 uses simple supervised MSE for both advantage and strategy targets.
|
|
This is enough for smoke testing, but not the intended training objective.
|
|
|
|
### Config System
|
|
|
|
Legacy config is split into:
|
|
|
|
1. rules config
|
|
2. network config
|
|
3. encoding config
|
|
4. traversal config
|
|
5. optimization config
|
|
6. memory config
|
|
7. evaluation config
|
|
8. checkpoint config
|
|
9. run config
|
|
|
|
It also supports YAML loading and experiment-level overrides.
|
|
|
|
Current v0 has only `DeepCFRConfig`.
|
|
|
|
## Runtime and Operations Gaps
|
|
|
|
### Checkpointing
|
|
|
|
Legacy checkpoints include:
|
|
|
|
1. config
|
|
2. Lost Cities rules config
|
|
3. iteration
|
|
4. input/action dimensions
|
|
5. advantage networks
|
|
6. strategy network
|
|
7. optimizer states
|
|
8. self-play league snapshots
|
|
9. latest and per-iteration checkpoint files
|
|
|
|
Current v0 has no checkpoint save/load.
|
|
|
|
### Evaluation Integration
|
|
|
|
Legacy implementation can load a strategy checkpoint as a bot and evaluate it
|
|
against supported opponents.
|
|
|
|
Supported legacy eval flow includes:
|
|
|
|
1. random opponent
|
|
2. safe heuristic opponent
|
|
3. passive discard opponent
|
|
4. noisy/safe variants through the evaluation layer
|
|
5. many detailed gameplay metrics
|
|
|
|
Current v0 does not evaluate during training and does not expose a strategy-net
|
|
bot adapter.
|
|
|
|
### CLI
|
|
|
|
Legacy implementation has command-line tools for:
|
|
|
|
1. training
|
|
2. evaluation
|
|
3. evaluation suites
|
|
4. status/progress
|
|
5. plotting and visualization
|
|
6. traversal benchmarking
|
|
7. imitation/pretraining experiments
|
|
8. policy-gradient fine-tuning experiments
|
|
|
|
Current v0 has no CLI.
|
|
|
|
### Multiprocessing Workers
|
|
|
|
Legacy traversal can run through multiprocessing worker batches:
|
|
|
|
1. worker count resolution, including `auto`
|
|
2. traversal chunking
|
|
3. frozen network state dict transfer
|
|
4. worker-local traversal
|
|
5. result merging
|
|
6. progress logging
|
|
7. hotspot profiling
|
|
|
|
Current v0 runs in-process only.
|
|
|
|
### Metrics and Logging
|
|
|
|
Legacy training writes:
|
|
|
|
1. `metrics.jsonl`
|
|
2. `runtime_progress.json`
|
|
3. `train.log`
|
|
4. traversal nodes/sec
|
|
5. cutoff rates
|
|
6. endpoint depth buckets
|
|
7. advantage and strategy losses
|
|
8. evaluation metrics
|
|
9. hotspot timing metrics
|
|
|
|
Current v0 returns a small `IterationMetrics` object.
|
|
|
|
## Encoding Gaps
|
|
|
|
Legacy encoding is much richer. It includes:
|
|
|
|
1. phase one-hot
|
|
2. current-player indicator
|
|
3. player id
|
|
4. hand slots with card-type one-hot and empty-slot flag
|
|
5. both players' expedition card counts
|
|
6. expedition lengths
|
|
7. last numeric rank per expedition
|
|
8. discard pile card counts
|
|
9. discard pile length
|
|
10. discard pile top card
|
|
11. public card counts
|
|
12. deck ratio
|
|
13. turn-count ratio
|
|
14. pending-discard one-hot
|
|
15. optional derived playability features
|
|
16. optional slot-aware playability features
|
|
|
|
Current v0 includes only a minimal subset:
|
|
|
|
1. phase flags
|
|
2. current player
|
|
3. encoded player
|
|
4. deck ratio
|
|
5. player hand slot features
|
|
6. legal action mask
|
|
|
|
The current encoding should be expanded before serious training runs.
|
|
|
|
## Self-play League Gap
|
|
|
|
Legacy implementation supports self-play league opponent selection:
|
|
|
|
1. current networks
|
|
2. recent snapshots
|
|
3. older snapshots
|
|
4. safe-heuristic anchor
|
|
5. snapshot interval
|
|
6. maximum snapshot count
|
|
|
|
Current v0 has no league or checkpoint-snapshot opponent sampling.
|
|
|
|
## Practical Priority
|
|
|
|
Recommended implementation order:
|
|
|
|
1. Implement real recursive Deep CFR traversal.
|
|
2. Add legal-mask-aware advantage and strategy memories/losses.
|
|
3. Expand encoding to include public board, discard, and score features.
|
|
4. Add checkpoint save/load.
|
|
5. Add strategy-net bot adapter and evaluation integration.
|
|
6. Add CLI for train/eval/smoke.
|
|
7. Add traversal stats and benchmark reporting.
|
|
8. Add multiprocessing workers only after the single-process algorithm is
|
|
correct.
|
|
|
|
The first three items are algorithm-critical. The rest are operationally useful
|
|
but should not block proving that the learning loop is correct.
|