coorl-lost-cities/docs/deep-cfr-v0-gap-vs-coolrl.md

Deep CFR v0 Gap vs Legacy coolrl

This document compares the current coolrl-lost-cities Deep CFR v0 smoke pipeline with the legacy Lost Cities Deep CFR implementation in ../coolrl.

The current implementation proves that Cython traversal primitives, PyTorch networks, memory collection, and a one-iteration smoke run can work together. It is not yet equivalent to the legacy implementation.

Current v0

Implemented:

1. Cython traversal primitives:
   • random_rollout_value
   • root_action_values
   • direct GameState C API use for legal actions and push/pop restoration
2. Minimal Cython information-state encoding.
3. Small PyTorch MLP.
4. Simple in-memory sample storage.
5. Minimal trainer that:
   • samples root action values with random rollouts
   • builds advantage-like targets
   • trains advantage networks and strategy network once
6. Smoke tests for traversal restoration and trainer execution.

This is a scaffold, not a complete Deep CFR algorithm.

Major Algorithm Gaps

Recursive Deep CFR Traversal

Legacy coolrl has recursive outcome-sampling traversal. Current v0 does not.

Missing:

1. Recursive traverse(state, traverser, iteration, depth) logic.
2. Terminal value handling at every node.
3. Traverser node vs opponent node behavior.
4. Node value calculation from sampled actions.
5. Instantaneous regret calculation at traverser nodes.
6. Strategy-memory collection at traverser and/or opponent nodes.
7. Depth cutoff and node-budget cutoff.

This is the highest-priority gap.

Network-driven Policies During Traversal

Current v0 does not use the advantage networks inside traversal. It estimates root action values with random rollouts.

Legacy flow:

encode information state
advantage network forward pass
regret matching over legal actions
sample action from policy
recurse
store regrets / strategy

Current flow:

enumerate root actions
random rollout from each child
train on resulting root targets

Outcome Sampling Controls

Legacy coolrl supports:

1. outcome_sampling_epsilon
2. sampled action probability correction
3. optional sampled value clipping
4. unsampled regret modes:
   • negative_node_value
   • zero

Current v0 has none of these.

Cutoff Values

Legacy traversal supports:

1. score-diff cutoff values
2. random rollout cutoff values
3. safe-heuristic rollout cutoff values
4. rollout max-step timeouts
5. cutoff stats

Current v0 only uses random rollouts at the root-action helper level.

Training and Memory Gaps

Reservoir Memory

Legacy implementation has separate AdvantageMemory and StrategyMemory with:

1. capacity limits
2. reservoir sampling
3. legal masks stored with each sample
4. batch sampling
5. sample merging from traversal workers

Current v0 stores simple TrainingSample objects in a list-like memory.

Legal-mask-aware Losses

Legacy advantage loss only trains legal action outputs:

masked MSE over legal actions

Legacy strategy loss uses masked policy learning:

masked logits -> log_softmax -> cross entropy against stored policy

Current v0 uses simple supervised MSE for both advantage and strategy targets. This is enough for smoke testing, but not the intended training objective.

Config System

Legacy config is split into:

1. rules config
2. network config
3. encoding config
4. traversal config
5. optimization config
6. memory config
7. evaluation config
8. checkpoint config
9. run config

It also supports YAML loading and experiment-level overrides.

Current v0 has only DeepCFRConfig.

Runtime and Operations Gaps

Checkpointing

Legacy checkpoints include:

1. config
2. Lost Cities rules config
3. iteration
4. input/action dimensions
5. advantage networks
6. strategy network
7. optimizer states
8. self-play league snapshots
9. latest and per-iteration checkpoint files

Current v0 has no checkpoint save/load.

Evaluation Integration

Legacy implementation can load a strategy checkpoint as a bot and evaluate it against supported opponents.

Supported legacy eval flow includes:

1. random opponent
2. safe heuristic opponent
3. passive discard opponent
4. noisy/safe variants through the evaluation layer
5. many detailed gameplay metrics

Current v0 does not evaluate during training and does not expose a strategy-net bot adapter.

CLI

Legacy implementation has command-line tools for:

1. training
2. evaluation
3. evaluation suites
4. status/progress
5. plotting and visualization
6. traversal benchmarking
7. imitation/pretraining experiments
8. policy-gradient fine-tuning experiments

Current v0 has no CLI.

Multiprocessing Workers

Legacy traversal can run through multiprocessing worker batches:

1. worker count resolution, including auto
2. traversal chunking
3. frozen network state dict transfer
4. worker-local traversal
5. result merging
6. progress logging
7. hotspot profiling

Current v0 runs in-process only.

Metrics and Logging

Legacy training writes:

1. metrics.jsonl
2. runtime_progress.json
3. train.log
4. traversal nodes/sec
5. cutoff rates
6. endpoint depth buckets
7. advantage and strategy losses
8. evaluation metrics
9. hotspot timing metrics

Current v0 returns a small IterationMetrics object.

Encoding Gaps

Legacy encoding is much richer. It includes:

1. phase one-hot
2. current-player indicator
3. player id
4. hand slots with card-type one-hot and empty-slot flag
5. both players' expedition card counts
6. expedition lengths
7. last numeric rank per expedition
8. discard pile card counts
9. discard pile length
10. discard pile top card
11. public card counts
12. deck ratio
13. turn-count ratio
14. pending-discard one-hot
15. optional derived playability features
16. optional slot-aware playability features

Current v0 includes only a minimal subset:

1. phase flags
2. current player
3. encoded player
4. deck ratio
5. player hand slot features
6. legal action mask

The current encoding should be expanded before serious training runs.

Self-play League Gap

Legacy implementation supports self-play league opponent selection:

1. current networks
2. recent snapshots
3. older snapshots
4. safe-heuristic anchor
5. snapshot interval
6. maximum snapshot count

Current v0 has no league or checkpoint-snapshot opponent sampling.

Practical Priority

Recommended implementation order:

1. Implement real recursive Deep CFR traversal.
2. Add legal-mask-aware advantage and strategy memories/losses.
3. Expand encoding to include public board, discard, and score features.
4. Add checkpoint save/load.
5. Add strategy-net bot adapter and evaluation integration.
6. Add CLI for train/eval/smoke.
7. Add traversal stats and benchmark reporting.
8. Add multiprocessing workers only after the single-process algorithm is correct.

The first three items are algorithm-critical. The rest are operationally useful but should not block proving that the learning loop is correct.