coorl-lost-cities/docs/deep-cfr-v0-gap-vs-coolrl.md

Deep CFR v0 Status vs Legacy coolrl

This document tracks the Lost Cities Deep CFR functionality in this repository against the legacy implementation in ../coolrl.

Implemented In This Repository

Traversal

Implemented:

1. Recursive traverse(state, traverser, iteration, depth) logic.
2. Terminal value handling.
3. Traverser vs opponent node behavior.
4. Advantage-network-driven policies during traversal.
5. Regret matching over legal actions.
6. Sampled action recursion with node-value calculation.
7. Instantaneous regret collection at traverser nodes.
8. Strategy-memory collection.
9. Depth and node-budget cutoffs.
10. Outcome-sampling epsilon.
11. Sampled action probability correction.
12. Optional sampled value clipping.
13. Unsampled regret modes:
    • negative_node_value
    • zero
14. Score-diff and rollout-based cutoff values.
15. Random and safe-heuristic cutoff rollout policies.
16. Deck-draw chance sampling with state restoration.

Training And Memory

Implemented:

1. PyTorch advantage networks.
2. PyTorch strategy network.
3. Legal-mask-aware advantage loss.
4. Masked strategy cross-entropy loss.
5. Reservoir memory with capacity limits.
6. Batch sampling.
7. Legal masks stored with samples.
8. Single-process traversal.
9. Multiprocessing traversal worker batches.
10. Worker result merge in the parent trainer process.

Encoding

Implemented information-state features:

1. Phase flags.
2. Current player.
3. Encoded player.
4. Deck ratio.
5. Player hand slot features.
6. Public expedition summaries for both players.
7. Public discard summaries.
8. Public card counts.
9. Total score and score diff features.
10. Turn ratio.
11. Pending-discard one-hot.
12. Legal action mask.

The encoding is still compact compared with the legacy feature set, but it now contains the key public board, discard, score, and legal-action information.

Runtime Operations

Implemented:

1. Checkpoint save/load.
2. Latest and per-iteration checkpoint files.
3. Config stored in checkpoints and config.json.
4. Strategy-net policy adapter.
5. Evaluation against registered classic bots.
6. Training CLI.
7. Evaluation CLI.
8. Traversal benchmark CLI.
9. Local run files:
   • config.json
   • metrics.jsonl
   • runtime_progress.json
   • train.log
10. Traversal benchmark metrics.
11. Self-play league snapshots.
12. Self-play league opponent selection from stored snapshots.
13. Safe-heuristic anchor opponent path.
14. Weighted current/recent/older/anchor self-play league buckets.
15. Safe-heuristic imitation pretraining.
16. Policy-gradient fine-tuning.
17. Single-vs-multiprocessing benchmark comparison.

Remaining Differences From Legacy coolrl

The main Deep CFR v0 gaps listed earlier are now implemented. Remaining differences are mostly experiment-system maturity and legacy-specific research extras.

Still smaller than legacy:

1. Config is YAML-first and nested through Pydantic, but only one smoke preset exists under configs/deep_cfr/.
2. Multiprocessing exists, but it is intentionally simple:
   • no progress callback per worker batch
   • no hotspot timing profile
3. Metrics logging exists, but no plotting/status command exists yet.
4. Checkpoint artifacts are local only; W&B artifact integration is not added.
5. Legacy visualization helpers are not ported.

These remaining items are not blockers for running and iterating on Deep CFR v0.

Suggested Next Steps

1. Add W&B/JSONL tracker abstraction on top of the existing local run files.
2. Add a status/plot command that reads metrics.jsonl.
3. Add worker progress logging and hotspot timing profile.
4. Add W&B checkpoint artifacts after checkpoint quality is stable.