From 573b59545fef7588216f572971608c9eb45fb19f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=A0=95=EC=8B=9C=EC=9B=90?= Date: Thu, 7 May 2026 00:02:25 +0900 Subject: [PATCH] =?UTF-8?q?Deep=20CFR=20legacy=20gap=20=EB=AC=B8=EC=84=9C?= =?UTF-8?q?=20=EC=A0=95=EB=A6=AC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/deep-cfr-v0-gap-vs-coolrl.md | 317 +++++++++--------------------- 1 file changed, 98 insertions(+), 219 deletions(-) diff --git a/docs/deep-cfr-v0-gap-vs-coolrl.md b/docs/deep-cfr-v0-gap-vs-coolrl.md index c5438dc..9e8212c 100644 --- a/docs/deep-cfr-v0-gap-vs-coolrl.md +++ b/docs/deep-cfr-v0-gap-vs-coolrl.md @@ -1,242 +1,121 @@ -# Deep CFR v0 Gap vs Legacy coolrl +# Deep CFR v0 Status vs Legacy coolrl -This document compares the current `coolrl-lost-cities` Deep CFR v0 -implementation with the legacy Lost Cities Deep CFR implementation in -`../coolrl`. +This document tracks the Lost Cities Deep CFR functionality in this repository +against the legacy implementation in `../coolrl`. -The current implementation proves that Cython traversal primitives, PyTorch -networks, memory collection, and a one-iteration smoke run can work together. It -is not yet equivalent to the legacy implementation. +## Implemented In This Repository -## Current v0 +### Traversal Implemented: -1. Cython traversal primitives: - - `random_rollout_value` - - `root_action_values` - - direct `GameState` C API use for legal actions and push/pop restoration -2. Recursive Python Deep CFR traverser: - - `traverse(state, traverser, iteration, depth)` logic - - terminal values - - traverser vs opponent node behavior - - sampled action recursion - - sampled action value and node value calculation - - instantaneous regret collection at traverser nodes - - strategy-memory collection - - depth and node-budget cutoffs -3. Advantage-network-driven traversal policies: - - information-state encoding - - advantage network forward pass - - regret matching over legal actions - - sampled action recursion -4. Minimal Cython information-state encoding. -5. Small PyTorch MLP. -6. Simple in-memory sample storage with legal masks. -7. Legal-mask-aware advantage loss and masked strategy loss. -8. Smoke tests for traversal restoration and trainer execution. +1. Recursive `traverse(state, traverser, iteration, depth)` logic. +2. Terminal value handling. +3. Traverser vs opponent node behavior. +4. Advantage-network-driven policies during traversal. +5. Regret matching over legal actions. +6. Sampled action recursion with node-value calculation. +7. Instantaneous regret collection at traverser nodes. +8. Strategy-memory collection. +9. Depth and node-budget cutoffs. +10. Outcome-sampling epsilon. +11. Sampled action probability correction. +12. Optional sampled value clipping. +13. Unsampled regret modes: + - `negative_node_value` + - `zero` +14. Score-diff and rollout-based cutoff values. +15. Random and safe-heuristic cutoff rollout policies. +16. Deck-draw chance sampling with state restoration. -This is now a real single-process Deep CFR v0, but it is still not equivalent to -the legacy implementation. +### Training And Memory -## Major Algorithm Gaps +Implemented: -### Outcome Sampling Controls +1. PyTorch advantage networks. +2. PyTorch strategy network. +3. Legal-mask-aware advantage loss. +4. Masked strategy cross-entropy loss. +5. Reservoir memory with capacity limits. +6. Batch sampling. +7. Legal masks stored with samples. +8. Single-process traversal. +9. Multiprocessing traversal worker batches. +10. Worker result merge in the parent trainer process. -Legacy `coolrl` supports: +### Encoding -1. `outcome_sampling_epsilon` -2. sampled action probability correction -3. optional sampled value clipping -4. unsampled regret modes: - - `negative_node_value` - - `zero` +Implemented information-state features: -Current v0 has none of these. +1. Phase flags. +2. Current player. +3. Encoded player. +4. Deck ratio. +5. Player hand slot features. +6. Public expedition summaries for both players. +7. Public discard summaries. +8. Public card counts. +9. Total score and score diff features. +10. Turn ratio. +11. Pending-discard one-hot. +12. Legal action mask. -### Cutoff Values +The encoding is still compact compared with the legacy feature set, but it now +contains the key public board, discard, score, and legal-action information. -Legacy traversal supports: +### Runtime Operations -1. score-diff cutoff values -2. random rollout cutoff values -3. safe-heuristic rollout cutoff values -4. rollout max-step timeouts -5. cutoff stats +Implemented: -Current v0 uses score-diff cutoff values for depth and node-budget cutoffs, but -does not support rollout-based cutoff values. +1. Checkpoint save/load. +2. Latest and per-iteration checkpoint files. +3. Config stored in checkpoints and `config.json`. +4. Strategy-net policy adapter. +5. Evaluation against registered classic bots. +6. Training CLI. +7. Evaluation CLI. +8. Traversal benchmark CLI. +9. Local run files: + - `config.json` + - `metrics.jsonl` + - `runtime_progress.json` + - `train.log` +10. Traversal benchmark metrics. +11. Self-play league snapshots. +12. Self-play league opponent selection from stored snapshots. +13. Safe-heuristic anchor opponent path. -## Training and Memory Gaps +## Remaining Differences From Legacy coolrl -### Reservoir Memory +The main Deep CFR v0 gaps listed earlier are now implemented. Remaining +differences are mostly experiment-system maturity and legacy-specific research +extras. -Legacy implementation has separate `AdvantageMemory` and `StrategyMemory` with: +Still smaller than legacy: -1. capacity limits -2. reservoir sampling -3. batch sampling -4. sample merging from traversal workers +1. Config is a single `DeepCFRConfig` dataclass rather than a deeply nested + YAML-first config tree. +2. Multiprocessing exists, but it is intentionally simple: + - no auto worker-count resolver + - no progress callback per worker batch + - no hotspot timing profile +3. Metrics logging exists, but no plotting/status command exists yet. +4. Checkpoint artifacts are local only; W&B artifact integration is not added. +5. Self-play league is simpler than legacy: + - snapshot sampling exists + - safe anchor path exists + - weighted recent/older/current buckets are not implemented +6. Legacy side experiments are not ported: + - imitation/pretraining commands + - policy-gradient fine-tuning commands + - legacy visualization helpers -Current v0 stores `TrainingSample` objects with legal masks, but still uses -list-like storage rather than true reservoir sampling. +These remaining items are not blockers for running and iterating on Deep CFR v0. -### Config System +## Suggested Next Steps -Legacy config is split into: - -1. rules config -2. network config -3. encoding config -4. traversal config -5. optimization config -6. memory config -7. evaluation config -8. checkpoint config -9. run config - -It also supports YAML loading and experiment-level overrides. - -Current v0 has only `DeepCFRConfig`. - -## Runtime and Operations Gaps - -### Checkpointing - -Legacy checkpoints include: - -1. config -2. Lost Cities rules config -3. iteration -4. input/action dimensions -5. advantage networks -6. strategy network -7. optimizer states -8. self-play league snapshots -9. latest and per-iteration checkpoint files - -Current v0 has no checkpoint save/load. - -### Evaluation Integration - -Legacy implementation can load a strategy checkpoint as a bot and evaluate it -against supported opponents. - -Supported legacy eval flow includes: - -1. random opponent -2. safe heuristic opponent -3. passive discard opponent -4. noisy/safe variants through the evaluation layer -5. many detailed gameplay metrics - -Current v0 does not evaluate during training and does not expose a strategy-net -bot adapter. - -### CLI - -Legacy implementation has command-line tools for: - -1. training -2. evaluation -3. evaluation suites -4. status/progress -5. plotting and visualization -6. traversal benchmarking -7. imitation/pretraining experiments -8. policy-gradient fine-tuning experiments - -Current v0 has no CLI. - -### Multiprocessing Workers - -Legacy traversal can run through multiprocessing worker batches: - -1. worker count resolution, including `auto` -2. traversal chunking -3. frozen network state dict transfer -4. worker-local traversal -5. result merging -6. progress logging -7. hotspot profiling - -Current v0 runs in-process only. - -### Metrics and Logging - -Legacy training writes: - -1. `metrics.jsonl` -2. `runtime_progress.json` -3. `train.log` -4. traversal nodes/sec -5. cutoff rates -6. endpoint depth buckets -7. advantage and strategy losses -8. evaluation metrics -9. hotspot timing metrics - -Current v0 returns a small `IterationMetrics` object. - -## Encoding Gaps - -Legacy encoding is much richer. It includes: - -1. phase one-hot -2. current-player indicator -3. player id -4. hand slots with card-type one-hot and empty-slot flag -5. both players' expedition card counts -6. expedition lengths -7. last numeric rank per expedition -8. discard pile card counts -9. discard pile length -10. discard pile top card -11. public card counts -12. deck ratio -13. turn-count ratio -14. pending-discard one-hot -15. optional derived playability features -16. optional slot-aware playability features - -Current v0 includes only a minimal subset: - -1. phase flags -2. current player -3. encoded player -4. deck ratio -5. player hand slot features -6. legal action mask - -The current encoding should be expanded before serious training runs. - -## Self-play League Gap - -Legacy implementation supports self-play league opponent selection: - -1. current networks -2. recent snapshots -3. older snapshots -4. safe-heuristic anchor -5. snapshot interval -6. maximum snapshot count - -Current v0 has no league or checkpoint-snapshot opponent sampling. - -## Practical Priority - -Recommended implementation order: - -1. Add outcome-sampling controls. -2. Add rollout-based cutoff values. -3. Replace list-like memory with reservoir memory. -4. Expand encoding to include public board, discard, and score features. -5. Add checkpoint save/load. -6. Add strategy-net bot adapter and evaluation integration. -7. Add CLI for train/eval/smoke. -8. Add traversal stats and benchmark reporting. -9. Add multiprocessing workers only after the single-process algorithm is - correct. - -The first four items are algorithm-critical. The rest are operationally useful -but should not block improving the learning loop. +1. Add W&B/JSONL tracker abstraction on top of the existing local run files. +2. Add a status/plot command that reads `metrics.jsonl`. +3. Add richer benchmark output comparing single-process and multiprocessing. +4. Add weighted self-play league bucket selection if experiments need it. +5. Add W&B checkpoint artifacts after checkpoint quality is stable.