Deep CFR v0 legacy gap 문서 추가

2026-05-06 23:18:24 +09:00
parent b5d82f99a1
commit 966cfa3a13
1 changed files with 284 additions and 0 deletions
@@ -0,0 +1,284 @@
+# Deep CFR v0 Gap vs Legacy coolrl
+
+This document compares the current `coolrl-lost-cities` Deep CFR v0 smoke
+pipeline with the legacy Lost Cities Deep CFR implementation in `../coolrl`.
+
+The current implementation proves that Cython traversal primitives, PyTorch
+networks, memory collection, and a one-iteration smoke run can work together. It
+is not yet equivalent to the legacy implementation.
+
+## Current v0
+
+Implemented:
+
+1. Cython traversal primitives:
+   - `random_rollout_value`
+   - `root_action_values`
+   - direct `GameState` C API use for legal actions and push/pop restoration
+2. Minimal Cython information-state encoding.
+3. Small PyTorch MLP.
+4. Simple in-memory sample storage.
+5. Minimal trainer that:
+   - samples root action values with random rollouts
+   - builds advantage-like targets
+   - trains advantage networks and strategy network once
+6. Smoke tests for traversal restoration and trainer execution.
+
+This is a scaffold, not a complete Deep CFR algorithm.
+
+## Major Algorithm Gaps
+
+### Recursive Deep CFR Traversal
+
+Legacy `coolrl` has recursive outcome-sampling traversal. Current v0 does not.
+
+Missing:
+
+1. Recursive `traverse(state, traverser, iteration, depth)` logic.
+2. Terminal value handling at every node.
+3. Traverser node vs opponent node behavior.
+4. Node value calculation from sampled actions.
+5. Instantaneous regret calculation at traverser nodes.
+6. Strategy-memory collection at traverser and/or opponent nodes.
+7. Depth cutoff and node-budget cutoff.
+
+This is the highest-priority gap.
+
+### Network-driven Policies During Traversal
+
+Current v0 does not use the advantage networks inside traversal. It estimates
+root action values with random rollouts.
+
+Legacy flow:
+
+```text
+encode information state
+advantage network forward pass
+regret matching over legal actions
+sample action from policy
+recurse
+store regrets / strategy
+```
+
+Current flow:
+
+```text
+enumerate root actions
+random rollout from each child
+train on resulting root targets
+```
+
+### Outcome Sampling Controls
+
+Legacy `coolrl` supports:
+
+1. `outcome_sampling_epsilon`
+2. sampled action probability correction
+3. optional sampled value clipping
+4. unsampled regret modes:
+   - `negative_node_value`
+   - `zero`
+
+Current v0 has none of these.
+
+### Cutoff Values
+
+Legacy traversal supports:
+
+1. score-diff cutoff values
+2. random rollout cutoff values
+3. safe-heuristic rollout cutoff values
+4. rollout max-step timeouts
+5. cutoff stats
+
+Current v0 only uses random rollouts at the root-action helper level.
+
+## Training and Memory Gaps
+
+### Reservoir Memory
+
+Legacy implementation has separate `AdvantageMemory` and `StrategyMemory` with:
+
+1. capacity limits
+2. reservoir sampling
+3. legal masks stored with each sample
+4. batch sampling
+5. sample merging from traversal workers
+
+Current v0 stores simple `TrainingSample` objects in a list-like memory.
+
+### Legal-mask-aware Losses
+
+Legacy advantage loss only trains legal action outputs:
+
+```text
+masked MSE over legal actions
+```
+
+Legacy strategy loss uses masked policy learning:
+
+```text
+masked logits -> log_softmax -> cross entropy against stored policy
+```
+
+Current v0 uses simple supervised MSE for both advantage and strategy targets.
+This is enough for smoke testing, but not the intended training objective.
+
+### Config System
+
+Legacy config is split into:
+
+1. rules config
+2. network config
+3. encoding config
+4. traversal config
+5. optimization config
+6. memory config
+7. evaluation config
+8. checkpoint config
+9. run config
+
+It also supports YAML loading and experiment-level overrides.
+
+Current v0 has only `DeepCFRConfig`.
+
+## Runtime and Operations Gaps
+
+### Checkpointing
+
+Legacy checkpoints include:
+
+1. config
+2. Lost Cities rules config
+3. iteration
+4. input/action dimensions
+5. advantage networks
+6. strategy network
+7. optimizer states
+8. self-play league snapshots
+9. latest and per-iteration checkpoint files
+
+Current v0 has no checkpoint save/load.
+
+### Evaluation Integration
+
+Legacy implementation can load a strategy checkpoint as a bot and evaluate it
+against supported opponents.
+
+Supported legacy eval flow includes:
+
+1. random opponent
+2. safe heuristic opponent
+3. passive discard opponent
+4. noisy/safe variants through the evaluation layer
+5. many detailed gameplay metrics
+
+Current v0 does not evaluate during training and does not expose a strategy-net
+bot adapter.
+
+### CLI
+
+Legacy implementation has command-line tools for:
+
+1. training
+2. evaluation
+3. evaluation suites
+4. status/progress
+5. plotting and visualization
+6. traversal benchmarking
+7. imitation/pretraining experiments
+8. policy-gradient fine-tuning experiments
+
+Current v0 has no CLI.
+
+### Multiprocessing Workers
+
+Legacy traversal can run through multiprocessing worker batches:
+
+1. worker count resolution, including `auto`
+2. traversal chunking
+3. frozen network state dict transfer
+4. worker-local traversal
+5. result merging
+6. progress logging
+7. hotspot profiling
+
+Current v0 runs in-process only.
+
+### Metrics and Logging
+
+Legacy training writes:
+
+1. `metrics.jsonl`
+2. `runtime_progress.json`
+3. `train.log`
+4. traversal nodes/sec
+5. cutoff rates
+6. endpoint depth buckets
+7. advantage and strategy losses
+8. evaluation metrics
+9. hotspot timing metrics
+
+Current v0 returns a small `IterationMetrics` object.
+
+## Encoding Gaps
+
+Legacy encoding is much richer. It includes:
+
+1. phase one-hot
+2. current-player indicator
+3. player id
+4. hand slots with card-type one-hot and empty-slot flag
+5. both players' expedition card counts
+6. expedition lengths
+7. last numeric rank per expedition
+8. discard pile card counts
+9. discard pile length
+10. discard pile top card
+11. public card counts
+12. deck ratio
+13. turn-count ratio
+14. pending-discard one-hot
+15. optional derived playability features
+16. optional slot-aware playability features
+
+Current v0 includes only a minimal subset:
+
+1. phase flags
+2. current player
+3. encoded player
+4. deck ratio
+5. player hand slot features
+6. legal action mask
+
+The current encoding should be expanded before serious training runs.
+
+## Self-play League Gap
+
+Legacy implementation supports self-play league opponent selection:
+
+1. current networks
+2. recent snapshots
+3. older snapshots
+4. safe-heuristic anchor
+5. snapshot interval
+6. maximum snapshot count
+
+Current v0 has no league or checkpoint-snapshot opponent sampling.
+
+## Practical Priority
+
+Recommended implementation order:
+
+1. Implement real recursive Deep CFR traversal.
+2. Add legal-mask-aware advantage and strategy memories/losses.
+3. Expand encoding to include public board, discard, and score features.
+4. Add checkpoint save/load.
+5. Add strategy-net bot adapter and evaluation integration.
+6. Add CLI for train/eval/smoke.
+7. Add traversal stats and benchmark reporting.
+8. Add multiprocessing workers only after the single-process algorithm is
+   correct.
+
+The first three items are algorithm-critical. The rest are operationally useful
+but should not block proving that the learning loop is correct.