coolguyandClaude Opus 4.7
004b913a7b
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
115 lines
5.7 KiB
Markdown
115 lines
5.7 KiB
Markdown
# Deep CFR v0: Subsystem Coverage vs. Legacy Reference
|
|
|
|
**Last verified:** 2026-05-07, commit `ad0be89`
|
|
|
|
Source: `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`
|
|
|
|
## Question
|
|
|
|
What does this repository's Deep CFR implementation cover relative to the legacy
|
|
`../coolrl` reference, and where are the intentional gaps?
|
|
|
|
Short answer: **all core Deep CFR subsystems are implemented; the gaps are
|
|
tooling, not correctness.** Traversal, training, encoding, memory,
|
|
checkpoints, evaluation, self-play league, imitation pretraining, and PG
|
|
fine-tuning are all present in the Cython/PyTorch package. Missing items
|
|
relative to legacy (preset configs, per-worker observability, plotters, W&B)
|
|
are deliberately deferred. The real remaining frontier is *performance*
|
|
(Python-boundary policy calls, recursive Cython traversal), not feature parity.
|
|
|
|
## Scope
|
|
|
|
The goal of this repo from the start was *not* legacy feature parity — it was
|
|
higher training throughput via Cython hot-paths and cleaner experiment
|
|
infrastructure. The gap document tracks where parity has been achieved and where
|
|
it has not.
|
|
|
|
## What is fully implemented
|
|
|
|
**Traversal** — all of the core Deep CFR traversal logic lives in
|
|
`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx`:
|
|
recursive traversal with traverser/opponent node dispatch, outcome-sampling
|
|
with epsilon exploration and importance-weight correction, optional value
|
|
clipping, unsampled-regret modes (`zero` and `negative_node_value`), depth and
|
|
node-budget cutoffs with score-diff and rollout-based terminal values, deck-draw
|
|
chance sampling with push/pop state restoration, and instantaneous regret and
|
|
strategy memory collection.
|
|
|
|
The old Python fallback `traverser.py` has been removed from the mainline; Cython
|
|
is the sole traversal path.
|
|
|
|
**Training and memory** — `trainer.py`, `memory.py`, `networks.py` provide PyTorch
|
|
advantage networks (one per player) and a strategy network, legal-mask-aware
|
|
advantage loss, masked strategy cross-entropy, reservoir sampling with capacity
|
|
limits, single-process and multiprocess worker batches with result merging in the
|
|
parent.
|
|
|
|
**Encoding** — `encoding.pyx` exposes both a Python wrapper and a C-level buffer
|
|
write path. The feature set covers: phase flags, current/traversing player, deck
|
|
ratio, hand slot features, public expeditions for both players, public discards,
|
|
public card counts, total score and score diff, turn ratio, pending-discard
|
|
one-hot, and legal action mask.
|
|
|
|
**Runtime operations** — checkpoint save/load (`checkpoints.py`), config stored
|
|
in checkpoints and `config.json`, strategy-net policy adapter, evaluation against
|
|
registered classic bots (`evaluate.py`), training CLI and evaluation CLI
|
|
(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
|
|
`runtime_progress.json` / `train.log` run artifacts, self-play league with
|
|
snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
|
|
heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`),
|
|
and policy-gradient fine-tuning (`policy_gradient.py`).
|
|
|
|
As of `ad0be89`, the package also includes `inference_server.py`,
|
|
`inference_client.py`, `inference_buffers.py`, `analyze.py`, `tracking.py`, and
|
|
`traversal_stats.py` — additions beyond the original plan that support batched
|
|
inference experiments and richer metrics collection.
|
|
|
|
## Intentional gaps (not blockers)
|
|
|
|
| Area | Legacy has | This repo | Notes |
|
|
|---|---|---|---|
|
|
| Config presets | Many experiments | Sparse YAML configs | Intentional — config-first, not preset-first |
|
|
| Multiprocess observability | Progress callbacks per worker, hotspot timing | Basic worker merge only | Low-priority tooling gap |
|
|
| Metrics visualization | Plot/status commands | `metrics.jsonl` only; no built-in plotter | `analyze.py` partially addresses this |
|
|
| Checkpoint artifacts | W&B integration | Local only | Not a correctness issue |
|
|
| Legacy visualization helpers | Present | Not ported | Not needed for training |
|
|
|
|
None of these gaps affect the correctness or usefulness of the training loop.
|
|
|
|
## Performance gap: the real remaining frontier
|
|
|
|
The original split from the legacy repo was motivated by traversal performance,
|
|
not feature parity. As of `ad0be89`:
|
|
|
|
- Game state mutation, legal-action generation, apply/undo, and cached scoring
|
|
run in Cython (`game.pyx` in the parent package).
|
|
- Information-state encoding and regret matching have Cython modules
|
|
(`encoding.pyx`, `cfr_math.pyx`).
|
|
- The full traversal loop runs through `traversal.pyx`.
|
|
- Policy inference and reservoir memory materialization still cross the Python
|
|
boundary (PyTorch call and NumPy buffer write).
|
|
- Traversal is still recursive inside Cython; an explicit iterative Cython
|
|
scheduler is a future optimization.
|
|
|
|
The batched inference path (`inference_server.py` / `inference_client.py`) is an
|
|
opt-in experiment toward reducing the Python boundary cost for policy calls, but
|
|
it is not the default training path.
|
|
|
|
The performance roadmap (C-level action enumeration, push/pop in tight loops,
|
|
batched memory writes, batched policy inference, and ultimately a Cython
|
|
iterative traversal scheduler) is the main remaining work, not legacy feature
|
|
parity.
|
|
|
|
## Practical implication
|
|
|
|
- Don't reach for the legacy repo to fill correctness gaps — there are none in
|
|
scope. Reach for it only for tooling references (preset YAMLs, plotting,
|
|
W&B wiring) where porting is deliberately deferred.
|
|
- New experiments should land in this repo's Cython traversal path; backporting
|
|
to the legacy traverser is not a goal.
|
|
- When prioritizing optimization work, target the Python boundary
|
|
(policy inference, NumPy buffer writes) before re-tuning anything already
|
|
fully in Cython — that is where the remaining throughput is hiding.
|
|
- Treat tooling-gap items in the table above as "open tickets, not blockers";
|
|
they should not gate training or eval work.
|