Codex deep diagnosis surfaced two real issues in our MCTS pipeline:
1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1)
instead of respecting MctsConfig.parallel_simulations. Standalone evals
(eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all
use this entry point, so all eval-time MCTS was running 1 sim per batch
regardless of the configured 64. Training was unaffected because it
goes through interleaved_self_play._run_search_jobs which respects the
config. Fix uses min(config.parallel_simulations, sims - completed).
2. use_rollout_value defaulted to True (config.py) but was never set in
the YAML. With this, _expand_with_prior returns the heuristic rollout
value and discards network_value, so the network value head is trained
from final game scores but its outputs are never fed back into MCTS
backups. This explains why mcts/value_prediction_error stays high
despite training -- learning the value head produces no behavioral
change because MCTS never reads it.
Now setting use_rollout_value=false in default.yaml so the network value
head closes the loop. Combined with the existing Dirichlet root noise +
heuristic rollout removal, this should give the network's value learning
actual leverage on action selection.
Also: updated test_search_visit_counts_match_with_parallel_simulations
to test the correct invariant (legal-action set match + total visit
count near n_sims) rather than literal visit-count equality, which was
only true under the previous bug.
Tests: 19/19 passing.
Key changes for ISMCTS speed and correctness:
- Cython port: HeuristicBot helpers (`heuristic_cy.pyx` + new `.pxd`) and
ISMCTS searcher (`mcts.pyx`) now run as cdef. Both share a fast
unified-action path through GameState's C interface to avoid Python
round-trips on hot rollout/tree-walk paths.
- Multi-process self-play and eval: `workers.py`, `eval_worker.py`,
`interleaved_self_play.py`, plus trainer wiring with ProcessPoolExecutor
+ spawn context. Eval inside `evaluate.py` is parallel per opponent.
- ISMCTS-specific eval (`evaluate.py`) runs MCTS at decision time so the
metric matches deploy mode; `evaluation.eval_with_mcts` flag preserves
backwards-compatible policy-only eval when needed.
- Trainer logs progress per phase (self-play start/done, eval per
opponent), and value loss is now scaled by `value_scale` so policy and
value losses sit on comparable magnitudes.
- Compact info-set key (`info_set.py`) using packed-struct format and
child-key reuse during MCTS descent to cut per-step canonicalization.
Tests: 19 ISMCTS suite passing, including parity (Cython-vs-Python
sequential, batched-vs-sequential visit counts, push/pop round-trip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap AlphaZeroNet with a logits-only view so IS-MCTS training evaluation can call evaluate_strategy_network and emit the same full diagnostic metric set as Deep CFR. Adds root prior capture and per-iteration MCTS entropy, value error, and policy-vs-search KL metrics.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml --set run.experiment_name=ismcts-metrics-smoke --set run.max_iterations=2 --set training.games_per_iter=2
Implements a proof-of-concept single-observer IS-MCTS trainer with AlphaZero-style policy/value network, determinization, replay, self-play, CLI configs, and focused tests. Mini acceptance run reaches positive random eval while keeping play_action_rate above the Deep CFR trap threshold.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml
Mirrors the discard_only plumbing pattern. Uses HeuristicBot() (default
balanced params) and converts the bot's phase-local action to unified via
state.to_unified_action. Recursive (Cython) path was already supported
and is unchanged.
Tests: smoke run + accept/reject validators. All 59 tests pass.
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The interleaved traversal scheduler's _regret_matching was hard-coded to
spread fallback policy uniformly across legal actions, regardless of the
configured regret_matching.all_negative_fallback. default.yaml has
shipped with all_negative_fallback: argmax_tiebreak since 618d5f8 based
on the 20-iter audit + 1000-iter empirical comparison in
docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md, but the
default scheduler was switched to interleaved in 09bbe7c, after which
the configured fallback mode silently no-op'd.
_regret_matching now takes fallback_mode and concentrates policy mass on
the lowest-index tied action when "argmax_tiebreak". Tiebreak is
deterministic; the Cython recursive traverser randomises ties using its
per-traverser RNG, which the batched policy does not have. Behaviour
matches the spirit of the recursive path (concentrate on best, do not
dilute uniformly).
Plumbed through BatchedPolicy, InterleavedTraversalConfig,
run_interleaved_traversal_batch, trainer.py, workers.py, and the
analyze_first_open_targets.py caller. Two unit tests added.
Also bumps default.yaml outcome_sampling_epsilon 0.2 -> 0.05. The
200-iter sweep in docs/plans/deep-cfr-selectivity.md section 1 showed
0.05 produced the best short-run safe_heuristic_strict score diff
(-40.01 vs -57.87 for 0.20). Recent experiments already used 0.05; the
default now matches actual experimental practice.
Neither change targets the diagnosed selection-bias bottleneck. They
align config intent with scheduler behaviour and make the default config
reproduce known-best knob settings out of the box.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ColorSharedNetwork previously sliced the input vector into n_colors equal
chunks (input_dim // n_colors). The slice boundaries do not align with the
actual encoding layout: adjacent slices contain phase flags, hand slots,
expedition state, scores, etc. mixed together. The "color-shared" encoder
was therefore sharing weights across semantically unrelated chunks, not
across per-color blocks. The single archived run that exercised this path
(2026-05-07_092137_color_shared_attention_1000iter) was killed at iter 41
and produced no eval data, so we have no measurement of whether a real
per-color architecture would help.
Adds compute_lost_cities_color_layout(input_dim) which returns explicit
per-color and common index lists for the standard Lost Cities encoding
(n_colors=5, hand_size=8, n_ranks=9). It recognises input_dim values
171, 219, 249, 297 across derived_playability and slot_aware_playability
flag combinations.
Per-color block (39 dims when derived_playability is on): both players'
expedition state for that color, discard top, public-histogram row,
pending-discard one-hot bit, legal-action draw-pile bit, and the
derived_playability per-color block. Slot-aware features are slot-major
and stay in common.
ColorSharedNetwork.forward now indexes per-color blocks via the layout
when input_dim matches a known schema. For other dims (unit tests,
non-Lost Cities use), it falls back to chunked slicing with a UserWarning
- preserves backward compatibility for tests but makes the legacy
behaviour visible.
No fair test of the new architecture was run as part of this commit.
Documented in docs/plans/deep-cfr-selectivity.md section 7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit of the Deep CFR information-state encoding identified two tiers of
non-pure features and removed both:
- Tier 3 (judgment): is_bad_open_candidate, open_risk_score,
is_safe_continuation. Same heuristic family used to label bad_open in
evaluation, embedded as model input.
- Tier 2 (projection): recoverable_score_no_bonus,
recoverable_margin_no_bonus, min_needed_to_break_even,
cards_needed_for_bonus, has_bonus_path. Mechanical but assumption-laden
("commit and play all currently-playable cards"). The no_bonus form is
asymmetric: it amplifies the immediate -20 penalty while truncating the
+20 bonus upside, biasing the model toward the same "don't open" basin
the diagnostics already flagged.
Input dim 365 -> 297. DERIVED_PLAYABILITY_PER_COLOR 19 -> 15;
SLOT_AWARE_PLAYABILITY_PER_SLOT 12 -> 6. Test shape assertions updated.
Also adds selectivity diagnostic infrastructure used to reach this point:
- traversal.outcome_unsampled_first_open_prior_alpha config field with
signed-prior overlay on unsampled first-open advantage targets (A1).
- analyze_first_open_counterfactual.py --post-policy to swap the
policy_player rollout policy and isolate selection bias (D1).
- analyze_first_open_followup.py to inspect post-forced-open behavior
(E2): same-color play vs discard counts, other-open rate, terminal
hand composition.
Findings recorded in docs/plans/deep-cfr-selectivity.md sections 3-6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds torch.autocast(fp16) + GradScaler around _train_advantage and
_train_strategy when run.use_amp=true and device=cuda. CPU/non-CUDA
falls back to fp32 no-op. Mitigations:
- scaler.unscale_(optimizer) before grad_clip.
- nonfinite-loss guard skips overflowing batches and counts them.
- diff.float().square() in advantage loss to avoid fp16 overflow.
- strategy mask/log_softmax kept in fp32.
New metrics: amp/grad_scale, amp/nonfinite_loss_count.
Tests: AMP CUDA smoke + CPU fallback in test_deep_cfr_trainer.py.
Bench: scripts/bench_amp_trainer.py micro-benches train phases under
synthetic replay memory. smoke.yaml result is fp32 3.22ms / AMP 3.92ms
(0.82×, regression). 100-iter A/B on default.yaml deliberately
skipped: smoke regression mirrors the 2026-05-07 torch.compile
regression dynamic (dispatch overhead > kernel benefit at this model
size) and re-confirming on the same size adds no information.
Default stays run.use_amp: false. Re-enable trigger documented in
docs/performance.md: hidden_size >= 1024 or num_layers >= 6, then run
the bench script + 100-iter A/B before flipping default.
Implements the central inference server pattern: a dedicated GPU
process owns advantage/strategy/league networks, batches policy
requests across traversal workers via shared-memory tensor pool, and
returns logits. Workers route forward calls through InferenceClient /
NetworkProxy when traversal.inference_backend == "server".
Default remains traversal.inference_backend: local. The server
backend regresses iter time ~3.8× on the inspected default config
(small-model dispatch + sync-blocking traversal capping realized
batch at ~num_workers=8 instead of the bs=64-256 needed to amortize
IPC overhead). Keeping the implementation behind the flag lets us
re-enable when (a) model size grows, (b) per-worker interleaved
traversal lands, or (c) eval becomes dominant — see
docs/performance.md "Option A Bench Result and Structural Ceiling"
for the full diagnosis.
Plumbing included:
- inference_buffers.py: shared-memory tensor pool with slot
management.
- inference_client.py: per-worker client + NetworkProxy adapter for
the existing traversal.pyx call sites.
- inference_server.py: spawn-context server process with
batch-window aggregation, weight sync, shutdown sentinel.
- bench_inference_backend.py: A/B between local and server backends
with eval/checkpoint disabled.
- test_inference_server.py: round-trip and integration tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython implementation in heuristic_cy.pyx achieves ~2.55× speedup on
opponent_act_seconds (200-game eval: 59.20s → 23.24s). Original Python
implementation preserved verbatim in heuristic_py.py as the equivalence
reference. Action-sequence equivalence is verified by
test_safe_heuristic_equivalence.py against seeded game corpora.
Key implementation notes:
- File-local wraparound=True override required for negative discard
indexing; Cython global wraparound=False would segfault.
- annotation_typing=False preserves verbatim Python semantics.
- _CachedState materializes hands/expeditions/discards/deck once per
act() call — this is the dominant performance win.
Further C-array optimization of _card_value_for_me /
_card_value_for_opponent / _color_commitment / _bonus_potential is
deferred. The current 2.55× delivers most of the dense-eval future
benefit; further work is gated on actually adopting denser eval
schedules (eval_every=5, games=1000).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adopt a 5-namespace scheme so wandb groups related metrics in the
sidebar and capture-group regex (eval/(random|safe_heuristic)/win_rate0
vs eval/(?:random|safe_heuristic)/win_rate0) controls panel splitting:
- loss/{advantage,strategy}
- samples/{advantage,strategy,advantage_player_N}
- memory/{advantage,strategy,advantage_player_N}
- time/{iteration_seconds,traversal_seconds,advantage_train_seconds,
strategy_train_seconds,evaluation_seconds,memory_add_seconds,
checkpoint_seconds,batch_tensor_seconds,nodes_per_second,
advantage_player_N_sample_seconds,strategy_sample_seconds}
- traversal/{nodes,terminals,depth_cutoffs,node_limit_cutoffs,
max_depth_reached,endpoints,avg_endpoint_depth,
endpoint_depth_bucket_*,regret_fallback_*,sampled_actions}
- eval/<opponent>/<metric> (3-level so opponent can be the capture group)
`iteration` keeps no namespace (it's the wandb step axis). Internal
TraversalStats.to_dict() and benchmark.py's standalone result dict
keep their flat names — only the trainer's emitted metrics are
remapped, with the traversal_*→traversal/* translation done at
insertion into runtime_metrics.
analyze.py updated to read the new keys (PlotSpec metrics, color map,
opponent_names parser, _first_existing_eval lookup). Tests updated for
the new eval_metrics dict keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make traversal progress and iteration-complete summary lines easier to
visually scan during long runs by leading with [i=N]. Drop the redundant
"iteration=N" kv from the body to keep lines short.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The avg-strategy 1000iter file (with the recent +traversals/+LR/+LCFR
changes) is the canonical "best-known" config. Renamed it to
default.yaml so users start from a single, obvious entry point and
override one field per ablation via --set. Other 12 configs moved to
configs/archive/ — kept for historical reproduction, not for active use.
- configs/deep_cfr/{default.yaml, smoke.yaml} are the only active configs
- experiment_name shortened to "deep-cfr-default" (was a long mouthful)
- AGENTS.md examples and Project Layout section rewritten around
default.yaml; ablation example shows the override-one-field pattern
- Tests pointed at the archived slot-playability config for the legacy
reproduction assertions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
If eval_every is positive but max_iterations falls before the next scheduled
eval iteration (including resume cases where current_iteration is already
past the last eval boundary), log a one-time warning at run start so the
user notices the misconfiguration. We deliberately do not force an
end-of-run eval, which would distort time budgets and reproducibility.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop checkpoint.directory from config — config defines what an experiment
is, not where its outputs go. The CLI now computes the run directory from
run.experiment_name plus a timestamp, defaulting to runs/tmp/ for
throwaway runs and runs/ when --keep is passed.
- Remove CheckpointConfig.directory and DeepCFRConfig.checkpoint_path
- DeepCFRTrainer takes run_dir: Path explicitly
- CLI: add --keep boolean; --resume requires an explicit path (no shortcut)
- Auto path: runs/[tmp/]<YYYY-MM-DD_HHMMSS>_<experiment_name-kebab>/
- Rename 13 configs to kebab-case; strip directory: lines; kebab their
experiment_name values
- Rewrite AGENTS.md training/run sections; document
archive/tmp/<flat> layout, --keep, kebab-case scope
- Update tests for new run_dir flow and dropped --resume shortcut
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove legacy aliases, rename max_hours to max_minutes, collapse the
four checkpoint save flags into save_every + save_latest, and change
defaults to safer values (opponent_policy=self_play_league,
device=auto, eval_every=50, max_depth=null). Migrate all archived
yaml configs and tests to the new schema.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror Deep CFR training metrics to W&B via a new WandbRunTracker
wired through CompositeRunTracker; wandb is an optional extra so
default installs and runs stay unchanged. Train CLI gains
--wandb/--wandb-project/--wandb-mode/--wandb-name/--wandb-tag, and
train() now closes the tracker in a finally block so runs finalize
even on early exit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add network.kind field to config (mlp/color_shared)
- Implement ColorSharedNetwork that:
- Splits input into 5 equal color blocks
- Encodes each block with shared weights
- Pools with mean/max aggregation
- Concatenates pooled embeddings with remainder
- Outputs same action logits as MLP
- Add ColorAttention for optional self-attention over color embeddings
- Add network.color_attention_layers and color_attention_heads config
- Maintain full backward compatibility (default kind=mlp)
- Add 28 comprehensive tests covering all architectures
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
coolguy
