Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck-
race rate, so the stalling that static opponents induced is gone. Duplicate match
eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both
seats, deal luck cancelling exactly.
Success criterion 1 does not pass. The carry probe is close to flat: expeditions
opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use
does move monotonically across all six carry levels, and in the right direction
(behind -> more multipliers), but the spread is 0.31 wagers.
Two diagnoses, one of which was mine and wrong:
- Residual potential shaping was NOT the cause. Annealing it fully to zero left
the probe just as flat.
- terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over
any realistic round margin, and maximising E[tanh] on a linear stretch is just
maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears
where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12
widens the wager spread 0.19 -> 0.31, which is the mechanism showing up.
The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is
a saturated dead zone with no gradient and the policy has learned nothing there.
The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to
be set from the scale rather than fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score
avg -85 to -97 vs three heuristic opponents. Bigger network alone did
not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without
rollout) but agent still loses every natural-end game.
Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not
break the loss pattern. Score average actually slightly worse than
512x3 baseline. So the bottleneck is not network capacity.
Going to the long-run experiment: AlphaZero-correct setup with the
network value loop closed. use_rollout_value=false means leaf Q comes
from network value head. Training signal: network value learns from
actual game outcomes; MCTS uses those values to pick actions; better
actions produce better outcomes; cycle closes.
100 iter previously gave essentially the same result as rollout=true
(comparing c3 to trapfix baseline). Both are too early in the AlphaZero
training curve. Standard AlphaZero papers train 1000s of iterations.
Going long: 1000 iter with the current config. Self-play ~9s/iter
without rollout, total wall ~150 min for the train phase.
Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval
and generate analyze.py plots to visualize trajectory.
Network kept at 768x4 since bigger capacity does not actively hurt.
c3 showed use_rollout_value=false alone is not the fix: without the
heuristic rollout safety net the random-init network value gives bad
MCTS Q early on, the agent plays more (PA 0.10 -> 0.22+) but eats more
-20 expedition penalties (score worsened from -57 to -97 avg).
Mathematical intuition: in Lost Cities, opening an expedition is a
20-point commitment. Break-even requires rank-sum × (handshakes+1) >= 20.
The model has to learn:
- which colors to open (based on hand handshake/high-rank holdings)
- when to commit vs discard
- card-ordering constraints (ascending only)
This is a moderately rich value function. 512x3 (~800k params, ~290
input dim) might be undersized. Test capacity hypothesis with 768x4
(~2M params) while keeping the rollout safety net so MCTS Q stays
competent.
Other params from c1 kept: c_puct=5, virtual_loss=5, dirichlet
α=0.3/ε=0.4, parallel_simulations=64, n_simulations=50.
Codex deep diagnosis surfaced two real issues in our MCTS pipeline:
1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1)
instead of respecting MctsConfig.parallel_simulations. Standalone evals
(eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all
use this entry point, so all eval-time MCTS was running 1 sim per batch
regardless of the configured 64. Training was unaffected because it
goes through interleaved_self_play._run_search_jobs which respects the
config. Fix uses min(config.parallel_simulations, sims - completed).
2. use_rollout_value defaulted to True (config.py) but was never set in
the YAML. With this, _expand_with_prior returns the heuristic rollout
value and discards network_value, so the network value head is trained
from final game scores but its outputs are never fed back into MCTS
backups. This explains why mcts/value_prediction_error stays high
despite training -- learning the value head produces no behavioral
change because MCTS never reads it.
Now setting use_rollout_value=false in default.yaml so the network value
head closes the loop. Combined with the existing Dirichlet root noise +
heuristic rollout removal, this should give the network's value learning
actual leverage on action selection.
Also: updated test_search_visit_counts_match_with_parallel_simulations
to test the correct invariant (legal-action set match + total visit
count near n_sims) rather than literal visit-count equality, which was
only true under the previous bug.
Tests: 19/19 passing.
50-iter sweep on default.yaml with stronger MCTS exploration:
- c_puct: 3.0 -> 5.0 (UCB weight, more exploration of low-prior actions)
- root_dirichlet_epsilon: 0.25 -> 0.4 (more noise injected at root prior)
Standalone eval at iter 50 (30 games/opponent, all natural-end, timeouts=0):
- vs heuristic-balanced: W=0/30 S=-70.5 (PA 0.15)
- vs heuristic-aggressive: W=2/30 S=-65.1 (PA 0.14) [+10, +4]
- vs heuristic-cautious: W=1/30 S=-48.0 (PA 0.14) [+2]
3 natural-end wins vs prev trapfix baseline iter 44 (which had 0 natural
wins + 1 timeout-tie). Stall trap fixed remains true (timeouts=0 in c1).
Trade-off observed: more exploration -> higher variance. Score avg vs
cautious worsened (-32 -> -48), but win events appeared. For the
non-terminal-win objective, exploration win > score-avg loss.
Next: commit to long run (300 iter) with these params before tuning more.
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw
from discard pile to extend deck) until max_steps timeout, then squeak by
on opponents' negative scores. All eval wins were from timeouts; agent
never won a naturally-terminating game. Self-play reinforced this because
timeout games still got a positive value target.
Fixes (no algorithm change, all MCTS hyperparameters or signal shaping):
- Dirichlet noise at root prior (AlphaZero standard, was missing):
mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion
mixes prior with Dirichlet(α). Callers in interleaved_self_play and
the internal evaluate_and_backup pass `not item.path`.
- Default config strengthens exploration on the 50-sim batched search:
c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new
root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25.
- Self-play timeout signal zeroed: `_finalize_context` sets v_target=0
if context.state is not terminal. Stops the network from learning
"stall = positive value".
New standalone evaluator:
- `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a
checkpoint, runs N games per opponent across a parallel pool, reports
win/score with 95% CIs plus per-game logging via --verbose. Defaults
cover heuristic-balanced/aggressive/cautious (rollout policy isn't in
the training-eval opponent list, so this is the natural way to compare
the trained policy against its rollout target).
Tests (19) still pass; .so rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Key changes for ISMCTS speed and correctness:
- Cython port: HeuristicBot helpers (`heuristic_cy.pyx` + new `.pxd`) and
ISMCTS searcher (`mcts.pyx`) now run as cdef. Both share a fast
unified-action path through GameState's C interface to avoid Python
round-trips on hot rollout/tree-walk paths.
- Multi-process self-play and eval: `workers.py`, `eval_worker.py`,
`interleaved_self_play.py`, plus trainer wiring with ProcessPoolExecutor
+ spawn context. Eval inside `evaluate.py` is parallel per opponent.
- ISMCTS-specific eval (`evaluate.py`) runs MCTS at decision time so the
metric matches deploy mode; `evaluation.eval_with_mcts` flag preserves
backwards-compatible policy-only eval when needed.
- Trainer logs progress per phase (self-play start/done, eval per
opponent), and value loss is now scaled by `value_scale` so policy and
value losses sit on comparable magnitudes.
- Compact info-set key (`info_set.py`) using packed-struct format and
child-key reuse during MCTS descent to cut per-step canonicalization.
Tests: 19 ISMCTS suite passing, including parity (Cython-vs-Python
sequential, batched-vs-sequential visit counts, push/pop round-trip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements a proof-of-concept single-observer IS-MCTS trainer with AlphaZero-style policy/value network, determinization, replay, self-play, CLI configs, and focused tests. Mini acceptance run reaches positive random eval while keeping play_action_rate above the Deep CFR trap threshold.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The interleaved traversal scheduler's _regret_matching was hard-coded to
spread fallback policy uniformly across legal actions, regardless of the
configured regret_matching.all_negative_fallback. default.yaml has
shipped with all_negative_fallback: argmax_tiebreak since 618d5f8 based
on the 20-iter audit + 1000-iter empirical comparison in
docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md, but the
default scheduler was switched to interleaved in 09bbe7c, after which
the configured fallback mode silently no-op'd.
_regret_matching now takes fallback_mode and concentrates policy mass on
the lowest-index tied action when "argmax_tiebreak". Tiebreak is
deterministic; the Cython recursive traverser randomises ties using its
per-traverser RNG, which the batched policy does not have. Behaviour
matches the spirit of the recursive path (concentrate on best, do not
dilute uniformly).
Plumbed through BatchedPolicy, InterleavedTraversalConfig,
run_interleaved_traversal_batch, trainer.py, workers.py, and the
analyze_first_open_targets.py caller. Two unit tests added.
Also bumps default.yaml outcome_sampling_epsilon 0.2 -> 0.05. The
200-iter sweep in docs/plans/deep-cfr-selectivity.md section 1 showed
0.05 produced the best short-run safe_heuristic_strict score diff
(-40.01 vs -57.87 for 0.20). Recent experiments already used 0.05; the
default now matches actual experimental practice.
Neither change targets the diagnosed selection-bias bottleneck. They
align config intent with scheduler behaviour and make the default config
reproduce known-best knob settings out of the box.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the central inference server pattern: a dedicated GPU
process owns advantage/strategy/league networks, batches policy
requests across traversal workers via shared-memory tensor pool, and
returns logits. Workers route forward calls through InferenceClient /
NetworkProxy when traversal.inference_backend == "server".
Default remains traversal.inference_backend: local. The server
backend regresses iter time ~3.8× on the inspected default config
(small-model dispatch + sync-blocking traversal capping realized
batch at ~num_workers=8 instead of the bs=64-256 needed to amortize
IPC overhead). Keeping the implementation behind the flag lets us
re-enable when (a) model size grows, (b) per-worker interleaved
traversal lands, or (c) eval becomes dominant — see
docs/performance.md "Option A Bench Result and Structural Ceiling"
for the full diagnosis.
Plumbing included:
- inference_buffers.py: shared-memory tensor pool with slot
management.
- inference_client.py: per-worker client + NetworkProxy adapter for
the existing traversal.pyx call sites.
- inference_server.py: spawn-context server process with
batch-window aggregation, weight sync, shutdown sentinel.
- bench_inference_backend.py: A/B between local and server backends
with eval/checkpoint disabled.
- test_inference_server.py: round-trip and integration tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The avg-strategy 1000iter file (with the recent +traversals/+LR/+LCFR
changes) is the canonical "best-known" config. Renamed it to
default.yaml so users start from a single, obvious entry point and
override one field per ablation via --set. Other 12 configs moved to
configs/archive/ — kept for historical reproduction, not for active use.
- configs/deep_cfr/{default.yaml, smoke.yaml} are the only active configs
- experiment_name shortened to "deep-cfr-default" (was a long mouthful)
- AGENTS.md examples and Project Layout section rewritten around
default.yaml; ablation example shows the override-one-field pattern
- Tests pointed at the archived slot-playability config for the legacy
reproduction assertions
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop checkpoint.directory from config — config defines what an experiment
is, not where its outputs go. The CLI now computes the run directory from
run.experiment_name plus a timestamp, defaulting to runs/tmp/ for
throwaway runs and runs/ when --keep is passed.
- Remove CheckpointConfig.directory and DeepCFRConfig.checkpoint_path
- DeepCFRTrainer takes run_dir: Path explicitly
- CLI: add --keep boolean; --resume requires an explicit path (no shortcut)
- Auto path: runs/[tmp/]<YYYY-MM-DD_HHMMSS>_<experiment_name-kebab>/
- Rename 13 configs to kebab-case; strip directory: lines; kebab their
experiment_name values
- Rewrite AGENTS.md training/run sections; document
archive/tmp/<flat> layout, --keep, kebab-case scope
- Update tests for new run_dir flow and dropped --resume shortcut
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove legacy aliases, rename max_hours to max_minutes, collapse the
four checkpoint save flags into save_every + save_latest, and change
defaults to safer values (opponent_policy=self_play_league,
device=auto, eval_every=50, max_depth=null). Migrate all archived
yaml configs and tests to the new schema.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strategy network (학습 중인 average policy)를 traversal opponent로 사용하는
새 옵션 추가. Deep CFR 이론적 수렴이 average strategy에 대한 보장이라는
점에 착안 — opponent_policy=network의 발산 문제를 완화할 수 있는지 실증.
구현:
- config: opponent_policy validator에 average_strategy 추가
- traversal.pyx: opponent_policy_id=3, strategy_network 인자, softmax 기반
policy 도출 (_policy_from_strategy_network)
- workers.py: TraversalWorkerBatch에 strategy_network state_dict 추가
- trainer.py: 직렬/병렬 traversal call에 strategy_network 전달
- 1000-iter 실험 config 추가 (opponent_policy=network와 동일 hyperparam)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
opponent_policy=network 설정으로 1000-iter 실험 두 개 (512x3, 1024x4)를
돌린 결과 두 실험 모두 policy collapse가 발생함을 확인. 큰 capacity는
plateau를 늘리지만 발산 자체를 막지 못함. 향후 학습은 self_play_league
기본값을 유지할 것을 권고.
- 실험에 사용한 config 두 개 추가
- 발견 분석 문서 추가 (원인, 비교, 권고)
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Add deep_cfr_color_shared_512x3.yaml: color_shared without attention
- Add deep_cfr_color_shared_attention_512x3.yaml: color_shared with 2-layer attention
- Both use 512 hidden size and 3 MLP layers
- Configs are ready for experimental training runs
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
coolguyandClaude Opus 4.8