rollout_steps was pinned to MAX_STEPS (400) while a round actually runs
~50 plies, and finished envs were only reset between updates. 83% of every
rollout was spent stepping already-done envs to produce masked-out zeros.
Measured at rollout_steps=400: active steps go 17.3% -> 100%, i.e. 5.8x the
learner actions per update for the same compute.
Resetting in-scan exposes three things that were previously benign:
- compute_gae bootstrapped truncated episodes from zero. That was safe only
because every episode used to terminate inside the scan; now episodes cross
the boundary, so thread V(s_T) through.
- rollout_metrics read final_env and summed rewards along the scan axis, both
of which assume one episode per slot. With several episodes per slot that
silently produces garbage, so aggregate at done boundaries instead.
- league assignments were redrawn only between updates, which would pin a slot
to one seat/opponent across every episode in a scan. Redraw them on reset.
Also anneal potential shaping against learner actions rather than padded scan
steps: the old accounting counted the dead steps, so a 5M-step anneal expired
within two updates of 250. Any earlier evidence that shaping does not help was
gathered with it effectively off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
The rival's policy now exposes a ranking, which drives both its own move and a
new HINT button: it highlights the card to play, where to put it, and where to
draw, with the model's confidence.
Undo and redo work per action rather than per turn — picking a card, choosing
its destination, and drawing are separate steps, as are the rival's moves — so a
finished game can be stepped back through from the result screen. The rival is
suspended while undone moves are pending, and PLAY FROM HERE resumes from the
reviewed position.
Cards now travel between zones instead of teleporting: a motion layer measures
each card's old and new position and animates the difference, flying cards out
of the deck face-down and flipping them over, and back into it on undo.
Also: the result screen gets a per-expedition score breakdown mirroring
human_play.py, and a rival card revealed by a discard-pile draw no longer
renders at full size in the card-back-sized rival hand row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
Every game is now dealt from a seed carried in the URL as `?seed=`, shown in
the menu, and re-dealable by typing it in, so a deal can be shared or replayed.
Seeds are hashed into a mulberry32 stream, independent of the Python shuffle
bank.
Cards previously teleported between zones: the only motion in the client was
the hover lift and the legal-target pulse. Cards are keyed by card id, so a
card that just moved into a zone mounts there and now animates in, with the
motion disabled under prefers-reduced-motion.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
Placing a card locked the turn in: the chosen card left the hand and no
affordance reverted the placement, forcing the move through. Clicking the
chosen destination again or pressing Escape now steps the selection back.
A failing ONNX inference left the AI's turn unadvanced, permanently
stalling the game. The AI turn now falls back to the heuristic policy.
Other fixes: the score plaque no longer covers hand cards (plaques become
compact chips at narrow widths and hand spacing tracks the viewport),
opponent cards drawn from a discard pile render face up, long expedition
stacks stay inside their lane, undo no longer bumps the generation counter
on empty history, and small viewports scroll instead of clipping.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
ismcts-bc-ceiling-2026-05-11 was filed under "Other"; promote it to a
proper SO-ISMCTS section between Deep CFR and Engine/Performance so the
catalog actually reflects the project's two algorithm families. As more
ISMCTS notes accrue they have an obvious home.
Summarizes the 13-cycle trap-exploration session: BC pretrain (heuristic
clone) is the self-play ceiling under our compute budget (1 GPU + 50
sims + 768x4 MLP). All variants (naive finetune, KL anchor, mirror
descent, mixed-opponent + opponent-aware search) either preserved BC
(~17-21/100 vs heuristic-cautious) or regressed to catastrophic
forgetting. The single largest improvement of the session — 4× win rate
on the same checkpoint — came from PUCT Q-value normalization at search
time, not from any learning change.
Records the mechanism (negative training signal from BC-vs-heuristic
games; search too shallow to find heuristic-beating moves), the
hypotheses we negated, and the dials left in code for future runs with
more compute.
C13/C14 cycles: heuristic-balanced bot plays a configurable fraction of
self-play games (training.mixed_opponent_fraction). Trainee turns are
stored as policy samples; opponent turns are taken by the bot directly
and not stored. When mcts.opponent_aware_search is set, the MCTS tree
also treats the opponent seat as that bot — opponent moves are applied
without expanding into the search tree, and all values are taken from
the traverser's perspective. This was Codex's top recommendation for
breaking the symmetric self-play weak fixed point.
Empirical: opponent-aware mixed self-play does NOT lift win-rate above
BC pretrain (vs heuristic-cautious 100-game eval):
C13 (mixed=0.5, no KL): 0/100 — catastrophic forgetting
C14 (mixed=0.2, KL beta=1): 17/100 — preserved BC, no improvement
BC pretrain baseline: 21/100
Combined with C10-C12 results, BC remains the ceiling under our
compute budget (1 GPU + 50 sims + 768x4 net). Code is left in place as
configurable dials for future runs with more compute.
Codex follow-up diagnostics identified two MCTS+training-loop issues that
together cap finetune-from-BC at the heuristic ceiling:
1. PUCT Q is in raw score units (~±100 for value_scale=100), but the
exploration bonus c_puct * prior * sqrt(N) / (1+n) is on order of 1-10
for our parameter ranges. Result: a single bad backup pushes q_eff
well below the bonus floor and that action is effectively never
visited again. With only 50 sims/move this is catastrophic for the
policy-improvement operator. Fix: divide q_eff by config.q_scale
(default 100, configurable) inside _select_action. Backups and value
targets remain in raw score units; only the selection signal is
normalized. AlphaZero canonical convention.
2. The current kl_anchor_beta path adds KL(current || ref) directly to
the loss. That preserves BC but prevents improvement (gradient
actively pulls policy back to reference). The standard regularized
policy improvement operator is to mix the target instead:
pi_target = softmax(alpha * log(pi_mcts) + (1-alpha) * log(pi_ref))
Anneal alpha from low (rely on BC) to high (rely on MCTS) over
training. Network learns to follow the regularized target, which
stays near BC early but lets MCTS-discovered improvements through
later.
Config additions:
- mcts.q_scale (default 100.0): PUCT Q divisor
- training.md_target_ref_ckpt: reference policy path (alternative to kl_anchor)
- training.md_target_alpha_start / _end / _iters: linear alpha schedule
Both Python mcts.py and Cython mcts.pyx updated; parity test passes.
Tests: 19/19.
Hypothesis: with normalized PUCT the network can actually explore and
exploit prior knowledge competently at 50 sims, and the mirror-descent
target lets self-play improvement happen while BC anchors the trajectory.
This is the operator-side fix that c9 (no anchor, collapsed) and c10/c11
(loss-side KL anchor, preserved-but-stuck) both missed.
Lets c6+ layer new exploration hyperparams on top of c5's learned
value head instead of restarting from random init. Saves ~60min per
cycle while preserving VPE-down trajectory observed in c5.
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score
avg -85 to -97 vs three heuristic opponents. Bigger network alone did
not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without
rollout) but agent still loses every natural-end game.
Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not
break the loss pattern. Score average actually slightly worse than
512x3 baseline. So the bottleneck is not network capacity.
Going to the long-run experiment: AlphaZero-correct setup with the
network value loop closed. use_rollout_value=false means leaf Q comes
from network value head. Training signal: network value learns from
actual game outcomes; MCTS uses those values to pick actions; better
actions produce better outcomes; cycle closes.
100 iter previously gave essentially the same result as rollout=true
(comparing c3 to trapfix baseline). Both are too early in the AlphaZero
training curve. Standard AlphaZero papers train 1000s of iterations.
Going long: 1000 iter with the current config. Self-play ~9s/iter
without rollout, total wall ~150 min for the train phase.
Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval
and generate analyze.py plots to visualize trajectory.
Network kept at 768x4 since bigger capacity does not actively hurt.
c3 showed use_rollout_value=false alone is not the fix: without the
heuristic rollout safety net the random-init network value gives bad
MCTS Q early on, the agent plays more (PA 0.10 -> 0.22+) but eats more
-20 expedition penalties (score worsened from -57 to -97 avg).
Mathematical intuition: in Lost Cities, opening an expedition is a
20-point commitment. Break-even requires rank-sum × (handshakes+1) >= 20.
The model has to learn:
- which colors to open (based on hand handshake/high-rank holdings)
- when to commit vs discard
- card-ordering constraints (ascending only)
This is a moderately rich value function. 512x3 (~800k params, ~290
input dim) might be undersized. Test capacity hypothesis with 768x4
(~2M params) while keeping the rollout safety net so MCTS Q stays
competent.
Other params from c1 kept: c_puct=5, virtual_loss=5, dirichlet
α=0.3/ε=0.4, parallel_simulations=64, n_simulations=50.
Codex flagged that mcts/value_prediction_error mathematically reconciles
with loss/value (MSE / value_scale^2 = 0.05) but the latter looks healthy
while the former says the value head is far off. To make this clearer in
W&B, expose:
- mcts/value_rmse — sqrt(MSE), in raw score units (interpretable)
- mcts/v_target_abs_mean — magnitude of |v_target|, indicates if game
outcomes are very lopsided (always negative for a losing agent)
- mcts/v_target_std — spread, low std means targets are saturated to
one end (e.g., always -100ish)
These let us see whether value head is failing because targets are
unlearnable variance, or just hard-to-predict, or because of saturation
at the value_scale=100 tanh boundary.
Tests: 19/19 passing.
Codex deep diagnosis surfaced two real issues in our MCTS pipeline:
1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1)
instead of respecting MctsConfig.parallel_simulations. Standalone evals
(eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all
use this entry point, so all eval-time MCTS was running 1 sim per batch
regardless of the configured 64. Training was unaffected because it
goes through interleaved_self_play._run_search_jobs which respects the
config. Fix uses min(config.parallel_simulations, sims - completed).
2. use_rollout_value defaulted to True (config.py) but was never set in
the YAML. With this, _expand_with_prior returns the heuristic rollout
value and discards network_value, so the network value head is trained
from final game scores but its outputs are never fed back into MCTS
backups. This explains why mcts/value_prediction_error stays high
despite training -- learning the value head produces no behavioral
change because MCTS never reads it.
Now setting use_rollout_value=false in default.yaml so the network value
head closes the loop. Combined with the existing Dirichlet root noise +
heuristic rollout removal, this should give the network's value learning
actual leverage on action selection.
Also: updated test_search_visit_counts_match_with_parallel_simulations
to test the correct invariant (legal-action set match + total visit
count near n_sims) rather than literal visit-count equality, which was
only true under the previous bug.
Tests: 19/19 passing.
Diagnosis: mcts/value_prediction_error stuck at 300-1000 (RMSE ~22 on score
range ±100), while loss/value stays at 0.05 because the loss divides
prediction and target by value_scale=100 (so MSE / 10000). Net effect: the
value head receives a tiny gradient relative to the policy head's
~1.75 cross-entropy loss, so it never learns to predict score scale well.
This adds a config knob to multiply the normalized value loss without
re-engineering the loss formula. value_loss_weight=50 recovers the raw
MSE magnitude (~2.5 vs policy loss ~1.75), giving the value head
comparable gradient signal.
50-iter sweep on default.yaml with stronger MCTS exploration:
- c_puct: 3.0 -> 5.0 (UCB weight, more exploration of low-prior actions)
- root_dirichlet_epsilon: 0.25 -> 0.4 (more noise injected at root prior)
Standalone eval at iter 50 (30 games/opponent, all natural-end, timeouts=0):
- vs heuristic-balanced: W=0/30 S=-70.5 (PA 0.15)
- vs heuristic-aggressive: W=2/30 S=-65.1 (PA 0.14) [+10, +4]
- vs heuristic-cautious: W=1/30 S=-48.0 (PA 0.14) [+2]
3 natural-end wins vs prev trapfix baseline iter 44 (which had 0 natural
wins + 1 timeout-tie). Stall trap fixed remains true (timeouts=0 in c1).
Trade-off observed: more exploration -> higher variance. Score avg vs
cautious worsened (-32 -> -48), but win events appeared. For the
non-terminal-win objective, exploration win > score-avg loss.
Next: commit to long run (300 iter) with these params before tuning more.
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw
from discard pile to extend deck) until max_steps timeout, then squeak by
on opponents' negative scores. All eval wins were from timeouts; agent
never won a naturally-terminating game. Self-play reinforced this because
timeout games still got a positive value target.
Fixes (no algorithm change, all MCTS hyperparameters or signal shaping):
- Dirichlet noise at root prior (AlphaZero standard, was missing):
mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion
mixes prior with Dirichlet(α). Callers in interleaved_self_play and
the internal evaluate_and_backup pass `not item.path`.
- Default config strengthens exploration on the 50-sim batched search:
c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new
root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25.
- Self-play timeout signal zeroed: `_finalize_context` sets v_target=0
if context.state is not terminal. Stops the network from learning
"stall = positive value".
New standalone evaluator:
- `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a
checkpoint, runs N games per opponent across a parallel pool, reports
win/score with 95% CIs plus per-game logging via --verbose. Defaults
cover heuristic-balanced/aggressive/cautious (rollout policy isn't in
the training-eval opponent list, so this is the natural way to compare
the trained policy against its rollout target).
Tests (19) still pass; .so rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Key changes for ISMCTS speed and correctness:
- Cython port: HeuristicBot helpers (`heuristic_cy.pyx` + new `.pxd`) and
ISMCTS searcher (`mcts.pyx`) now run as cdef. Both share a fast
unified-action path through GameState's C interface to avoid Python
round-trips on hot rollout/tree-walk paths.
- Multi-process self-play and eval: `workers.py`, `eval_worker.py`,
`interleaved_self_play.py`, plus trainer wiring with ProcessPoolExecutor
+ spawn context. Eval inside `evaluate.py` is parallel per opponent.
- ISMCTS-specific eval (`evaluate.py`) runs MCTS at decision time so the
metric matches deploy mode; `evaluation.eval_with_mcts` flag preserves
backwards-compatible policy-only eval when needed.
- Trainer logs progress per phase (self-play start/done, eval per
opponent), and value loss is now scaled by `value_scale` so policy and
value losses sit on comparable magnitudes.
- Compact info-set key (`info_set.py`) using packed-struct format and
child-key reuse during MCTS descent to cut per-step canonicalization.
Tests: 19 ISMCTS suite passing, including parity (Cython-vs-Python
sequential, batched-vs-sequential visit counts, push/pop round-trip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Loss section: add ISMCTS policy/value twin-axis panel alongside the
Deep CFR advantage/strategy panel. Both render only when their keys
exist; the unused side shows "No data".
- Memory Size: add memory/replay alongside memory/advantage and
memory/strategy.
- New MCTS section (analysis_09_mcts.png): visit-count entropy, value
prediction error, policy/MCTS KL — three iter-time scalars emitted by
IsMctsTrainer. Auto-skips on Deep CFR runs (no data).
Wrap AlphaZeroNet with a logits-only view so IS-MCTS training evaluation can call evaluate_strategy_network and emit the same full diagnostic metric set as Deep CFR. Adds root prior capture and per-iteration MCTS entropy, value error, and policy-vs-search KL metrics.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml --set run.experiment_name=ismcts-metrics-smoke --set run.max_iterations=2 --set training.games_per_iter=2
Mini Lost Cities (3색 5랭크) 50 iter / 20 eval games:
- score_diff vs random +33, vs heuristic_cautious -5.7 (45% 승률)
- play_action_rate 13~31% (Deep CFR의 0~2% 대비 명확)
- trap escaped on mini, "long-horizon credit assignment" 가설 지지
Codex commit e69f316으로 ISMCTS 구현 완료. Full game scale-up이
다음 후보 (100 iter ETA 3시간 추정).
Implements a proof-of-concept single-observer IS-MCTS trainer with AlphaZero-style policy/value network, determinization, replay, self-play, CLI configs, and focused tests. Mini acceptance run reaches positive random eval while keeping play_action_rate above the Deep CFR trap threshold.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml
R2 167 iter 시점 trap signature 명확히 R0/R1과 동질 (play_action_rate
≈ 0%, positive expedition 0.02/game, bad_open 0.98). 세 가지 opponent
환경(self / passive / competent) 모두에서 같은 결함 확정. opponent
dimension closed.
Doc §11에 R2 결과 + 세 런 비교표 + opponent 변경으로 풀 수 없음
결론 + 다음 방향(SO-ISMCTS 정공법) 기록.
coolguyandClaude Opus 4.8