ismcts-bc-ceiling-2026-05-11 was filed under "Other"; promote it to a
proper SO-ISMCTS section between Deep CFR and Engine/Performance so the
catalog actually reflects the project's two algorithm families. As more
ISMCTS notes accrue they have an obvious home.
Summarizes the 13-cycle trap-exploration session: BC pretrain (heuristic
clone) is the self-play ceiling under our compute budget (1 GPU + 50
sims + 768x4 MLP). All variants (naive finetune, KL anchor, mirror
descent, mixed-opponent + opponent-aware search) either preserved BC
(~17-21/100 vs heuristic-cautious) or regressed to catastrophic
forgetting. The single largest improvement of the session — 4× win rate
on the same checkpoint — came from PUCT Q-value normalization at search
time, not from any learning change.
Records the mechanism (negative training signal from BC-vs-heuristic
games; search too shallow to find heuristic-beating moves), the
hypotheses we negated, and the dials left in code for future runs with
more compute.
C13/C14 cycles: heuristic-balanced bot plays a configurable fraction of
self-play games (training.mixed_opponent_fraction). Trainee turns are
stored as policy samples; opponent turns are taken by the bot directly
and not stored. When mcts.opponent_aware_search is set, the MCTS tree
also treats the opponent seat as that bot — opponent moves are applied
without expanding into the search tree, and all values are taken from
the traverser's perspective. This was Codex's top recommendation for
breaking the symmetric self-play weak fixed point.
Empirical: opponent-aware mixed self-play does NOT lift win-rate above
BC pretrain (vs heuristic-cautious 100-game eval):
C13 (mixed=0.5, no KL): 0/100 — catastrophic forgetting
C14 (mixed=0.2, KL beta=1): 17/100 — preserved BC, no improvement
BC pretrain baseline: 21/100
Combined with C10-C12 results, BC remains the ceiling under our
compute budget (1 GPU + 50 sims + 768x4 net). Code is left in place as
configurable dials for future runs with more compute.
Codex follow-up diagnostics identified two MCTS+training-loop issues that
together cap finetune-from-BC at the heuristic ceiling:
1. PUCT Q is in raw score units (~±100 for value_scale=100), but the
exploration bonus c_puct * prior * sqrt(N) / (1+n) is on order of 1-10
for our parameter ranges. Result: a single bad backup pushes q_eff
well below the bonus floor and that action is effectively never
visited again. With only 50 sims/move this is catastrophic for the
policy-improvement operator. Fix: divide q_eff by config.q_scale
(default 100, configurable) inside _select_action. Backups and value
targets remain in raw score units; only the selection signal is
normalized. AlphaZero canonical convention.
2. The current kl_anchor_beta path adds KL(current || ref) directly to
the loss. That preserves BC but prevents improvement (gradient
actively pulls policy back to reference). The standard regularized
policy improvement operator is to mix the target instead:
pi_target = softmax(alpha * log(pi_mcts) + (1-alpha) * log(pi_ref))
Anneal alpha from low (rely on BC) to high (rely on MCTS) over
training. Network learns to follow the regularized target, which
stays near BC early but lets MCTS-discovered improvements through
later.
Config additions:
- mcts.q_scale (default 100.0): PUCT Q divisor
- training.md_target_ref_ckpt: reference policy path (alternative to kl_anchor)
- training.md_target_alpha_start / _end / _iters: linear alpha schedule
Both Python mcts.py and Cython mcts.pyx updated; parity test passes.
Tests: 19/19.
Hypothesis: with normalized PUCT the network can actually explore and
exploit prior knowledge competently at 50 sims, and the mirror-descent
target lets self-play improvement happen while BC anchors the trajectory.
This is the operator-side fix that c9 (no anchor, collapsed) and c10/c11
(loss-side KL anchor, preserved-but-stuck) both missed.
Lets c6+ layer new exploration hyperparams on top of c5's learned
value head instead of restarting from random init. Saves ~60min per
cycle while preserving VPE-down trajectory observed in c5.
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score
avg -85 to -97 vs three heuristic opponents. Bigger network alone did
not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without
rollout) but agent still loses every natural-end game.
Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not
break the loss pattern. Score average actually slightly worse than
512x3 baseline. So the bottleneck is not network capacity.
Going to the long-run experiment: AlphaZero-correct setup with the
network value loop closed. use_rollout_value=false means leaf Q comes
from network value head. Training signal: network value learns from
actual game outcomes; MCTS uses those values to pick actions; better
actions produce better outcomes; cycle closes.
100 iter previously gave essentially the same result as rollout=true
(comparing c3 to trapfix baseline). Both are too early in the AlphaZero
training curve. Standard AlphaZero papers train 1000s of iterations.
Going long: 1000 iter with the current config. Self-play ~9s/iter
without rollout, total wall ~150 min for the train phase.
Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval
and generate analyze.py plots to visualize trajectory.
Network kept at 768x4 since bigger capacity does not actively hurt.
c3 showed use_rollout_value=false alone is not the fix: without the
heuristic rollout safety net the random-init network value gives bad
MCTS Q early on, the agent plays more (PA 0.10 -> 0.22+) but eats more
-20 expedition penalties (score worsened from -57 to -97 avg).
Mathematical intuition: in Lost Cities, opening an expedition is a
20-point commitment. Break-even requires rank-sum × (handshakes+1) >= 20.
The model has to learn:
- which colors to open (based on hand handshake/high-rank holdings)
- when to commit vs discard
- card-ordering constraints (ascending only)
This is a moderately rich value function. 512x3 (~800k params, ~290
input dim) might be undersized. Test capacity hypothesis with 768x4
(~2M params) while keeping the rollout safety net so MCTS Q stays
competent.
Other params from c1 kept: c_puct=5, virtual_loss=5, dirichlet
α=0.3/ε=0.4, parallel_simulations=64, n_simulations=50.
Codex flagged that mcts/value_prediction_error mathematically reconciles
with loss/value (MSE / value_scale^2 = 0.05) but the latter looks healthy
while the former says the value head is far off. To make this clearer in
W&B, expose:
- mcts/value_rmse — sqrt(MSE), in raw score units (interpretable)
- mcts/v_target_abs_mean — magnitude of |v_target|, indicates if game
outcomes are very lopsided (always negative for a losing agent)
- mcts/v_target_std — spread, low std means targets are saturated to
one end (e.g., always -100ish)
These let us see whether value head is failing because targets are
unlearnable variance, or just hard-to-predict, or because of saturation
at the value_scale=100 tanh boundary.
Tests: 19/19 passing.
Codex deep diagnosis surfaced two real issues in our MCTS pipeline:
1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1)
instead of respecting MctsConfig.parallel_simulations. Standalone evals
(eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all
use this entry point, so all eval-time MCTS was running 1 sim per batch
regardless of the configured 64. Training was unaffected because it
goes through interleaved_self_play._run_search_jobs which respects the
config. Fix uses min(config.parallel_simulations, sims - completed).
2. use_rollout_value defaulted to True (config.py) but was never set in
the YAML. With this, _expand_with_prior returns the heuristic rollout
value and discards network_value, so the network value head is trained
from final game scores but its outputs are never fed back into MCTS
backups. This explains why mcts/value_prediction_error stays high
despite training -- learning the value head produces no behavioral
change because MCTS never reads it.
Now setting use_rollout_value=false in default.yaml so the network value
head closes the loop. Combined with the existing Dirichlet root noise +
heuristic rollout removal, this should give the network's value learning
actual leverage on action selection.
Also: updated test_search_visit_counts_match_with_parallel_simulations
to test the correct invariant (legal-action set match + total visit
count near n_sims) rather than literal visit-count equality, which was
only true under the previous bug.
Tests: 19/19 passing.
Diagnosis: mcts/value_prediction_error stuck at 300-1000 (RMSE ~22 on score
range ±100), while loss/value stays at 0.05 because the loss divides
prediction and target by value_scale=100 (so MSE / 10000). Net effect: the
value head receives a tiny gradient relative to the policy head's
~1.75 cross-entropy loss, so it never learns to predict score scale well.
This adds a config knob to multiply the normalized value loss without
re-engineering the loss formula. value_loss_weight=50 recovers the raw
MSE magnitude (~2.5 vs policy loss ~1.75), giving the value head
comparable gradient signal.
50-iter sweep on default.yaml with stronger MCTS exploration:
- c_puct: 3.0 -> 5.0 (UCB weight, more exploration of low-prior actions)
- root_dirichlet_epsilon: 0.25 -> 0.4 (more noise injected at root prior)
Standalone eval at iter 50 (30 games/opponent, all natural-end, timeouts=0):
- vs heuristic-balanced: W=0/30 S=-70.5 (PA 0.15)
- vs heuristic-aggressive: W=2/30 S=-65.1 (PA 0.14) [+10, +4]
- vs heuristic-cautious: W=1/30 S=-48.0 (PA 0.14) [+2]
3 natural-end wins vs prev trapfix baseline iter 44 (which had 0 natural
wins + 1 timeout-tie). Stall trap fixed remains true (timeouts=0 in c1).
Trade-off observed: more exploration -> higher variance. Score avg vs
cautious worsened (-32 -> -48), but win events appeared. For the
non-terminal-win objective, exploration win > score-avg loss.
Next: commit to long run (300 iter) with these params before tuning more.
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw
from discard pile to extend deck) until max_steps timeout, then squeak by
on opponents' negative scores. All eval wins were from timeouts; agent
never won a naturally-terminating game. Self-play reinforced this because
timeout games still got a positive value target.
Fixes (no algorithm change, all MCTS hyperparameters or signal shaping):
- Dirichlet noise at root prior (AlphaZero standard, was missing):
mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion
mixes prior with Dirichlet(α). Callers in interleaved_self_play and
the internal evaluate_and_backup pass `not item.path`.
- Default config strengthens exploration on the 50-sim batched search:
c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new
root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25.
- Self-play timeout signal zeroed: `_finalize_context` sets v_target=0
if context.state is not terminal. Stops the network from learning
"stall = positive value".
New standalone evaluator:
- `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a
checkpoint, runs N games per opponent across a parallel pool, reports
win/score with 95% CIs plus per-game logging via --verbose. Defaults
cover heuristic-balanced/aggressive/cautious (rollout policy isn't in
the training-eval opponent list, so this is the natural way to compare
the trained policy against its rollout target).
Tests (19) still pass; .so rebuilt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Key changes for ISMCTS speed and correctness:
- Cython port: HeuristicBot helpers (`heuristic_cy.pyx` + new `.pxd`) and
ISMCTS searcher (`mcts.pyx`) now run as cdef. Both share a fast
unified-action path through GameState's C interface to avoid Python
round-trips on hot rollout/tree-walk paths.
- Multi-process self-play and eval: `workers.py`, `eval_worker.py`,
`interleaved_self_play.py`, plus trainer wiring with ProcessPoolExecutor
+ spawn context. Eval inside `evaluate.py` is parallel per opponent.
- ISMCTS-specific eval (`evaluate.py`) runs MCTS at decision time so the
metric matches deploy mode; `evaluation.eval_with_mcts` flag preserves
backwards-compatible policy-only eval when needed.
- Trainer logs progress per phase (self-play start/done, eval per
opponent), and value loss is now scaled by `value_scale` so policy and
value losses sit on comparable magnitudes.
- Compact info-set key (`info_set.py`) using packed-struct format and
child-key reuse during MCTS descent to cut per-step canonicalization.
Tests: 19 ISMCTS suite passing, including parity (Cython-vs-Python
sequential, batched-vs-sequential visit counts, push/pop round-trip).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Loss section: add ISMCTS policy/value twin-axis panel alongside the
Deep CFR advantage/strategy panel. Both render only when their keys
exist; the unused side shows "No data".
- Memory Size: add memory/replay alongside memory/advantage and
memory/strategy.
- New MCTS section (analysis_09_mcts.png): visit-count entropy, value
prediction error, policy/MCTS KL — three iter-time scalars emitted by
IsMctsTrainer. Auto-skips on Deep CFR runs (no data).
Wrap AlphaZeroNet with a logits-only view so IS-MCTS training evaluation can call evaluate_strategy_network and emit the same full diagnostic metric set as Deep CFR. Adds root prior capture and per-iteration MCTS entropy, value error, and policy-vs-search KL metrics.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml --set run.experiment_name=ismcts-metrics-smoke --set run.max_iterations=2 --set training.games_per_iter=2
Mini Lost Cities (3색 5랭크) 50 iter / 20 eval games:
- score_diff vs random +33, vs heuristic_cautious -5.7 (45% 승률)
- play_action_rate 13~31% (Deep CFR의 0~2% 대비 명확)
- trap escaped on mini, "long-horizon credit assignment" 가설 지지
Codex commit e69f316으로 ISMCTS 구현 완료. Full game scale-up이
다음 후보 (100 iter ETA 3시간 추정).
Implements a proof-of-concept single-observer IS-MCTS trainer with AlphaZero-style policy/value network, determinization, replay, self-play, CLI configs, and focused tests. Mini acceptance run reaches positive random eval while keeping play_action_rate above the Deep CFR trap threshold.
Tests: uv run python -m pytest tests/games/classic/ismcts/ -x; uv run python -m pytest tests/games/classic/test_deep_cfr_trainer.py -x; uv run lost-cities-ismcts train --config configs/ismcts/mini.yaml
R2 167 iter 시점 trap signature 명확히 R0/R1과 동질 (play_action_rate
≈ 0%, positive expedition 0.02/game, bad_open 0.98). 세 가지 opponent
환경(self / passive / competent) 모두에서 같은 결함 확정. opponent
dimension closed.
Doc §11에 R2 결과 + 세 런 비교표 + opponent 변경으로 풀 수 없음
결론 + 다음 방향(SO-ISMCTS 정공법) 기록.
Mirrors the discard_only plumbing pattern. Uses HeuristicBot() (default
balanced params) and converts the bot's phase-local action to unified via
state.to_unified_action. Recursive (Cython) path was already supported
and is unchanged.
Tests: smoke run + accept/reject validators. All 59 tests pass.
이전 레포(../coolrl) commit 5a98855, da0d4cd에서 시도된
regret_matching_epsilon 튜닝(eps1e3 → eps1e4)이 zero-pit 대응책이었고
1e-4가 best로 채택되어 현재 default 0.0001로 남아있음을 doc에 기록.
이 lever는 이미 당겨진 상태이며 R1의 새 zero-pit은 다른 원인
(opponent=discard_only가 만든 game-theoretic 균형)이라는 점 명시.
R1 결론: trap 깨지지 않음. 자기참조 회로는 충분조건이 아니라 trap 강화
요인일 뿐. opponent를 fixed external로 바꾸자 모델이 zero-pit으로
collapse — 핵심 결함은 model 자체의 credit assignment 실패임이 확정.
R2 방향 = curriculum (작은 게임 → 큰 게임)을 후속 후보로 등록.
Replace the three explicit Cython-generated paths with src/**/*.c so
new .pyx files anywhere under src/ are automatically covered (the
recent bots/heuristic_cy.pyx surfaced this gap; its generated
heuristic_cy.c was showing up as untracked).
Also ignore .compute.lock (local flock file used by training/benchmarks
to coordinate GPU access) and .claude/ (Claude Code session metadata
directory). Both are local-only artifacts and should never be checked
in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The interleaved traversal scheduler's _regret_matching was hard-coded to
spread fallback policy uniformly across legal actions, regardless of the
configured regret_matching.all_negative_fallback. default.yaml has
shipped with all_negative_fallback: argmax_tiebreak since 618d5f8 based
on the 20-iter audit + 1000-iter empirical comparison in
docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md, but the
default scheduler was switched to interleaved in 09bbe7c, after which
the configured fallback mode silently no-op'd.
_regret_matching now takes fallback_mode and concentrates policy mass on
the lowest-index tied action when "argmax_tiebreak". Tiebreak is
deterministic; the Cython recursive traverser randomises ties using its
per-traverser RNG, which the batched policy does not have. Behaviour
matches the spirit of the recursive path (concentrate on best, do not
dilute uniformly).
Plumbed through BatchedPolicy, InterleavedTraversalConfig,
run_interleaved_traversal_batch, trainer.py, workers.py, and the
analyze_first_open_targets.py caller. Two unit tests added.
Also bumps default.yaml outcome_sampling_epsilon 0.2 -> 0.05. The
200-iter sweep in docs/plans/deep-cfr-selectivity.md section 1 showed
0.05 produced the best short-run safe_heuristic_strict score diff
(-40.01 vs -57.87 for 0.20). Recent experiments already used 0.05; the
default now matches actual experimental practice.
Neither change targets the diagnosed selection-bias bottleneck. They
align config intent with scheduler behaviour and make the default config
reproduce known-best knob settings out of the box.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ColorSharedNetwork previously sliced the input vector into n_colors equal
chunks (input_dim // n_colors). The slice boundaries do not align with the
actual encoding layout: adjacent slices contain phase flags, hand slots,
expedition state, scores, etc. mixed together. The "color-shared" encoder
was therefore sharing weights across semantically unrelated chunks, not
across per-color blocks. The single archived run that exercised this path
(2026-05-07_092137_color_shared_attention_1000iter) was killed at iter 41
and produced no eval data, so we have no measurement of whether a real
per-color architecture would help.
Adds compute_lost_cities_color_layout(input_dim) which returns explicit
per-color and common index lists for the standard Lost Cities encoding
(n_colors=5, hand_size=8, n_ranks=9). It recognises input_dim values
171, 219, 249, 297 across derived_playability and slot_aware_playability
flag combinations.
Per-color block (39 dims when derived_playability is on): both players'
expedition state for that color, discard top, public-histogram row,
pending-discard one-hot bit, legal-action draw-pile bit, and the
derived_playability per-color block. Slot-aware features are slot-major
and stay in common.
ColorSharedNetwork.forward now indexes per-color blocks via the layout
when input_dim matches a known schema. For other dims (unit tests,
non-Lost Cities use), it falls back to chunked slicing with a UserWarning
- preserves backward compatibility for tests but makes the legacy
behaviour visible.
No fair test of the new architecture was run as part of this commit.
Documented in docs/plans/deep-cfr-selectivity.md section 7.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Audit of the Deep CFR information-state encoding identified two tiers of
non-pure features and removed both:
- Tier 3 (judgment): is_bad_open_candidate, open_risk_score,
is_safe_continuation. Same heuristic family used to label bad_open in
evaluation, embedded as model input.
- Tier 2 (projection): recoverable_score_no_bonus,
recoverable_margin_no_bonus, min_needed_to_break_even,
cards_needed_for_bonus, has_bonus_path. Mechanical but assumption-laden
("commit and play all currently-playable cards"). The no_bonus form is
asymmetric: it amplifies the immediate -20 penalty while truncating the
+20 bonus upside, biasing the model toward the same "don't open" basin
the diagnostics already flagged.
Input dim 365 -> 297. DERIVED_PLAYABILITY_PER_COLOR 19 -> 15;
SLOT_AWARE_PLAYABILITY_PER_SLOT 12 -> 6. Test shape assertions updated.
Also adds selectivity diagnostic infrastructure used to reach this point:
- traversal.outcome_unsampled_first_open_prior_alpha config field with
signed-prior overlay on unsampled first-open advantage targets (A1).
- analyze_first_open_counterfactual.py --post-policy to swap the
policy_player rollout policy and isolate selection bias (D1).
- analyze_first_open_followup.py to inspect post-forced-open behavior
(E2): same-color play vs discard counts, other-open rate, terminal
hand composition.
Findings recorded in docs/plans/deep-cfr-selectivity.md sections 3-6.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second survey processed the remaining 12 archives via gemini after
the first batch of 3 was accepted. 12 drafts, 0 skips, 0 errors.
Every draft carries a deterministic Last-verified header
(2026-05-08, commit 5c221fb) thanks to the post-processing fix
landed in the previous commit. All 12 accepted into docs/research/
verbatim:
deep-cfr-evaluation-profile-plan
deep-cfr-legacy-experiment-reproduction
deep-cfr-legacy-runtime-comparison
deep-cfr-performance-experiments
deep-cfr-profile-advantage-memory-split
deep-cfr-profile
deep-cfr-regret-fallback-audit
deep-cfr-v0-gap-vs-coolrl
deep-cfr-v0-plan
fast-engine-next-optimizations
post-a-optimization-calculus
test-coverage-notes
docs/archive/ is now fully covered: every entry either has a
research counterpart by stem or by tail-match.
Also extracts _dispatch_one and adds --parallel N to
scripts/librarian_survey.py. ThreadPoolExecutor over the per-archive
work is safe because subprocess.run is network-bound (no GIL fight)
and each thread writes to its own output filename. Default stays
1 (sequential); --parallel 4 is the recommended speedup for large
surveys. The two surveys above ran sequentially; future runs can
opt in.
Plan declares librarian closed for new feature work. MEMORY drift
fixup and duplicate-merge modes stay deferred until a real input
surfaces. Stage 1 (5 deterministic checks) and Stage 2 (promote +
survey) remain operational.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Spot-check of the three drafts gemini produced in the --max 3
survey smoke test: all cited file paths exist, line numbers and
function names land within 1-2 lines of actual symbols
(game.pyx:217 cdef class GameState, evaluate.py:220 batched-entropy
block, trainer.py:892 _evaluate_parallel, action_distribution at
evaluate.py:238 with cited code at line 250 inside it). Numbers
cross-checked against archives match. Conclusions preserved.
The one systemic weakness was the Last-verified commit field:
gemini left a `<short-hash>` placeholder, a literal `HEAD`, or
omitted the commit entirely depending on the call. Fixed in two
places:
1. Manually patched the three drafts before acceptance and copied
them into docs/research/.
2. Added _current_commit_sha and _post_process_draft helpers to
both librarian_survey.py and librarian_promote.py. The drafts
now go through `**Last verified:**` line normalization that
substitutes today's date and `git rev-parse --short HEAD`
before being written to disk. Future runs converge
deterministically.
Net: docs/research/ gains classic-port-notes.md,
deep-cfr-batched-evaluation.md, and deep-cfr-evaluation-profile.md.
12 archive entries remain unprocessed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/librarian_survey.py walks docs/archive/*.md and dispatches
every entry without a research counterpart through the same prompt
assembly as librarian_promote.py. Outputs land under
runs/tmp/librarian-survey-<timestamp>/, classified into:
<stem>.md draft, ready to copy into docs/research/
<stem>.SKIP.txt LLM's one-line "not promotable" reason
<stem>.ERROR.txt CLI stderr if the call itself failed
Counterpart detection uses exact stem match plus a tail-match
heuristic so research notes that intentionally drop a domain prefix
still suppress their archive. Verified against the current tree:
docs/archive/deep-cfr-opponent-policy-network-divergence-* is
correctly recognized as already covered by
docs/research/opponent-policy-network-divergence.md.
--dry-run lists candidates and suggested research targets without
calling the LLM. --max N caps processed archives per run, useful as
a cost guard. Sequential dispatch; one LLM call per archive.
Plan updated to mark survey mode complete.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/librarian_promote.py gains a --accept flag that copies the
generated draft to the suggested docs/research/ target in the same
invocation. This is explicit per-invocation opt-in, not auto-apply:
the operator types --accept knowingly, the cp is still a deliberate
acceptance decision, just expressed in one command instead of two.
docs/research/option-a-bench-result.md is the first promoted note,
generated by gemini from
docs/archive/option-a-bench-result-2026-05-07.md and accepted
verbatim. Spot-check verified that cited file:line locations match
current source (traversal.pyx:473-475 and
inference_server.py:226-228 carry the cited code), the Last-verified
header reflects today's date and HEAD, and the durable conclusion
(sync-blocking policy boundary as the structural ceiling, not IPC
plumbing) is preserved.
This closes the end-to-end loop the librarian was designed for:
oversize check surfaced docs/performance.md, the routing pass split
durable analysis into the archive entry, the promote dispatcher
turned the archive entry into a research draft, and the human
accept step landed it as a tracked research note. Took one LLM call.
Removes the now-resolved ignore-list entry for
docs/research/option-a-bench-result.md (the file exists; the
forward reference is real).
scripts/librarian.sh exits 0 against the working tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/librarian_promote.py is the first Stage 2 piece: a
vendor-agnostic LLM dispatcher that drafts a docs/research/ note
from a given docs/archive/ entry. It assembles the prompt by
stitching scripts/librarian-prompt.md (system) onto the archive
body with a "draft a research note per the rules above" task
instruction, then shells out to the CLI selected by LIBRARIAN_LLM
({claude|codex|gemini}; default claude). The LLM's stdout is
captured to runs/tmp/librarian-promote-<timestamp>-draft.md for
human review — the script never writes into docs/research/ itself.
Refusal cases:
- path not under docs/archive/
- target docs/research/<stem>.md already exists (after stripping any
-YYYY-MM-DD suffix)
- archive missing
If the LLM judges the archive non-promotable, it is instructed to
return a single SKIP: <reason> line instead of fabricating a draft.
--show-prompt prints the assembled prompt without invoking the LLM,
useful for inspecting what would be sent.
Plan updated: Stage 2 v1 marked complete; remaining Stage 2 work
(MEMORY drift fixup, duplicate-merge, survey mode) catalogued.
Next concrete step is a smoke test against one real archive entry.
Adds one ignore-list entry for docs/research/option-a-bench-result.md
which appears in the plan as a hypothetical accept target.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs/performance.md grew to 914 lines because dated experiments and
design analyses kept getting appended instead of routed to
docs/archive/ and docs/research/ as AGENTS.md prescribes. The
oversize check from librarian Stage 1 surfaced the file; this commit
acts on that finding by extracting the parts that belong elsewhere
and trimming the source to a focused current-state reference.
Extracts (verbatim from the original prose, with cross-link headers
and a brief routing note added at top):
- docs/archive/deep-cfr-performance-experiments-2026-05-07.md
bundles torch.compile (regression), AMP (regression), GPU forward
profiling (decision support), and Option B interleaved traversal
(pass) — same date, same theme.
- docs/research/batched-traversal-inference-decision.md captures the
durable A vs B vs C rationale with a closing "Outcome" pointer to
the post-bench archive doc.
- docs/archive/post-a-optimization-calculus-2026-05-07.md preserves
the forward-looking sequencing recorded pre-bench.
- docs/archive/option-a-bench-result-2026-05-07.md preserves the
regression diagnosis and re-enable criteria.
docs/performance.md is now 345 lines, holds sections 1–9 (current
runtime / bottleneck / device / AMP status / batching / eval /
TensorRT / priorities), and ends with a "See Also" linking the four
extracts.
Also reworded the AGENTS.md soft-cap rule from a bare "~500-line
soft cap" to clarify the intent: the cap is a *routing trigger* (is
content piling up that should live in archive/research?), not a
split mandate. Reduces the risk of future agents shredding a useful
doc just to satisfy a number.
scripts/librarian.sh now exits 0 against the working tree.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the last two deterministic Stage 1 checks and wires them into
scripts/librarian.sh:
- librarian_check_stale_plans.py: per docs/plans/*.md, uses
`git log -1 --format=%cs` to get the last commit date and flags
plans untouched in 60+ days. Plans should be either active or
archived; long silence is drift.
- librarian_check_memory_drift.py: validates the user-memory dir
(~/.claude/projects/<slug>/memory). Each MEMORY.md index line
must point at a real file with frontmatter (name, description,
type ∈ {user, feedback, project, reference}); no orphaned memory
files. Memory dir is derived from repo root for portability.
Both run clean against current state.
Stage 1 declared complete. Promotable-archive and duplicate-prose
detection are moved to Stage 2 in the plan because both need LLM
judgment to avoid false positives — not pattern matching.
Next concrete step is the open Stage 1 finding from yesterday:
docs/performance.md at 914 lines, due for a split into sub-topic
notes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Third Stage 1 piece: scripts/librarian_check_oversize.py walks
non-archive markdown files and flags any over the 500-line soft cap
declared in AGENTS.md. Wired into scripts/librarian.sh.
Caught one real finding on first run: docs/performance.md at 914
lines. Splitting it into sub-topic notes is a separate cleanup task
— surfaced for the user, not auto-applied.
Updates docs/plans/librarian.md Progress + sets the next concrete
step to a stale-plan checker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documents two coordination mechanisms agents need to know about on
every turn:
- Compute Lock: train and speed-benchmark commands grab a shared
.compute.lock so multiple agents on one machine don't trample each
other's GPU/CPU runs. Eval and analyze stay lock-free.
- Librarian: scripts/librarian.sh runs the Stage 1 doc lints (lychee
link integrity + file:line citation parity) and should run before
committing doc changes. Full design in docs/plans/librarian.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single entry point that runs every Stage 1 check in order and
aggregates exit codes — both checks always run so users see all
findings in one pass. Two checks wired up today (lychee link
integrity, file:line citations), more land incrementally as Stage 1
grows.
Removes scripts/librarian.sh from the ignore list now that the file
exists. Updates docs/plans/librarian.md Progress + Next Step.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/librarian_check_citations.py extracts file:line references from
inline-code spans across docs/**/*.md and verifies each path exists
(and, if cited with a line number, is within range). Skips
docs/archive/ and docs/plans/archive/ which are read-only by policy.
Caught one real drift in docs/research/optimization_sequencing.md: the
note pointed at docs/plans/amp_trainer.md, which had moved into
docs/plans/archive/.
scripts/librarian-ignore.txt holds fnmatch globs for citations that are
intentionally future-tense (planned files described in the plan docs
themselves). Used sparingly so the checker stays useful as a drift
signal.
Updates docs/plans/librarian.md Progress + Next Step. Next concrete
step is a thin scripts/librarian.sh orchestrator over both checkers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the full librarian design (two-layer architecture, three-stage
pipeline, vendor-agnostic via LIBRARIAN_LLM env, propose-only / no
auto-apply) in docs/plans/librarian.md. Lands the first concrete Stage 1
piece: scripts/librarian_check_links.py, a lychee --offline wrapper
ported from ~/dev/coolrl/src/coolrl/dev/check_doc_links.py.
Also moves the librarian prompt from .claude/agents/ (Claude Code only)
to scripts/librarian-prompt.md so any CLI can load it as a system
prompt later. Fixes one stale README link the new checker caught:
docs/classic-port-notes.md → docs/archive/classic-port-notes.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a "Where to write what" table and 5 writing rules so every agent
sees the same doc-placement policy at the top of each session, instead
of the rules living only inside the librarian subagent prompt.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add an experiment-only explicit-continuation traversal prototype that batches policy requests, records CPU/CUDA parity results, and updates the Option B plan with Phase 0/1 evidence.
Co-Authored-By: Codex <codex@openai.com>
Archive implemented AMP, Option A inference-server, and Cython heuristic plans. Add the active Option B interleaved traversal plan and update model-size/torch.compile plans to reflect the current traversal scheduling conclusion.
Co-Authored-By: Codex <codex@openai.com>
Add a microbench that separates game/encoding overhead, single-request policy boundary overhead, and batched PyTorch forward lower bounds. Record CPU/CUDA results and link the finding from the Julia port evaluation.
Co-Authored-By: Codex <codex@openai.com>
진단 가설/개입/측정 섹션은 docs/research/lost_cities_selectivity.md
에 verbatim으로 보존되어 있어 ideas.md 쪽 사본은 stale 위험 + 중복
유지 비용만 남는다. ideas.md는 brainstorm 인덱스로 축소하고, research/
의 세 thread를 단일 진입점으로 정리.
기존의 "All-negative fallback 가설 — 검증됨, 부분 풀림" 같은 stale
표현도 함께 사라진다 (research doc에선 여전히 open hypothesis로
다뤄지고 있음).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the dependency graph between pending levers (model size,
Option B, AMP, compile, TensorRT, Option A re-enable, Julia port) and
the rule that infrastructure optimization precedes the model-size
experiment because every future training run benefits from the
infrastructure speedup, not just the one keystone experiment.
ideas.md gets a third Active Research Threads pointer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Decision-in-advance for criteria 3 (multi-thread scaling), 4 (Flux/CUDA
MLP forward), and 5 (real-game-state slice). Each criterion lists PASS
/ PARTIAL / FAIL bands with concrete numerical thresholds, plus a
decision rule that maps {3, 4, 5} outcomes to a single action: port,
hybrid (Julia traversal + PyTorch networks), or stay on Python/Cython
and pursue Option B instead.
Cost-of-being-wrong asymmetry stated explicitly: port is months,
staying is zero work, so the GO bar is deliberately above 50% and the
STAY bar is permissive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
experiments/julia_cfr_toy/ ports a synthetic CFR external-sampling
traversal to both Julia and Cython for direct head-to-head
measurement of the actual project hot-path pattern (recursive
tree + mutable regret state + branch-heavy legal-action logic).
Headline result (2026-05-07, single run): Julia ~1.9× faster than
Cython on this pattern, 0 MB allocation, 0% GC time. Root regret
parity ε ≤ 1e-9. The GC-pause concern that was the main argument
against Julia adoption did not materialize. Cython's 21.3 MB
allocation suggests its implementation can be tightened, so the
honest gap window is roughly 1.3×–1.9×.
Multi-thread scaling (bench_cfr_threaded.jl): 2.44× wall-clock at 8T
but only 31% efficiency — inconclusive, likely a toy-size artifact
(per-thread workload too small to amortize dispatch). A heavier
per-thread workload sweep is the remaining decisive test.
docs/research/julia_port_evaluation.md captures this evidence
alongside the earlier safe-heuristic single-thread parity result
and lists the remaining decision criteria (multi-thread scaling with
heavier workload, Flux.jl+CUDA.jl coverage, real-game-state slice).
Do not commit to porting until multi-thread scaling is conclusively
settled.
ideas.md gets a second Active Research Threads pointer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mechanism-level analysis of the slot_aware_playability iter 240
plateau (opened_colors 4.95+, bad_open_rate 88-92%, calibration
gap 6-9 → 2-4). Captures four expert consultations with diagnostic
hypotheses, intervention catalog (architectural / training-dynamics
/ game-specific), measurement plan, and a comparison table across
the four sources.
Key new directions surfaced:
- Current vs average vs league policy separation (Deep CFR average
strategy is the convergence target, not advantage current).
- All-negative fallback as Deep CFR ablation lever.
- Empirical r̃ partitioning by action class.
- Tabular Lost Cities oracle as a clean test of "is 5-color the
game-theoretic answer or an approximation artifact".
- Entry-gate target defined from traversal counterfactual values
instead of heuristic labels.
ideas.md gets an "Active Research Threads" pointer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OpenSpiel's Deep CFR records strategy samples at opponent nodes during
the traverser's tree walk; storing on traverser nodes under external
sampling drops the ρ_p reach factor and biases the average-policy
estimate. Reject that config combination at load time and add a research
note deriving why outcome sampling is unaffected while external is not.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five notes covering outcome-sampling target correctness, package
architecture, v0 feature-parity vs legacy, opponent-policy network
divergence, and regret-matching fallback audit. Four are derived from
archive sources (cited via Source: lines); outcome-sampling-target is
a fresh write-up and serves as the style template.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex and other coding subagents have been creating branches like
experiments/foo and feature/bar without being asked. This fragments
review, hides work from the user, and requires manual cleanup. The
project intentionally develops on main with frequent small commits.
AGENTS.md adds an explicit "Git Branching Policy" section: no
checkout -b, switch -c, branch <name>, or PR-from-new-branch unless
the user asks for it in the current task. Includes a pass-through
clause so this propagates to subagents the main agent spawns.
CLAUDE.md adds a one-line pointer with the same pass-through note,
since CLAUDE.md mandates AGENTS.md is read at session start.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds torch.autocast(fp16) + GradScaler around _train_advantage and
_train_strategy when run.use_amp=true and device=cuda. CPU/non-CUDA
falls back to fp32 no-op. Mitigations:
- scaler.unscale_(optimizer) before grad_clip.
- nonfinite-loss guard skips overflowing batches and counts them.
- diff.float().square() in advantage loss to avoid fp16 overflow.
- strategy mask/log_softmax kept in fp32.
New metrics: amp/grad_scale, amp/nonfinite_loss_count.
Tests: AMP CUDA smoke + CPU fallback in test_deep_cfr_trainer.py.
Bench: scripts/bench_amp_trainer.py micro-benches train phases under
synthetic replay memory. smoke.yaml result is fp32 3.22ms / AMP 3.92ms
(0.82×, regression). 100-iter A/B on default.yaml deliberately
skipped: smoke regression mirrors the 2026-05-07 torch.compile
regression dynamic (dispatch overhead > kernel benefit at this model
size) and re-confirming on the same size adds no information.
Default stays run.use_amp: false. Re-enable trigger documented in
docs/performance.md: hidden_size >= 1024 or num_layers >= 6, then run
the bench script + 100-iter A/B before flipping default.
The experiment finds a model size where (a) learning-curve gains
justify compute, and (b) forward time is large enough to amortize
AMP/compile/TensorRT overhead. Outcome gates re-enabling those four
deferred optimizations and Option A.
Tested grid: hidden={512,768,1024,1536} × layers={3,4,6,8} subset.
200 iterations per config on home (RTX 3090), single seed initially,
second seed for boundary configs. Eval cadence held at default.
Decision tree included for: success → recommend new default and
trigger downstream plans; null → document and stay; expensive-but-
better → defer until AMP/compile/TRT land.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
performance.md additions:
- Batched Traversal Inference design decision (A vs B vs C with
rationale).
- Option A bench result and structural ceiling (realized batch ~7.2,
IPC overhead exceeds GPU gain at small model size).
- Post-A optimization calculus: why compile/TensorRT remain
iter-neutral today and become meaningful only after model growth
and/or denser eval. Sequencing matters; do not retest these on the
current small model.
- Free-threaded Python (3.13t/3.14t) note: cleanest endpoint in
principle, but PyTorch maturity + Cython nogil audit cost block
near-term adoption.
docs/plans/ (4 plan documents for Codex execution):
- batched_traversal_inference_server.md (executed; deferred).
- amp_trainer.md.
- torch_compile.md.
- cython_safe_heuristic_bots.md (executed; first-pass landed).
docs/reports/ (3 cost reports):
- cost_pytorch_free_threaded_2026-05-07.md: WAIT 3-6 months;
PyTorch wheels exist but our Cython is the gating cost.
- cost_cython_nogil_audit_2026-05-07.md: medium effort, traversal.pyx
carries 90% of blockers; Steps 1-3 (cfr_math/encoding nogil
keywords, TraversalStats cdef class) are safe and cheap, Steps
4-6 wait for triggers.
- cost_pytorch_cuda_multithread_2026-05-07.md: risky;
optimizer.step / load_state_dict race silently with concurrent
forward; per-thread default streams unset means naive threading
serializes on default stream anyway.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements the central inference server pattern: a dedicated GPU
process owns advantage/strategy/league networks, batches policy
requests across traversal workers via shared-memory tensor pool, and
returns logits. Workers route forward calls through InferenceClient /
NetworkProxy when traversal.inference_backend == "server".
Default remains traversal.inference_backend: local. The server
backend regresses iter time ~3.8× on the inspected default config
(small-model dispatch + sync-blocking traversal capping realized
batch at ~num_workers=8 instead of the bs=64-256 needed to amortize
IPC overhead). Keeping the implementation behind the flag lets us
re-enable when (a) model size grows, (b) per-worker interleaved
traversal lands, or (c) eval becomes dominant — see
docs/performance.md "Option A Bench Result and Structural Ceiling"
for the full diagnosis.
Plumbing included:
- inference_buffers.py: shared-memory tensor pool with slot
management.
- inference_client.py: per-worker client + NetworkProxy adapter for
the existing traversal.pyx call sites.
- inference_server.py: spawn-context server process with
batch-window aggregation, weight sync, shutdown sentinel.
- bench_inference_backend.py: A/B between local and server backends
with eval/checkpoint disabled.
- test_inference_server.py: round-trip and integration tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cython implementation in heuristic_cy.pyx achieves ~2.55× speedup on
opponent_act_seconds (200-game eval: 59.20s → 23.24s). Original Python
implementation preserved verbatim in heuristic_py.py as the equivalence
reference. Action-sequence equivalence is verified by
test_safe_heuristic_equivalence.py against seeded game corpora.
Key implementation notes:
- File-local wraparound=True override required for negative discard
indexing; Cython global wraparound=False would segfault.
- annotation_typing=False preserves verbatim Python semantics.
- _CachedState materializes hands/expeditions/discards/deck once per
act() call — this is the dominant performance win.
Further C-array optimization of _card_value_for_me /
_card_value_for_opponent / _color_commitment / _bonus_potential is
deferred. The current 2.55× delivers most of the dense-eval future
benefit; further work is gated on actually adopting denser eval
schedules (eval_every=5, games=1000).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Measured DeepCFRMLP forward at bs={1,4,16,64,256,1024} on RTX 3090.
Per-state cost drops 232× from bs=1 (80 µs) to bs=256 (0.34 µs) while
per-call latency stays near 90 µs through bs=256. Policy-call supply
from a real run is ~368 states per traversal and ~200k per iteration,
well above the bs=64–256 plateau, so batched inference is not
supply-limited. GPU forward is not the limiter once batching exists.
Verdict: Optimization Priorities #5 (batched traversal inference) is
worth pursuing. End-to-end gain will still be bounded by encoding and
worker-GPU coordination overhead.
- scripts/profile_gpu_forward.py: standalone profiling script
- docs/performance.md: new "GPU forward profiling for batched traversal"
experiment section with table, supply estimate, and verdict
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Make the link from the failed torch.compile experiment to Optimization
Priorities #5 explicit, so a future revisit happens at the right time
(once compile is on the dominant phase, not just trainer optimization
steps).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
coolguy
