coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Compare commits

base: coolguy/coorl-lost-cities:experimental/inference-server-prototype
Branches Tags
coolguy/coorl-lost-cities:autonomous/trap-exploration coolguy/coorl-lost-cities:main coolguy/coorl-lost-cities:experimental/inference-server-prototype
...
compare: coolguy/coorl-lost-cities:0c83824243594c50093e2fac970528e60d5347cb
Branches Tags
coolguy/coorl-lost-cities:main coolguy/coorl-lost-cities:autonomous/trap-exploration coolguy/coorl-lost-cities:experimental/inference-server-prototype
55 Commits
experimental/inference-server-prototype ... 0c83824243
Author SHA1 Message Date
coolguyandClaude Opus 4.8 0c83824243 Correct the exploiter budget: it was a quarter of the target, not half 
The exploiters ran 32.5M learner actions, not the 65.5M I recorded -- they train
one seat, so a 250x1024 run yields half of what the same shape gives the
both-seat self-play trainer. Against targets trained on 131M and 122.6M, that
makes the attacker roughly 4x underfunded.

Which means the absolute number does not support "ours is only 22.8%
exploitable". It supports exactly one claim: at a matched budget, league gives up
more. Whether the ordering survives a properly funded attacker is now the open
question, so the script takes --updates and --batch-games to run it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 05:50:58 +09:00
coolguyandClaude Opus 4.8 e51d14d0fb Measure exploitability: ours is harder to farm than league 
Winning the head-to-head says a policy is strong on average, not that it is hard
to beat. So freeze each policy, train a fresh one from scratch whose only job is
to beat that policy, and see how far it gets.

Same exploiter budget (250 updates x batch 1024, 65.5M learner actions):

  ours (match stack, 131M)   exploiter reaches 0.2278 [0.219, 0.237]
  league (web-deployed)      exploiter reaches 0.3213 [0.311, 0.331]

League gives up 9.4 more points to a dedicated attacker. Both sit far from 0.5, so
neither is a pushover -- but the caveat left open by the head-to-head is now closed
in our favour on both axes: stronger on average (0.6094) and harder to exploit.

Worth noting against expectation: league was trained *with* an exploiter structure
and we ran pure self-play, and we still come out less exploitable. Whatever the
league machinery buys, it did not buy that here.

The number is a lower bound -- a bigger exploiter would find more -- so it only
means anything as a like-for-like comparison, which is how it is used.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 05:37:55 +09:00
coolguyandClaude Opus 4.8 cc12b058e5 Re-measure both-seat vs privileged critic at the same budget 
The 39.3M ablation was the critic's worst ground. At a matched 131M budget:
both-seat training is worth +17.1 points of win rate, the privileged critic
+3.7 -- a 4.6x gap. And they behave differently: both-seat is flat across scale
(+16.8 at 39.3M, +17.1 at 131M) while the critic swings (-1.6 to +3.7).

The mechanism explains it. Both-seat doubles the data at the same compute --
self-play already played those plies with the same network. The critic adds no
data, only accuracy, and a privileged value trunk has more to fit, so it grades
badly until there is enough to fit it with.

Fable ranked the critic first among missing ideas and both-seat fourth. That was
exactly backwards.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 05:26:38 +09:00
coolguyandClaude Opus 4.8 4d1c4473b6 Restore the privileged critic: its ablation flips with scale 
The 39.3M ablation said the privileged critic hurt (switching it off won 0.5160
[0.505, 0.527]). Head to head at 131M, both sides trained identically, it says the
opposite: off *loses*, 0.4633 [0.453, 0.474]. Against league the critic-on model
scores 0.6094 and the critic-off one 0.5526.

The critic earns its keep once there is enough data to fit it -- at 39.3M the
privileged value trunk is underfit and only adds advantage noise. Defaulted back
on, with the small-scale number kept in the docstring as a warning: an ablation at
a budget you do not intend to ship can invert.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 04:01:40 +09:00
coolguyandClaude Opus 4.8 1f51fb8149 Ablate the match stack: both-seat training carries it, the privileged critic hurts 
Each piece switched off in turn, trained at identical compute, then played against
the full stack over 8192 duplicate matches. Below 0.5 means the removed piece was
doing work.

- both seats: 0.3317 [0.322, 0.342]. The biggest single contributor. Half of it is
  simply sample count -- dropping the opponent seat halves the learner actions per
  update -- but that is the point: self-play already produced those plies with the
  same network, and the old trainer stop_gradiented them away.
- match observation: 0.4751 [0.464, 0.486]. Small but real. Since carry itself
  contributes almost nothing (rounds decompose), most of this is likely the
  single-round observation defects being fixed: to_move, the deck clock, and the
  score_diff scale.
- privileged critic: 0.5160 [0.505, 0.527] -- turning it OFF makes the agent
  significantly STRONGER. Fable called this the biggest missing idea; it is wrong.
  A critic that knows the deck fits V(full state), which is not
  E[return | masked obs], so the advantage picks up a component the actor cannot
  act on. From the actor's side that is noise, not variance reduction. Asymmetric
  critics hurting under partial observability is a known failure mode.

Defaulted off accordingly. (Reusing it as a PIMC leaf evaluator may still stand --
that is a separate claim from using it to train the policy.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 03:49:24 +09:00
coolguyandClaude Opus 4.8 4392ec255a Record the old-vs-new head to head in the real three-round game 
The question was what any of this actually improved over the training method
that existed. Duplicate matches, 8192 a piece, same three deals and coins from
both seats:

- At 39.3M learner actions the match stack beats the Phase 0a gate agent
  (0.5842) which had 411M -- 10.5x the data. Sample efficiency is the headline.
- At 39.3M it *loses* to the league policy (0.3142). That is a budget gap, not a
  strength gap: league had 122.6M plus a league/exploiter structure.
- Scaled to a matched budget (131M vs league's 122.6M) it wins: 0.6094
  (CI 0.599-0.620), +22.0 points.

So: same compute, stronger agent, measured on the actual game.

Caveat kept honest in the plan -- league was trained with exploiters, and we have
measured average strength, not exploitability. "Harder to exploit" is not shown.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 03:30:23 +09:00
coolguyandClaude Opus 4.8 8860f62030 Pay the match total densely and drop the tanh terminal reward 
The user asked why we were not simply maximising the three-round total, and they
were right. Each ply now pays the points by which it moved the running match
difference; at gamma=1 that telescopes to the final total, so the objective is
exactly the rulebook's, handed out every ply instead of once 150 plies later.

Three measurements killed the tanh design:

- Rounds are independent (corr(m1,m2)=0.004, corr(m1+m2,m3)=0.05), so a reward
  linear in the total decomposes the match into three independent rounds and
  carry enters the objective nowhere. The only coupling, the start-player rule,
  is worth +0.73 +/- 0.84 points -- indistinguishable from zero.
- Risk attitude, the one thing tanh buys, is worthless here. A policy made to
  gamble when it trails by 20 entering round three *loses* to a greedy clone over
  6144 duplicate matches (0.482); gambling only at -40 breaks even (0.498). A
  marginal wager buys about +1.7 sigma for -2 to -3 expected points. Ceiling on
  the whole carry-conditioning idea: under one win-rate point.
- Head to head over 10,000 duplicate matches at equal compute, the linear reward
  *beats* tanh(total/12): 0.5859 (CI 0.576-0.596), +20.3 points. Dropping it is
  not merely free, it is better -- not because of risk, but because tanh hands a
  ~150-ply match one saturated +/-1 and leaves all credit assignment to the critic.

The flat carry probe was not exploration collapse: sampled play still opens 5.00
expeditions, entropy settles at 1.36 nats (3.9 effective actions), and the critic
reads carry cleanly (round-three values run -0.87 to +0.86, monotone). The signal
was there; there was nothing to buy with it.

Criterion 1 (a monotone carry response) comes off the gate accordingly -- the
optimal response barely exists in this game. carry stays in the observation: it
costs nothing and the start-player rule keys off it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 03:07:18 +09:00
coolguyandClaude Opus 4.8 f170fcdcfd Run the classic self-play trainer and record what the carry probe says 
Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck-
race rate, so the stalling that static opponents induced is gone. Duplicate match
eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both
seats, deal luck cancelling exactly.

Success criterion 1 does not pass. The carry probe is close to flat: expeditions
opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use
does move monotonically across all six carry levels, and in the right direction
(behind -> more multipliers), but the spread is 0.31 wagers.

Two diagnoses, one of which was mine and wrong:

- Residual potential shaping was NOT the cause. Annealing it fully to zero left
  the probe just as flat.
- terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over
  any realistic round margin, and maximising E[tanh] on a linear stretch is just
  maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears
  where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12
  widens the wager spread 0.19 -> 0.31, which is the mechanism showing up.

The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is
a saturated dead zone with no gradient and the policy has learned nothing there.
The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to
be set from the scale rather than fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 02:44:04 +09:00
coolguyandClaude Opus 4.8 9ba5a07a9d Train on matches: match observation, asymmetric critic, both seats, match reward 
Phases 2-4 land together because they all rewrite the same rollout, and doing
them in sequence would mean writing it three times.

Observation adds the four things a match policy cannot play without: carry as a
scalar *and* a binned one-hot (round three is a threshold problem -- "win by 41
or lose" plays nothing like "win by 39" -- and the old score_diff divided by
MAX_ABS_SCORE=780, squashing a decisive 50-point lead to 0.06); the round index;
whose turn it is, which the single-round observation never carried even though
the critic is trained on opponent-turn states; and the deck clock, since a round
ends on the last deck draw and players bend that parity by drawing from discard
piles. Live points per colour are split by hand / discard pile / unseen, because
a discard pile is public and recoverable.

The critic is asymmetric: it gets the opponent's hand and the deck in order, on
a separate trunk so none of it can reach the logits. A test pins that down --
perturbing the privileged input leaves the policy logits bit-identical while
moving the value. Deal luck is what makes a match-terminal reward hard to learn
from, and a state-value baseline may condition on anything action-independent.

Both seats now train. Self-play ran one network on both sides and stop_gradiented
the opponent, throwing away half of every game; each ply now emits a transition
per seat, folded into the batch so each seat keeps an independent GAE chain.

Reward is the match: rounds one and two only bank into carry, and round three
pays tanh(total / terminal_scale). Potential shaping on the running total covers
the early sparsity and anneals out.

match_eval adds the two measurements the plan turns on: duplicate match play
(same deals and coins from both seats -- self-play scores exactly 0.500 with zero
mean lead, so the mirroring cancels deal luck exactly) and the carry probe. On an
untrained net the probe is flat: 4.97 expeditions opened at a 60-point deficit
and at a 60-point lead alike. Breaking that flat line is success criterion 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 02:23:50 +09:00
coolguyandClaude Opus 4.8 62b755c3f0 Retract the 120-ply round cap and record the missing wiring step 
The cap item existed so a three-round match would fit inside a 400-step scan.
Phase 0a's truncation bootstrap removed that constraint -- scan length is a free
parameter now -- and the same phase showed the cap is the thing that turns
stalled games into losses: 20.6% of cap-hit games lost, 99.4% of all losses. A
120-ply cap would put *more* games into the wall and make "freeze the round
while ahead" easier, so the round cap stays at 400 and max_steps_rate stays a
watched metric.

Also records what the plan never listed: ppo.py, gates.py and league.py all take
a single-round State and still have to be wired to MatchState. That is the
largest remaining piece of Phase 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 02:09:49 +09:00
coolguyandClaude Opus 4.8 7cd299dd1e Add the classic three-round match layer 
Wraps the single-round engine rather than changing it: engine.py is the rules
oracle the TypeScript client is differential-tested against, and the round
itself is unchanged by the match. Only two rules live up here, both from the
Kosmos rulebook -- three rounds decided on the summed total, and "the player who
has more points begins" the next one. That is not alternating, and it is not
what reset_from_order hardcoded, so it takes a first_player argument now.

The rulebook says nothing about an exact tie, so the starter falls back to a
coin flip. A deterministic tie-break would give one seat a standing edge in
symmetric self-play and the agent would learn to steer for it. Round one needs
no special case: carry is (0, 0) there, so the tie branch already yields the
coin, which is exactly the rulebook's arbitrary "oldest player begins".

All randomness -- three deals and three coins -- is drawn in match_reset and
stored in the state, so match_step stays deterministic and needs no PRNG key
threaded through every rollout, eval, and gate body. It also makes a mirrored
match (same deals, seats swapped, same coins) a pure seat relabel, which the
antithetic pairing later depends on.

Tests cover the deck clock (44 deck draws a round, discard draws extend it),
carry banking each round exactly once, the start-player rule across all three
branches, a fair round-one coin, mirror symmetry, and that the running total
does not jump across a round boundary -- the last one matters because potential
shaping will be built on it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 02:05:50 +09:00
coolguyandClaude Opus 4.8 47d0e803f5 Record the Phase 0a gate result and the MAX_STEPS artifact it exposed 
At equal compute (250 updates either way, 5.8x the samples) the new loop beats
the baseline against heuristic_expert: 0.4525 -> 0.5071 win rate, -4.28 ->
-0.56 mean score diff. Expert ends rounds at their natural length, so that is
a clean read on card play.

It appears to *lose* ground against heuristic_balanced (0.9866 -> 0.9292), but
that is entirely the MAX_STEPS=400 cap. Drawing from a discard pile does not
deplete the deck, so a round can be stalled indefinitely; against a weak
opponent the extra turns are worth points, and with 5.8x the samples the agent
learns the exploit harder (greedy rounds run 225 plies). Splitting 2048 games:
naturally-ended games lose 0.08%, cap-hit games lose 20.62%, and 99.4% of all
losses land in cap-hit games. The board freezes mid-expedition and the -20s
stand.

Self-play does not have this failure mode -- league runs converge to 53.6-ply
rounds and never hit the cap -- which is the regime the 3-round work targets.
Noted in the plan: static heuristic anchors are unusable as gates, and the
per-round cap needs care in Phase 1.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 01:57:03 +09:00
coolguyandClaude Opus 4.8 d1c2cf628e Reset envs inside the rollout scan and fix the metrics it breaks 
rollout_steps was pinned to MAX_STEPS (400) while a round actually runs
~50 plies, and finished envs were only reset between updates. 83% of every
rollout was spent stepping already-done envs to produce masked-out zeros.
Measured at rollout_steps=400: active steps go 17.3% -> 100%, i.e. 5.8x the
learner actions per update for the same compute.

Resetting in-scan exposes three things that were previously benign:

- compute_gae bootstrapped truncated episodes from zero. That was safe only
  because every episode used to terminate inside the scan; now episodes cross
  the boundary, so thread V(s_T) through.
- rollout_metrics read final_env and summed rewards along the scan axis, both
  of which assume one episode per slot. With several episodes per slot that
  silently produces garbage, so aggregate at done boundaries instead.
- league assignments were redrawn only between updates, which would pin a slot
  to one seat/opponent across every episode in a scan. Redraw them on reset.

Also anneal potential shaping against learner actions rather than padded scan
steps: the old accounting counted the dead steps, so a 5M-step anneal expired
within two updates of 250. Any earlier evidence that shaping does not help was
gathered with it effectively off.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
 2026-07-15 01:27:02 +09:00
coolguyandClaude Opus 4.8 8b7348cbce Add hints, per-action undo/redo, and card motion 
The rival's policy now exposes a ranking, which drives both its own move and a
new HINT button: it highlights the card to play, where to put it, and where to
draw, with the model's confidence.

Undo and redo work per action rather than per turn — picking a card, choosing
its destination, and drawing are separate steps, as are the rival's moves — so a
finished game can be stepped back through from the result screen. The rival is
suspended while undone moves are pending, and PLAY FROM HERE resumes from the
reviewed position.

Cards now travel between zones instead of teleporting: a motion layer measures
each card's old and new position and animates the difference, flying cards out
of the deck face-down and flipping them over, and back into it on undo.

Also: the result screen gets a per-expedition score breakdown mirroring
human_play.py, and a rival card revealed by a discard-pile draw no longer
renders at full size in the card-back-sized rival hand row.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
 2026-07-14 21:33:53 +09:00
coolguyandClaude Opus 4.8 be5226bd3b Add seeded deals and card movement animations 
Every game is now dealt from a seed carried in the URL as `?seed=`, shown in
the menu, and re-dealable by typing it in, so a deal can be shared or replayed.
Seeds are hashed into a mulberry32 stream, independent of the Python shuffle
bank.

Cards previously teleported between zones: the only motion in the client was
the hover lift and the legal-target pulse. Cards are keyed by card id, so a
card that just moved into a zone mounts there and now animates in, with the
motion disabled under prefers-reduced-motion.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
 2026-07-14 21:10:14 +09:00
coolguyandClaude Opus 4.8 299c788545 Fix web client interaction and layout bugs 
Placing a card locked the turn in: the chosen card left the hand and no
affordance reverted the placement, forcing the move through. Clicking the
chosen destination again or pressing Escape now steps the selection back.

A failing ONNX inference left the AI's turn unadvanced, permanently
stalling the game. The AI turn now falls back to the heuristic policy.

Other fixes: the score plaque no longer covers hand cards (plaques become
compact chips at narrow widths and hand spacing tracks the viewport),
opponent cards drawn from a discard pile render face up, long expedition
stacks stay inside their lane, undo no longer bumps the generation counter
on empty history, and small viewports scroll instead of clipping.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LmyprzuzanXRhpomc3Ga1i
 2026-07-14 21:04:27 +09:00
coolguy e47db5eb1e Show placed cards before drawing 2026-07-14 20:37:59 +09:00
coolguy 891cc113f0 Align web client with pygame table 2026-07-14 20:28:47 +09:00
coolguy ef4b9d82b0 Focus project on JAX PPO 2026-07-14 20:09:03 +09:00
coolguy 79273f7eb3 Add on-device web client 2026-07-14 19:49:03 +09:00
coolguy 9517fd3ba8 Document JAX PPO model capacity experiment 2026-07-12 15:21:09 +09:00
coolguy f854d97764 Add match replay and JSONL export 2026-07-12 15:14:24 +09:00
coolguy 7e481ca064 Add polished Lost Cities table GUI 2026-07-12 14:58:59 +09:00
coolguy 3b86f81bf6 Add JAX PPO opponent to classic GUI 2026-07-12 04:14:02 +09:00
coolguy 19560029b5 Add final cycle report and human play CLI 2026-07-06 00:14:24 +09:00
coolguy 16cc31676c Add diminishing returns diagnostic report 2026-07-05 18:32:47 +09:00
coolguy fa9a1c5286 Add gates 1-2 audit and repair workflow 2026-07-05 17:55:22 +09:00
coolguy 94e9ac1854 Add JAX PPO league self-play v1 2026-07-05 06:33:22 +09:00
coolguy 6037b650f3 Add JAX PPO ladder v2 expert pass 2026-07-05 02:06:22 +09:00
coolguy 7fe0bfdcfe Add JAX PPO ladder verification pass 2026-07-05 00:17:49 +09:00
coolguy bb52ef9ec1 Document JAX PPO ladder results 2026-07-04 23:19:41 +09:00
coolguy 4c0c2e9add Reset finished JAX PPO environments 2026-07-04 22:33:51 +09:00
coolguy 9e27f42f27 Add JAX PPO static-opponent trainer 2026-07-04 22:30:16 +09:00
coolguy 768f74693d Plan JAX PPO static-opponent ladder 2026-07-04 22:14:37 +09:00
coolguy b37b841eef Record CUDA JAX throughput 2026-07-04 21:56:24 +09:00
coolguy 72893250ec Record full JAX differential verification 2026-07-04 20:39:01 +09:00
coolguy 30ccc3cf41 Batch JAX differential verification 2026-07-04 19:42:05 +09:00
coolguy f872204b13 Document JAX engine and benchmark 2026-07-04 19:38:09 +09:00
coolguy ac54f98189 Add JAX engine verification tests 2026-07-04 19:38:04 +09:00
coolguy 1d7758b7d3 Add JAX Lost Cities rules engine 2026-07-04 19:37:27 +09:00
coolguy 8f380928e6 docs(research): split SO-ISMCTS into its own group in catalog 
ismcts-bc-ceiling-2026-05-11 was filed under "Other"; promote it to a
proper SO-ISMCTS section between Deep CFR and Engine/Performance so the
catalog actually reflects the project's two algorithm families. As more
ISMCTS notes accrue they have an obvious home.
 2026-05-12 00:22:10 +09:00
coolguy 2a8a082a12 Catalog research notes for new agents 
Give new agents a single entrypoint into accumulated research, and point AGENTS.md at that catalog.
 2026-05-11 21:04:44 +09:00
coolguy 44b8faba3d docs(research): SO-ISMCTS BC ceiling write-up from 2026-05-11 autonomous session 
Summarizes the 13-cycle trap-exploration session: BC pretrain (heuristic
clone) is the self-play ceiling under our compute budget (1 GPU + 50
sims + 768x4 MLP). All variants (naive finetune, KL anchor, mirror
descent, mixed-opponent + opponent-aware search) either preserved BC
(~17-21/100 vs heuristic-cautious) or regressed to catastrophic
forgetting. The single largest improvement of the session — 4× win rate
on the same checkpoint — came from PUCT Q-value normalization at search
time, not from any learning change.

Records the mechanism (negative training signal from BC-vs-heuristic
games; search too shallow to find heuristic-beating moves), the
hypotheses we negated, and the dials left in code for future runs with
more compute.
 2026-05-11 20:45:10 +09:00
coolguy cba6caee2f Add mixed-opponent self-play with opponent-aware MCTS 
C13/C14 cycles: heuristic-balanced bot plays a configurable fraction of
self-play games (training.mixed_opponent_fraction). Trainee turns are
stored as policy samples; opponent turns are taken by the bot directly
and not stored. When mcts.opponent_aware_search is set, the MCTS tree
also treats the opponent seat as that bot — opponent moves are applied
without expanding into the search tree, and all values are taken from
the traverser's perspective. This was Codex's top recommendation for
breaking the symmetric self-play weak fixed point.

Empirical: opponent-aware mixed self-play does NOT lift win-rate above
BC pretrain (vs heuristic-cautious 100-game eval):
  C13 (mixed=0.5, no KL):     0/100 — catastrophic forgetting
  C14 (mixed=0.2, KL beta=1): 17/100 — preserved BC, no improvement
  BC pretrain baseline:       21/100

Combined with C10-C12 results, BC remains the ceiling under our
compute budget (1 GPU + 50 sims + 768x4 net). Code is left in place as
configurable dials for future runs with more compute.
 2026-05-11 20:42:13 +09:00
coolguy b9fc5693a4 Normalize PUCT Q + add mirror-descent policy target 
Codex follow-up diagnostics identified two MCTS+training-loop issues that
together cap finetune-from-BC at the heuristic ceiling:

1. PUCT Q is in raw score units (~±100 for value_scale=100), but the
   exploration bonus c_puct * prior * sqrt(N) / (1+n) is on order of 1-10
   for our parameter ranges. Result: a single bad backup pushes q_eff
   well below the bonus floor and that action is effectively never
   visited again. With only 50 sims/move this is catastrophic for the
   policy-improvement operator. Fix: divide q_eff by config.q_scale
   (default 100, configurable) inside _select_action. Backups and value
   targets remain in raw score units; only the selection signal is
   normalized. AlphaZero canonical convention.

2. The current kl_anchor_beta path adds KL(current || ref) directly to
   the loss. That preserves BC but prevents improvement (gradient
   actively pulls policy back to reference). The standard regularized
   policy improvement operator is to mix the target instead:
     pi_target = softmax(alpha * log(pi_mcts) + (1-alpha) * log(pi_ref))
   Anneal alpha from low (rely on BC) to high (rely on MCTS) over
   training. Network learns to follow the regularized target, which
   stays near BC early but lets MCTS-discovered improvements through
   later.

Config additions:
- mcts.q_scale (default 100.0): PUCT Q divisor
- training.md_target_ref_ckpt: reference policy path (alternative to kl_anchor)
- training.md_target_alpha_start / _end / _iters: linear alpha schedule

Both Python mcts.py and Cython mcts.pyx updated; parity test passes.
Tests: 19/19.

Hypothesis: with normalized PUCT the network can actually explore and
exploit prior knowledge competently at 50 sims, and the mirror-descent
target lets self-play improvement happen while BC anchors the trajectory.
This is the operator-side fix that c9 (no anchor, collapsed) and c10/c11
(loss-side KL anchor, preserved-but-stuck) both missed.
 2026-05-11 16:31:02 +09:00
coolguy d850070ed4 Add KL anchor to BC reference policy in trainer 
Self-play drift fix: regularize loss with KL(current || BC_reference).
Config: training.kl_anchor_ckpt + training.kl_anchor_beta. Loaded once
at trainer init, frozen. KL computed over legal actions only.
Hypothesis: appropriate beta keeps pretrained competence during self-play
finetune, escaping the c9 catastrophic forgetting.
 2026-05-11 15:24:14 +09:00
coolguy 9fdfa88b23 Add lost-cities-ismcts pretrain: behavior-clone heuristic into network 2026-05-11 13:36:28 +09:00
coolguy 33c44c708e Add --resume-from for warm-starting training from a checkpoint 
Lets c6+ layer new exploration hyperparams on top of c5's learned
value head instead of restarting from random init. Saves ~60min per
cycle while preserving VPE-down trajectory observed in c5.
 2026-05-11 10:41:08 +09:00
coolguy 0d35341bbe Cycle 5 setup: long run with use_rollout_value=false + 768x4 network 
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score
avg -85 to -97 vs three heuristic opponents. Bigger network alone did
not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without
rollout) but agent still loses every natural-end game.

Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not
break the loss pattern. Score average actually slightly worse than
512x3 baseline. So the bottleneck is not network capacity.

Going to the long-run experiment: AlphaZero-correct setup with the
network value loop closed. use_rollout_value=false means leaf Q comes
from network value head. Training signal: network value learns from
actual game outcomes; MCTS uses those values to pick actions; better
actions produce better outcomes; cycle closes.

100 iter previously gave essentially the same result as rollout=true
(comparing c3 to trapfix baseline). Both are too early in the AlphaZero
training curve. Standard AlphaZero papers train 1000s of iterations.
Going long: 1000 iter with the current config. Self-play ~9s/iter
without rollout, total wall ~150 min for the train phase.

Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval
and generate analyze.py plots to visualize trajectory.

Network kept at 768x4 since bigger capacity does not actively hurt.
 2026-05-11 07:56:35 +09:00
coolguy a501a93223 Cycle 4: revert use_rollout_value=true, scale network 512x3 -> 768x4 
c3 showed use_rollout_value=false alone is not the fix: without the
heuristic rollout safety net the random-init network value gives bad
MCTS Q early on, the agent plays more (PA 0.10 -> 0.22+) but eats more
-20 expedition penalties (score worsened from -57 to -97 avg).

Mathematical intuition: in Lost Cities, opening an expedition is a
20-point commitment. Break-even requires rank-sum × (handshakes+1) >= 20.
The model has to learn:
  - which colors to open (based on hand handshake/high-rank holdings)
  - when to commit vs discard
  - card-ordering constraints (ascending only)

This is a moderately rich value function. 512x3 (~800k params, ~290
input dim) might be undersized. Test capacity hypothesis with 768x4
(~2M params) while keeping the rollout safety net so MCTS Q stays
competent.

Other params from c1 kept: c_puct=5, virtual_loss=5, dirichlet
α=0.3/ε=0.4, parallel_simulations=64, n_simulations=50.
 2026-05-11 07:03:45 +09:00
coolguy 200129d16d Add diagnostic value-head metrics: rmse, target stats 
Codex flagged that mcts/value_prediction_error mathematically reconciles
with loss/value (MSE / value_scale^2 = 0.05) but the latter looks healthy
while the former says the value head is far off. To make this clearer in
W&B, expose:

- mcts/value_rmse — sqrt(MSE), in raw score units (interpretable)
- mcts/v_target_abs_mean — magnitude of |v_target|, indicates if game
  outcomes are very lopsided (always negative for a losing agent)
- mcts/v_target_std — spread, low std means targets are saturated to
  one end (e.g., always -100ish)

These let us see whether value head is failing because targets are
unlearnable variance, or just hard-to-predict, or because of saturation
at the value_scale=100 tanh boundary.

Tests: 19/19 passing.
 2026-05-11 06:43:50 +09:00
coolguy 8b7ed66ffd Cycle 3 prep: fix search() ignoring parallel_simulations + flip use_rollout_value=false 
Codex deep diagnosis surfaced two real issues in our MCTS pipeline:

1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1)
   instead of respecting MctsConfig.parallel_simulations. Standalone evals
   (eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all
   use this entry point, so all eval-time MCTS was running 1 sim per batch
   regardless of the configured 64. Training was unaffected because it
   goes through interleaved_self_play._run_search_jobs which respects the
   config. Fix uses min(config.parallel_simulations, sims - completed).

2. use_rollout_value defaulted to True (config.py) but was never set in
   the YAML. With this, _expand_with_prior returns the heuristic rollout
   value and discards network_value, so the network value head is trained
   from final game scores but its outputs are never fed back into MCTS
   backups. This explains why mcts/value_prediction_error stays high
   despite training -- learning the value head produces no behavioral
   change because MCTS never reads it.

Now setting use_rollout_value=false in default.yaml so the network value
head closes the loop. Combined with the existing Dirichlet root noise +
heuristic rollout removal, this should give the network's value learning
actual leverage on action selection.

Also: updated test_search_visit_counts_match_with_parallel_simulations
to test the correct invariant (legal-action set match + total visit
count near n_sims) rather than literal visit-count equality, which was
only true under the previous bug.

Tests: 19/19 passing.
 2026-05-11 06:42:20 +09:00
coolguy be0c1a8d62 Add training.value_loss_weight (default 1.0) for value loss reweighting 
Diagnosis: mcts/value_prediction_error stuck at 300-1000 (RMSE ~22 on score
range ±100), while loss/value stays at 0.05 because the loss divides
prediction and target by value_scale=100 (so MSE / 10000). Net effect: the
value head receives a tiny gradient relative to the policy head's
~1.75 cross-entropy loss, so it never learns to predict score scale well.

This adds a config knob to multiply the normalized value loss without
re-engineering the loss formula. value_loss_weight=50 recovers the raw
MSE magnitude (~2.5 vs policy loss ~1.75), giving the value head
comparable gradient signal.
 2026-05-11 06:34:05 +09:00
coolguy 169d4dcb14 Cycle 1: c_puct 3->5, dirichlet_eps 0.25->0.4 produced first natural-end wins 
50-iter sweep on default.yaml with stronger MCTS exploration:
- c_puct: 3.0 -> 5.0 (UCB weight, more exploration of low-prior actions)
- root_dirichlet_epsilon: 0.25 -> 0.4 (more noise injected at root prior)

Standalone eval at iter 50 (30 games/opponent, all natural-end, timeouts=0):
- vs heuristic-balanced:    W=0/30 S=-70.5  (PA 0.15)
- vs heuristic-aggressive:  W=2/30 S=-65.1  (PA 0.14)  [+10, +4]
- vs heuristic-cautious:    W=1/30 S=-48.0  (PA 0.14)  [+2]

3 natural-end wins vs prev trapfix baseline iter 44 (which had 0 natural
wins + 1 timeout-tie). Stall trap fixed remains true (timeouts=0 in c1).

Trade-off observed: more exploration -> higher variance. Score avg vs
cautious worsened (-32 -> -48), but win events appeared. For the
non-terminal-win objective, exploration win > score-avg loss.

Next: commit to long run (300 iter) with these params before tuning more.
 2026-05-11 05:53:54 +09:00
coolguyandClaude Opus 4.7 f289997c1c Add Dirichlet root noise + standalone eval CLI, fix self-play stall trap 
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw
from discard pile to extend deck) until max_steps timeout, then squeak by
on opponents' negative scores. All eval wins were from timeouts; agent
never won a naturally-terminating game. Self-play reinforced this because
timeout games still got a positive value target.

Fixes (no algorithm change, all MCTS hyperparameters or signal shaping):

- Dirichlet noise at root prior (AlphaZero standard, was missing):
  mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion
  mixes prior with Dirichlet(α). Callers in interleaved_self_play and
  the internal evaluate_and_backup pass `not item.path`.
- Default config strengthens exploration on the 50-sim batched search:
  c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new
  root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25.
- Self-play timeout signal zeroed: `_finalize_context` sets v_target=0
  if context.state is not terminal. Stops the network from learning
  "stall = positive value".

New standalone evaluator:
- `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a
  checkpoint, runs N games per opponent across a parallel pool, reports
  win/score with 95% CIs plus per-game logging via --verbose. Defaults
  cover heuristic-balanced/aggressive/cautious (rollout policy isn't in
  the training-eval opponent list, so this is the natural way to compare
  the trained policy against its rollout target).

Tests (19) still pass; .so rebuilt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 05:17:32 +09:00
173 changed files with 36206 additions and 561 deletions
Expand all files Collapse all files
.github/workflows/deploy-web.yml
+46
View File
@@ -0,0 +1,46 @@
name: Deploy web client
on:
push:
branches: [main]
paths:
- "web/**"
- ".github/workflows/deploy-web.yml"
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
concurrency:
group: github-pages
cancel-in-progress: true
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
cache: npm
cache-dependency-path: web/package-lock.json
- run: npm ci
working-directory: web
- run: npm run build -- --base=/${{ github.event.repository.name }}/
working-directory: web
- uses: actions/upload-pages-artifact@v3
with:
path: web/dist
deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
.gitignore
+9
View File
@@ -27,3 +27,12 @@ tools/julia/
.pytest_cache
.ruff_cache
# Web dependencies and locally exported models. The verified public browser
# policy below is the one exception: it is deliberately served as a static asset.
web/node_modules/
web/*.tsbuildinfo
web/public/models/*.onnx
web/public/models/*.json
!web/public/models/jax-ppo.onnx
!web/public/models/jax-ppo.json
.gitlab-ci.yml
+18
View File
@@ -0,0 +1,18 @@
stages:
- deploy
deploy-pages:
stage: deploy
image: node:22
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
changes:
- web/**/*
- .gitlab-ci.yml
script:
- cd web
- npm ci
- npm run build -- --base="${CI_PAGES_URL}/"
- mv dist ../public
pages:
publish: public
AGENTS.md
+61 -405
View File
@@ -1,442 +1,98 @@
# AGENTS.md
This repository is managed with `uv`. Use `uv run ...` for commands so the
project environment and Cython extensions are built/loaded consistently.
This repository is managed with `uv`. Use `uv run ...` so the project
environment and Cython extensions are built and loaded consistently.
## Project Layout
## Active project path
- `src/coolrl_lost_cities/games/classic/game.pyx`: Cython Lost Cities engine.
- `src/coolrl_lost_cities/games/classic/deep_cfr/`: Deep CFR training,
traversal, evaluation, analysis, and CLI code.
- `configs/deep_cfr/`: active Deep CFR YAML configs (kebab-case filenames).
Currently holds two:
- `default.yaml`: the canonical "best-known" baseline. Start here, then
override fields via `--set` for experiments/ablations.
- `smoke.yaml`: 1-iter sanity check for the training loop.
- `configs/archive/`: retired/historical configs. Don't modify; reference
if you need to reproduce an old run.
- `runs/`: generated training runs. Gitignored, may be a symlink to larger
storage. Layout:
- `runs/archive/`: past runs. **Do not modify or delete.**
- `runs/tmp/`: smoke, tests, throwaway. Free to `rm -rf` anytime.
- `runs/<YYYY-MM-DD_HHMMSS>_<kebab-name>/`: real experiments (flat).
Promote a `runs/<...>` directory to `runs/archive/` with a manual `mv`
once analysis is complete.
- `docs/`: profiling notes, migration notes, and experiment documentation.
The active product and research path is **JAX + PPO**:
## Core Commands
- `src/lost_cities_jax/`: pure JAX rules, observations, PPO, evaluation,
league, and human-play tools.
- `configs/jax_ppo/`: active YAML configurations.
- `web/`: static TypeScript client with the final JAX PPO ONNX policy.
- `web/public/models/jax-ppo.onnx`: verified browser policy. Keep its manifest
in sync and do not replace it without running the export validation.
Run lint:
Historical Deep CFR and ISMCTS implementations live under
`src/coolrl_lost_cities/games/classic/`. Their configs are in `legacy/`; they
are reproduction-only and must not be used as the default for new work. See
`docs/legacy.md`.
## Core commands
```bash
uv run ruff check .
```
Run all tests:
```bash
uv run pytest -q tests/lost_cities_jax
uv run pytest -q
uv run lost-cities-jax-ppo --help
```
Run focused Deep CFR tests:
Web checks:
```bash
uv run pytest -q tests/games/classic/test_deep_cfr_trainer.py
cd web
npm test
npm run build
```
Run the CLI through the console script:
## JAX PPO workflow
CPU smoke runs write disposable artifacts under `runs/tmp/`:
```bash
uv run lost-cities-deep-cfr --help
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
```
Equivalent module form:
GPU training must hold the shared compute lock. Use a real `tmux` session for
long user-observable jobs:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli --help
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
--config configs/jax_ppo/balanced.yaml \
--set run.artifact_root=runs/jax-ppo
```
## Deep CFR Training
The CLI auto-derives the run directory from `run.experiment_name` plus a
timestamp. By default runs land under `runs/tmp/`; pass `--keep` for a real
experiment that should live under `runs/`.
Smoke / throwaway run (lands in `runs/tmp/`):
Evaluation is deterministic and does not need the lock:
```bash
uv run lost-cities-deep-cfr train --config configs/deep_cfr/smoke.yaml
# → runs/tmp/<YYYY-MM-DD_HHMMSS>_smoke/
uv run lost-cities-jax-ppo eval \
--config configs/jax_ppo/balanced.yaml \
--checkpoint runs/jax-ppo/<run>/latest \
--opponent heuristic_balanced --games 10000 --duplicate
```
Real experiment (lands in `runs/`):
Generated artifacts under `runs/` are gitignored. Do not modify or delete
`runs/archive/`.
## Static browser policy
The checked-in ONNX policy is small enough for static distribution. To replace
it, export a validated checkpoint and rebuild the web app:
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep
# → runs/<YYYY-MM-DD_HHMMSS>_deep-cfr-default/
uv run --with onnx scripts/export_jax_ppo_onnx.py \
--checkpoint /path/to/checkpoint \
--output web/public/models/jax-ppo.onnx
cd web && npm test && npm run build
```
Variant / ablation (override one field; keep slug informative):
Commit the ONNX file and its JSON manifest together. The app must continue to
use a base-relative model URL so static subpath hosts work.
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--keep \
--set training_weighting.mode=none \
--set run.experiment_name=ablation-no-lcfr
# → runs/<YYYY-MM-DD_HHMMSS>_ablation-no-lcfr/
```
## Documentation
Short fixed-iteration run:
- Active plans: `docs/plans/`.
- Retired plans and historical evidence: `docs/plans/archive/`,
`docs/archive/`, and `docs/research/`.
- Do not edit archival documents; add a new dated note or active plan instead.
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--set run.max_iterations=100 \
--set checkpoint.save_every=0
```
When changing docs, run `scripts/librarian.sh` to validate Markdown links and
`file:line` citations.
Resume (path required, no shortcut):
## Git policy
```bash
uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--resume runs/<YYYY-MM-DD_HHMMSS>_<slug>/latest.pt
```
When `--resume` is given, the trainer reuses the resumed checkpoint's parent
directory; no new timestamped folder is created.
Useful train controls:
- `--keep`: real experiment, write under `runs/` (default is `runs/tmp/`).
- `--resume PATH`: resume from a specific checkpoint. `PATH` is required.
- `--set PATH=VALUE`: override config fields. Repeatable, parses values as
YAML (e.g. `--set traversal.num_workers=4`, `--set run.max_minutes=null`).
Common `--set` overrides:
- `--set run.device=cuda`: set the trainer device.
- `--set run.experiment_name=foo-v2`: change the slug used in the run dir
name (kebab-case).
- `--set checkpoint.exact_resume=true`: require checkpoint config compatibility.
- `--set checkpoint.save_latest=false --set checkpoint.save_every=0`:
disable checkpoint writes.
- `--set checkpoint.save_every=0`: keep only `latest.pt` (no archives).
- `--set checkpoint.save_every=N`: archive every N iterations.
## Naming Conventions
- **Directory names, run dirs, config filenames, `experiment_name` values**:
kebab-case (`deep-cfr-color-shared-512x3.yaml`,
`runs/2026-05-08_103045_color-attn-v2/`).
- **YAML keys, Python identifiers, config field names**: snake_case
(unchanged: `hidden_size`, `traversals_per_player`, `experiment_name`).
- The CLI converts `run.experiment_name` to a kebab slug when building the
run directory, so values may contain spaces or mixed case.
## Long Runs
Run long jobs in a real `tmux` session so the user can attach and stop them.
Do not rely on Codex command sessions for long user-observable training runs.
Start a long unbounded run:
```bash
tmux new-session -s coolrl-deepcfr-unbounded \
-c /home/coolguy/dev/coolrl-lost-cities \
'uv run lost-cities-deep-cfr train \
--config configs/deep_cfr/default.yaml \
--set run.max_iterations=null \
--set run.max_minutes=null \
--keep'
```
Attach later:
```bash
tmux attach -t coolrl-deepcfr-unbounded
```
Detach without stopping:
```text
Ctrl+B, D
```
Stop training:
```text
Ctrl+C
```
Follow logs from another terminal:
```bash
tail -f runs/<YYYY-MM-DD_HHMMSS>_<slug>/train.log
```
The unbounded config intentionally has:
```yaml
run:
max_iterations: null
max_minutes: null
checkpoint:
save_every: 100
```
`latest.pt` is updated continuously; archive checkpoints are written every 100
iterations. If disk is tight, set `--set checkpoint.save_every=0` (keep only
`latest.pt`) or increase `save_every`.
## Compute Lock
여러 에이전트가 한 머신을 공유하므로, train과 속도 벤치마크는
repo 루트 `.compute.lock`을 잡고 실행한다:
```bash
flock -n .compute.lock uv run lost-cities-deep-cfr train ...
```
승률/점수만 뽑는 eval과 `analyze`는 결정적이라 락 불필요.
## Evaluation And Analysis
Evaluate a checkpoint:
```bash
uv run lost-cities-deep-cfr eval \
--checkpoint runs/<run-dir>/latest.pt \
--opponent random \
--games 100 \
--device cpu
```
Save evaluation game records:
```bash
uv run lost-cities-deep-cfr eval \
--checkpoint runs/<run-dir>/latest.pt \
--opponent random \
--games 100 \
--device cpu \
--save-games runs/<run-dir>/eval_random_games.json
```
Generate analysis plots from `metrics.jsonl`:
```bash
uv run lost-cities-deep-cfr analyze \
--run runs/<run-dir>
```
Write plots to a separate directory:
```bash
uv run lost-cities-deep-cfr analyze \
--run runs/<run-dir> \
--output-dir runs/<run-dir>/analysis
```
The analyzer reads `metrics.jsonl` and writes PNG files grouped by diagnostic
section. Opponents are compared within each plot using fixed colors. The
`lost-cities-deep-cfr analyze` subcommand uses the analyzer default smoothing
window, currently 1 iteration (no smoothing), and supports `--max-iteration`.
For smoothing controls, run the analyzer module directly:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.analyze \
--run runs/<run-dir> \
--smoothing-window 5
```
Use `--no-smoothing` to force no moving average.
Current output files:
- `analysis_01_loss.png`
- `analysis_02_match.png`
- `analysis_03_action.png`
- `analysis_04_gameflow.png`
- `analysis_05_open_quality.png`
- `analysis_06_expedition_outcomes.png`
- `analysis_07_calibration.png`
- `analysis_08_traversal.png`
- `analysis_09_selectivity.png`
- `analysis_final_eval_summary.png`
## Runtime Artifacts
Each training run writes:
- `metrics.jsonl`: structured metrics, one completed iteration per line.
- `train.log`: human-readable timestamped logs.
- `runtime_progress.json`: latest progress snapshot.
- `latest.pt`: latest checkpoint.
- `iteration_*.pt`: archive checkpoints when enabled.
- `config.json`: resolved config for the run.
If a run is stopped mid-iteration, the in-progress iteration may not appear in
`metrics.jsonl`. Analyze the latest completed metric row.
## Weights & Biases (Optional)
Metrics can be mirrored to W&B. `wandb` is an optional extra; default
installs and runs do not require it.
Install:
```bash
uv sync --extra wandb
```
Run with W&B:
```bash
# Offline: no login, writes to <run_dir>/wandb/offline-run-*/
uv run lost-cities-deep-cfr train --config <...> --wandb --wandb-mode offline
# Online: requires `uv run wandb login` once, then real-time upload
uv run lost-cities-deep-cfr train --config <...> --wandb
```
W&B data is stored **per run** at `<run_dir>/wandb/`, not at a global
`runs/wandb/`. Each training run gets its own subfolder, so moving or
deleting a run directory carries its W&B data along with it.
Sync offline runs to wandb.ai later:
```bash
wandb sync runs/<run-dir>/wandb/offline-run-*
```
Flags:
- `--wandb`: enable W&B mirroring.
- `--wandb-project <name>`: defaults to `coolrl-lost-cities`.
- `--wandb-name <name>`: W&B run name; defaults to `run.experiment_name`.
- `--wandb-mode {online,offline,disabled}`: default `online`.
- `--wandb-group <name>`: group related runs from one experiment/hypothesis.
- `--wandb-job-type <type>`: role of this run, e.g. `train`, `eval`, `sweep`, or `smoke`.
- `--wandb-tag <tag>`: tag the run; repeatable.
W&B is purely additive — `metrics.jsonl` remains the source of truth, and
`analyze` reads `metrics.jsonl`, not W&B. Disabling W&B never breaks
training, resume, or analysis.
### Groups, notes, and tags
Use `--wandb-group` for the experiment or hypothesis family, e.g.
`model-size-grid-2026-05-08` or `strict-curriculum-v1`. Use `--wandb-name`
for the individual run, e.g. `512x3-seed79`, and `--wandb-job-type` for the
run role (`train`, `eval`, `sweep`, `smoke`).
Use `--wandb-notes` for the run's *purpose* (free-form prose) and `--wandb-tag`
for *categories you might filter on later* (short kebab-case keywords,
repeatable). Otherwise use them however you like. Just avoid:
- Tags that duplicate `config` (`lr-1e-4`, `traversal-280`) — W&B already
indexes config fields.
- Tags that are unique per run (`test-1`, `2026-05-07`) — that's the run
name and timestamp's job.
- Tag-as-sentence (`tested-bigger-traversal-with-lcfr`) — that belongs in
`--wandb-notes`.
**Notes length**: 3–5 lines, commit-message-body length. Should answer
*why* (hypothesis), *what* (key config delta), and *baseline* (run/iter
to compare against). Long analyses go in `docs/` and are linked from
notes; don't paste them in.
### Comparing two runs
Default is **sequential, single seed**. Run baseline first, then the
treatment with exactly one config change, both with the same `run.seed`.
Put both in the same W&B group (e.g. `--wandb-group lr-bump-v1`) and optionally
tag both with a shared hypothesis tag (e.g. `--wandb-tag lr-bump`) so they show
up together in W&B's Compare Runs view.
Do **not** run multiple seeds per condition unless explicitly asked —
that doubles or quintuples wall-clock and isn't the default protocol.
Single-seed comparison is enough to surface a signal; multi-seed is a
follow-up to confirm it.
Do **not** run two trainings in parallel on the same GPU — VRAM/SM
contention slows both unevenly and breaks the comparison.
## Docs & Experiment Workflow
### Where to write what
| 기록 내용 | 쓸 곳 |
| --- | --- |
| 진행 중인 계획/가설 | `docs/plans/<topic>.md` (1 주제 1 파일) |
| 끝난 계획 | `docs/plans/archive/` (수동 `mv`) |
| 날짜 박힌 실험 기록 | `docs/archive/<name>-YYYY-MM-DD.md` (immutable) |
| 항구적 알고리즘 노트 | `docs/research/<name>.md` (`Last verified:` 헤더) |
| 비용/프로파일 | `docs/reports/<name>-YYYY-MM-DD.md` (dated) |
| 스크래치 / 연구 스레드 인덱스 | `ideas.md` |
### Rules
- `docs/archive/`, `runs/archive/`는 read-only.
- archive 본문 복붙 금지 — 대신 `Source:` 링크 + distill.
- `file.py:NN` 인용은 작성 시점에 `rg`로 검증.
- 한 주제 한 파일 — `foo-v2.md` 만들지 말 것.
- 파일당 ~500줄 soft cap — 강제 분할이 아니라 "다른 위치(archive/research)로
가야 할 내용이 누적됐는지" routing 점검 트리거. 초과 시 dated 실험은
`docs/archive/`로, 항구적 분석은 `docs/research/`로 보내고 본 파일은
현재 상태 reference만 남긴다.
## Notes For Future Agents
- Prefer `rg`/`rg --files` for search.
- Use `apply_patch` for manual edits.
- Do not commit generated run artifacts from `runs/`.
- Cython-generated `.c` files are gitignored; edit `.pyx`/`.pxd` sources.
- Before committing, run `uv run ruff check .` and at least the relevant pytest
subset. For Deep CFR changes, run
`uv run pytest -q tests/games/classic/test_deep_cfr_trainer.py`.
- When changing docs, run `scripts/librarian.sh` to lint markdown link
integrity and `file:line` code citations across `docs/**`. See
`docs/plans/librarian.md` for the full design.
## Git Branching Policy (READ THIS)
**DO NOT create new git branches unless the user explicitly asks for one in
the current task. Work on whatever branch is currently checked out (default
`main`). This is a hard rule — no exceptions for "safety", "isolation",
"experiments", "work-in-progress", or any other self-justified reason.**
Why this rule exists:
- This project intentionally develops on `main` with frequent small commits.
- Auto-created branches like `experiments/foo`, `feature/bar`, `wip/baz`
fragment review, hide work from the user, and require manual cleanup.
- The user has not authorized branch creation as a default behavior. If
they want a branch, they will say so explicitly ("make a branch", "PR
this", "isolate this in a branch", etc.).
What you MUST do instead:
- Make commits directly on the currently checked-out branch.
- If you think a branch is justified, **stop and ask the user first** —
do not preemptively create one.
- If a tool or subagent invocation auto-suggests creating a branch
(e.g. PR-style workflows), refuse the branch creation step and commit
to the current branch.
- If you find yourself already on a non-default branch you didn't expect,
stop and ask the user — do not switch, do not create a new one, do not
reset.
Forbidden without explicit user instruction:
- `git checkout -b <name>`
- `git switch -c <name>`
- `git branch <name>`
- `gh pr create` from an auto-created branch
- Any worktree creation that implicitly creates a new branch
This rule applies to the main agent and to every subagent or tool the main
agent invokes. Pass it through in subagent prompts when delegating
git-touching work.
Do not create git branches unless the user explicitly asks in the current
task. Work on the checked-out branch. Before committing, run `uv run ruff
check .` and the relevant tests. Never commit generated run artifacts.
README.md
+262 -18
View File
@@ -1,38 +1,282 @@
# coolrl-lost-cities
Focused Lost Cities extraction from the legacy `coolrl` repository.
JAX PPO training and on-device browser play for the two-player card game Lost
Cities. The project provides a pure JAX rules engine, PPO training and
evaluation tools, and a static web client that runs the shipped final policy
with WebGPU when available and WebAssembly otherwise.
The current implementation starts with the classic two-player card game:
The active path is **JAX + PPO**. Deep CFR and ISMCTS are retained only as
historical research implementations; see [legacy notes](docs/legacy.md).
- classic 5-expedition rules by default
- Python/Cython game engine
- env wrapper
- random, discard-only, and safe-heuristic bots
- core rule, scoring, mask, env, canonical-state, bot, and GUI smoke tests
## Quick start
Training code, Deep CFR, learned-policy evaluation, GUI, and web client are
intentionally outside the first port.
## Development
Install the project, then run a complete CPU-sized sanity check:
```bash
uv run pytest tests/games/classic
uv run lost-cities-classic
uv sync
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
```
For future GUI work, install the optional GUI dependencies:
The final command prints a run directory containing `latest`, `config.json`,
and `metrics.jsonl` under `runs/tmp/jax-ppo-artifacts/`.
## Train and evaluate a PPO policy
The committed configurations describe the opponent and training budget. For a
GPU run, use CUDA JAX and keep generated artifacts outside git:
```bash
uv sync --extra gui
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
--config configs/jax_ppo/balanced.yaml \
--set run.artifact_root=runs/jax-ppo
```
Run the classic pygame GUI:
Evaluate a saved checkpoint against a fixed opponent. Duplicate evaluation
swaps seats over the same shuffled games:
```bash
uv run lost-cities-classic-gui --mode pvc --bot safe-heuristic
uv run lost-cities-jax-ppo eval \
--config configs/jax_ppo/balanced.yaml \
--checkpoint runs/jax-ppo/<run>/latest \
--opponent heuristic_balanced \
--games 10000 \
--duplicate
```
The GUI uses the in-process Cython game engine.
Run `uv run lost-cities-jax-ppo --help` for training against saved opponents,
league runs, gates, human-play logs, and evaluation variants.
## Browser client
The final verified JAX PPO policy is shipped as a 3.1 MB static ONNX asset.
No server or local checkpoint is required to play it:
```bash
cd web
npm ci
npm run dev
```
`npm run build` produces a fully static site. The model path is deployment-base
aware, so the build can be served from GitHub Pages, GitLab Pages, or a normal
web root. Pushes to `main` deploy that build to both configured Pages hosts.
See [web/README.md](web/README.md) for model replacement, tests, and the
cross-runtime parity fixture.
## JAX Rules Engine
`lost_cities_jax` is a standalone pure rules simulator for one two-player
Lost Cities round. It does not contain neural networks, PPO, CFR, MCTS, match
wrappers, bots, or rule variants.
Public API:
```python
from lost_cities_jax import (
OBS_DIM,
N_ACTIONS,
State,
batched_legal_mask,
batched_obs,
batched_reset,
batched_step,
board_score,
legal_action_mask,
observation,
reset,
reset_from_order,
score,
step,
)
```
Core functions are pure JAX functions:
- `reset(rng) -> State`
- `reset_from_order(deck_order) -> State`
- `legal_action_mask(state) -> bool[96]`
- `step(state, action) -> (State, float32[2], bool)`
- `score(state) -> float32[2]`
- `board_score(state) -> float32[2]`
- `observation(state, player) -> float32[454]`
The batched exports are `jax.jit(jax.vmap(...))` wrappers. Illegal actions and
done-state actions are defined as no-op transitions with zero reward; training
code should still sample only from `legal_action_mask`.
### Rule Summary
One round uses 60 cards: five colors, each with three handshakes and ranks
2 through 10. Each player starts with eight cards, then each ply must place one
hand card to the matching expedition or discard pile and draw one card from the
deck or a discard pile. A player may not draw the card they just discarded.
Expedition numbers must be strictly increasing. Handshakes may be played only
before any number in that color. The round ends immediately when the final deck
card is drawn, or at `MAX_STEPS == 400`; forced termination is scored exactly
like natural termination.
Scoring per player/color:
```text
empty column: 0
non-empty: (sum(number ranks) - 20) * (1 + handshake_count)
bonus: +20 if total column length >= 8, not multiplied
```
### Encodings
Cards:
| Field | Encoding |
| --- | --- |
| `card_id` | `color * 12 + slot` |
| `color` | `0..4` |
| `slot 0..2` | handshake |
| `slot 3..11` | ranks `2..10`, with `rank = slot - 1` |
Actions (`N_ACTIONS == 96`):
```text
action_id = hand_slot * 12 + place_type * 6 + draw_source
hand_slot = 0..7, current player's hand sorted by card_id
place_type = 0 play, 1 discard
draw_source = 0 deck, 1..5 discard pile color 0..4
```
Observation (`OBS_DIM == 454`):
- 60 cards x 7 one-hot channels:
my hand, my board, opponent board, discard top, discard non-top,
opponent public hand, unknown.
- 34 scalar features:
remaining deck `/44`, opponent unknown hand count `/8`, step count `/400`,
current-player then opponent `col_top /10`, `col_hs /3`, `col_len /12`,
and current board score difference `(player - opponent) /780`.
### Verification
```bash
uv run pytest -q tests/lost_cities_jax
uv run pytest -q
uv run ruff check .
```
Large differential profiles:
```bash
# CI profile: 100,000 random legal-policy games
CI=1 uv run pytest -q tests/lost_cities_jax/test_differential.py
# Full profile: 1,000,000 random legal-policy games
uv run pytest -q tests/lost_cities_jax/test_differential.py --full
```
Observed differential results on 2026-07-04 with CPU JAX backend:
```text
CI=1 ... test_differential.py
1 passed in 247.61s (0:04:07)
... test_differential.py --full
1 passed in 2514.14s (0:41:54)
elapsed=41:54.48
```
Throughput benchmark:
```bash
flock -n .compute.lock uv run python benchmarks/throughput.py
# Optional CUDA check without making CUDA a project dependency:
flock -n .compute.lock uv run --with 'jax[cuda12]' python benchmarks/throughput.py
```
Measured on 2026-07-04 with CPU JAX backend:
```text
backend=cpu
batch_size=8192
steps=256
elapsed_sec=4.655526
steps_per_sec=450465.14
```
Measured on 2026-07-04 with CUDA JAX backend on RTX 3090, using the optional
`uv run --with 'jax[cuda12]' ...` command:
```text
backend=gpu
batch_size=8192
steps=256
elapsed_sec=0.530039
steps_per_sec=3956598.38
```
### DECISIONS.md
- Explicit `deck_order` dealing uses the first eight cards for player 0 and
the next eight for player 1. The remaining cards are drawn from index 16.
This is equivalent under a uniform shuffle and is fixed by tests.
- After a legal terminal transition, `to_move` is advanced to the next player,
but `done=True` makes all later steps complete no-ops.
- Terminal reward is emitted only on the transition that reaches `done=True`.
Done-state no-op steps return zero reward.
- Observation scalar normalization is implementation-defined as documented
above and locked by the exported `OBS_DIM`.
## JAX PPO Static-Opponent Ladder
The first training stack above `lost_cities_jax` is exposed as:
```bash
uv run lost-cities-jax-ppo --help
```
CPU smoke:
```bash
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
```
GPU training uses optional CUDA JAX, keeping CUDA wheels out of the default
project dependency set:
```bash
tmux new-session -s coolrl-jax-ppo-discard \
-c /home/coolguy/dev/coolrl-lost-cities \
"flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
--config configs/jax_ppo/discard-only.yaml"
```
Random-policy baseline vs `discard_only`, measured on 2026-07-04 with
8192 games x 400 plies on GPU:
```text
return_mean=-55.582763671875
game_length_mean=69.6085205078125
max_steps_rate=0.0
play_action_rate=0.28854578733444214
opened_colors_mean=4.942626953125
positive_expeditions_mean=0.41796875
```
Static-opponent gate results, measured on 2026-07-04 with 10,000 fixed shuffles
and duplicate seat-swapped evaluation:
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Large PPO artifacts are written under
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
Generated checkpoints and evaluation JSON are not committed to git. The full
run summary is in
[docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md](docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md).
## Basic Usage
benchmarks/throughput.py
+56
View File
@@ -0,0 +1,56 @@
from __future__ import annotations
import argparse
import time
from functools import partial
import jax
import jax.numpy as jnp
from lost_cities_jax.engine import legal_action_mask, reset, step
@partial(jax.jit, static_argnames=("steps",))
def rollout(states, rng, *, steps: int):
def body(carry, _):
state, key = carry
key, action_key = jax.random.split(key)
mask = jax.vmap(legal_action_mask)(state)
logits = jnp.where(mask, 0.0, -1.0e9)
actions = jax.random.categorical(action_key, logits, axis=1).astype(jnp.int32)
state, _, _ = jax.vmap(step, in_axes=(0, 0))(state, actions)
return (state, key), None
(states, rng), _ = jax.lax.scan(body, (states, rng), xs=None, length=steps)
return states, rng
def main() -> None:
parser = argparse.ArgumentParser(description="Lost Cities JAX random-policy throughput")
parser.add_argument("--batch-size", type=int, default=8192)
parser.add_argument("--steps", type=int, default=256)
parser.add_argument("--warmup-steps", type=int, default=32)
args = parser.parse_args()
key = jax.random.PRNGKey(0)
reset_keys = jax.random.split(key, args.batch_size)
states = jax.jit(jax.vmap(reset))(reset_keys)
warm_states, key = rollout(states, jax.random.PRNGKey(1), steps=args.warmup_steps)
jax.tree_util.tree_leaves(warm_states)[0].block_until_ready()
start = time.perf_counter()
states, _ = rollout(states, key, steps=args.steps)
jax.tree_util.tree_leaves(states)[0].block_until_ready()
elapsed = time.perf_counter() - start
transitions = args.batch_size * args.steps
print(f"backend={jax.default_backend()}")
print(f"batch_size={args.batch_size}")
print(f"steps={args.steps}")
print(f"elapsed_sec={elapsed:.6f}")
print(f"steps_per_sec={transitions / elapsed:.2f}")
if __name__ == "__main__":
main()
configs/jax_ppo/balanced.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-balanced
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
opponent:
name: heuristic_balanced
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/cautious.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-cautious
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
opponent:
name: heuristic_cautious
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/discard-only.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-discard-only
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
opponent:
name: discard_only
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/gate3-exploiter.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-gate3-exploiter
seed: 20260705
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/verification
opponent:
name: gate3_checkpoint_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/gates-1-2-exploiter-long-random.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: gates-1-2-long-random-exploiter
seed: 20260705
learner_seat: 0
total_updates: 1200
log_every: 10
checkpoint_every: 100
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
opponent:
name: league_v1_update_500_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 2000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/gates-1-2-exploiter-replay.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: gates-1-2-replay-exploiter
seed: 20260705
learner_seat: 0
total_updates: 900
log_every: 10
checkpoint_every: 100
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
opponent:
name: league_v1_update_500_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 0.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 0
evaluation:
games: 2000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/gates-1-2-exploiter-smoke.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: gates-1-2-smoke-exploiter
seed: 20260705
learner_seat: 0
total_updates: 1
log_every: 1
checkpoint_every: 1
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/exploiters
opponent:
name: league_v1_update_500_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8
rollout_steps: 16
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 1
minibatches: 2
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 2
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 2
configs/jax_ppo/gates-1-2-exploiter-warmstart.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: gates-1-2-warmstart-gate3-exploiter
seed: 20260705
learner_seat: 0
total_updates: 900
log_every: 10
checkpoint_every: 100
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
opponent:
name: league_v1_update_500_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 0.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 0
evaluation:
games: 2000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/gates-1-2-smoke.yaml
+35
View File
@@ -0,0 +1,35 @@
run:
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke
report_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/report.md
summary_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/summary.jsonl
shuffle_bank_seed: 20260704
gpu_budget_hours: 1.0
target:
config: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500
gate1:
games: 2
batch_games: 2
delta_games: 2
delta_target_events: 2
delta_pairs: 2
delta_batch_events: 1
gate2:
eval_games: 2
pass_threshold: 0.55
exploiters:
- name: long_random_smoke
config: configs/jax_ppo/gates-1-2-exploiter-smoke.yaml
repair:
max_cycles: 1
league_template: configs/jax_ppo/league-smoke.yaml
experiment_name: jax-ppo-gates-1-2-repair-smoke
artifact_subdir: gate2c
evaluation_games: 2
evaluation_batch_games: 2
guard_expert_ci_low: -1000.0
guard_max_steps_rate: 1.0
configs/jax_ppo/gates-1-2.yaml
+44
View File
@@ -0,0 +1,44 @@
run:
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2
report_path: docs/reports/gates-1-2-2026-07-05.md
summary_path: docs/reports/gates-1-2-2026-07-05-summary.jsonl
shuffle_bank_seed: 20260704
gpu_budget_hours: 8.0
target:
config: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500
gate1:
games: 2000
batch_games: 8192
delta_games: 200
delta_target_events: 500
delta_pairs: 64
delta_batch_events: 16
gate2:
eval_games: 2000
pass_threshold: 0.55
exploiters:
- name: long_random
config: configs/jax_ppo/gates-1-2-exploiter-long-random.yaml
notes: random init + shaping anneal, extended budget
- name: warmstart_gate3
config: configs/jax_ppo/gates-1-2-exploiter-warmstart.yaml
resume: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
notes: ladder v2 gate-3 warm start, shaping disabled
- name: replay_exploiter
config: configs/jax_ppo/gates-1-2-exploiter-replay.yaml
resume: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest
notes: league v1 exploiter warm start, shaping disabled
repair:
max_cycles: 3
league_template: configs/jax_ppo/league-v1.yaml
experiment_name: jax-ppo-gates-1-2-repair
artifact_subdir: gate2c
evaluation_games: 2000
evaluation_batch_games: 8192
guard_expert_ci_low: 0.0
guard_max_steps_rate: 0.02
configs/jax_ppo/ladder-v2-balanced.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-ladder-v2-balanced
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
opponent:
name: heuristic_balanced
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/ladder-v2-discard-only.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-ladder-v2-discard-only
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
opponent:
name: discard_only
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/ladder-v2-expert.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-ladder-v2-expert
seed: 20260704
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
opponent:
name: heuristic_expert
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/ladder-v2-exploiter.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-ladder-v2-exploiter
seed: 20260705
learner_seat: 0
total_updates: 250
log_every: 1
checkpoint_every: 10
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
opponent:
name: ladder_v2_gate3_checkpoint_frozen
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5000000
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
configs/jax_ppo/league-smoke.yaml
+77
View File
@@ -0,0 +1,77 @@
base_config: configs/jax_ppo/ladder-v2-expert.yaml
warm_start_checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
run:
experiment_name: jax-ppo-league-smoke
seed: 20260705
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke
league:
cycles: 1
league_updates_per_cycle: 1
snapshot_interval_updates: 1
mirror_probability: 0.5
uniform_mix: 0.1
stalling_anchor_floor: 0.02
stalling_anchor_cap: 0.05
pool_max_size: 8
max_active_pool_members: 7
exploiter_member_fraction_cap: 0.3333333333333333
wall_clock_hours: 1.0
success_exploiter_win_rate: 0.60
stagnation_window: 5
stagnation_min_delta: 0.02
evaluation:
games: 2
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 2
recent_snapshot_evals: 1
exploiter:
config: configs/jax_ppo/ladder-v2-exploiter.yaml
updates: 1
guards:
expert_ci_low: -1000.0
max_steps_rate: 1.0
tracking:
tracked_summary_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke/summary.jsonl
report_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke/report.md
anchors:
- name: discard_only
kind: static
policy_name: discard_only
anchor: true
stalling: true
- name: heuristic_balanced
kind: static
policy_name: heuristic_balanced
anchor: true
- name: heuristic_cautious
kind: static
policy_name: heuristic_cautious
anchor: true
stalling: true
- name: heuristic_expert
kind: static
policy_name: heuristic_expert
anchor: true
- name: ladder_v2_gate1_discard
kind: checkpoint
config: configs/jax_ppo/ladder-v2-discard-only.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest
anchor: true
- name: ladder_v2_gate2_balanced
kind: checkpoint
config: configs/jax_ppo/ladder-v2-balanced.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest
anchor: true
- name: ladder_v2_gate3_expert
kind: checkpoint
config: configs/jax_ppo/ladder-v2-expert.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
anchor: true
configs/jax_ppo/league-v1.yaml
+77
View File
@@ -0,0 +1,77 @@
base_config: configs/jax_ppo/ladder-v2-expert.yaml
warm_start_checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
run:
experiment_name: jax-ppo-league-v1
seed: 20260705
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league
league:
cycles: 5
league_updates_per_cycle: 500
snapshot_interval_updates: 250
mirror_probability: 0.5
uniform_mix: 0.1
stalling_anchor_floor: 0.02
stalling_anchor_cap: 0.05
pool_max_size: 24
max_active_pool_members: 12
exploiter_member_fraction_cap: 0.3333333333333333
wall_clock_hours: 6.0
success_exploiter_win_rate: 0.60
stagnation_window: 5
stagnation_min_delta: 0.02
evaluation:
games: 10000
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 8192
recent_snapshot_evals: 3
exploiter:
config: configs/jax_ppo/ladder-v2-exploiter.yaml
updates: 250
guards:
expert_ci_low: 0.0
max_steps_rate: 0.02
tracking:
tracked_summary_path: docs/reports/league-v1-2026-07-05-summary.jsonl
report_path: docs/reports/league-v1-2026-07-05.md
anchors:
- name: discard_only
kind: static
policy_name: discard_only
anchor: true
stalling: true
- name: heuristic_balanced
kind: static
policy_name: heuristic_balanced
anchor: true
- name: heuristic_cautious
kind: static
policy_name: heuristic_cautious
anchor: true
stalling: true
- name: heuristic_expert
kind: static
policy_name: heuristic_expert
anchor: true
- name: ladder_v2_gate1_discard
kind: checkpoint
config: configs/jax_ppo/ladder-v2-discard-only.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest
anchor: true
- name: ladder_v2_gate2_balanced
kind: checkpoint
config: configs/jax_ppo/ladder-v2-balanced.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest
anchor: true
- name: ladder_v2_gate3_expert
kind: checkpoint
config: configs/jax_ppo/ladder-v2-expert.yaml
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
anchor: true
configs/jax_ppo/match-selfplay.yaml
+50
View File
@@ -0,0 +1,50 @@
# Classic Lost Cities: three rounds, self-play.
#
# rollout_steps is deliberately shorter than a match. Early self-play stalls
# rounds badly (untrained matches run ~900 plies), and the env carries across
# updates, so a long match simply spans several rollouts -- the GAE truncation
# bootstrap keeps that unbiased. Sizing the scan to the worst-case match instead
# would blow up the rollout tensors, which are already doubled by training both
# seats and widened by the critic's 681-dim privileged view.
run:
experiment_name: match-selfplay
seed: 20260715
learner_seat: 0
total_updates: 300
log_every: 10
checkpoint_every: 50
artifact_root: runs/jax-ppo-match
opponent:
name: discard_only # unused: self-play
network:
hidden_size: 512
num_layers: 3
ppo:
batch_games: 512
rollout_steps: 256
gamma: 1.0
gae_lambda: 0.97 # a match is ~3x a round; 0.95 reaches too little of it
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 32
reward:
terminal_scale: 50.0
# Counted in learner actions now, not padded scan steps. Held slightly above
# zero: the match-terminal signal alone is one bounded number per ~160 plies.
potential_shaping_initial: 1.0
potential_shaping_final: 0.05
potential_shaping_anneal_steps: 20000000
evaluation:
games: 2000
duplicate: true
shuffle_bank_seed: 20260715
batch_games: 512
configs/jax_ppo/smoke.yaml
+40
View File
@@ -0,0 +1,40 @@
run:
experiment_name: jax-ppo-smoke
seed: 20260704
learner_seat: 0
total_updates: 1
log_every: 1
checkpoint_every: 1
artifact_root: runs/tmp/jax-ppo-artifacts
opponent:
name: discard_only
network:
hidden_size: 64
num_layers: 2
ppo:
batch_games: 64
rollout_steps: 64
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 1
minibatches: 4
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 100000
evaluation:
games: 128
duplicate: true
shuffle_bank_seed: 20260704
batch_games: 64
docs/legacy.md
+25
View File
@@ -0,0 +1,25 @@
# Legacy research stacks
The supported project path is the JAX rules engine and JAX PPO tooling exposed
by `lost-cities-jax-ppo`. The historical Deep CFR and ISMCTS implementations
are preserved for research reproduction, but they are not current training
recipes and no longer have public console-script entry points.
## What is preserved
- Deep CFR sources: `src/coolrl_lost_cities/games/classic/deep_cfr/`
- ISMCTS sources: `src/coolrl_lost_cities/games/classic/ismcts/`
- Their retired configurations: `legacy/deep-cfr/configs/` and
`legacy/ismcts/configs/`
- Earlier plans and dated findings: `docs/plans/archive/`, `docs/archive/`,
and `docs/research/`
These files remain in the repository so a historical experiment can be read or
reproduced against its original commit. They should not be chosen for new
models, benchmarks, or product work.
## Current workflow
Start from `configs/jax_ppo/`, run `lost-cities-jax-ppo`, and use the browser
client in `web/`. The root [README](../README.md) contains the short training,
evaluation, and static-web recipes.
docs/performance.md
+8 -5
View File
@@ -1,7 +1,10 @@
# Deep CFR Performance Notes
# Legacy Deep CFR Performance Notes
This document tracks current runtime bottlenecks for the active Deep CFR
training path. The numbers below are observational, not a benchmark contract.
> Historical record only. The supported training stack is JAX PPO; see the
> root [README](../README.md) and [legacy notes](legacy.md).
This document records the Deep CFR runtime bottlenecks observed before the JAX
PPO transition. The numbers below are historical, not a benchmark contract.
## Current Default Runtime
@@ -11,7 +14,7 @@ Source run:
runs/tmp/2026-05-07_171535_deep-cfr-default/metrics.jsonl
```
The run used `configs/deep_cfr/default.yaml` with CUDA enabled. At the time of
The run used `legacy/deep-cfr/configs/default.yaml` with CUDA enabled. At the time of
inspection, completed metrics covered iterations 70 through 95. The training
process was still running, so later rows may differ.
@@ -88,7 +91,7 @@ This gives 560 traversals per iteration, split into 70 worker batches.
## Device Use
The trainer constructs the advantage and strategy networks on `run.device`.
`configs/deep_cfr/default.yaml` sets:
`legacy/deep-cfr/configs/default.yaml` sets:
```yaml
run:
docs/plans/deep-cfr-selectivity.md → docs/plans/archive/deep-cfr-selectivity.md
View File
docs/plans/librarian.md → docs/plans/archive/librarian.md
+1 -1
View File
@@ -117,7 +117,7 @@ operator applies the patch.
- ✅ AGENTS.md "Docs & Experiment Workflow" section landed
(commit `09d5815`, 2026-05-07).
- ✅ Plan drafted at `docs/plans/librarian.md` (this file).
- ✅ Plan drafted at `docs/plans/archive/librarian.md` (this file).
- ✅ Prompt moved: `.claude/agents/librarian.md` →
`scripts/librarian-prompt.md`. Claude-specific subagent registration
removed.
docs/plans/model_size_experiment.md → docs/plans/archive/model_size_experiment.md
+7
View File
@@ -1,5 +1,12 @@
# Plan: Model-Size Experiment (Keystone for Model-Scale Optimizations)
> **⚠️ 스택 주의 — 이 문서는 Deep CFR / PyTorch 스택 전용이다.**
> `input_dim=365`, `legacy/deep-cfr/configs/`, `DeepCFRMLP` 기준으로 쓰였다.
> 현행 JAX PPO 스택(`OBS_DIM=454`, `configs/jax_ppo/`, `ActorCritic`)에는
> **적용되지 않는다.** PPO 쪽 모델 크기 문제는
> [jax-ppo-model-size-ab.md](../jax-ppo-model-size-ab.md)와
> [../../reports/jax-ppo-model-capacity-2026-07-12.md](../../reports/jax-ppo-model-capacity-2026-07-12.md)를 볼 것.
**Status:** Ready to execute
**Owner:** Operator (runs grid on `home`); Codex (adds configs and runner script)
**Background:** See `docs/performance.md` → "Post-A Optimization Calculus",
docs/plans/option_b_interleaved_traversal.md → docs/plans/archive/option_b_interleaved_traversal.md
View File
docs/plans/torch_compile.md → docs/plans/archive/torch_compile.md
View File
docs/plans/jax-ppo-model-size-ab.md
+100
View File
@@ -0,0 +1,100 @@
# Plan: JAX PPO Model-Size A/B
**Status:** Ready to execute
**Priority:** Low — 진단상 용량은 병목이 아닐 가능성이 높다. 그러나 PPO 스택에서
한 번도 측정된 적이 없어 정황증거뿐이다. 이 A/B의 목적은 **스케일링 라인을 열지,
닫을지 확정**하는 것이다.
**Background:** [docs/reports/jax-ppo-model-capacity-2026-07-12.md](../reports/jax-ppo-model-capacity-2026-07-12.md)
> 이 계획은 은퇴한 Deep CFR/PyTorch 스택용
> [model_size_experiment.md](archive/model_size_experiment.md)를 **대체하지 않는다** —
> 그쪽은 다른 스택(`input_dim=365`, `legacy/deep-cfr/configs/`) 이야기다. 서로 무관하다.
## 가설
`hidden_size=512, num_layers=3` (808K 파라미터)은 구 스택에서 복사돼 온 상속값이며
PPO에서 검증된 적이 없다. 셀프플레이 샘플:파라미터 비가 약 2,028:1이므로
용량을 키우면 **응수 품질(expert 앵커)은 오르지만, exploitability 바닥(~0.54)은
움직이지 않을 것**이다.
이 예측이 맞으면 → 병목은 용량이 아니라 셀프플레이 스킴이며, 스케일링 라인을
닫고 리그/착취 구조 쪽에 투자한다.
틀리면 (exploitability가 유의하게 내려가면) → 스케일링이 열린다.
## 실험 설계
**2점 A/B. 그리드 아님.** 그리드는 이 우선순위에 비해 과하다.
| 이름 | hidden_size | num_layers | 파라미터 |
| --- | ---: | ---: | ---: |
| `size-512x3` (baseline) | 512 | 3 | 808,033 |
| `size-1024x4` (treatment) | 1024 | 4 | 3,714,145 |
- 두 config는 `network` 블록과 `run.experiment_name`만 다르고 **나머지는
final candidate와 byte-identical**이어야 한다. 기준 config:
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/main_ppo_config.json`
- **같은 `run.seed`**, 순차 실행 (AGENTS.md "Comparing two runs" 규약).
같은 GPU에서 두 학습을 병렬로 돌리지 말 것.
- 리그 **1 사이클**, 500 updates.
## 판정 기준 (중요)
**H2H로 판정하지 말 것.** 더 큰 모델이 더 작은 모델을 이기는 건 거의 자명하고,
그건 우리가 알고 싶은 게 아니다. 두 축으로만 본다:
1. **Expert 앵커** — `heuristic_expert` 상대 duplicate 평가
(shuffle bank seed 20260704, 2,000쌍). *응수 품질*을 본다.
2. **Exploitability** — 각 체크포인트에 대해 exploiter를 새로 학습시켜 승률 측정.
`long_random_shaping` 프로토콜(1200 updates, shaping anneal, warm start
`random`) 하나로 충분하다.
[diminishing-returns-2026-07-05.md](../reports/diminishing-returns-2026-07-05.md)의
기존 측정과 **같은 프로토콜 안에서만** 비교할 것 — 프로토콜이 다르면 섞지 않는다.
### 결정 트리
| Expert 앵커 | Exploiter 승률 | 결론 | 조치 |
| --- | --- | --- | --- |
| 유의하게 상승 | **변화 없음** (~0.54 유지) | **예측대로.** 용량은 병목이 아니다 | 스케일링 라인 종료. 리그/착취 구조로 이동. 1024×4를 default로 올릴지는 비용 대비 선택 사항 |
| 유의하게 상승 | **유의하게 하락** | 가설 기각. 용량이 실제로 제약이었다 | `network` default를 1024×4로 올리고, 2048×4까지 한 점 더 확장 |
| 변화 없음 | 변화 없음 | 용량 무관 확정 | 스케일링 라인 종료 |
| 하락 | — | 최적화 문제 (LR/KL 등이 큰 모델에 안 맞음) | default 유지. 스케일 재시도 전에 하이퍼파라미터 점검 |
"유의하게"는 Wilson CI(승률) / score CI(점수차) 비중첩 기준. 기존 리포트들이
쓰는 것과 동일하다.
## 비용
리포트 기준 리그 1사이클 ≈ 1.5시간 GPU (512×3). 1024×4는 파라미터 4.6배이나
병목이 롤아웃(엔진 시뮬)이라 학습 시간이 4.6배가 되지는 않는다.
exploiter 학습이 별도로 붙는다.
`.compute.lock`을 잡고 돌릴 것 (AGENTS.md "Compute Lock"):
```bash
flock -n .compute.lock <train command>
```
## Non-goals
- 그리드 스윕(768×4, 1536×8 등) — 2점으로 충분하다. 결과가 애매할 때만 확장.
- 아키텍처 변경 (residual, transformer, per-color 공유 인코더 등) — 별도 주제.
- 인코딩 변경 (`OBS_DIM=454` 고정).
- 하이퍼파라미터 동시 변경 — 한 번에 하나만 바꾼다.
## Definition of done
산출될 파일 (아직 존재하지 않음):
```text
configs/jax_ppo/size-512x3.yaml
configs/jax_ppo/size-1024x4.yaml
docs/reports/jax-ppo-model-size-ab-<YYYY-MM-DD>.md
```
- [ ] 위 두 config 추가 (final candidate config 기준, `network` /
`run.experiment_name`만 차이)
- [ ] 두 학습 순차 실행, 같은 seed, `.compute.lock` 사용
- [ ] 두 체크포인트 expert 앵커 duplicate 평가
- [ ] 두 체크포인트 exploiter 학습 + 승률 측정 (`long_random_shaping`)
- [ ] 결과표 + 결정 트리 적용 결과를 위 dated 리포트에 기록
- [ ] 결정 트리에 따라: `network` default 갱신 **또는** 스케일링 라인 종료를 문서화
docs/plans/jax-ppo-static-opponents.md
+378
View File
@@ -0,0 +1,378 @@
# Plan: JAX PPO Static-Opponent Ladder
**Status:** Static-opponent ladder passed on 2026-07-04.
**Owner:** Codex implements; operator reviews training gates.
**Scope:** A compact PPO training stack on top of `lost_cities_jax`, using GPU
via optional CUDA JAX execution.
## Goal
Build the first learning layer above the JAX Lost Cities rules engine:
1. A batched rollout driver using `batched_reset`, `batched_step`,
`batched_legal_mask`, and `batched_obs`.
2. Three static pure-JAX opponent policies:
`discard_only`, `heuristic_balanced`, and `heuristic_cautious`.
3. A 3 x 512 MLP actor-critic with 96-action policy logits and scalar value.
4. Standard PPO training against one static opponent at a time.
5. Duplicate evaluation on a fixed 10,000-deck shuffle bank.
6. Orbax checkpoints, metrics logging, and a CLI:
`lost-cities-jax-ppo train --config ...`.
The near-term objective is not league self-play. It is to pass the
static-opponent diagnostic ladder and preserve enough artifacts that a later
self-play failure can be localized cleanly.
## Non-Goals
- No league self-play, snapshot pool, Elo, or opponent matchmaking in this
phase.
- No MCTS, CFR, search, or neural opponent ensemble.
- No multi-round match wrapper.
- No rule variants, expanded colors, or extra players.
- No large binary checkpoints committed to git. Generated artifacts are stored
outside the code repo, with small manifests and summaries committed.
## GPU Execution
Keep the project dependency portable. Do not make CUDA wheels mandatory in
`pyproject.toml`.
Use optional CUDA JAX for training and GPU benchmarks:
```bash
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
--config configs/jax_ppo/discard-only.yaml
```
The CPU path must still work for tests:
```bash
uv run pytest -q tests/lost_cities_jax
```
Current benchmark evidence on RTX 3090:
```text
backend=gpu
batch_size=8192
steps=256
steps_per_sec=3956598.38
```
## Implementation Shape
Prefer a small number of files while keeping testable boundaries:
```text
src/lost_cities_jax/
ppo.py # config, network, rollout, PPO update, train loop
opponents.py # pure-JAX static opponent policy functions
ppo_cli.py # argparse CLI
configs/jax_ppo/
discard-only.yaml
balanced.yaml
cautious.yaml
```
If the PPO implementation remains readable in one main file, keep it there.
Split only when a file becomes hard to test or review.
Dependencies likely needed:
- `flax` for model modules and train state.
- `optax` for Adam and PPO losses.
- `orbax-checkpoint` for checkpointing.
Add them through `uv add`, not pip/conda/poetry.
## Static Opponents
All opponent policies are pure JAX functions:
```python
policy_fn(state: State, player: jax.Array) -> jax.Array # int32 action
```
They must sample no Python-side randomness. If tie-breaking needs randomness,
pass a JAX key explicitly:
```python
policy_fn(state, player, rng) -> action
```
Policies:
- `discard_only`: always discard a legal hand slot and draw from deck when
legal. This opponent should make free expedition building easy.
- `heuristic_balanced`: prefer legal plays that improve expedition prospects,
avoid obviously toxic openings, draw useful discard tops when available.
- `heuristic_cautious`: stricter opening threshold, fewer negative expedition
commitments, more conservative discard/draw behavior.
Opponent policies are not learning targets. They are diagnostic fixtures.
## Rollout Driver
Run 8192 games in parallel by default.
The learner controls one fixed seat per rollout batch. The opponent occupies
the other seat. Because games alternate turns, each environment step chooses:
- learner action from the actor policy when `state.to_move == learner_seat`;
- static opponent action otherwise.
Done states remain no-op through the engine, so the rollout can keep a fixed
time axis. Use `MAX_STEPS == 400` as the scan length unless a shorter config
value is explicitly introduced.
The first milestone is random-policy rollout with the full dashboard. This is
not throwaway; it defines the baseline canaries.
Log canaries:
- episode return;
- score difference;
- game length distribution;
- `max_steps` termination rate;
- `play_action_rate`;
- opened color count;
- positive expedition count per game;
- average entropy under legal-action masking.
Record the random-policy baseline in `README.md` before PPO training starts.
## PPO Details
Network:
- Input: `OBS_DIM` observation vector.
- Body: MLP 512 -> 512 -> 512, ReLU.
- Policy head: 96 logits.
- Value head: scalar.
- Illegal actions are masked to a large negative value before sampling and
before log-prob/loss computation.
Training defaults:
```yaml
ppo:
batch_games: 8192
rollout_steps: 400
gamma: 1.0
gae_lambda: 0.95
clip_epsilon: 0.2
entropy_coef: 0.01
value_coef: 0.5
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
potential_shaping_final: 0.0
potential_shaping_anneal_steps: 5_000_000
```
Terminal reward:
```text
tanh((learner_score - opponent_score) / terminal_scale)
```
Potential shaping:
```text
coef(t) * (board_score_diff_after - board_score_diff_before)
```
The shaping coefficient anneals linearly from `initial` to `final`. Expose all
four shaping fields in config. This shaping exists to prevent the previous
play-action-rate collapse by giving immediate credit for board progress; its
schedule is a controlled experiment variable, not a hidden constant.
## Evaluation Gates
Evaluation uses a fixed shuffle bank:
- Generate 10,000 explicit `deck_order` permutations from a fixed seed.
- For each deck, play twice:
- learner as player 0, opponent as player 1;
- opponent as player 0, learner as player 1.
- Aggregate duplicate-pair results.
Report:
- win rate;
- Wilson confidence interval;
- mean score difference;
- mean game length;
- positive expedition count per game;
- opened colors;
- `play_action_rate`.
Gates:
| Gate | Opponent | Pass condition | Diagnosis if failed |
| --- | --- | --- | --- |
| 1 | `discard_only` | win rate >= 90% and >= 2 positive expeditions/game | Audit reward pipeline, observations, and masks. Self-play is irrelevant. |
| 2 | `heuristic_balanced` | mean score difference > 0 | Learner works; tune shaping schedule and batch/variance. |
| 3 | `heuristic_cautious` | mean score difference > 0 | Same as gate 2; static ladder still not cleared. |
Only after all three gates pass should league self-play begin.
## 2026-07-04 Gate Results
All three static-opponent gates passed on an RTX 3090 using optional CUDA JAX:
```bash
uv run --with 'jax[cuda12]' lost-cities-jax-ppo train --config <config>
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval --config <config> \
--checkpoint <run>/latest --games 10000 --duplicate --output <eval.json>
```
Training used `batch_games=8192`, `rollout_steps=400`, `total_updates=250`,
`seed=20260704`, and artifact root
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
The implementation commit used for the runs was `4c0c2e9`.
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Artifact directories:
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/`
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/`
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/`
Final training rollout canaries:
| Opponent | Final train return_mean | Final play_action_rate | Final max_steps_rate |
| --- | ---: | ---: | ---: |
| `discard_only` | 0.99281 | 0.59007 | 0.00000 |
| `heuristic_balanced` | 0.93545 | 0.37737 | 0.00415 |
| `heuristic_cautious` | 0.97101 | 0.37473 | 0.00391 |
The balanced and cautious policies win decisively but produce longer games than
the discard-only diagnostic. That is not a gate failure, but the next training
phase should keep game length and forced-end rate as canaries.
## Artifact Policy
Generated training output stays out of git. Use:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/
```
Per gate, preserve:
- checkpoint directory;
- resolved config;
- duplicate evaluation JSON;
- metrics JSONL;
- short summary markdown with command, git commit, seed, and pass/fail result.
Commit only small, durable metadata to the code repo:
- config files;
- evaluator/training code;
- README baseline and gate summaries;
- optional dated summary under docs/reports.
If binary artifact tracking is later required, add DVC/Git LFS explicitly
instead of committing large checkpoint files directly.
## Work Order
### Phase 1: Rollout Dashboard
1. Add static opponent policies.
2. Add random-policy batched rollout.
3. Log canary metrics:
returns, game length, max-step rate, `play_action_rate`, opened colors,
positive expeditions.
4. Run CPU tests and one GPU random rollout.
5. Record the random baseline in README.
Exit criteria:
- deterministic smoke rollout passes;
- canary metrics look finite and stable;
- `play_action_rate` and game length are logged before any PPO code is judged.
### Phase 2: PPO Against `discard_only`
1. Add MLP actor-critic and masked action sampling.
2. Add GAE and PPO update.
3. Add Orbax checkpoints and resume.
4. Add CLI and the discard-only config under configs/jax_ppo.
5. Train on GPU.
First-run checks:
- with shaping enabled, `play_action_rate` rises above random baseline within
the first few hundred thousand environment steps;
- average game length converges near natural 44-ply games;
- `max_steps` termination rate goes to zero.
Then anneal shaping and run duplicate evaluation for gate 1.
### Phase 3: Static Ladder
1. Reuse the same config and checkpoint flow.
2. Change only the opponent config for `heuristic_balanced`.
3. Train/evaluate until gate 2 passes or diagnostics point to variance.
4. Repeat for `heuristic_cautious`.
No self-play work starts before gate 3 passes.
## Commands
Smoke rollout:
```bash
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/discard-only.yaml
```
GPU training in tmux:
```bash
tmux new-session -s coolrl-jax-ppo-discard \
-c /home/coolguy/dev/coolrl-lost-cities \
"flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
--config configs/jax_ppo/discard-only.yaml"
```
Duplicate evaluation:
```bash
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
--config configs/jax_ppo/discard-only.yaml \
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/latest \
--opponent discard_only \
--games 10000 \
--duplicate \
--output /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/eval_duplicate.json
```
## Risks
- The engine is fast on GPU, but Python logging/evaluation can dominate if
metrics are copied every step. Aggregate in JAX and transfer per rollout.
- Static heuristics can accidentally become too weak or too strong. Keep them
deterministic and versioned by config.
- Potential shaping can teach score-chasing artifacts if it never anneals.
Treat the coefficient schedule as part of the experiment identity.
- Checkpoint volume can grow quickly. Store large artifacts under `/mnt/2tbhdd`
and keep only `latest` plus gate checkpoints unless a run is explicitly
archived.
## Current Next Action
The static-opponent ladder is complete. The next eligible phase is snapshot-pool
league self-play design, using the three passed checkpoints as initial anchors.
docs/plans/lost-cities-classic-3round.md
+583
View File
@@ -0,0 +1,583 @@
# Plan: 클래식 3라운드 로스트시티 에이전트
**Status:** Ready to execute (Fable 2차 검토 반영 완료)
**Priority:** High — 현재 스택은 **단판(1라운드)** 게임을 학습하는데, 원작 로스트시티는
**3라운드 합산**이 승패를 가른다. 즉 지금 에이전트는 애초에 다른 게임을 배우고 있다.
**Background:** Fable 설계 검토 2회 (2026-07-15), 인간 대 AI 기보
(`data/human-play/game-records.jsonl`, 계속 증가 중)
## 목표
**원작 규칙 그대로의 로스트시티에서 가장 강한 에이전트.** 레거시 코드 호환은 고려하지 않는다.
env/obs/보상/학습 루프 재작성 모두 허용.
## 확정된 규칙 (룰북 원문 검증 완료)
코스모스 공식 영문 룰북(691820-02) 및 Board Game Arena 구현체로 교차 확인:
- 3라운드를 두고 **누적 총점**이 높은 쪽이 매치 승리.
- 1라운드 선공: "가장 나이 많은 사람" → 게임 내적으로는 **임의**.
- **2·3라운드 선공: 누적 점수가 더 많은 쪽** ("The player who has more points begins.")
— 번갈아 가는 것이 아니다.
- **정확한 동점일 때의 선공은 룰북에 없다.** 공식 룰북·BGA 모두 침묵.
→ **무작위(동전 던지기)로 정한다.** 결정론적 규칙은 대칭 제로섬 셀프플레이에
자리(seat) 비대칭을 주입해 착취 가능한 구멍이 된다.
점수 계산(`(랭크합 − 20) × (1 + 악수) + (8장 이상 +20)`)과 "방금 버린 카드는 즉시
회수 불가"(`just_discarded`)는 **현재 엔진이 이미 정확하다**
(`engine.py:244`, `engine.py:124`).
### 선공이 중요한 이유: 덱 시계
라운드는 **덱의 마지막 카드를 뽑는 순간** 끝난다. 카드는 놓은 뒤에 뽑으므로 마지막
카드를 뽑은 쪽은 그 카드를 쓰지 못한다. 총 턴 수가 홀수면 **선공이 한 장 더 놓는다**.
총 턴 수 = 44 + (버림패 드로우 횟수)이므로 **플레이어가 홀짝을 조작할 수 있다**
(실측 게임당 버림패 드로우 ≈6.6회). 선공권이 누적 점수에 달려 있으므로, 2라운드
마진에는 **점수를 넘어 3라운드 선공권이라는 추가 가치**가 붙는다.
### 측정된 사실: stalling은 정적 상대 착취이지 균형이 아니다 (Phase 1 사이징의 근거)
버림패 드로우는 덱을 줄이지 않으므로, **양쪽이 계속 버림패만 뒤지면 라운드가 끝나지
않는다.** 기존 학습 로그를 보면 이 착취가 실제로 학습된다:
| 학습 체제 | 라운드 길이 추이 | 400수 상한 도달 |
|---|---|---|
| 정적 휴리스틱 상대 (`balanced.yaml`) | 82 → **116수** (eval greedy **168수**) | 0.4~0.7% |
| **셀프플레이 (league)** | 71 → **53.6수로 수렴** | **0%** |
정적 약체 상대에게는 턴을 늘릴수록 원정 깊이·8장 보너스에서 강자가 이득이라 질질 끄는
것이 합리적이다. 그러나 **셀프플레이에서는 상대도 똑같이 끌 수 있어 상쇄되고, 라운드가
자연 길이(≈54수 = 44 덱 드로우 + ~10 버림패 드로우)로 수렴한다.**
**결론:** 우리의 목표 체제는 셀프플레이/league이므로 **3라운드 매치 ≈ 160수**로 잡으면
되고, 기존 `rollout_steps=400` 안에 여유롭게 들어간다. 단 두 가지를 유의한다:
1. **정적 상대로 3라운드를 학습시키면 매치가 350~500수까지 늘어날 수 있다** — 정적 상대
워밍업 단계를 쓴다면 스캔 길이를 따로 잡아야 한다.
2. **라운드 상한(~120수)은 그 자체가 착취 가능한 구멍이다.** 앞선 쪽이 상한까지 끌면
라운드가 유리한 상태로 얼어붙는다. `max_steps_rate`를 게이트 지표로 계속 감시할 것
(셀프플레이에서는 0이어야 정상).
## 현재 코드의 결함 (Fable 2회 교차 검증 — 8건 전부 사실 확인)
1. **env가 1라운드짜리** — `State`에 라운드/누적 점수 없음 (`types.py:46`),
`to_move=0` 하드코딩 (`engine.py:63`).
2. **롤아웃의 87%가 죽은 스텝** — 한 라운드는 실측 평균 50.6수(최대 56)인데
`rollout_steps = MAX_STEPS = 400` (`ppo.py:66`)이고, 스캔 **안에는 리셋이 없다**
(`reset_done_envs`는 업데이트 사이에서만, `ppo.py:275`).
3. **GAE 부트스트랩 초기값이 0** (`ppo.py:691`). 지금은 모든 에피소드가 스캔 안에서 끝나
무해하나, **in-scan 리셋을 켜는 순간 편향**이 된다.
4. **shaping 어닐링 회계 버그** — `update * batch_games * rollout_steps` (`ppo.py:222`)로
**죽은 스텝까지 세어서**, 5M 어닐링이 250 업데이트 중 2번째에 끝난다. league는 shaping을
아예 끈다 (`league.py:592`).
→ **"shaping은 불필요하다"는 과거 결론이 있다면 무효다. 켜진 적이 없다.**
5. **obs에 `to_move`가 없다.** 그런데 value loss는 `active` 마스크로 학습되어 (`ppo.py:624`)
**상대 차례 상태에서도** 가치를 맞추라고 요구한다. critic이 차례를 구분할 단서가
`step_count / 400`뿐 — MLP에게 부동소수점에서 홀짝을 뽑으라는 요구다.
6. **점수차 정규화가 ÷780** (`MAX_ABS_SCORE`, `obs.py:68`). ±50점 차가 ±0.06 → 사실상
안 보인다.
7. **셀프플레이에서 상대 자리 결정을 전부 버린다** (`stop_gradient`, `ppo.py:425`) —
의사결정의 절반을 낭비.
8. **평가가 단판 승률** (`gates.py`) — 3라운드 개편의 성패를 측정할 자가 없다.
## 핵심 설계 결정
### PRNG: 모든 무작위성을 `reset()`에서 미리 뽑는다 (필수)
라운드 전환 리셔플과 동점 코인플립을 `step()` 안에서 샘플링하면 `step`이 확률적이 되어
서명이 바뀌고, 모든 rollout/eval/gates 바디에 키를 꿰어야 하며 duplicate 미러링이 꼬인다.
**`reset()`에서 3라운드 덱 순서 전부(3×60), 동점 코인플립 비트 2개, 1라운드 선공 비트
1개를 미리 샘플링해 `State`에 저장한다.** `step()`은 결정론을 유지하고 라운드 전환은
다음 덱 슬라이스로 스위치만 한다. 덤: **미러 매치(같은 3딜 + 자리 교대 + 같은 코인플립
비트)가 공짜로 따라온다** — Phase 5의 antithetic 페어링이 그대로 성립.
### 라운드 전환 vs 매치 리셋의 분리
- **라운드 전환은 엔진 `step()` 내부**에서 (`done=False` 유지, carry 갱신, 덱 슬라이스 전환)
- **매치 done 리셋은 rollout 바디**에서 (in-scan auto-reset)
이러면 두 메커니즘이 깨끗이 분리된다. 단 위의 pre-sampled 덱 설계가 전제다.
- `State`에 **라운드 스텝 카운터와 매치 스텝 카운터를 둘 다** 둔다 (forced-done은 라운드
단위, obs 정규화는 라운드 상대 진행도 + round 원-핫).
- 중간 라운드 forced-done(라운드 상한 초과) 시 **자연 종료와 동일하게 라운드 종료 → 전환**.
### 보상 스케일에 대한 해소 (혼동 방지)
`sign()`에 가까운 작은 scale은 **틀린 선택**이다 — 150수에 걸쳐 모든 수가 동일한 ±1을
받아 크레딧 할당이 전부 critic에 떠넘겨지고, 5점 차 패배와 80점 차 패배의 그래디언트가
같아진다. **scale 30~50의 tanh가 절충점**이다: |마진| ≳ 60에서는 `sign`을 근사하면서
크레딧 그래디언트를 보존한다. 대신 5점 승과 80점 승을 여전히 구분하므로 순수 승률
최적화 대비 **마진 쪽으로 약간 왜곡**된다 — 이 잔여 왜곡은 Phase 5의 후기 fine-tune으로
scale을 낮춰서(예: 50 → 25) 제거한다.
명시적 risk 항은 **넣지 않는다.** carry + round를 obs에 넣은 종료 보상이 올바른 리스크
태도(뒤지면 도박, 앞서면 잠금)를 자동으로 유도한다.
## 체크리스트
### Phase 0a — 학습 루프 수정 (최우선, 단판 체제에서 검증 가능)
- [x] **GAE 부트스트랩 수정**: 잘린 에피소드를 `V(s_final)`로 부트스트랩 (`ppo.py:691`).
- [x] **in-scan auto-reset**: 롤아웃 스캔 안에서 done env 리셋 → 죽은 스텝 제거.
**실측(rollout_steps=400, batch_games=64, 3 업데이트):** main은 활성 17.3%
(learner 액션 2,221/업데이트), 수정 후 **활성 100% (12,842/업데이트) — 동일 연산량에
샘플 5.8배.** 활성 비율은 게임 길이에 비례하므로, 학습 초기 정책(~69수)에서 5.8배,
학습된 정책(~50수)이면 8배에 가까워진다. 3라운드 매치(~150수)가 되면 약 2.7배로
줄지만 그때는 스캔 전체가 실제 매치로 채워진다.
- [x] **메트릭 파이프라인 재작업 (필수, 놓치면 조용히 오염됨)**: `rollout_metrics`는
`final_env`에서 통계를 읽고 `episode_return = jnp.sum(transitions.reward, axis=0)`은
**env당 에피소드 1개를 가정**한다 (`ppo.py:1069`). in-scan 리셋 후에는 final_env가
에피소드 중간이고 여러 에피소드의 보상이 합산된다 → **metrics.jsonl 전체가 쓰레기가
된다.** done 경계에서 누적하는 에피소드 단위 집계로 바꿀 것.
- [x] **league assignments를 scan carry로**: `learner_seat`/`opponent_index`/`use_mirror`가
업데이트 사이에서만 재샘플링된다 (`ppo.py:568`). in-scan 리셋을 켜면 리셋 시점에
스캔 내부에서 재샘플링해야 한다 — 안 하면 자리/상대가 에피소드 간 고정되어 편향.
- [x] **shaping 어닐링 회계 수정**: 패딩된 스캔 스텝이 아니라 **learner 액션 수** 기준
(`ppo.py:222`). `shaping_coef`는 호스트에서 계산해 jit 함수에 넘기므로
(`ppo.py:221`), 직전 업데이트의 액션 수를 호스트에서 누적하는 **1-업데이트 지연**
구조가 된다 (무해).
- [x] **회귀 게이트 — 통과 (단, 게이트 설계를 고쳐야 했다).**
**(1) "동일 learner-액션 예산"은 잘못된 게이트였다.** PPO 진척도는 샘플 수와
**옵티마이저 스텝 수** 둘 다에 달려 있다. 샘플을 맞추면(64 vs 250 업데이트) 새 코드는
그래디언트 스텝이 1/4이 되어 덜 수렴한다. 업데이트당 연산량은 동일하므로
**올바른 게이트는 250 대 250**(같은 컴퓨트·같은 그래디언트 스텝, 샘플만 5.8배)이다.
**(2) 동일 컴퓨트 게이트 결과 (2만판 duplicate eval):**
| 앵커 | 지표 | 베이스라인 | Phase 0a |
|---|---|---|---|
| **expert** (강함) | 승률 | 0.4525 | **0.5071** |
| **expert** | 평균 점수차 | −4.28 | **−0.56** |
| balanced (약함) | 승률 | 0.9866 | 0.9292 |
| balanced | 평균 점수차 | 116.8 | **128.1** |
학습 곡선은 **모든 업데이트 지점에서** 새 코드가 위(최종 return 0.963 vs 0.935).
**(3) balanced 승률 하락은 전부 `MAX_STEPS=400` 아티팩트다.** 새 정책 2,048판 분해:
| | 판수 | 패배율 |
|---|---|---|
| 자연 종료 | 1,214 | **0.08%** |
| 400수 상한 도달 | 834 (40.7%) | **20.62%** |
**전체 패배의 99.4%가 상한에 부딪힌 판에서 발생.** 새 정책은 자연 종료 게임에서
**99.92% 승률**이다. 샘플이 5.8배라 stalling 착취를 더 깊이 배웠고(greedy 225수),
그 결과 40%가 인위적 벽에 박아 원정 미완성인 채 얼어붙는다.
**결론:** Phase 0a는 성공. expert(=stalling을 허용하지 않는 앵커)에서 45%→51%로
실제 실력이 올랐다.
### Phase 1 — 3라운드 env
**구현: `src/lost_cities_jax/match.py` (커밋 `7cd299d`).** 단판 엔진을 고치지 않고 **감쌌다**
— `engine.py`는 TS 클라이언트와의 차분 테스트가 지키는 규칙 오라클이고, 라운드 자체는
매치가 생겨도 변하지 않는다. `MatchState`가 내부에 단판 `State`를 든다.
- [x] `MatchState`: `round`(단판 State), `deck_orders`(3×60), `coin_flips`(3,),
`round_idx`, `carry`(2, 플레이어별 누적 점수), `done`.
별도 매치 스텝 카운터는 두지 않았다 — 매치는 3라운드가 끝나면 종료하므로 종료 판정에
불필요하고, 라운드 진행도는 내부 `State.step_count`(라운드마다 리셋)가 이미 준다.
- [x] `match_step()`: 덱 소진 시 라운드 < 3이면 carry 갱신 → 다음 덱 슬라이스 → `round_idx += 1`,
`done` 유지. 라운드 == 3이면 `done = True`. **보상은 3라운드 끝에서만.**
- [x] **선공 규칙**: `carry[0] > carry[1]` → p0, `<` → p1, `==` → pre-sampled 코인플립.
1라운드는 특수 케이스가 **필요 없다** — carry가 (0,0)이라 동점 분기가 자동으로 코인을
뽑고, 그게 룰북의 "가장 나이 많은 사람"(= 임의)과 정확히 같다.
- [x] **PRNG**: 딜 3벌 + 코인 3개를 `match_reset`에서 전부 미리 뽑아 상태에 저장.
`match_step`은 결정론적이라 키를 rollout/eval/gates에 꿸 필요가 없고, **미러 매치가
자리 라벨 교체만으로 성립**한다 (Phase 5 antithetic 페어링이 여기 의존).
- [x] ~~라운드당 스텝 상한을 400 → ~120으로 분리~~ — **철회한다. 라운드 상한은 400을 유지.**
**이 항목의 원래 명분이 사라졌다.** 근거는 "3라운드(~360수)가 400스텝 스캔에 들어가게"
였는데, **Phase 0a의 GAE 절단 부트스트랩이 스캔 길이 제약을 없앴다.** 잘린 매치는
올바르게 부트스트랩되므로 스캔 길이는 이제 자유 파라미터다.
**그리고 상한을 낮추면 아티팩트가 악화된다.** Phase 0a 실측: 400수 벽에 부딪힌 판의
20.6%가 패배로 뒤집혔고 **전체 패배의 99.4%가 거기서 발생**했다. 상한을 120으로 내리면
**더 많은 판이 벽에 박고**, "앞선 쪽이 상한까지 끌어 라운드를 얼린다"는 착취가 훨씬
쉬워진다. 셀프플레이에서는 라운드가 54수로 수렴해 120이든 400이든 안 걸리므로,
**덜 걸리는 쪽을 두고 `max_steps_rate`를 감시**하는 것이 맞다.
- [ ] **학습 파이프라인 배선** (원래 계획서에서 누락된 항목): `ppo.py`의 rollout/eval,
`gates.py`, `league.py`가 전부 단판 `State`를 받는다. `MatchState`를 받도록 배선한다.
- 매치용 obs 래퍼: 우선 `observation(state.round, player)`로 carry-blind하게 연결
(carry/round 피처는 Phase 3에서 추가).
- `match_legal_action_mask`, `match_step`, `match_score`로 교체.
- 스캔 길이는 자유 파라미터가 되었으므로 셀프플레이 매치 길이(~160수) 기준으로 잡되,
잘려도 부트스트랩이 처리한다.
- [ ] **league 풀 재구축**: 기존 스냅샷 멤버는 obs 변경으로 전부 무효화된다 — Phase 3 이후로
미룬다 (obs가 확정되기 전에 풀을 다시 채우면 두 번 일한다).
- [x] **테스트 12개** (`tests/lost_cities_jax/test_match.py`): 라운드당 덱 드로우 = 44 +
버림패 드로우만큼 연장, `carry`가 각 라운드를 정확히 한 번씩 적립, 선공 규칙 3분기,
1라운드 코인 공정성(≈50/50), 라운드 경계에서 누적 점수 불연속 없음(PBRS 전제),
미러 매치 대칭성, 보상은 3라운드 끝에서만 지급.
### Phase 0b — 측정 자 (첫 장기 3라운드 학습 **전에** 착지)
- [ ] **매치 단위 평가**: 3딜 전부 미러링 + 자리 교대 + **동일 코인플립 비트**,
매치 승률 + Wilson CI, 평균 총 마진. `gates.py`/`league.py`의 단판 기준을 대체.
**`gates.py:884`, `gates.py:961`의 `MAX_STEPS` 길이 단판 스캔 2곳 포팅 포함** —
exploiter 학습·평가 경로 전체가 3라운드로 가야 성공 기준 3을 잴 수 있다.
- [ ] **carry 조건부 프로브**: carry ∈ {−60, −25, −1, +1, +25, +60}을 주입한 3라운드 시작
위치에서 승률·행동 변화(원정 개수, 악수 비율, 덱 레이스 비율) 측정.
- [ ] **shuffle bank 포맷 확장**: 덱과 함께 코인플립·선공 비트도 뽑도록. 안 그러면
`jax.random.PRNGKey(0)` 고정 eval(`ppo.py:1025`)의 재현성이 매치 정의와 얽힌다.
- [ ] **legacy obs 버전 보존**: 구 정책을 3라운드 매치에 투입해 베이스라인을 재려면 구
obs(454차원)로 추론해야 하는데, `checkpoint_policy_from_params`는 전역 `observation`을
호출한다 (`ppo.py:802`) → obs 개편 후 구 체크포인트는 **로드 자체가 실패**한다.
policy 로더가 체크포인트별 obs 버전을 받도록 할 것. **성공 기준 2의 분모가 여기 달렸다.**
### Phase 2 — 보상
- [ ] 종료 보상: 3라운드 끝에서만 `tanh(총_점수차 / scale)`, **scale = 30~50**
(근거는 위 "보상 스케일에 대한 해소" 절).
- [ ] 마진 shaping 부활: `Φ(s) = carry + 현재 보드 점수차`, **작은 계수**(종료 보상 스케일의
0.05~0.2). 현재의 계수 1.0 원점수 shaping은 ±1 종료 보상보다 10~30배 크다.
learner 스텝 기준으로 어닐링하되 **후반까지 정확히 0으로 내리지 않는다.**
- [ ] γ=1.0에서 `Φ(s') − Φ(s)`는 **올바른 PBRS 형태다** (γ 누락 아님 — 검토에서 확인).
### Phase 3 — 관측
- [ ] `carry`: **÷75 스칼라 + 구간 원-핫**(약 9구간). 3라운드 정책은 "1점만 더" 임계값
근처에서 급격히 꺾여야 한다. 기존 `score_diff`의 ÷780 정규화도 같이 고친다.
- [ ] `round_idx` 원-핫 + 남은 라운드 수.
- [ ] **`to_move` 비트** + "내가 이번 라운드 선공인가" + "현재 홀짝에서 마지막 덱 카드를
누가 뽑는가"(덱 시계).
- [ ] 색깔별 **살아있는 점수 3분할**: 내 `col_top` 위로 아직 나올 수 있는 점수를
(a) **내 손패**, (b) **버림패 더미**(공개돼 있고 회수 가능 — 빠뜨리기 쉬움),
(c) **미공개**(덱 ∪ 상대 은닉 손패)로 나눠 넣는다. 상대에 대해서도 동일
(상대 `col_top`은 공개). MLP가 7×60 채널에서 뽑아내기 어려운 비선형 집계이고,
모든 개시/연장/차단 판단을 좌우한다.
### Phase 4 — 학습 효율
- [ ] **전지적 critic (CTDE)**: critic에만 상대 손패 + 덱 구성을 준다. 행동과 무관한
정보이므로 정책 그래디언트를 편향시키지 않는다. 딜 운 분산을 정면으로 깎는다.
**가치 경로를 정책 트렁크에서 분리해야 한다** (현재 공유 트렁크, `ppo.py:104`) —
특권 정보가 정책 로짓으로 새면 안 된다.
→ 이 critic은 나중에 **PIMC 탐색의 리프 평가기로 그대로 재활용**된다.
- [ ] **양쪽 자리 학습**: `stop_gradient`된 상대 자리 전이(`ppo.py:425`)도 학습에 쓴다
(샘플 효율 2배). 같은 게임의 두 자리는 반상관이므로 같은 배치에 두고 advantage
정규화에 맡긴다.
- [ ] `gae_lambda` 재검토: 0.95는 150수 지평에서 너무 짧다(중반 수 직접 가중치 0.02).
전지적 critic이 있으면 유지, 없으면 0.97~0.99. **λ=1은 금지**(딜 분산).
- [ ] `entropy_coef` **스윕으로 재결정**. (주의: "0.01이 원점수 shaping 기준으로 잡혔다"는
추론은 **틀렸다** — league는 shaping을 끄고 학습했으므로 0.01은 이미 ±1 tanh 체제에서
동작해온 값이다. 다만 3라운드에서 보상 빈도가 1/150로 희석되고 작은 shaping이
추가되므로 재튜닝 자체는 타당하다. **근거 없이 10배 낮추면 과소탐색으로 직행한다.**)
### Phase 5 (나중) — 최종 강함
- [ ] **후기 fine-tune**: 학습 말미에 종료 보상 scale을 낮춰(50 → 25) 마진 왜곡을 제거하고
순수 승률 쪽으로 당긴다.
- [ ] 페어드 antithetic 딜(같은 3딜 + 자리 교대 + 같은 코인플립 비트)을 **학습에** 도입.
단순 포함이 아니라 **쌍으로 묶어** control variate로 써야 효과가 있다.
(Phase 1의 pre-sampled PRNG 설계 덕에 사실상 공짜.)
- [ ] MMD식 정규화 셀프플레이 (loss에 ~20줄) — 2인 제로섬 근사 내시 보험.
- [ ] **추론 시점 탐색 (PIMC / ISMCTS)** — 최종 강함의 가장 큰 이득. 로스트시티는 블러핑
경제가 없는 저기만성 불완전정보 게임이라 결정화 탐색이 잘 맞는다. raw net 대
net+search 맞대결로 측정.
- [ ] 네트워크 용량 A/B (512×3 → 1024×3 또는 residual) — **파이프라인 변경이 끝난 뒤에.**
## 범위 밖 (명시적 동결)
- **웹 클라이언트/ONNX는 별도 계획 전까지 레거시 단판 모델로 동결한다.** obs 개편 즉시
export 파이프라인(`scripts/export_jax_ppo_onnx.py:70`, manifest `observation_size: 454`),
TS obs 빌더, TS 단판 엔진이 전부 비호환이 된다. 이 선언이 없으면 실행 중 스코프가
웹 재작성으로 샌다.
- **Deep CFR 복귀** — 이미 BC 천장을 쳤고, 3라운드는 트리만 키운다. PPO+league를 학습
백본으로 유지한다.
- 레거시 호환을 위한 타협.
## 성공 기준
0. **(Phase 0a 게이트)** 루프 수정 후 단판 체제에서 동일 learner-액션 예산으로 기존 anchor
성적 재현 ≥ 동등. **없으면 auto-reset 버그가 3라운드 결과에 섞여 원인 분리가 불가능해진다.**
1. **carry 프로브에서 행동이 단조롭게 변한다** — carry 6개 수준에 걸쳐 원정 개수/악수 비율의
**단조 추세**(CI 포함). ("행동이 변한다"는 노이즈로도 통과 가능하므로 단조성으로 정의.)
2. **매치 승률이 구 정책(3라운드에 그대로 투입)을 이긴다** — 페어드 매치 **≥ 1만 쌍**,
매치 승률 **Wilson 하한 > 0.5** *및* 평균 총 마진 **CI 하한 > 0**.
3. 착취자(exploiter) 승률이 악화되지 않는다.
4. **(anchor 비회귀)** 휴리스틱 anchor 상대 매치 승률·라운드당 마진이 구 정책 대비 악화되지
않는다. **기준 2의 구멍을 막는 항목**: carry-blind인 구 정책만 상대로 이기는 것은
**카드 플레이가 퇴보해도 carry 착취만으로 달성 가능**하다.
**단, anchor 선택에 주의:** Phase 0a에서 실증됐듯 **약한 anchor(`heuristic_balanced`)는
무한 stalling을 허용해서 신호를 오염시킨다** — 승률이 실력이 아니라 상한 도달률을 잰다.
**`heuristic_expert`처럼 stalling을 허용하지 않는 anchor**(게임이 자연 길이로 끝남)만
실력 지표로 쓰고, 약한 anchor는 **자연 종료 게임만 필터링해서** 보거나 `max_steps_rate`를
함께 보고할 것.
---
## 실행 결과 (2026-07-15, 커밋 `9ba5a07`)
### 동작하는 것
- **셀프플레이가 stalling을 스스로 제거한다.** 매치가 146.8수(라운드 ≈49수)로 수렴하고
덱 레이스 비율 91%. Phase 0a에서 정적 상대가 225수까지 끌던 것과 대조된다.
- **미러 매치 평가가 정확하다.** 셀프플레이 duplicate 승률 0.4968, **평균 lead 정확히 0.0** —
같은 3딜 + 같은 코인 + 자리 교대가 딜 운을 완전히 상쇄한다. 미완료 0%.
- **전지적 critic이 격리돼 있다.** 특권 입력을 흔들어도 정책 로짓은 비트 단위로 동일하고,
가치만 움직인다 (테스트로 고정).
### 성공 기준 1 — **약하게만 충족. 사실상 미달.**
carry 프로브(3라운드 시작 시 carry 주입)에서:
| | scale 50 | scale 25 | scale 12 |
|---|---|---|---|
| 악수 스프레드 (carry −60 → +60) | 0.19 | 0.08 | **0.31** |
| 원정 개수 | 5.00 고정 | 4.99 고정 | **5.00 고정** |
- **악수 사용은 6개 carry 수준 전체에서 단조**로 움직인다 (뒤지면 배수 베팅 ↑). 방향은 맞다.
- 그러나 **원정 개수는 carry와 무관하게 5.00에 붙어 있다.** 굳은 습관이지 조건부 플레이가 아니다.
- 효과 크기가 작다.
### 확정된 진단
1. **shaping 가설은 틀렸다.** `potential_shaping_final=0.0`으로 완전히 꺼도 프로브는 평평했다.
2. **원인은 `terminal_scale`이다.** carry = −60에서 `tanh((margin − 60)/50)`의 인자는 현실적
마진 범위에서 **거의 선형**이고, 선형 구간에서 `E[tanh]` 최대화는 `E[margin]` 최대화와 같다
→ 도박할 이유가 없다. 위험 추구는 tanh가 **강하게 볼록한** 구간에서만 나오며 scale을 낮춰야
그 구간에 들어간다. scale 50 → 12에서 스프레드가 0.19 → 0.31로 커진 것이 이를 확인한다.
3. **프로브 설계 결함:** scale 12에서 `tanh(60/12) ≈ 1.0`이라 **±60은 포화 = 그래디언트 0**.
거기서 정책은 학습된 바가 없다. **의미 있는 측정 구간은 `|carry| ≲ 2 × terminal_scale`.**
프로브 수준을 scale에 맞춰 재설계해야 한다.
### 다음에 할 일
- [ ] **프로브 재설계**: carry 수준을 `terminal_scale`에 맞춰 잡는다 (포화 구간 측정 금지).
- [ ] **학습 중 carry 분포 확인**: 셀프플레이에서 3라운드 진입 시 carry가 실제로 얼마나
퍼지는가. 좁으면 정책이 큰 deficit을 본 적이 없다는 뜻이고, 그게 진짜 원인일 수 있다.
- [ ] **"항상 5색 개시"가 정상인지 검증**: 인간 기보에서 AI는 4.81, 인간은 4.19를 열었다.
5.00 고정은 의심스럽다. 셀프플레이 균형인지, 탐색 붕괴인지 (엔트로피 3.5 → 1.03).
- [ ] scale 스케줄(50 → 12 후기 fine-tune)이 처음부터 12로 학습하는 것보다 나은지 A/B.
---
## 판정: 보상은 **선형 총점**이다 (2026-07-15, Fable 3차 검토 + 실측)
사용자 제안("그냥 3판 총점이 크기만 하면 되는 것 아니냐")이 **맞았다. tanh 종료 보상은
오버엔지니어링이었다.**
### 근거 1 — 분해 논증 (실측)
보상이 총점에 선형이면 라운드가 독립이므로 3라운드 게임이 3개의 독립 단판으로 분해된다.
셀프플레이 2,048매치 실측: `corr(m1, m2) = 0.004`, `corr(m1+m2, m3) = 0.05`.
라운드를 잇는 유일한 고리인 **선공 어드밴티지는 +0.73 ± 0.84점** — 0과 구분 불가.
### 근거 2 — 리스크 태도는 값어치가 없다 (실측)
Fable이 리스크 태도를 직접 구현해 greedy 클론과 duplicate 6,144판 맞대결:
| 도박 정책 | 매치 승률 |
|---|---|
| 3R에서 20점 이상 뒤지면 온도 샘플링 | **0.482** (진다) |
| 40점 이상 뒤질 때만 | 0.498 (본전) |
**일부러 도박을 시켜도 지거나 본전.** 로스트시티의 분산 레버(한계 악수 ≈ Δσ +1.7에 마진
−2~3점)가 근본적으로 약해서, 분산을 사는 비용이 볼록성 이득을 먹는다.
carry 조건부 플레이의 가치 상한: **1승점 미만.**
### 근거 3 — 맞대결에서 단순한 쪽이 **이겼다**
duplicate 10,000판 (같은 3딜 + 같은 코인 + 자리 교대), 동일 컴퓨트 300 업데이트:
| A | B | A 승률 | A 평균 총점차 |
|---|---|---|---|
| **선형 총점** | tanh(총점/12) | **0.5859** (CI 0.576–0.596) | **+20.3점** |
버려도 손해가 없는 게 아니라 **버리니 더 강해졌다.** 이유는 리스크가 아니라 **신호 밀도**다:
tanh는 ~150수 매치에 포화된 ±1 하나를 주고, 선형은 매 수마다 그 수가 총점차를 움직인 만큼을
준다. γ=1에서 후자의 합이 정확히 최종 총점차로 telescoping되므로 **목적함수는 동일한데 크레딧
할당만 150배 조밀**하다.
### 근거 4 — 탐색 붕괴가 아니었다
프로브가 평평했던 이유는 탐색 붕괴가 아니다. 샘플링 프로브에서도 원정 5.00 ± 0.05,
엔트로피 1.36나트(유효 행동 3.9개)로 정상 수렴. 그리고 **critic은 carry를 완벽히 읽고 있었다**
(3R 시작 가치 −0.87 → +0.86 단조). 신호는 있었고, **정책이 그걸로 살 수 있는 물건이
없었을 뿐**이다.
### 최종 설계
```python
# match_ppo.py 롤아웃 바디
reward0 = jnp.where(active, (after - before) / cfg.reward.terminal_scale, 0.0)
```
- **성공 기준 1(carry 프로브 단조성)은 게이트에서 제거한다.** 최적 반응 자체가 이 게임에서
거의 존재하지 않는다는 것이 측정 결과다. 기준 2·4(맞대결 + anchor 비회귀)가 옳은 자다.
- `carry`는 obs에 **남긴다** (비용 0, 선공 규칙이 키로 쓰는 상태, critic이 잘 읽음).
### 목적함수와 무관하게 살아남은 것 (전부 순이득)
in-scan auto-reset + GAE 절단 부트스트랩(샘플 5.8배), 메트릭 재작업, **비대칭 CTDE critic**,
**양쪽 좌석 학습**, **duplicate 매치 평가**, pre-sampled PRNG/미러 설계, 그리고 match_obs의
단판 결함 수정분(`to_move`, 덱 시계, ÷780 → ÷75, 살아있는 점수 3분할).
**죽은 것은 tanh 종료 보상과 carry 구간 원-핫뿐이다.**
---
## 신구 대결: 진짜 3라운드 클래식에서 (2026-07-15)
**질문: "기존 학습 방법 대비 실제로 뭐가 나아졌나?"**
duplicate 매치 8,192판(같은 3딜 + 같은 코인 + 자리 교대), 진짜 3라운드 게임:
| 신(매치 스택) | 상대 | 신 승률 | 신 평균 총점차 | 상대 learner 액션 |
|---|---|---|---|---|
| 39.3M | 기존 baseline (정적 상대 학습) | 0.6077 | +19.5 | 104M |
| 39.3M | Phase 0a 게이트 (루프 수정만) | **0.5842** | +16.0 | **411M (10.5배)** |
| 39.3M | league (셀프플레이, 웹 배포판) | 0.3142 | −37.0 | 122.6M (3.1배) |
| **131M** | **league (셀프플레이, 웹 배포판)** | **0.6094** (CI 0.599–0.620) | **+22.0** | 122.6M |
**결론:**
1. **샘플 효율이 크게 올랐다.** 39.3M짜리가 **411M(10.5배)짜리를 이긴다.**
2. **동일 예산에서 기존 최강(league)을 이긴다** — 131M vs 122.6M에서 0.6094.
3. 39.3M에서 league에 졌던 것(0.3142)은 **약해서가 아니라 예산이 1/3이어서**였다.
**남은 단서:** league는 **착취자(exploiter) 구조**를 포함한 런이다. 평균 강함은 우리가 이기지만
**exploitability는 아직 재지 않았다.** "덜 착취당한다"는 증명되지 않았다.
---
## 기여도 A/B (2026-07-15)
각 조각을 하나씩 끄고 동일 컴퓨트(300 업데이트, batch 512)로 학습시킨 뒤,
full 스택과 duplicate 매치 8,192판 맞대결. **0.5 미만 = 그 조각이 기여하고 있었다.**
| 제거한 것 | full 대비 승률 | 95% CI | 평균 총점차 | 판정 |
|---|---|---|---|---|
| **양쪽 좌석 학습** | **0.3317** | [0.322, 0.342] | −28.6 | **압도적 기여** |
| 매치 관측 (carry/to_move/덱시계/살아있는점수) | 0.4751 | [0.464, 0.486] | −4.0 | 유의하게 기여 |
| **전지적 critic (CTDE)** | **0.5160** | [0.505, 0.527] | +2.6 | **오히려 해가 된다** |
### 1. 양쪽 좌석 학습 — 가장 큰 기여
끄면 승률이 0.33으로 무너진다. 같은 업데이트 수에서 learner 액션이 **정확히 절반**이 되므로
상당 부분은 샘플 수 효과다 — 그러나 **그게 요점이다.** 셀프플레이에서 상대 좌석의 수는 같은
네트워크가 둔 것인데 기존 트레이너는 `stop_gradient`로 버렸다. **공짜로 데이터가 2배**가 된다.
### 2. 매치 관측 — 작지만 실재
0.4751 (CI 상한 0.486 < 0.5). `to_move`, 덱 시계, ÷780 → ÷75 스케일 수정, 살아있는 점수
3분할이 합쳐서 약 2.5%p 값어치. carry 자체는 (분해 논증대로) 거의 기여하지 않으므로,
이 이득의 대부분은 **단판 obs의 결함 수정분**으로 보인다.
### 3. 전지적 critic — **Fable의 최우선 권고가 틀렸다**
Fable은 이것을 "가장 큰 누락 아이디어"로 꼽았다. 실측은 **반대**다: 끄면 오히려
**0.5160 (CI 하한 0.505 > 0.5)** 으로 유의하게 **더 강해진다.**
그럴듯한 이유: 특권 critic은 덱을 알기 때문에 가치를 아주 잘 맞추지만, 그 결과
`V(전체상태) ≠ E[리턴 | 마스킹된 관측]`이 되어, **어드밴티지에 정책이 통제할 수 없는 성분**이
섞인다. 액터 입장에서 그건 분산 감소가 아니라 노이즈다. 부분관측 환경에서 비대칭 critic이
해가 되는 것은 알려진 현상이다.
**조치: 전지적 critic을 기본에서 끈다.** (PIMC 탐색의 리프 평가기로 재활용한다는 계획은
별개로 유효할 수 있으나, 정책 학습용으로는 손해다.)
### 종합: 실제로 값어치 있었던 것
1. **in-scan auto-reset + GAE 절단 부트스트랩** — 샘플 5.8배, expert 0.4525 → 0.5071
2. **양쪽 좌석 학습** — 데이터 2배, 단독으로 승률 0.33 → 0.5
3. **선형 총점 보상** — tanh 대비 0.5859 (사용자 제안)
4. **매치 관측 결함 수정** — 약 2.5%p
5. ~~전지적 critic~~ — **해가 된다. 끈다.**
### 정정: 전지적 critic의 가치는 **스케일에 따라 뒤집힌다**
위 A/B는 39.3M learner 액션 규모에서 돌렸다. 실전 규모(131M)에서 동일 설정으로 둘을 직접
맞붙이면 **결론이 반대로 나온다.**
| 규모 | critic OFF의 승률 (critic ON 상대) | 판정 |
|---|---|---|
| 39.3M (A/B 규모) | **0.5160** [0.505, 0.527] | OFF가 낫다 |
| **131M (실전 규모)** | **0.4633** [0.453, 0.474] | **ON이 낫다** |
**전지적 critic은 그것을 학습시킬 데이터가 충분해질 때 값어치를 한다.** 39.3M에서는 특권
정보를 쓰는 큰 가치 트렁크를 제대로 못 맞춰서 어드밴티지에 노이즈만 얹었고, 131M에서는
제대로 맞춰서 분산 감소가 실현된다.
**교훈: 배포할 예산이 아닌 규모에서 ablation을 돌리면 결론이 뒤집힐 수 있다.**
`privileged_critic` 기본값은 **ON으로 되돌린다.**
### 최종 순위 (진짜 3라운드 클래식, duplicate 8,192판)
| 모델 | learner 액션 | vs league |
|---|---|---|
| **매치 스택 (critic ON)** | 131M | **0.6094** (+22.0점) |
| 매치 스택 (critic OFF) | 131M | 0.5526 (+10.9점) |
| league (기존 최강, 웹 배포판) | 122.6M | — |
### 기여도 재측정: 동일 규모(131M)에서
39.3M A/B는 critic에게 가장 불리한 지점이었다. 두 조각을 **같은 131M 예산**에서 다시 쟀다:
| 조각 | full의 승률 (그 조각 뺀 버전 상대) | 순이득 |
|---|---|---|
| **양쪽 좌석 학습** | 0.6707 | **+17.1 %p** |
| **전지적 critic** | 0.5367 | **+3.7 %p** |
**양쪽 좌석이 약 4.6배 크다.** 그리고 성격이 다르다:
| 규모 | 양쪽 좌석 | 전지적 critic |
|---|---|---|
| 39.3M | +16.8 %p | **−1.6 %p** |
| 131M | **+17.1 %p** | +3.7 %p |
**양쪽 좌석은 규모와 무관하게 안정적**(+17%p). **critic은 규모를 탄다.**
이유: 양쪽 좌석은 **데이터를 2배로** 만든다(65.5M → 131M, 같은 컴퓨트에서). critic은 데이터를
안 늘리고 **채점 정확도만** 올리는데, 코치가 배울 게 많아져서 데이터가 부족하면 오히려
엉터리 채점을 한다.
**Fable은 critic을 "가장 큰 누락 아이디어" 1순위로, 양쪽 좌석을 4순위로 꼽았다. 정확히 거꾸로였다.**
---
## Exploitability: 우리가 league보다 **덜 착취당한다** (2026-07-15)
**남겨뒀던 단서를 해소했다.** 평균 실력에서 이기는 것과 "파먹을 약점이 없는 것"은 다른
속성이다. 가위바위보에서 바위를 60% 내는 놈은 아무한테나 이길 수도 있지만, 그 습관을
알아챈 놈에게는 매번 진다.
**측정 방법:** 정책을 얼려놓고, **오직 그놈만 이기도록 특화된 새 정책을 처음부터 학습**시킨다
(`src/lost_cities_jax/exploit.py`). 착취자가 도달한 승률이 곧 그 정책이 못 막아낸 습관의 크기다.
**동일 착취자 예산 (250 업데이트 × batch 1024 = 32.5M learner 액션):**
| 얼려놓은 정책 | 착취자 승률 | 95% CI |
|---|---|---|
| **우리 매치 스택 (131M)** | **0.2278** | [0.219, 0.237] |
| league (웹 배포판, 122.6M) | **0.3213** | [0.311, 0.331] |
**league가 9.4%p 더 털린다.** 그리고 둘 다 0.5에서 한참 멀다 — 어느 쪽도 만만한 상대는 아니다.
### 최종 정리: 두 축 모두에서 이긴다
| 축 | 결과 |
|---|---|
| **평균 실력** (정면 대결) | 0.6094, +22.0점 ✅ |
| **약점 적음** (exploitability) | 0.2278 vs 0.3213 ✅ |
### 단서
1. **하한선이다.** 더 세거나 더 오래 학습한 착취자는 더 찾아낼 수 있다. **절대값이 아니라
동일 예산에서의 비교로만 의미가 있다.**
2. **착취자 예산이 목표의 약 1/4에 불과하다 (32.5M vs 131M).** 약한 공격자다.
따라서 **"우리는 22.8%밖에 안 털린다"고 말할 수 없다** — 그건 이 약한 공격자 기준일 뿐이다.
살아남는 주장은 **"동일 예산에서 league가 더 털린다"** 하나뿐이다.
착취자를 목표와 같은 예산(131M)으로 키워서 **순서가 유지되는지** 확인해야 한다.
3. league는 단판 정책이라 3라운드 게임에선 다소 제 물이 아니다 — 다만 carry가 무의미하다는
것이 이미 증명됐으므로 큰 불리함은 아니다.
**흥미로운 점:** 우리는 **순수 셀프플레이**이고 league는 **착취자 구조를 학습에 넣은** 런인데,
그런데도 우리가 덜 착취당한다. 착취자 구조가 exploitability를 낮춰줄 것이라는 기대가
이 게임에서는 확인되지 않았다.
docs/plans/lost-cities-league-selfplay-spec.md
+139
View File
@@ -0,0 +1,139 @@
# Lost Cities JAX PPO Snapshot-Pool League Spec
**Status:** ready for implementation after ladder v2.
**Last updated:** 2026-07-05.
**Warm start:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`.
## Preconditions
The static-opponent ladder v2 has passed:
- `discard_only`: win rate 1.0000, mean score diff +182.0670.
- `heuristic_balanced`: win rate 0.9409, mean score diff +119.9250.
- `heuristic_expert`: win rate 0.8382, mean score diff +43.3099, score-diff
CI95 lower bound +42.7065.
Canary warnings remain:
- Gate-3 opened colors are 4.5509/game, not the desired 2-3 range.
- The v2 gate-3 checkpoint is exploitable: a 250-update PPO exploiter reached
win rate 0.83135 and mean score diff +54.7721 against it.
## Anchor Semantics
Anchors are not certificates of strength. They serve two narrower purposes:
- Fix the Elo scale so curves remain comparable over time.
- Provide style diversity so the league does not train only against recent
policy snapshots.
Strength certification is tracked on separate axes:
- Duplicate mean score difference against `heuristic_expert`.
- Exploiter win-rate trend under the fixed exploiter protocol.
- Later human play, once the automated diagnostics are stable.
## Initial Pool
Permanent anchors are never removed:
- Static anchors: `discard_only`, `heuristic_balanced`, `heuristic_cautious`,
`heuristic_expert`.
- Learned ladder anchors:
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest`
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest`
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`
The learning policy starts from the v2 gate-3 checkpoint. Shaping coefficient is
fixed at 0 by default. If warm-start collapse is observed, the implementation
may expose a config switch for random initialization plus shaping anneal, but
that is not the default path.
## Opponent Sampling
For each episode:
- Randomize the learner seat.
- With probability 50%, play mirror self-play against the current policy copy.
- With probability 50%, sample from the pool.
Pool sampling is PFSP-like:
- Track recent win rate versus each pool member.
- Weight non-anchor snapshots by `(1 - win_rate)^2`.
- Mix in a small uniform component to avoid starvation.
- Give every permanent anchor only a small floor probability, such as 2-3% per
anchor, so stalling or weak anchors do not waste most throughput.
- Allocate the remaining probability mass to snapshots according to the PFSP
weights.
## Snapshot Lifecycle
- Add the current policy to the pool at a configurable interval.
- If the pool size cap is reached, remove the oldest non-anchor snapshot.
- Never remove permanent anchors.
- Store large checkpoints and raw evaluation JSON under
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/`.
- Store one-line tracked JSON summaries under `docs/reports/` so results do
not disappear with artifact cleanup.
## Evaluation
At each snapshot interval, evaluate the current policy with fixed shuffle-bank
duplicate play against:
- The six ladder anchors.
- `heuristic_expert`.
- A configurable set of recent snapshots.
Update logistic Elo from the pairwise results, using the heuristic anchors as
fixed reference points. Elo is a scale diagnostic, not a strength certificate.
Run the exploiter protocol periodically against the current best checkpoint:
- Randomly initialized PPO.
- Same 250-update budget used in ladder v2 unless explicitly overridden.
- Duplicate evaluation versus the frozen target checkpoint.
- Track exploiter win rate and mean score difference over time.
## Canary Metrics
Log these metrics at every evaluation:
- Opened colors per game.
- Play action rate.
- Mean and quantile game length.
- Max-steps rate.
- Positive expeditions per game.
- Duplicate score-diff distribution against `heuristic_expert`.
Canaries are diagnostics only. Do not put them directly into the reward.
## Regression Guards
- `heuristic_expert` replaces `heuristic_cautious` as the main regression
opponent.
- If win rate versus `heuristic_expert` falls below 90% of the v2 gate-3 value,
flag the snapshot.
- If max-steps rate rises above the gate-3 baseline of 0.0 by a material
amount, flag the snapshot.
- If exploiter win rate rises above the v2 baseline of 0.83135, flag the
snapshot.
## Stop Conditions
Stop the league run when either:
- Elo is statistically flat over a configured recent-snapshot window and
exploiter win rate is at or below the configured threshold, for example 0.55.
- The wall-clock budget is exhausted.
## Decisions
- The earlier `heuristic_cautious` remains a permanent anchor for scale and
style diversity only. It is a stalling anchor and no longer certifies
strength.
- `discard_only` is also kept only as a low-probability anchor and regression
sanity check.
- The league spec file was not present in the repository when this update was
requested, so this tracked plan file is the authoritative spec location.
docs/reports/diminishing-returns-2026-07-05-expert.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 66 KiB

docs/reports/diminishing-returns-2026-07-05-exploiter.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 89 KiB

docs/reports/diminishing-returns-2026-07-05-h2h.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 57 KiB

docs/reports/diminishing-returns-2026-07-05-summary.jsonl
+17
View File
@@ -0,0 +1,17 @@
{"current": "league_v1_update_250", "current_label": "league v1 update 250", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_league_v1_update_250_vs_ladder_v2_gate3.json", "max_steps_rate": 0.0, "mean_game_length": 57.4935, "mean_score_diff": 42.87575, "opened_colors_per_game": 4.911, "play_action_rate": 0.6949448449578737, "positive_expeditions_per_game": 2.56825, "previous": "ladder_v2_gate3", "previous_label": "ladder v2 gate3", "score_diff_ci95_high": 44.43765342421665, "score_diff_ci95_low": 41.313846575783344, "score_diff_std": 50.40064352917072, "wilson_high": 0.8135690832007875, "wilson_low": 0.7888523724955226, "win_rate": 0.8015}
{"current": "league_v1_update_500", "current_label": "league v1 update 500", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_league_v1_update_500_vs_league_v1_update_250.json", "max_steps_rate": 0.0, "mean_game_length": 50.7755, "mean_score_diff": 5.20975, "opened_colors_per_game": 4.9415, "play_action_rate": 0.7485131356099098, "positive_expeditions_per_game": 2.351, "previous": "league_v1_update_250", "previous_label": "league v1 update 250", "score_diff_ci95_high": 6.664905283640598, "score_diff_ci95_low": 3.7545947163594007, "score_diff_std": 46.95601635366284, "wilson_high": 0.5558977420904201, "wilson_low": 0.525024543003114, "win_rate": 0.5405}
{"current": "repair_c01_update_500", "current_label": "repair c01 update 500", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_repair_c01_update_500_vs_league_v1_update_500.json", "max_steps_rate": 0.0, "mean_game_length": 49.2915, "mean_score_diff": 3.64125, "opened_colors_per_game": 4.9395, "play_action_rate": 0.7722021038110018, "positive_expeditions_per_game": 2.43225, "previous": "league_v1_update_500", "previous_label": "league v1 update 500", "score_diff_ci95_high": 5.0464015232057395, "score_diff_ci95_low": 2.23609847679426, "score_diff_std": 45.34245839244679, "wilson_high": 0.5476742803350019, "wilson_low": 0.5167638355728124, "win_rate": 0.53225}
{"checkpoint": "ladder_v2_gate3", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_ladder_v2_gate3.json", "label": "ladder v2 gate3", "max_steps_rate": 0.0, "mean_game_length": 61.398, "mean_score_diff": 43.456, "opened_colors_per_game": 4.55925, "play_action_rate": 0.6680377805945128, "positive_expeditions_per_game": 2.61475, "score_diff_ci95_high": 44.81720694691559, "score_diff_ci95_low": 42.094793053084416, "score_diff_std": 43.92442262256479, "stage_index": 0, "wilson_high": 0.8490930348441887, "wilson_low": 0.8262583814919731, "win_rate": 0.838}
{"checkpoint": "league_v1_update_250", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_league_v1_update_250.json", "label": "league v1 update 250", "max_steps_rate": 0.0, "mean_game_length": 56.52425, "mean_score_diff": 26.42325, "opened_colors_per_game": 4.54775, "play_action_rate": 0.6422064732439839, "positive_expeditions_per_game": 2.19725, "score_diff_ci95_high": 27.654455989944754, "score_diff_ci95_low": 25.192044010055245, "score_diff_std": 39.729456538778834, "stage_index": 1, "wilson_high": 0.7570405983993863, "wilson_low": 0.7299916729968844, "win_rate": 0.74375}
{"checkpoint": "league_v1_update_500", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_league_v1_update_500.json", "label": "league v1 update 500", "max_steps_rate": 0.0, "mean_game_length": 53.29175, "mean_score_diff": 13.967, "opened_colors_per_game": 4.84725, "play_action_rate": 0.6934619506966774, "positive_expeditions_per_game": 2.14075, "score_diff_ci95_high": 15.054322005906956, "score_diff_ci95_low": 12.879677994093045, "score_diff_std": 35.086502770569304, "stage_index": 2, "wilson_high": 0.6609204517986065, "wilson_low": 0.6312989110391558, "win_rate": 0.64625}
{"checkpoint": "repair_c01_update_500", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_repair_c01_update_500.json", "label": "repair c01 update 500", "max_steps_rate": 0.0, "mean_game_length": 54.61975, "mean_score_diff": 19.654, "opened_colors_per_game": 4.7015, "play_action_rate": 0.6708126417051431, "positive_expeditions_per_game": 2.1505, "score_diff_ci95_high": 20.81558482366198, "score_diff_ci95_low": 18.49241517633802, "score_diff_std": 37.482869759149246, "stage_index": 3, "wilson_high": 0.7068566468456556, "wilson_low": 0.6782734877661645, "win_rate": 0.69275}
{"budget": "same-budget phase-1 exploiter", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 54.7721, "phase": "ladder_v2", "protocol": "short_random_shaping", "source": "docs/reports/ladder-v2-2026-07-05-summary.jsonl", "target": "ladder_v2_gate3", "warmstart": "random", "win_rate": 0.83135}
{"budget": "league v1 cycle exploiter", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 0.6818, "phase": "league_v1", "protocol": "league_cycle_exploiter", "source": "docs/reports/league-v1-2026-07-05-summary.jsonl", "target": "league_v1_update_250", "warmstart": "random", "win_rate": 0.50225}
{"budget": "same exploiter vs final", "delta_from_previous_same_protocol": -0.031299999999999994, "event": "exploiter_trajectory", "mean_score_diff": -2.63775, "phase": "league_v1", "protocol": "league_cycle_exploiter", "source": "docs/reports/league-v1-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "random", "win_rate": 0.47095}
{"budget": "1200 updates, shaping anneal", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 10.95125, "phase": "gates_1_2_original", "protocol": "long_random_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "random", "win_rate": 0.587}
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 11.38075, "phase": "gates_1_2_original", "protocol": "warmstart_gate3_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "ladder_v2_gate3", "win_rate": 0.59}
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 10.42525, "phase": "gates_1_2_original", "protocol": "replay_exploiter_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "league_v1_cycle_1_exploiter", "win_rate": 0.582}
{"budget": "1200 updates, shaping anneal", "delta_from_previous_same_protocol": -0.04799999999999993, "event": "exploiter_trajectory", "mean_score_diff": 6.27925, "phase": "repair_c01", "protocol": "long_random_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "random", "win_rate": 0.539}
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": -0.040999999999999925, "event": "exploiter_trajectory", "mean_score_diff": 7.0285, "phase": "repair_c01", "protocol": "warmstart_gate3_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "ladder_v2_gate3", "win_rate": 0.549}
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": -0.042749999999999955, "event": "exploiter_trajectory", "mean_score_diff": 6.10225, "phase": "repair_c01", "protocol": "replay_exploiter_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "league_v1_cycle_1_exploiter", "win_rate": 0.53925}
{"artifact_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05", "elapsed_seconds": 15.667906627990305, "event": "diminishing_returns_complete", "report": "docs/reports/diminishing-returns-2026-07-05.md"}
docs/reports/diminishing-returns-2026-07-05.md
+65
View File
@@ -0,0 +1,65 @@
# Diminishing Returns Diagnostic - 2026-07-05
신규 학습 없이 기존 체크포인트와 평가 롤아웃만 사용했다. 모든 duplicate 평가는 셔플 뱅크 seed 20260704, 2,000쌍(4000 games) 기준이다.
Raw artifacts: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05`
## Checkpoints
| Order | Name | Config | Checkpoint |
| ---: | --- | --- | --- |
| 0 | `ladder_v2_gate3` | `configs/jax_ppo/ladder-v2-expert.yaml` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest` |
| 1 | `league_v1_update_250` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250` |
| 2 | `league_v1_update_500` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500` |
| 3 | `repair_c01_update_500` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500` |
## 1. Adjacent Head-to-Head
![Adjacent H2H](diminishing-returns-2026-07-05-h2h.png)
| Later | Earlier | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `league_v1_update_250` | `ladder_v2_gate3` | 0.8015 | [0.7889, 0.8136] | +42.8757 | [+41.3138, +44.4377] | 4.9110 | 0.0000 |
| `league_v1_update_500` | `league_v1_update_250` | 0.5405 | [0.5250, 0.5559] | +5.2097 | [+3.7546, +6.6649] | 4.9415 | 0.0000 |
| `repair_c01_update_500` | `league_v1_update_500` | 0.5323 | [0.5168, 0.5477] | +3.6412 | [+2.2361, +5.0464] | 4.9395 | 0.0000 |
## 2. External Anchor Trajectory
![Expert anchor](diminishing-returns-2026-07-05-expert.png)
| Checkpoint | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `ladder_v2_gate3` | 0.8380 | [0.8263, 0.8491] | +43.4560 | [+42.0948, +44.8172] | 4.5592 | 0.0000 |
| `league_v1_update_250` | 0.7438 | [0.7300, 0.7570] | +26.4232 | [+25.1920, +27.6545] | 4.5477 | 0.0000 |
| `league_v1_update_500` | 0.6462 | [0.6313, 0.6609] | +13.9670 | [+12.8797, +15.0543] | 4.8472 | 0.0000 |
| `repair_c01_update_500` | 0.6927 | [0.6783, 0.7069] | +19.6540 | [+18.4924, +20.8156] | 4.7015 | 0.0000 |
Recent expert-anchor slope: `+5.6870` points (league_v1_update_500 -> repair_c01_update_500).
## 3. Exploitability Trajectory
![Exploitability](diminishing-returns-2026-07-05-exploiter.png)
프로토콜이 다른 피탈률은 한 곡선에 섞지 않았다. `Delta`는 같은 프로토콜 안에서만 계산했다.
| Phase | Target | Protocol | Budget | Warm start | Win rate | Delta same protocol | Mean diff | Source |
| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
| `ladder_v2` | `ladder_v2_gate3` | `short_random_shaping` | same-budget phase-1 exploiter | `random` | 0.83135 | n/a | +54.7721 | [ladder-v2-2026-07-05-summary.jsonl](ladder-v2-2026-07-05-summary.jsonl) |
| `league_v1` | `league_v1_update_250` | `league_cycle_exploiter` | league v1 cycle exploiter | `random` | 0.50225 | n/a | +0.6818 | [league-v1-2026-07-05-summary.jsonl](league-v1-2026-07-05-summary.jsonl) |
| `league_v1` | `league_v1_update_500` | `league_cycle_exploiter` | same exploiter vs final | `random` | 0.47095 | -0.03130 | -2.6378 | [league-v1-2026-07-05-summary.jsonl](league-v1-2026-07-05-summary.jsonl) |
| `gates_1_2_original` | `league_v1_update_500` | `long_random_shaping` | 1200 updates, shaping anneal | `random` | 0.58700 | n/a | +10.9512 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
| `gates_1_2_original` | `league_v1_update_500` | `warmstart_gate3_no_shaping` | 900 updates, shaping 0 | `ladder_v2_gate3` | 0.59000 | n/a | +11.3808 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
| `gates_1_2_original` | `league_v1_update_500` | `replay_exploiter_no_shaping` | 900 updates, shaping 0 | `league_v1_cycle_1_exploiter` | 0.58200 | n/a | +10.4253 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
| `repair_c01` | `repair_c01_update_500` | `long_random_shaping` | 1200 updates, shaping anneal | `random` | 0.53900 | -0.04800 | +6.2793 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
| `repair_c01` | `repair_c01_update_500` | `warmstart_gate3_no_shaping` | 900 updates, shaping 0 | `ladder_v2_gate3` | 0.54900 | -0.04100 | +7.0285 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
| `repair_c01` | `repair_c01_update_500` | `replay_exploiter_no_shaping` | 900 updates, shaping 0 | `league_v1_cycle_1_exploiter` | 0.53925 | -0.04275 | +6.1022 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
## Judgment
추가 학습 1사이클(~1.5시간 GPU)의 기대 개선은 인접 H2H 최근 이득 +3.64점, expert 앵커 최근 변화 +5.69점, 동일 프로토콜 exploiter 평균 피탈률 감소 0.044에 따라 대략 +3.6~+5.7점 또는 피탈률 -0.044 내외 수준으로 추정되며, 수확체감 구간에 진입했다. 근거는 최근 H2H 이득이 첫 리그 전이보다 작고, expert 앵커 성능이 v1 update 250 이후 하락/회복을 반복하며, 보수 후 exploiter 최악값이 0.549로 게이트 바로 아래에 머문다는 세 측정의 일치다. 단, repair가 expert 앵커를 유의하게 회복시킨 점은 별도 긍정 신호다.
## Notes
- H2H와 expert 앵커 평가는 이번 작업에서 새로 실행했다.
- Exploiter 표는 지금까지 생성된 리포트/JSONL의 기존 측정값만 재정리했다.
- 이 작업에서는 신규 학습, 봇 수정, 체크포인트 수정이 없었다.
docs/reports/final-cycles-and-human-play-2026-07-05-summary.jsonl
+8
View File
@@ -0,0 +1,8 @@
{"cycle": 1, "event": "final_cycle_league", "league_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/league_config.yaml", "league_run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "worst_protocol_used": "warmstart_gate3"}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 919.0, "max_steps_rate": 0.0005, "mean_game_length": 65.69525, "mean_score_diff": 55.799, "notes": "random init + shaping anneal, 1200 updates", "opened_colors_per_game": 4.92425, "play_action_rate": 0.5457588946038896, "positive_expeditions_per_game": 2.7735, "resume": null, "score_diff_ci95_high": 57.963152016622004, "score_diff_ci95_low": 53.634847983377995, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 22.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter", "wilson_high": 0.7776365926294233, "wilson_low": 0.7513553821486798, "win_rate": 0.76475, "wins": 3059.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 961.0, "max_steps_rate": 0.00075, "mean_game_length": 65.13875, "mean_score_diff": 53.30925, "notes": "ladder v2 gate-3 warm start, shaping disabled, 900 updates", "opened_colors_per_game": 4.90075, "play_action_rate": 0.5480372138722999, "positive_expeditions_per_game": 2.71225, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 55.52225515395847, "score_diff_ci95_low": 51.09624484604153, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 18.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter", "wilson_high": 0.7683246594694194, "wilson_low": 0.741685544731188, "win_rate": 0.75525, "wins": 3021.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 866.0, "max_steps_rate": 0.00025, "mean_game_length": 65.467, "mean_score_diff": 59.351, "notes": "league v1 cycle-1 exploiter warm start, shaping disabled, 900 updates", "opened_colors_per_game": 4.989, "play_action_rate": 0.5882132820136853, "positive_expeditions_per_game": 2.7905, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 61.580508518304526, "score_diff_ci95_low": 57.12149148169547, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 17.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter", "wilson_high": 0.7918317711217939, "wilson_low": 0.7661323798009079, "win_rate": 0.77925, "wins": 3117.0}
{"cycle": 1, "event": "final_cycle_battery_judgment", "passed": false, "threshold": 0.52, "worst_exploiter": "replay_exploiter", "worst_win_rate": 0.77925}
{"cycle": 1, "event": "final_cycle_guard_expert", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/guard_vs_heuristic_expert.json", "losses": 852.0, "max_steps_rate": 0.0, "mean_game_length": 60.38125, "mean_score_diff": 36.08475, "opened_colors_per_game": 4.15225, "passed": true, "play_action_rate": 0.5845773004045489, "positive_expeditions_per_game": 2.05975, "score_diff_ci95_high": 37.5625235448534, "score_diff_ci95_low": 34.6069764551466, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 33.0, "wilson_high": 0.7913426526326875, "wilson_low": 0.7656224577333041, "win_rate": 0.77875, "wins": 3115.0}
{"current": "final_cycle_01", "current_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "cycle": 1, "event": "final_cycle_h2h", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/h2h_final_cycle_01_vs_repair_c01_update_500.json", "losses": 1987.0, "max_steps_rate": 0.0, "mean_game_length": 49.54275, "mean_score_diff": 0.35425, "opened_colors_per_game": 4.91275, "play_action_rate": 0.7646239835347768, "positive_expeditions_per_game": 2.422, "previous": "repair_c01_update_500", "previous_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "score_diff_ci95_high": 1.7891388254913125, "score_diff_ci95_low": -1.0806388254913126, "ties": 51.0, "wilson_high": 0.505993762454951, "wilson_low": 0.47502446696755335, "win_rate": 0.4905, "wins": 1962.0}
{"elapsed_seconds": 17320.886204754002, "event": "final_cycles_complete", "final_candidate": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate", "final_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "source_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "stop_reason": "h2h_stagnation"}
docs/reports/final-cycles-and-human-play-2026-07-05.md
+83
View File
@@ -0,0 +1,83 @@
# Final Cycles and Human Play - 2026-07-05
## Part A - Closing Reinforcement Cycles
Raw artifacts: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05`
Stop reason: `h2h_stagnation`
Final candidate: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate`
Final candidate config: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json`
The initial 4 hour GPU budget was treated as an estimate after user confirmation,
not a hard cutoff. The run completed cycle 1 fully, then stopped by the written
H2H stagnation rule. Cycle 2 was not started.
Bottom line: the closing reinforcement cycle did not produce a materially better
policy. The cycle-1 snapshot passed the expert guard, but the strengthened
exploiter battery got substantially worse than the incoming baseline target:
worst exploiter win rate was 0.7792, far above the 0.5200 success threshold. The
adjacent H2H against `repair_c01_update_500` was statistically indistinguishable
from zero, so this recipe is stopped here. Per instruction, this report does not
recommend more training with the same observation/network/PPO recipe.
### Expert Guard
| Cycle | Passed | Win rate | Mean diff | CI low | Opened colors | Max-step |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | True | 0.7788 | +36.0847 | +34.6070 | 4.1523 | 0.0000 |
### Adjacent H2H
| Cycle | Current | Previous | Win rate | Mean diff | Score CI |
| ---: | --- | --- | ---: | ---: | ---: |
| 1 | `final_cycle_01` | `repair_c01_update_500` | 0.4905 | +0.3543 | [-1.0806, +1.7891] |
### Strengthened Exploiter Battery
| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |
| ---: | --- | ---: | ---: | ---: | ---: | ---: |
| 1 | `long_random` | 0.7648 | +55.7990 | +53.6348 | 4.9242 | 0.0005 |
| 1 | `warmstart_gate3` | 0.7552 | +53.3092 | +51.0962 | 4.9008 | 0.0008 |
| 1 | `replay_exploiter` | 0.7792 | +59.3510 | +57.1215 | 4.9890 | 0.0003 |
### Battery Judgment
| Cycle | Worst exploiter | Worst win rate | Threshold | Passed |
| ---: | --- | ---: | ---: | ---: |
| 1 | `replay_exploiter` | 0.7792 | 0.5200 | False |
## Part B - Human Play Interface
Start a single game:
```bash
uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \
--seat 0
```
Start a duplicate set with one shared shuffle and swapped seats:
```bash
uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \
--seat 0 --duplicate
```
Summarize logged human games:
```bash
uv run lost-cities-jax-ppo human-play summarize \
--log-dir /mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/
```
Move syntax: `play R7 draw deck`, `discard G3 draw Y`, or `play RHS draw deck`.
The renderer shows only the human hand, both boards, all public discard piles, deck count, and current board score differential. Opponent hand and deck order are not rendered.
Every game is appended to `/mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/games.jsonl` with deck seed/index, full action list, AI top-3 policy actions/probabilities, value outputs, scoring breakdown, and optional human comment.
Validation performed before final report:
- `uv run ruff check .` passed.
- `uv run pytest -q` passed: 257 passed, 1 skipped.
- `uv run lost-cities-jax-ppo play --help` passed.
- `uv run lost-cities-jax-ppo human-play summarize --log-dir /tmp/nonexistent-human-play-log` passed.
- `scripts/librarian.sh` found no link or code-citation errors; it still exits non-zero on the known pre-existing `docs/plans/archive/deep-cfr-selectivity.md` 500-line soft cap.
docs/reports/gates-1-2-2026-07-05-summary.jsonl
+19
View File
@@ -0,0 +1,19 @@
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_expert_cap3.json", "learner": "expert_cap2", "losses": 2102.0, "max_steps_rate": 0.0, "mean_game_length": 46.549, "mean_score_diff": -1.89325, "opened_colors_per_game": 1.77925, "opponent": "expert_cap3", "play_action_rate": 0.2414775169804832, "positive_expeditions_per_game": 0.94175, "score_diff_ci95_high": -1.4203123965597364, "score_diff_ci95_low": -2.366187603440264, "score_diff_std": 15.261096936571331, "ties": 132.0, "wilson_high": 0.45693730157215046, "wilson_low": 0.42617495329274474, "win_rate": 0.4415, "wins": 1766.0}
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_expert_capN.json", "learner": "expert_cap2", "losses": 2092.0, "max_steps_rate": 0.0, "mean_game_length": 46.354, "mean_score_diff": -1.88075, "opened_colors_per_game": 1.7815, "opponent": "expert_capN", "play_action_rate": 0.24223387949830538, "positive_expeditions_per_game": 0.93725, "score_diff_ci95_high": -1.4070318876291175, "score_diff_ci95_low": -2.3544681123708826, "score_diff_std": 15.286282970338549, "ties": 130.0, "wilson_high": 0.45994507591745637, "wilson_low": 0.4291614222877006, "win_rate": 0.4445, "wins": 1778.0}
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap3_vs_expert_capN.json", "learner": "expert_cap3", "losses": 1927.0, "max_steps_rate": 0.0, "mean_game_length": 45.35975, "mean_score_diff": 0.0285, "opened_colors_per_game": 2.21525, "opponent": "expert_capN", "play_action_rate": 0.3101634934459303, "positive_expeditions_per_game": 1.20625, "score_diff_ci95_high": 0.5258735279417411, "score_diff_ci95_low": -0.46887352794174114, "score_diff_std": 16.04961324366783, "ties": 137.0, "wilson_high": 0.49949486790031544, "wilson_low": 0.4685358342849551, "win_rate": 0.484, "wins": 1936.0}
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_league_v1_update_500.json", "learner": "expert_cap2", "losses": 2871.0, "max_steps_rate": 0.00025, "mean_game_length": 55.372, "mean_score_diff": -20.88625, "opened_colors_per_game": 1.9735, "opponent": "league_v1_update_500", "play_action_rate": 0.22526015834941476, "positive_expeditions_per_game": 1.09725, "score_diff_ci95_high": -19.787867065349186, "score_diff_ci95_low": -21.984632934650815, "score_diff_std": 35.44342491958145, "ties": 39.0, "wilson_high": 0.2865114339331819, "wilson_low": 0.2589251127636324, "win_rate": 0.2725, "wins": 1090.0}
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap3_vs_league_v1_update_500.json", "learner": "expert_cap3", "losses": 2590.0, "max_steps_rate": 0.0, "mean_game_length": 53.52525, "mean_score_diff": -14.35125, "opened_colors_per_game": 2.7385, "opponent": "league_v1_update_500", "play_action_rate": 0.320897958422324, "positive_expeditions_per_game": 1.52675, "score_diff_ci95_high": -13.256641796659636, "score_diff_ci95_low": -15.445858203340364, "score_diff_std": 35.32161912528796, "ties": 50.0, "wilson_high": 0.3548273870268761, "wilson_low": 0.32547963482582853, "win_rate": 0.34, "wins": 1360.0}
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_capN_vs_league_v1_update_500.json", "learner": "expert_capN", "losses": 2585.0, "max_steps_rate": 0.0, "mean_game_length": 53.29175, "mean_score_diff": -13.967, "opened_colors_per_game": 3.004, "opponent": "league_v1_update_500", "play_action_rate": 0.3410701746172932, "positive_expeditions_per_game": 1.56475, "score_diff_ci95_high": -12.879677994093045, "score_diff_ci95_low": -15.054322005906956, "score_diff_std": 35.086502770569304, "ties": 49.0, "wilson_high": 0.3563415432992672, "wilson_low": 0.3269626002235683, "win_rate": 0.3415, "wins": 1366.0}
{"ci95_high": 0.3559042062015777, "ci95_low": -0.32834170620157765, "deltas_npy": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/delta_open_deltas.npy", "event": "gate1_delta_open", "histogram": {"counts": [107, 1643, 5209, 7192, 10627, 5447, 1662, 113], "edges": [-200.0, -100.0, -50.0, -20.0, 0.0, 20.0, 50.0, 100.0, 200.0]}, "histogram_png": "docs/reports/gates-1-2-delta-open-hist.png", "judgment": "near_zero", "mean_delta": 0.01378125, "pairs": 64, "quantiles": {"max": 149.0, "min": -167.0, "p05": -52.0, "p25": -17.0, "p50": 0.0, "p75": 17.0, "p95": 51.0}, "samples": 32000, "states": 500, "std_delta": 31.225479356323433}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter/latest", "event": "gate2_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1610.0, "max_steps_rate": 0.0, "mean_game_length": 50.74375, "mean_score_diff": 10.95125, "notes": "random init + shaping anneal, extended budget", "opened_colors_per_game": 4.99575, "play_action_rate": 0.8010700772505124, "positive_expeditions_per_game": 2.53625, "resume": null, "score_diff_ci95_high": 12.423541316332845, "score_diff_ci95_low": 9.478958683667155, "score_diff_std": 47.50897440589277, "ties": 42.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter", "wilson_high": 0.6021679468487495, "wilson_low": 0.5716651100188423, "win_rate": 0.587, "wins": 2348.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter/latest", "event": "gate2_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1605.0, "max_steps_rate": 0.0, "mean_game_length": 50.8715, "mean_score_diff": 11.38075, "notes": "ladder v2 gate-3 warm start, shaping disabled", "opened_colors_per_game": 4.9945, "play_action_rate": 0.798003736847281, "positive_expeditions_per_game": 2.5195, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 12.872234578510747, "score_diff_ci95_low": 9.889265421489254, "score_diff_std": 48.128316645747006, "ties": 35.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter", "wilson_high": 0.6051483732088389, "wilson_low": 0.5746789269990148, "win_rate": 0.59, "wins": 2360.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter/latest", "event": "gate2_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1638.0, "max_steps_rate": 0.0, "mean_game_length": 51.4305, "mean_score_diff": 10.42525, "notes": "league v1 exploiter warm start, shaping disabled", "opened_colors_per_game": 4.99775, "play_action_rate": 0.7904596181514173, "positive_expeditions_per_game": 2.54225, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 11.913719629678091, "score_diff_ci95_low": 8.936780370321909, "score_diff_std": 48.0310280688623, "ties": 34.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter", "wilson_high": 0.5971992743148633, "wilson_low": 0.5666433769856256, "win_rate": 0.582, "wins": 2328.0}
{"event": "gate2_judgment", "passed": false, "threshold": 0.55, "worst_exploiter": "warmstart_gate3", "worst_win_rate": 0.59}
{"elapsed_seconds": 11425.08623591601, "event": "gates_complete", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2"}
{"cycle": 1, "event": "gate2c_league_cycle", "league_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/league_config.yaml", "league_run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "worst_protocol": "warmstart_gate3"}
{"cycle": 1, "event": "gate2c_guard_expert", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/guard_vs_heuristic_expert.json", "losses": 1188.0, "max_steps_rate": 0.0, "mean_game_length": 54.61975, "mean_score_diff": 19.654, "opened_colors_per_game": 4.7015, "passed": true, "play_action_rate": 0.6708126417051431, "positive_expeditions_per_game": 2.1505, "score_diff_ci95_high": 20.81558482366198, "score_diff_ci95_low": 18.49241517633802, "score_diff_std": 37.482869759149246, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "ties": 41.0, "wilson_high": 0.7068566468456556, "wilson_low": 0.6782734877661645, "win_rate": 0.69275, "wins": 2771.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1810.0, "max_steps_rate": 0.0, "mean_game_length": 51.4185, "mean_score_diff": 6.27925, "notes": "random init + shaping anneal, extended budget", "opened_colors_per_game": 4.99425, "play_action_rate": 0.7816023796562719, "positive_expeditions_per_game": 2.52, "resume": null, "score_diff_ci95_high": 7.76660187726116, "score_diff_ci95_low": 4.79189812273884, "score_diff_std": 47.99495961530356, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 34.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter", "wilson_high": 0.5544028831760525, "wilson_low": 0.5235222802473508, "win_rate": 0.539, "wins": 2156.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1763.0, "max_steps_rate": 0.0, "mean_game_length": 50.434, "mean_score_diff": 7.0285, "notes": "ladder v2 gate-3 warm start, shaping disabled", "opened_colors_per_game": 4.9935, "play_action_rate": 0.8006960073764884, "positive_expeditions_per_game": 2.4845, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 8.490821414519859, "score_diff_ci95_low": 5.5661785854801415, "score_diff_std": 47.18725831288327, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 41.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter", "wilson_high": 0.5643659496573598, "wilson_low": 0.5335400249002497, "win_rate": 0.549, "wins": 2196.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1798.0, "max_steps_rate": 0.0, "mean_game_length": 51.116, "mean_score_diff": 6.10225, "notes": "league v1 exploiter warm start, shaping disabled", "opened_colors_per_game": 4.99425, "play_action_rate": 0.7845902569268536, "positive_expeditions_per_game": 2.50025, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 7.583110013023179, "score_diff_ci95_low": 4.62138998697682, "score_diff_std": 47.785475385853, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 45.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter", "wilson_high": 0.5546520360903574, "wilson_low": 0.523772647611401, "win_rate": 0.53925, "wins": 2157.0}
{"cycle": 1, "event": "gate2c_judgment", "passed": true, "threshold": 0.55, "worst_exploiter": "warmstart_gate3", "worst_win_rate": 0.549}
{"elapsed_seconds_total": 28945.184503759025, "event": "gates_repair_complete", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2"}
docs/reports/gates-1-2-2026-07-05.md
+67
View File
@@ -0,0 +1,67 @@
# Gates 1-2 Report - 2026-07-05
**Run dir:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2`.
**Target:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500`.
## Gate 1A - Policy Class Tournament
| Learner | Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `expert_cap2` | `expert_cap3` | 0.4415 | -1.8933 | -2.3662 | 1.7792 | 0.0000 |
| `expert_cap2` | `expert_capN` | 0.4445 | -1.8807 | -2.3545 | 1.7815 | 0.0000 |
| `expert_cap3` | `expert_capN` | 0.4840 | +0.0285 | -0.4689 | 2.2153 | 0.0000 |
## Gate 1A - Variants vs League v1
| Learner | Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `expert_cap2` | `league_v1_update_500` | 0.2725 | -20.8863 | -21.9846 | 1.9735 | 0.0003 |
| `expert_cap3` | `league_v1_update_500` | 0.3400 | -14.3513 | -15.4459 | 2.7385 | 0.0000 |
| `expert_capN` | `league_v1_update_500` | 0.3415 | -13.9670 | -15.0543 | 3.0040 | 0.0000 |
## Gate 1B - Delta Open Audit
- States: 500
- Paired samples: 32000
- Mean delta: +0.0138
- CI95: [-0.3283, +0.3559]
- Judgment: `near_zero`
- Histogram: `docs/reports/gates-1-2-delta-open-hist.png`
## Gate 1C - Selectivity Judgment
통념 기각/미결: capN은 집중 변형보다 유의하게 나쁘지 않고, 리그 정책의 4번째+ 오픈 delta는 0 근처다. selectivity는 현재 주요 성능 병목으로 보이지 않는다.
## Gate 2A - Strengthened Exploiter Battery
| Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step | Run |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| `long_random` | 0.5870 | +10.9512 | +9.4790 | 4.9958 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter` |
| `warmstart_gate3` | 0.5900 | +11.3808 | +9.8893 | 4.9945 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter` |
| `replay_exploiter` | 0.5820 | +10.4253 | +8.9368 | 4.9977 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter` |
## Gate 2B - Robustness Judgment
보수 필요: worst exploiter `warmstart_gate3` win rate 0.5900 vs threshold 0.5500.
## Gate 2C - Conditional Repair League
| Cycle | Worst protocol used | Guard pass | Battery worst | Passed | Checkpoint |
| ---: | --- | ---: | ---: | ---: | --- |
| 1 | `warmstart_gate3` | True | 0.5490 | True | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500` |
| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |
| ---: | --- | ---: | ---: | ---: | ---: | ---: |
| 1 | `long_random` | 0.5390 | +6.2793 | +4.7919 | 4.9943 | 0.0000 |
| 1 | `warmstart_gate3` | 0.5490 | +7.0285 | +5.5662 | 4.9935 | 0.0000 |
| 1 | `replay_exploiter` | 0.5393 | +6.1022 | +4.6214 | 4.9943 | 0.0000 |
## Human Play Recommendation
조건부 예: 강화 exploiter 관문은 통과했다. 다만 selectivity 관문이 미결/부분 지지이면 인간 대전은 실력 인증이 아니라 행동 양식 진단으로 시작해야 한다.
## Decisions
- Existing `heuristic_expert` remains unchanged. The cap variants add only a hard `max_open_colors` gate around new-color openings.
- `expert_capN` means no hard cap; EV thresholds and the existing soft concentration penalties are retained.
- Warm-started exploiters use shaping coefficient 0 to measure target-specific exploitation without reintroducing early shaping rewards.
docs/reports/gates-1-2-delta-open-hist.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

docs/reports/jax-ppo-model-capacity-2026-07-12.md
+119
View File
@@ -0,0 +1,119 @@
# JAX PPO Model-Capacity Diagnostic - 2026-07-12
"MLP를 더 키우면 아직 얻을 게 남아 있나?"에 대한 진단이다.
신규 학습 없이 기존 체크포인트, `league_metrics.jsonl`, 소스만 읽어 작성했다.
새로 실행한 학습·평가·벤치마크는 없다.
Subject: final candidate
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate`
Metrics: `.../final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/league_metrics.jsonl` (500 updates)
## 요약
용량이 **현재의 병목일 가능성은 낮다.** 다만 PPO 스택에서 모델 크기는
**한 번도 검증된 적이 없으므로**, 이 판단은 아직 측정으로 뒷받침되지 않았다.
단발 A/B로 결론을 확정할 것을 권한다 —
[docs/plans/jax-ppo-model-size-ab.md](../plans/jax-ppo-model-size-ab.md).
## 1. 현재 크기는 근거 없는 상속값
`configs/jax_ppo/` 의 실 학습 config는 **전부** `hidden_size: 512, num_layers: 3`
이다 (`smoke.yaml`만 64×2). final candidate도 동일하다
(`main_ppo_config.json` → `network: {hidden_size: 512, num_layers: 3}`).
`hidden_size`를 다룬 기존 문서는 전부 은퇴한 Deep CFR/PyTorch 스택 것이다
(`input_dim=365`, `legacy/deep-cfr/configs/` 기준). 특히
[docs/plans/model_size_experiment.md](../plans/archive/model_size_experiment.md)는
**현 JAX PPO 스택과 무관하다.**
→ 512×3은 실험으로 고른 값이 아니라 구 스택에서 복사돼 온 값이다.
## 2. 데이터 대비 모델이 매우 작다 (스케일링에 유리한 신호)
| 항목 | 값 |
| --- | ---: |
| `OBS_DIM` (`types.py:27`) | 454 |
| `N_ACTIONS` | 96 |
| 512×3 파라미터 수 | **808,033** |
| 1024×4 파라미터 수 | 3,714,145 |
| 2048×4 파라미터 수 | 13,719,649 |
| 최종 사이클 소비 transition | **1,638,400,000** |
| 샘플 : 파라미터 | **약 2,028 : 1** |
transition 수는 `rollout_steps=400 × batch_games=8192 × updates=500`.
셀프플레이라 데이터는 사실상 무한하다. 샘플:파라미터 2,000:1은 **데이터 제약이
아니라 용량/최적화 제약** 구간의 전형이며, 보통 이런 영역에서 스케일링이 먹힌다.
0.8M 파라미터는 절대적으로도 작다.
여기까지만 보면 "키우면 이득"이다. 그러나 3절이 반대 방향을 가리킨다.
## 3. 학습 로그에 용량 부족의 지문이 없다
최종 사이클 500 업데이트 구간 평균:
| update | value_loss | entropy_mean | approx_kl | return_mean | return_std |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0–50 | 0.0468 | 1.1159 | 0.0433 | 0.2696 | 0.6299 |
| 50–100 | 0.0462 | 1.1167 | 0.0438 | 0.2799 | 0.6275 |
| 100–200 | 0.0461 | 1.1241 | 0.0441 | 0.2822 | 0.6316 |
| 200–300 | 0.0459 | 1.1143 | 0.0437 | 0.2857 | 0.6306 |
| 300–400 | 0.0460 | 1.1239 | 0.0424 | 0.2900 | 0.6323 |
| 400–500 | 0.0463 | 1.1163 | 0.0423 | 0.2916 | 0.6340 |
세 가지를 읽을 수 있다.
**(a) 크리틱은 이미 잘 맞춘다.** `value_loss`는 순수 MSE다
(`ppo.py:624`: `masked_mean((mb_returns - value) ** 2, active_weight)`).
`return_std ≈ 0.634` → 에피소드 리턴 분산 ≈ 0.402. MSE 0.0463과 비교하면
**설명분산 ≈ 0.88**.
> 주의 — 이 0.88은 **상한 추정치**다. `return_std`는 에피소드 리턴의 표준편차이고
> (`ppo.py:1072`: `episode_return = jnp.sum(transitions.reward, axis=0)`),
> `value_loss`의 타깃은 GAE 리턴(`mb_returns`)이다. final candidate는 shaping이
> 0이고 `gamma=1.0`이라 두 분포가 가깝지만, `gae_lambda=0.95`의 부트스트랩이
> 타깃 분산을 축소시키므로 실제 설명분산은 이보다 낮을 수 있다.
용량이 모자란 네트워크는 loss가 **높은 지점에서** 정체한다. 여기는 **낮은
지점에서** 정체한다. 잔여 오차 상당 부분은 불완전정보 + 덱 셔플에서 오는
**환원 불가능한 분산**으로 보인다.
**(b) 엔트로피는 계수가 잡아둔 평형이다.** `entropy_mean`이 1.114–1.124에서
500 업데이트 내내 미동도 없다. `entropy_coef=0.01`과 정책 그래디언트가 이룬
평형이지 용량 한계가 아니다. **파라미터를 늘려도 이 값은 안 변한다.**
(불완전정보에서 혼합 전략은 정상이므로 이 자체가 결함은 아니다.)
**(c) 죽은 게 아니라 느린 것이다.** `return_mean`은 0.2696 → 0.2916으로
여전히 오르고 있다. `approx_kl`도 0.042–0.044로 유지되어 정책이 계속 움직인다.
수렴해서 멈춘 상태가 아니다.
## 4. 실제 천장은 exploitability이고, 이건 용량 문제가 아니다
[diminishing-returns-2026-07-05.md](diminishing-returns-2026-07-05.md) 기준,
final 계열을 상대로 새로 학습시킨 exploiter가 **여전히 승률 0.539–0.549로 이긴다**
(`repair_c01_update_500`, 3개 프로토콜).
불완전정보 게임에서 이 잔여 착취가능성은 **게임이론적 문제**다. PPO 셀프플레이는
내쉬로 수렴하지 않고 전략공간을 순환한다. MLP를 키우면 *같은 순환 역학 안에서
더 나은 응수*를 찾을 뿐, 이 0.54 바닥 자체를 무너뜨리지는 못한다.
## 판정
- 스케일링이 **소폭 이득**을 줄 가능성은 있다 (2절: 샘플:파라미터 2,000:1).
- 그러나 **용량 부족의 직접 증거는 없다** (3절: 낮은 지점에서 평평한 value_loss,
계수가 고정한 엔트로피).
- 그리고 **모델을 실제로 가두고 있는 것은 exploitability**이며, 이건 스케일로
풀리지 않는다 (4절).
따라서 우선순위는 낮다. 다만 **한 번도 측정한 적이 없다**는 사실 때문에 위
판단은 전부 정황증거다. 결론을 확정하려면 단발 A/B가 필요하다 →
[docs/plans/jax-ppo-model-size-ab.md](../plans/jax-ppo-model-size-ab.md).
## Notes
- 이 작업에서 신규 학습, config 변경, 체크포인트 수정은 없었다.
- 파라미터 수는 `ActorCritic(hidden, layers).init()` 후 리프 크기 합으로 계산했다
(`ppo.py:104`).
- GUI 영향 참고: 현재 CPU 추론 약 50ms/수. 1024×4로 키워도 150–200ms 수준이라
`lost-cities-play` 대국에는 지장이 없다.
docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md
+68
View File
@@ -0,0 +1,68 @@
# JAX PPO Static-Opponent Ladder Report - 2026-07-04
**Status:** PASS.
**Implementation commit:** `4c0c2e9`.
**Hardware:** RTX 3090 with optional CUDA JAX via `uv run --with 'jax[cuda12]'`.
**Artifact root:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
## Protocol
Each opponent was trained from scratch with the same PPO configuration:
- `batch_games=8192`
- `rollout_steps=400`
- `total_updates=250`
- `gamma=1.0`
- `gae_lambda=0.95`
- `clip_epsilon=0.2`
- `entropy_coef=0.01`
- `seed=20260704`
Evaluation used a fixed 10,000-deck shuffle bank and duplicate play: each deck
was played once with the learner in seat 0 and once with the learner in seat 1,
for 20,000 evaluated games per gate. Win rates include Wilson 95% intervals.
## Results
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean game length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Gate 1 passed both required conditions: win rate >= 90% and at least two
positive expeditions per game. Gates 2 and 3 passed their mean-score-difference
condition.
## Artifacts
| Opponent | Run directory | Evaluation JSON |
| --- | --- | --- |
| `discard_only` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` | `eval_discard_only_duplicate.json` |
| `heuristic_balanced` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` | `eval_balanced_duplicate.json` |
| `heuristic_cautious` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` | `eval_cautious_duplicate.json` |
The random rollout baseline against `discard_only` is stored at:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/random_baseline_discard_only.json
```
## Canary Notes
The random-policy baseline against `discard_only` had `play_action_rate=0.28855`
and only `0.41797` positive expeditions per game. The trained discard-only
policy ended duplicate evaluation at `play_action_rate=0.64598` and `3.2783`
positive expeditions per game, so the intended anti-collapse signal is present.
Balanced and cautious checkpoints pass their score gates decisively, but their
duplicate evaluations are longer: mean game lengths are `167.81450` and
`185.38690`. The final training rollout forced-end rates were low but nonzero
for those opponents (`0.00415` and `0.00391`). Treat game length and forced-end
rate as canaries in the next self-play phase.
## Next Step
The static ladder has cleared. The next phase can start snapshot-pool league
self-play, using these three passed checkpoints as initial anchors and keeping
duplicate evaluation as the regression gate.
docs/reports/ladder-v2-2026-07-05-summary.jsonl
+1
View File
@@ -0,0 +1 @@
{"date":"2026-07-05","status":"pass_with_canary_warnings","expert_bot":{"mirror_max_steps_rate":0.0,"mirror_opened_colors_per_game":2.256,"mirror_play_action_rate":0.3117858091947914,"vs_discard_mean_score_diff":8.9716,"vs_balanced_mean_score_diff":54.36725,"vs_cautious_mean_score_diff":51.9397},"gates":{"discard_only":{"win_rate":1.0,"wilson_low":0.9998079639438954,"mean_score_diff":182.067,"opened_colors_per_game":4.2163,"max_steps_rate":0.0},"heuristic_balanced":{"win_rate":0.9409,"wilson_low":0.9375464241082592,"mean_score_diff":119.925,"opened_colors_per_game":4.9862,"max_steps_rate":0.1859},"heuristic_expert":{"win_rate":0.83815,"wilson_low":0.8329806740267004,"mean_score_diff":43.3099,"score_diff_ci95_low":42.706527573078894,"opened_colors_per_game":4.5509,"max_steps_rate":0.0}},"exploiter":{"target":"ladder_v2_gate3","win_rate":0.83135,"wilson_low":0.8260970642435626,"mean_score_diff":54.7721,"score_diff_ci95_low":53.98710920050297,"opened_colors_per_game":4.99135,"max_steps_rate":0.0001},"artifact_root":"/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/"}
docs/reports/ladder-v2-2026-07-05.md
+114
View File
@@ -0,0 +1,114 @@
# JAX PPO Ladder v2 Report - 2026-07-05
**Status:** PASS for the ordered ladder gates; canary warnings remain.
**Hardware:** RTX 3090 via `uv run --with 'jax[cuda12]'`.
**Artifact roots:**
- Expert bot: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/`
- Ladder v2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/`
## Protocol
Part A added a pure-JAX `heuristic_expert` opponent and evaluated it with a
fixed shuffle bank plus duplicate play. Part B trained PPO from random
initialization for each gate with the same 250-update configuration used by the
original ladder, changing only the opponent and artifact root:
- Gate 1: `discard_only`
- Gate 2: `heuristic_balanced`
- Gate 3: `heuristic_expert`
All evaluations below used 10,000 deck orders with duplicate seat-swapped play,
for 20,000 evaluated games.
## Part A - Expert Bot Gate
| Check | Result | Win rate | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Expert mirror, 1,000 duplicate pairs | PASS | 0.4815 | 0.0000 | [-0.6956, +0.6956] | 45.1560 | 0.0000 | 2.2560 | 0.3118 |
| Expert vs `discard_only` | PASS | 0.6643 | +8.9716 | [+8.7886, +9.1546] | 45.2125 | 0.0000 | 1.8971 | 0.2828 |
| Expert vs `heuristic_balanced` | PASS | 0.9961 | +54.3673 | [+54.0791, +54.6554] | 64.9181 | 0.0000 | 2.8003 | 0.2728 |
| Expert vs `heuristic_cautious` | PASS | 0.9972 | +51.9397 | [+51.6960, +52.1834] | 75.2329 | 0.0000 | 2.6090 | 0.2240 |
The new script bot is non-stalling in mirror play, symmetric under duplicate
seat swap, and beats all three previous static bots by a clear positive score
margin. Its own opened-color profile is in the intended 2-3 color band.
## Part B - Ladder v2
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.0000 [0.9998, 1.0000] | +182.0670 | [+181.6390, +182.4950] | 82.1558 | 0.0000 | 4.2163 | 0.6398 | 3.1845 |
| 2 | `heuristic_balanced` | PASS | 0.9409 [0.9375, 0.9441] | +119.9250 | [+118.9950, +120.8550] | 151.2468 | 0.1859 | 4.9862 | 0.2937 | 3.5316 |
| 3 | `heuristic_expert` | PASS | 0.8382 [0.8330, 0.8432] | +43.3099 | [+42.7065, +43.9133] | 61.4025 | 0.0000 | 4.5509 | 0.6661 | 2.6121 |
Gate 1 passed the original win-rate and positive-expedition conditions. Gate 2
passed the mean-score-difference condition. Gate 3 passed the v2 condition:
duplicate mean score difference is positive and its 95% confidence lower bound
is above zero.
## Canary Findings
The ladder did not produce the desired 2-3 color selectivity in the learned PPO
policies. The gate-3 policy still opens 4.5509 colors/game against the expert.
This is lower than the original gate-3 checkpoint but still far outside the
target band. Treat the selectivity hypothesis as not confirmed.
Gate 2 also has a high forced-end canary: `max_steps_rate=0.1859` in duplicate
evaluation versus `heuristic_balanced`. It passed its score gate, but the
balanced opponent still permits long games and 5-color farming.
## Exploiter Baseline
After gate 3 passed, a new PPO exploiter was trained from random initialization
against the frozen v2 gate-3 checkpoint with the same 250-update budget and
duplicate-evaluated against that checkpoint.
| Target checkpoint | Exploiter win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Old gate-3 checkpoint | 0.6712 [0.6646, 0.6776] | +24.1371 | n/a | 48.0017 | 0.0000 | 4.8971 | 0.7461 | 2.1287 |
| New v2 gate-3 checkpoint | 0.8314 [0.8261, 0.8365] | +54.7721 | [+53.9871, +55.5571] | 60.8744 | 0.0001 | 4.9914 | 0.7028 | 2.8921 |
The new gate-3 checkpoint beats `heuristic_expert`, but is more exploitable
under this protocol than the old checkpoint. That does not invalidate the gate,
but it makes exploiter reduction a primary objective for the league phase.
## Old vs New Gate-3 Canary Comparison
| Metric | Old gate-3 vs cautious | New gate-3 vs expert | Interpretation |
| --- | ---: | ---: | --- |
| Mean score diff | +142.8993 | +43.3099 | New gate is harder and less suspiciously clean. |
| Win rate | 0.9596 | 0.8382 | New result is less inflated. |
| Opened colors | 4.9896 | 4.5509 | Improved, but still not selective enough. |
| Max-step rate | not recorded in old eval | 0.0000 | New expert evaluation does not stall. |
| Play action rate | 0.2763 | 0.6661 | New policy plays much more actively. |
| Exploiter win rate | 0.6712 | 0.8314 | New checkpoint is currently more exploitable. |
## Artifacts
| Item | Path |
| --- | --- |
| Expert mirror JSON | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/expert-mirror-1000-duplicate.json` |
| Expert vs old bots JSONs | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/` |
| Gate 1 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/` |
| Gate 2 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/` |
| Gate 3 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/` |
| Exploiter run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_014713_jax-ppo-ladder-v2-exploiter/` |
## Decisions
- The requested `lost-cities-league-selfplay-spec.md` file was not present in
the repository. I created `docs/plans/lost-cities-league-selfplay-spec.md` as
the tracked league spec location, following the repository docs routing for
active plans.
- Gate 2's high max-step rate is treated as a canary warning rather than a
gate failure because the v2 instruction keeps gate 2's original score-based
pass condition.
## Next
Proceed to league implementation only with the updated interpretation: anchors
fix the Elo scale and provide style diversity, but they do not certify strength.
The league's main success criterion should be lowering exploiter win rate from
the v2 baseline of `0.83135` while preserving positive duplicate performance
against `heuristic_expert`.
docs/reports/league-v1-2026-07-05-summary.jsonl
+18
View File
@@ -0,0 +1,18 @@
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 914.3801500956258, "event": "snapshot_eval", "games": 20000, "losses": 96.0, "max_steps_rate": 0.0, "mean_game_length": 71.11445, "mean_score_diff": 125.14285, "opened_colors_per_game": 4.4374, "opponent": "discard_only", "opponent_kind": "static", "play_action_rate": 0.6509964599542043, "positive_expeditions_per_game": 2.9703, "score_diff_ci95_high": 125.74118659681464, "score_diff_ci95_low": 124.54451340318535, "snapshot": "league_c01_u000250", "ties": 7.0, "update": 250, "wilson_high": 0.9957514261405391, "wilson_low": 0.9937585157747342, "win_rate": 0.99485, "wins": 19897.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 637.8568047136434, "event": "snapshot_eval", "games": 20000, "losses": 479.0, "max_steps_rate": 0.0231, "mean_game_length": 84.2736, "mean_score_diff": 100.8997, "opened_colors_per_game": 4.8582, "opponent": "heuristic_balanced", "opponent_kind": "static", "play_action_rate": 0.49003797546566286, "positive_expeditions_per_game": 2.6421, "score_diff_ci95_high": 101.6030587557229, "score_diff_ci95_low": 100.1963412442771, "snapshot": "league_c01_u000250", "ties": 17.0, "update": 250, "wilson_high": 0.9772657590483588, "wilson_low": 0.9729517298839193, "win_rate": 0.9752, "wins": 19504.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 522.2171364943674, "event": "snapshot_eval", "games": 20000, "losses": 931.0, "max_steps_rate": 0.1264, "mean_game_length": 139.4069, "mean_score_diff": 116.66435, "opened_colors_per_game": 4.9623, "opponent": "heuristic_cautious", "opponent_kind": "static", "play_action_rate": 0.3169406240961486, "positive_expeditions_per_game": 3.0465, "score_diff_ci95_high": 117.46585342388337, "score_diff_ci95_low": 115.86284657611662, "snapshot": "league_c01_u000250", "ties": 12.0, "update": 250, "wilson_high": 0.9557015978586921, "wilson_low": 0.9498244750852939, "win_rate": 0.95285, "wins": 19057.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 186.84131209092277, "event": "snapshot_eval", "games": 20000, "losses": 4910.0, "max_steps_rate": 0.0, "mean_game_length": 56.74285, "mean_score_diff": 26.7149, "opened_colors_per_game": 4.5478, "opponent": "heuristic_expert", "opponent_kind": "static", "play_action_rate": 0.6407205946284102, "positive_expeditions_per_game": 2.20505, "score_diff_ci95_high": 27.262035512864674, "score_diff_ci95_low": 26.167764487135326, "snapshot": "league_c01_u000250", "ties": 177.0, "update": 250, "wilson_high": 0.7516379759261593, "wilson_low": 0.7395676767594765, "win_rate": 0.74565, "wins": 14913.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 816.73245801838, "event": "snapshot_eval", "games": 20000, "losses": 178.0, "max_steps_rate": 0.03575, "mean_game_length": 124.3454, "mean_score_diff": 163.52095, "opened_colors_per_game": 4.99725, "opponent": "ladder_v2_gate1_discard", "opponent_kind": "checkpoint", "play_action_rate": 0.36905599758881236, "positive_expeditions_per_game": 3.6202, "score_diff_ci95_high": 164.3139031956684, "score_diff_ci95_low": 162.7279968043316, "snapshot": "league_c01_u000250", "ties": 2.0, "update": 250, "wilson_high": 0.9922178307682814, "wilson_low": 0.9895935898246238, "win_rate": 0.991, "wins": 19820.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 90.70646999257946, "event": "snapshot_eval", "games": 20000, "losses": 7275.0, "max_steps_rate": 0.0, "mean_game_length": 49.7744, "mean_score_diff": 16.1786, "opened_colors_per_game": 4.7712, "opponent": "ladder_v2_gate2_balanced", "opponent_kind": "checkpoint", "play_action_rate": 0.7223761262392399, "positive_expeditions_per_game": 2.14855, "score_diff_ci95_high": 16.80698121804704, "score_diff_ci95_low": 15.550218781952958, "snapshot": "league_c01_u000250", "ties": 172.0, "update": 250, "wilson_high": 0.6343247760445404, "wilson_low": 0.6209261971503363, "win_rate": 0.62765, "wins": 12553.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 246.30450760214026, "event": "snapshot_eval", "games": 20000, "losses": 3778.0, "max_steps_rate": 5e-05, "mean_game_length": 57.6601, "mean_score_diff": 43.7116, "opened_colors_per_game": 4.9119, "opponent": "ladder_v2_gate3_expert", "opponent_kind": "checkpoint", "play_action_rate": 0.6936249212033888, "positive_expeditions_per_game": 2.57955, "score_diff_ci95_high": 44.41070179719779, "score_diff_ci95_low": 43.012498202802206, "snapshot": "league_c01_u000250", "ties": 122.0, "update": 250, "wilson_high": 0.8104321748965465, "wilson_low": 0.7994506831092297, "win_rate": 0.805, "wins": 16100.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 630.8113677090081, "event": "snapshot_eval", "games": 20000, "losses": 494.0, "max_steps_rate": 0.0, "mean_game_length": 69.40915, "mean_score_diff": 107.6151, "opened_colors_per_game": 4.88455, "opponent": "discard_only", "opponent_kind": "static", "play_action_rate": 0.6697188054808169, "positive_expeditions_per_game": 2.9962, "score_diff_ci95_high": 108.30476814184044, "score_diff_ci95_low": 106.92543185815956, "snapshot": "league_c01_u000500", "ties": 22.0, "update": 500, "wilson_high": 0.9763077989189339, "wilson_low": 0.9719100740854563, "win_rate": 0.9742, "wins": 19484.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 560.1723021956866, "event": "snapshot_eval", "games": 20000, "losses": 740.0, "max_steps_rate": 0.0271, "mean_game_length": 82.88495, "mean_score_diff": 88.5699, "opened_colors_per_game": 4.93005, "opponent": "heuristic_balanced", "opponent_kind": "static", "play_action_rate": 0.5006923105759661, "positive_expeditions_per_game": 2.53425, "score_diff_ci95_high": 89.24524287663027, "score_diff_ci95_low": 87.89455712336974, "snapshot": "league_c01_u000500", "ties": 25.0, "update": 500, "wilson_high": 0.9643207034494167, "wilson_low": 0.9590019512527707, "win_rate": 0.96175, "wins": 19235.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 455.44923699476067, "event": "snapshot_eval", "games": 20000, "losses": 1336.0, "max_steps_rate": 0.16445, "mean_game_length": 148.75985, "mean_score_diff": 94.1284, "opened_colors_per_game": 4.9832, "opponent": "heuristic_cautious", "opponent_kind": "static", "play_action_rate": 0.28929498432939504, "positive_expeditions_per_game": 2.8439, "score_diff_ci95_high": 94.9311528056073, "score_diff_ci95_low": 93.3256471943927, "snapshot": "league_c01_u000500", "ties": 19.0, "update": 500, "wilson_high": 0.9356506507735038, "wilson_low": 0.9286833340559937, "win_rate": 0.93225, "wins": 18645.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 111.06146709621818, "event": "snapshot_eval", "games": 20000, "losses": 6672.0, "max_steps_rate": 0.0, "mean_game_length": 53.33205, "mean_score_diff": 14.2458, "opened_colors_per_game": 4.8409, "opponent": "heuristic_expert", "opponent_kind": "static", "play_action_rate": 0.6926969797194702, "positive_expeditions_per_game": 2.16835, "score_diff_ci95_high": 14.735340147409243, "score_diff_ci95_low": 13.756259852590755, "snapshot": "league_c01_u000500", "ties": 236.0, "update": 500, "wilson_high": 0.6611596974601753, "wilson_low": 0.647980924991277, "win_rate": 0.6546, "wins": 13092.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 683.7735272455216, "event": "snapshot_eval", "games": 20000, "losses": 373.0, "max_steps_rate": 0.0419, "mean_game_length": 123.1991, "mean_score_diff": 141.3215, "opened_colors_per_game": 4.99905, "opponent": "ladder_v2_gate1_discard", "opponent_kind": "checkpoint", "play_action_rate": 0.3599053985607506, "positive_expeditions_per_game": 3.56165, "score_diff_ci95_high": 142.1693389362133, "score_diff_ci95_low": 140.47366106378666, "snapshot": "league_c01_u000500", "ties": 10.0, "update": 500, "wilson_high": 0.9826591308155324, "wilson_low": 0.9788561881093119, "win_rate": 0.98085, "wins": 19617.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 118.72098430949596, "event": "snapshot_eval", "games": 20000, "losses": 6562.0, "max_steps_rate": 5e-05, "mean_game_length": 49.765, "mean_score_diff": 20.6296, "opened_colors_per_game": 4.9362, "opponent": "ladder_v2_gate2_balanced", "opponent_kind": "checkpoint", "play_action_rate": 0.771107881039547, "positive_expeditions_per_game": 2.2327, "score_diff_ci95_high": 21.259362634675494, "score_diff_ci95_low": 19.999837365324506, "snapshot": "league_c01_u000500", "ties": 148.0, "update": 500, "wilson_high": 0.6710116102233363, "wilson_low": 0.6579252099142052, "win_rate": 0.6645, "wins": 13290.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 229.5355020840115, "event": "snapshot_eval", "games": 20000, "losses": 4093.0, "max_steps_rate": 5e-05, "mean_game_length": 55.7303, "mean_score_diff": 38.96295, "opened_colors_per_game": 4.96905, "opponent": "ladder_v2_gate3_expert", "opponent_kind": "checkpoint", "play_action_rate": 0.7273133036132009, "positive_expeditions_per_game": 2.5661, "score_diff_ci95_high": 39.63449150676593, "score_diff_ci95_low": 38.291408493234066, "snapshot": "league_c01_u000500", "ties": 119.0, "update": 500, "wilson_high": 0.7949949671932197, "wilson_low": 0.7836938823375068, "win_rate": 0.7894, "wins": 15788.0}
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 30.163477370688103, "event": "snapshot_eval", "games": 20000, "losses": 8961.0, "max_steps_rate": 0.0001, "mean_game_length": 50.86105, "mean_score_diff": 5.5848, "opened_colors_per_game": 4.94555, "opponent": "league_c01_u000250", "opponent_kind": "checkpoint", "play_action_rate": 0.7468741704926414, "positive_expeditions_per_game": 2.3624, "score_diff_ci95_high": 6.236320508695035, "score_diff_ci95_low": 4.933279491304966, "snapshot": "league_c01_u000500", "ties": 173.0, "update": 500, "wilson_high": 0.5501945130045903, "wilson_low": 0.536388856672951, "win_rate": 0.5433, "wins": 10866.0}
{"cycle": 1, "eval_json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/eval_vs_league_c01_u000250_duplicate.json", "event": "exploiter_eval", "exploiter": "exploiter_c1", "exploiter_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "games": 20000, "losses": 9759.0, "max_steps_rate": 0.0, "mean_game_length": 48.76485, "mean_score_diff": 0.6818, "opened_colors_per_game": 4.99735, "play_action_rate": 0.7938232266689632, "positive_expeditions_per_game": 2.5895, "score_diff_ci95_high": 1.283273913015642, "score_diff_ci95_low": 0.08032608698435773, "target": "league_c01_u000250", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "ties": 196.0, "wilson_high": 0.5091783515083876, "wilson_low": 0.4953207843293599, "win_rate": 0.50225, "wins": 10045.0}
{"cycle": 1, "eval_json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/eval_vs_league_c01_u000500_duplicate.json", "event": "exploiter_final_eval", "exploiter": "exploiter_c1", "exploiter_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "games": 20000, "losses": 10376.0, "max_steps_rate": 0.0, "mean_game_length": 48.23185, "mean_score_diff": -2.63775, "opened_colors_per_game": 4.9971, "play_action_rate": 0.797814966505273, "positive_expeditions_per_game": 2.569, "score_diff_ci95_high": -2.039857230939478, "score_diff_ci95_low": -3.2356427690605223, "target": "league_c01_u000500", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "ties": 205.0, "wilson_high": 0.4778727301883191, "wilson_low": 0.46403842710654053, "win_rate": 0.47095, "wins": 9419.0}
{"cycles_completed": 1, "event": "league_complete", "final_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1", "stop_reason": "success", "updates": 500}
docs/reports/league-v1-2026-07-05.md
+53
View File
@@ -0,0 +1,53 @@
# League v1 Report - 2026-07-05
**Status:** `success`.
**Run dir:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1`.
**Final checkpoint:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500`.
## Summary
- Updates: 500
- Cycles completed: 1
- Success threshold: exploiter win rate <= 0.60 and expert CI low > 0.00
## Exploiter Series
| Type | Cycle | Win rate | Mean diff | Opened colors | Max-step | Target |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| cycle-target | 1 | 0.5022 | +0.6818 | 4.9973 | 0.0000 | `league_c01_u000250` |
| final-checkpoint | 1 | 0.4709 | -2.6378 | 4.9971 | 0.0000 | `league_c01_u000500` |
## Expert Evaluation Series
| Cycle | Update | Win rate | Mean diff | CI low | Opened colors | Max-step |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| 1 | 250 | 0.7457 | +26.7149 | +26.1678 | 4.5478 | 0.0000 |
| 1 | 500 | 0.6546 | +14.2458 | +13.7563 | 4.8409 | 0.0000 |
## Final Anchor Table
| Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step | Elo est. |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| `discard_only` | 0.9742 | +107.6151 | +106.9254 | 4.8845 | 0.0000 | +630.8 |
| `heuristic_balanced` | 0.9617 | +88.5699 | +87.8946 | 4.9300 | 0.0271 | +560.2 |
| `heuristic_cautious` | 0.9323 | +94.1284 | +93.3256 | 4.9832 | 0.1645 | +455.4 |
| `heuristic_expert` | 0.6546 | +14.2458 | +13.7563 | 4.8409 | 0.0000 | +111.1 |
| `ladder_v2_gate1_discard` | 0.9808 | +141.3215 | +140.4737 | 4.9991 | 0.0419 | +683.8 |
| `ladder_v2_gate2_balanced` | 0.6645 | +20.6296 | +19.9998 | 4.9362 | 0.0001 | +118.7 |
| `ladder_v2_gate3_expert` | 0.7894 | +38.9629 | +38.2914 | 4.9691 | 0.0001 | +229.5 |
| `league_c01_u000250` | 0.5433 | +5.5848 | +4.9333 | 4.9455 | 0.0001 | +30.2 |
## Plots
- `docs/reports/league-v1-exploiter-win-rate.png`
- `docs/reports/league-v1-opened-colors.png`
- `docs/reports/league-v1-elo-estimate.png`
## Hypothesis Read
부분 지지: 피탈률 목표는 한 사이클 만에 달성했다. 다만 expert 상대 오픈 색은 4.55에서 4.84로 증가해, 적대적 압력이 selectivity를 유도한다는 하위 가설은 아직 지지되지 않는다.
## Decisions
- Training uses a PFSP active subset when the pool exceeds `max_active_pool_members=12`. The full pool is retained for lifecycle and evaluation; the subset keeps single-GPU cycle time within budget.
- Stalling anchors are capped between the configured floor and cap during PFSP sampling.
docs/reports/league-v1-elo-estimate.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 26 KiB

docs/reports/league-v1-exploiter-win-rate.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

docs/reports/league-v1-opened-colors.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 24 KiB

docs/reports/verification-pass-2026-07-04-transcripts.txt
+13166
View File
File diff suppressed because it is too large Load Diff
docs/reports/verification-pass-2026-07-04.md
+191
View File
@@ -0,0 +1,191 @@
# Verification Pass - 2026-07-04
**Status:** FAIL - stop before league self-play.
**Implementation start commit:** `bb52ef9`.
**Primary checkpoint under audit:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/latest`.
**Verification artifacts:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/`.
## Decision
Do not start snapshot-pool league self-play from the current ladder artifacts.
The duplicate/Wilson protocol is reproducible, and the obvious seat-perspective
bug was not found, but the cautious anchor is not a trustworthy permanent Elo
anchor: it stalls heavily in mirror play and gives the trained policy a very
clean win while the trained policy opens almost all colors.
## Transcript Audit
Transcript file:
```text
docs/reports/verification-pass-2026-07-04-transcripts.txt
```
Rendered protocol: 10 fixed shuffle-bank deck orders, duplicate seat-swapped,
for 20 total games of gate-3 checkpoint vs `heuristic_cautious`.
Summary from the transcript dump:
| Metric | Value |
| --- | ---: |
| Games | 20 |
| Agent mean score diff | +144.500 |
| Agent score diff min/max | -93 / +230 |
| Agent opened colors mean | 5.000 |
| Mean game length | 159.200 |
| Length min / p50 / p95 / max | 63 / 109 / 400 / 400 |
| Max-steps rate | 0.150 |
| Handshake play events | 86 |
| Cautious openings | 78 |
| Cautious low openings below rank 7 | 3 |
| Cautious discards immediately playable by opponent | 162 |
Interpretation: the sample confirms the concern behind this pass. The agent is
not winning by a restrained 2-3 color expert pattern; it opens all 5 colors on
average and still wins by a wide margin. The cautious policy is also leaking
many immediately useful discard tops.
## Protocol Recheck
The original ladder evaluation was audited in code. The `eval` path uses:
- explicit shuffle banks generated from `evaluation.shuffle_bank_seed`;
- duplicate seat-swapped loops over learner seat 0 and learner seat 1;
- Wilson confidence intervals from aggregate wins.
The protocol itself is not the main failure. The failure is that the original
gate summary omitted `max_steps_rate` and score-diff distribution canaries.
Recomputed gate-3 duplicate evaluation with distribution canaries:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/gate3_reverification_distribution.json
```
| Metric | Value |
| --- | ---: |
| Games | 20,000 |
| Win rate | 0.95960 |
| Wilson 95% | [0.95678, 0.96224] |
| Mean score diff | +142.9178 |
| Score diff p05 / p50 / p95 | +21 / +152 / +227 |
| Opened colors/game | 4.98965 |
| Positive expeditions/game | 4.09470 |
| Play action rate | 0.27639 |
| Mean game length | 185.3179 |
| Game length p50 / p95 / max | 115 / 400 / 400 |
| Max-steps rate | 0.24070 |
The 24.07% forced-end rate is a high-severity canary failure for using this
gate as a clean league baseline.
## Perspective And Mirror Tests
New tests added:
- hand-authored heuristic behavior checks for weak unopened hands, strong
balanced opens, and P1 own-board perspective;
- static-policy duplicate mirror checks for all three heuristic policies;
- gate-3 checkpoint duplicate self-mirror check, skipped on CPU-only runs and
executed under CUDA JAX locally.
Commands run:
```bash
uv run pytest -q tests/lost_cities_jax/test_ppo_stack.py
uv run --with 'jax[cuda12]' pytest -q \
tests/lost_cities_jax/test_ppo_stack.py::test_gate3_checkpoint_duplicate_self_mirror_score_diff_is_zero
```
Results:
- CPU focused tests: 11 passed, 1 skipped.
- CUDA checkpoint self-mirror test: 1 passed.
Mirror evaluation artifacts:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_discard_only.json
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_heuristic_balanced.json
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_heuristic_cautious.json
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_gate3_checkpoint.json
```
| Mirror policy | Mean diff | Win/loss symmetry | Max-steps rate | Play action rate | Mean length |
| --- | ---: | --- | ---: | ---: | ---: |
| `discard_only` | 0.0 | exact | 0.0000 | 0.0000 | 44.0000 |
| `heuristic_balanced` | 0.0 | exact | 0.4683 | 0.0422 | 231.1210 |
| `heuristic_cautious` | 0.0 | exact | 0.8956 | 0.0053 | 368.8435 |
| gate-3 checkpoint | 0.0 | exact | 0.0000 | 0.7839 | 45.4292 |
Interpretation: the broad P0/P1 perspective bug is unlikely. Duplicate mirror
score differences cancel exactly. The serious issue is heuristic quality:
`heuristic_cautious` mirror play almost never opens expeditions and reaches
forced termination in 89.56% of games.
## Exploiter Baseline
Frozen opponent: gate-3 checkpoint. New exploiter: random init PPO, same 8192 x
400 rollout shape, 250 updates, shaping annealed by the standard schedule.
Training run:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_235948_jax-ppo-gate3-exploiter/
```
Final train-row canaries:
| Metric | Value |
| --- | ---: |
| return_mean | +0.24118 |
| play_action_rate | 0.71362 |
| opened_colors_mean | 4.91016 |
| positive_expeditions_mean | 2.07922 |
| game_length_mean | 49.12524 |
| max_steps_rate | 0.00000 |
Duplicate exploiter evaluation:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/exploiter_vs_gate3_duplicate.json
```
| Metric | Value |
| --- | ---: |
| Games | 20,000 |
| Exploiter win rate | 0.67115 |
| Wilson 95% | [0.66461, 0.67763] |
| Mean score diff | +24.1371 |
| Score diff p05 / p50 / p95 | -50 / +21 / +110 |
| Opened colors/game | 4.89710 |
| Positive expeditions/game | 2.12865 |
| Play action rate | 0.74612 |
| Mean game length | 48.0017 |
| Max-steps rate | 0.00000 |
This is now the measured exploitability baseline. It is not by itself a stop
condition, but it reinforces that the current gate-3 checkpoint is not a
finished robust policy.
## Findings
| Severity | Finding | Evidence | Consequence |
| --- | --- | --- | --- |
| High | `heuristic_cautious` is a stalling anchor, not a strong cautious baseline. | Mirror max-steps rate 89.56%, play action rate 0.0053. | Do not use it as a permanent Elo anchor without fixing/replacing it. |
| High | Gate-3 result is contaminated by forced-end games and over-opening. | Gate-3 max-steps rate 24.07%, opened colors/game 4.98965. | The ladder PASS remains reproducible, but it should not become a league baseline. |
| Medium | Cautious discards many immediately playable cards to the opponent. | Transcript sample: 162 such discards in 20 games. | The agent may be exploiting discard leakage rather than learning robust play. |
| Medium | A same-budget PPO exploiter beats the gate-3 checkpoint. | Exploiter duplicate win rate 67.115%, mean diff +24.1371. | League success should reduce this number, but Phase 2 should wait for anchor repair. |
| Low | No broad seat perspective bug found. | Static and checkpoint duplicate mirrors have exact zero mean diff. | Seat symmetry is not the likely explanation for the ladder result. |
## Stop Condition
Phase 1 did not pass cleanly. Per the work order, Phase 2 is not started.
Recommended next work:
1. Replace or repair `heuristic_balanced` and `heuristic_cautious` so mirror
play has near-zero forced-end rate and realistic play/open rates.
2. Add `max_steps_rate` and game-length quantiles to every gate report.
3. Rerun the static ladder gates with the repaired anchors before initializing
a snapshot-pool league.
docs/research/README.md
+47
View File
@@ -0,0 +1,47 @@
# Research Notes Catalog
This file catalogs permanent research notes in `docs/research/`. `AGENTS.md` is
the workflow document for deciding where new documentation belongs.
## Deep CFR
- [Lost Cities Deep CFR selectivity ideas](lost_cities_selectivity.md) — Consolidates selectivity failure hypotheses, diagnostics, and interventions before expensive architecture changes. (2026-05-10)
- [Deep CFR 512x3 2000-iteration baseline analysis](deep-cfr-baseline-2000-analysis.md) — Finds advantage loss improving while evaluated average-policy quality degrades, requiring current-vs-average diagnostics. (2026-05-08)
- [Batched and parallel evaluation in Deep CFR](deep-cfr-batched-evaluation.md) — Batching games and parallelizing opponents cuts CUDA evaluation overhead after entropy stays on device. (2026-05-08)
- [Deep CFR evaluation profiling](deep-cfr-evaluation-profile-plan.md) — Defines runtime counters that separate policy inference, encoding, opponent logic, and engine costs. (2026-05-08)
- [Deep CFR evaluation performance](deep-cfr-evaluation-profile.md) — Shows serial batch-size-one CUDA evaluation is slower than CPU until evaluation is batched. (2026-05-08)
- [Deep CFR legacy parity and hyperparameter mapping](deep-cfr-legacy-experiment-reproduction.md) — Maps legacy features, architecture, and traversal knobs needed for meaningful reproduction comparisons. (2026-05-08)
- [Deep CFR runtime: legacy vs. current implementation](deep-cfr-legacy-runtime-comparison.md) — Attributes current speedups to Cython traversal, batched evaluation, and opponent-parallel execution. (2026-05-08)
- [Deep CFR performance optimization and scaling](deep-cfr-performance-experiments.md) — Concludes small models regress under AMP or compile; interleaved traversal is the real win. (2026-05-08)
- [Advantage memory split performance optimization](deep-cfr-profile-advantage-memory-split.md) — Splitting advantage reservoirs by player removes linear sample filtering and stabilizes iteration time. (2026-05-08)
- [Deep CFR performance profile: advantage training bottlenecks](deep-cfr-profile.md) — Identifies shared-memory player filtering as the original advantage-training scaling bottleneck. (2026-05-08)
- [Deep CFR regret matching fallback and early over-opening](deep-cfr-regret-fallback-audit.md) — Shows uniform all-negative fallback frequently fires early and over-samples expedition-opening actions. (2026-05-08)
- [Deep CFR reproducibility policy](deep-cfr-reproducibility-policy.md) — Sets debug versus research reproducibility expectations and required matched-seed reporting practice. (2026-05-08)
- [Deep CFR reproducibility](deep-cfr-reproducibility.md) — Traces same-seed multi-worker divergence to completion-order-sensitive reservoir insertion and sampling. (2026-05-08)
- [Deep CFR v0: architectural parity and performance gaps](deep-cfr-v0-gap-vs-coolrl.md) — Confirms functional parity while naming synchronous recursive policy inference as the scaling bottleneck. (2026-05-08)
- [Deep CFR Cython traversal architecture](deep-cfr-v0-plan.md) — Explains mutation-based Cython traversal, chance sampling, and non-leaking information-state encoding. (2026-05-08)
- [Option A benchmark and structural ceiling](option-a-bench-result.md) — Shows centralized inference regressed because sync-blocking recursion capped realized GPU batch size. (2026-05-08)
- [Post-A optimization calculus](post-a-optimization-calculus.md) — Defers compile and TensorRT until model scale or evaluation density makes inference compute-bound. (2026-05-08)
- [Deep CFR package architecture and design rationale](deep-cfr-architecture.md) — Documents the Cython/Python module split that follows traversal hot paths versus orchestration. (2026-05-07)
- [Batched traversal inference decision](batched-traversal-inference-decision.md) — Records the initial Option A rationale and why later benchmarks redirected work toward interleaving. (2026-05-07)
- [Deep CFR v0 subsystem coverage vs. legacy reference](deep-cfr-v0-feature-parity.md) — Establishes that remaining legacy gaps are tooling and performance, not core algorithm correctness. (2026-05-07)
- [Opponent policy: network vs. self-play league](opponent-policy-network-divergence.md) — Explains why a live network opponent collapses while snapshot leagues preserve stationarity. (2026-05-07)
- [Outcome-sampling MCCFR advantage target](outcome-sampling-target.md) — Defends zero unsampled-action targets as the textbook importance-weighted outcome-sampling estimator. (2026-05-07)
- [Regret-matching all-negative fallback](regret-matching-fallback.md) — Keeps uniform as the safe default while documenting argmax-tiebreak's unsettled early-training effects. (2026-05-07)
- [Strategy memory recording location](strategy-memory-location.md) — Concludes traverser-node strategy samples are fine for outcome sampling, but external sampling needs OpenSpiel flags. (2026-05-07)
## SO-ISMCTS
- [SO-ISMCTS BC ceiling](ismcts-bc-ceiling-2026-05-11.md) — Finds behavior cloning remains the ceiling under current search and compute budgets. (2026-05-11)
## Engine / Performance
- [Classic game port architecture](classic-port-notes.md) — Describes the standalone Cython classic engine as the stable rules layer for all consumers. (2026-05-08)
- [Lost Cities match record v1](lost-cities-match-record-v1.md) — Defines the versioned JSONL format for complete hidden-state replay and post-game analysis. (2026-07-12)
- [Fast engine optimization architecture](fast-engine-next-optimizations.md) — Prioritizes C-level APIs, contiguous allocation, and zero-copy extraction for high-throughput RL. (2026-05-08)
- [Optimization sequencing](optimization_sequencing.md) — Orders runtime, traversal, model-scale, and inference optimizations to avoid invalidating experiments. (2026-05-07)
## Other
- [Julia port evaluation](julia_port_evaluation.md) — Rejects a full Julia port for now because GPU MLP inference misses the threshold. (2026-05-10)
- [Test coverage strategy for Python and Cython modules](test-coverage-notes.md) — Recommends Python-first coverage and isolated Cython tracing to protect performance artifacts. (2026-05-08)
docs/research/deep-cfr-performance-experiments.md
+2 -2
View File
@@ -54,5 +54,5 @@ The 8-worker interleaved path is more effective than a single-process CUDA path
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`: Implementation of AMP and training loops.
- `src/coolrl_lost_cities/games/classic/deep_cfr/networks.py`: `DeepCFRMLP` architecture.
- `configs/deep_cfr/default.yaml`: Configuration for interleaved scheduler.
- `scripts/profile_gpu_forward.py`: GPU forward pass micro-benchmarks.
- `legacy/deep-cfr/configs/default.yaml`: Configuration for interleaved scheduler.
- `scripts/profile_gpu_forward.py`: GPU forward pass micro-benchmarks.
docs/research/deep-cfr-v0-gap-vs-coolrl.md
+2 -2
View File
@@ -14,7 +14,7 @@ The core algorithmic components have been ported to Cython to ensure C-level per
- `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring.
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine.
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling.
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference.
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:406`: `def encode_info_state` generates the information-state feature vectors required for network inference.
- `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives.
## Analysis
@@ -39,4 +39,4 @@ Until this transition is made, optimization efforts should focus on reducing the
- `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`: Original status and gap analysis.
- `docs/research/deep-cfr-v0-feature-parity.md`: Detailed subsystem coverage report.
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
docs/research/ismcts-bc-ceiling-2026-05-11.md
+169
View File
@@ -0,0 +1,169 @@
# SO-ISMCTS BC Ceiling — 2026-05-11 Autonomous Session
**Last verified:** 2026-05-11, commit `cba6cae` (branch `autonomous/trap-exploration`)
## Short answer
Under our current compute budget (1 GPU, 12 CPU cores, 50 MCTS sims/move,
768x4 MLP), **behavior-cloning the heuristic-balanced bot is the ceiling**.
Across 13 self-play training variants, no run cleared the BC baseline of
21/100 wins vs `heuristic-cautious` in 100-game evaluation. Every variant
either preserved BC (KL anchor, mirror-descent target) or regressed toward
catastrophic forgetting (naive finetune, high-fraction mixed opponent).
The single largest improvement of the session came from PUCT Q-value
normalization at the *search* level, not from any learning change.
## Headline numbers (vs heuristic-cautious, 100 games, n_sims = 16)
| Run | Setup | W/100 | Notes |
|-------------------|------------------------------------------------|------:|-------|
| BC pretrain | 5k heuristic-vs-heuristic games, 20 epochs CE+MSE | 21 | baseline |
| C9 naive finetune | BC + plain self-play (no regularizer) | 0 | catastrophic forgetting |
| C10 KL β=1.0 | BC + self-play + KL(current ‖ BC) | ~21 | preserved BC, no improvement |
| C11 KL β=0.3 | weaker anchor | ~21 | preserved BC, no improvement |
| C12 mirror desc. | target = softmax(α log π_mcts + (1−α) log π_BC) | 19 | preserved BC, no improvement |
| C13 mixed=0.5 | 50 % games vs heuristic-balanced, opponent-aware MCTS, no KL | 0 | forgetting (worse than naive) |
| C14 mixed=0.2 | mixed-opponent + opponent-aware + KL β=1.0 | 17 | preserved BC, no improvement |
CIs (Wilson 95 %) overlap across all "preserved BC" rows; the 17–22 band
is statistically indistinguishable from the BC baseline.
## What actually moved the needle: PUCT Q normalization
`mcts.pyx _select_action` previously used the raw score-unit Q:
```
score = q_eff + c_puct * prior * sqrt(N) / (1 + n)
```
With `value_scale = 100` (Lost Cities score units), a single backup could
swing `q_eff` by ±100, while the exploration bonus is ~1–10. A noisy value
at the root permanently buried low-prior actions before they could be
explored.
Fix (`b9fc569`): divide Q by `q_scale` (defaults to 100) before scoring:
```
score = q_eff / q_scale + c_puct * prior * sqrt(N) / (1 + n)
```
Replaying the exact same BC checkpoint with this fix took win rate vs
heuristic-cautious from **5/100 → 21/100** — a 4× improvement from a
~10-line search change, with no retraining. Worth holding onto as the
load-bearing finding of the session.
## Hypotheses we negated
1. **Symmetric self-play eventually escapes the weak fixed point.**
Random-init + KL-free self-play ran for 100s of iterations across
C1–C8 without exceeding the noise floor (0–25 wins, all CIs overlap
each other and zero).
2. **Mixed-opponent self-play (Codex top pick) breaks the weak
equilibrium.** With opponent-aware MCTS so the search distribution
reflects the real opponent (per Codex's "pitfall" warning), C13
regressed to 0/100. The training signal from vs-heuristic games is
structurally negative — BC cannot beat the heuristic, so every mixed
sample is a loss, and the gradient labels every BC action as bad.
C14 cut the fraction to 0.2 and added a strong KL anchor (β = 1.0),
which preserved BC but did not lift it.
3. **Deeper search compensates for weak learning.** Increasing
`n_simulations` from 50 → 200 on the BC checkpoint *reduced* wins
vs `heuristic-balanced` from 28/64 → 14/64 in earlier probing.
Deeper search amplifies the network's preferences, including its
weaker ones, without supplying new information.
4. **A different regularizer would let self-play improve on BC.**
KL anchor (β ∈ {0.3, 1.0}) and mirror-descent target mixing (α
annealed 0.3 → 0.8) both kept the network glued to BC. Neither
supplied a positive gradient to walk away from it.
## Why BC is the ceiling — the mechanism
Self-play seeded from a strong heuristic faces a structural trap:
- BC has internalized the heuristic. Two BC copies playing each other
produce a near-symmetric outcome distribution; the visit counts at
most nodes give little policy-improvement signal beyond what BC
already encodes.
- Against the real heuristic, BC loses systematically (the heuristic
beats its own clone in approx. 79 % of games at our scale). The
resulting training signal is uniformly negative; learning that
signal pushes the policy *away* from BC without pointing anywhere
productive.
- With 50 MCTS sims/move on a 768x4 network, the search cannot
reliably *find* moves that beat the heuristic. So the only way out
of the trap — discovering a positive improvement direction —
is closed by the search-depth budget.
The result is consistent with the standard SO-ISMCTS picture: π_weak
(the symmetric weak fixed point) sits at roughly BC strength, π_Nash
is unreachable at this compute, and every variant we tried collapses
onto π_weak.
## Things left as configurable dials (no behavior change at defaults)
The `autonomous/trap-exploration` branch leaves the following in place
for future runs with more compute:
- `MctsConfig.q_scale` — PUCT Q normalization (defaults to 100, keep).
- `MctsConfig.root_dirichlet_alpha / epsilon` — AlphaZero exploration noise.
- `MctsConfig.opponent_aware_search` — when true, MCTS treats the
opponent seat as an external bot (skips tree expansion on opponent
turns, traverser-centered values).
- `TrainingConfig.mixed_opponent_fraction` — 0 disables (pure self-play).
- `TrainingConfig.mixed_opponent_bot` — bot name from
`coolrl_lost_cities.games.classic.bots.registry`.
- `TrainingConfig.kl_anchor_ckpt` / `kl_anchor_beta` — frozen reference
network for `KL(current ‖ ref)` regularization.
- `TrainingConfig.md_target_ref_ckpt` / `md_target_alpha_*` — mirror-
descent policy target with annealed mixing.
- `lost-cities-ismcts pretrain` — heuristic behavior cloning subcommand.
- `lost-cities-ismcts eval --ckpt … --n-sims N --games N --device cpu`
— standalone evaluator with Wilson CIs (`eval_checkpoint.py`).
## What would be worth trying with more compute
Not implemented here. These are the directions that the mechanism above
*does not rule out*:
- **Deeper search at training time** (n_sims ≫ 200, e.g. 800–1600).
Enough simulations should eventually surface a heuristic-beating
action somewhere in the search tree; that's a positive gradient.
- **Population training with frozen snapshots.** Periodically snapshot
the trainer and route 10–20 % of self-play games against the snapshot
pool. Combined with opponent-aware search, this gives a stationary
diverse-opponent gradient without the all-negative-signal problem of
pure heuristic mixing.
- **Value-weighted replay.** Prioritize high-error samples in the
buffer so the value head sees the cases where it disagrees with the
search rollout.
- **Larger / better-shaped networks.** 768x4 MLP may simply lack the
capacity to represent the conjunctions Lost Cities needs (color ×
expedition × hand composition). Attention or factored heads could be
worth probing.
## Code references
- Search-side: `src/coolrl_lost_cities/games/classic/ismcts/mcts.pyx`
(`_select_action`, `prepare_simulation`, `_expand_with_prior`).
- Python parity: `src/coolrl_lost_cities/games/classic/ismcts/mcts.py`.
- Mixed-opponent / opponent-aware wiring:
`src/coolrl_lost_cities/games/classic/ismcts/interleaved_self_play.py`.
- Regularization (KL anchor, mirror-descent) and metrics:
`src/coolrl_lost_cities/games/classic/ismcts/trainer.py`.
- BC pretrain: `src/coolrl_lost_cities/games/classic/ismcts/pretrain.py`.
- Eval CLI with Wilson CIs:
`src/coolrl_lost_cities/games/classic/ismcts/eval_checkpoint.py`.
- BC checkpoint (load with `--resume-from`):
`runs/pretrain/heuristic_balanced_5kg_20ep.pt` (5 k games, 20 epochs).
## Related memory
- `opponent-policy-network-divergence.md` — the Deep CFR analogue:
using the live network as its own opponent breaks stationarity and
diverges. The SO-ISMCTS picture here is the same family of failure:
bootstrapping from oneself does not provide a positive learning
signal.
docs/research/lost-cities-match-record-v1.md
+42
View File
@@ -0,0 +1,42 @@
# Lost Cities Match Record v1
Last verified: 2026-07-12
`coolrl.lost-cities.match.v1` is the canonical portable record for one complete
or in-progress classic Lost Cities match. Files use UTF-8 JSON Lines (`.jsonl`)
so metadata can be inspected without loading the full replay and steps can be
streamed in order.
## Row 1: metadata
The first row has `type: "metadata"` and `format:
"coolrl.lost-cities.match.v1"`. It records creation time, classic rules,
initial seed, human seat, opponent identity, and completion status. Producers
may add fields; v1 readers must ignore unknown metadata fields.
## Rows 2+: steps
Every remaining row has `type: "step"` and a contiguous zero-based `index`.
Step 0 is the initial position and has null `actor`, `phase_before`, and
`action_id`. Later steps contain:
- `actor`: player that took the action (`0` or `1`).
- `phase_before`: `card` or `draw`.
- `action_id`: the classic engine unified action ID.
- `state`: complete `GameState.to_snapshot()` output after the action.
- `public_hands`: cards known to be in each player's hand because they were
drawn from a discard pile. Entries use `{color, value, count}`; wagers have
value 0 and numbered cards use their printed value.
Full state snapshots intentionally include hidden hands and deck order. A match
record is therefore suitable for post-game analysis and deterministic replay,
but must not be exposed to a player during a live match.
## Compatibility
Readers must reject an unknown `format` value rather than guessing. New
optional fields may be added within v1. Any incompatible change to action IDs,
state encoding, or required row semantics requires `v2`.
The reference reader, writer, and validator are in
`src/coolrl_lost_cities/games/classic/match_record.py`.
docs/research/optimization_sequencing.md
+2 -2
View File
@@ -6,10 +6,10 @@
| Lever | 어디서 nail되는지 | 현재 상태 |
| --- | --- | --- |
| **Model size growth** (hidden ≥ 1024 / layers ≥ 6) | `docs/plans/model_size_experiment.md` | 인프라 미설치 |
| **Model size growth** (hidden ≥ 1024 / layers ≥ 6) | `docs/plans/archive/model_size_experiment.md` | 인프라 미설치 |
| **Option B** (per-worker interleaved traversal) | plan 미작성 | 미시작 |
| **AMP** trainer | `docs/plans/archive/amp_trainer.md` (구현 됨, default off) | 모델 키운 후 재측정 |
| **torch.compile** trainer | `docs/plans/torch_compile.md` | 모델 키운 후 재측정 |
| **torch.compile** trainer | `docs/plans/archive/torch_compile.md` | 모델 키운 후 재측정 |
| **TensorRT** inference | plan 미작성 | 모델 키운 + eval dense 시점 |
| **Option A re-enable** | 코드 있음 (default off) | 모델 키운 후 또는 Option B 후 |
| **Julia port** | `docs/research/julia_port_evaluation.md` | Torch.jl 결과 대기 중. Flux FAIL. |
docs/research/option-a-bench-result.md
+2 -2
View File
@@ -19,7 +19,7 @@ with torch.inference_mode():
advantages = networks[player](x).squeeze(0).detach().cpu().numpy().astype(np.float32)
```
When `traversal.inference_backend` is set to `server` in `configs/deep_cfr/default.yaml`, the `networks[player]` call is intercepted by a `NetworkProxy` (instantiated in `workers.py`, around line 91). This proxy posts a request to the `InferenceServer` and blocks until a response is received via a per-slot event.
When `traversal.inference_backend` is set to `server` in `legacy/deep-cfr/configs/default.yaml`, the `networks[player]` call is intercepted by a `NetworkProxy` (instantiated in `workers.py`, around line 91). This proxy posts a request to the `InferenceServer` and blocks until a response is received via a per-slot event.
The server's batching logic in `src/coolrl_lost_cities/games/classic/deep_cfr/inference_server.py` (around line 221) reports the realized batch size:
@@ -46,7 +46,7 @@ The "structural ceiling" is that `batch_window_us` and `max_batch` tuning cannot
## Practical implication
Option A is deferred for the current small MLP models (512x3). The `local` backend remains the default in `configs/deep_cfr/default.yaml`.
Option A is deferred for the current small MLP models (512x3). The `local` backend remains the default in `legacy/deep-cfr/configs/default.yaml`.
To unlock the projected GPU gains, the traversal must be restructured to drive batch sizes up. This leads to two primary paths:
1. **Option B (Interleaved Traversal):** Refactor the Cython traversal into a state machine that can advance multiple traversals concurrently per worker. Each worker would suspend at a policy call, batch its own requests, and resume continuations once the results return.
docs/research/post-a-optimization-calculus.md
+2 -2
View File
@@ -9,7 +9,7 @@ Why do architectural optimizations like `torch.compile` and TensorRT integration
## Code reference
The current baseline configuration is defined in `configs/deep_cfr/default.yaml`:
The historical baseline configuration is defined in `legacy/deep-cfr/configs/default.yaml`:
```yaml
network:
@@ -51,4 +51,4 @@ Avoid premature optimization with `torch.compile` or TensorRT on the current sma
- `docs/archive/deep-cfr-performance-experiments-2026-05-07.md` (Small-model regression data)
- `docs/archive/option-a-bench-result-2026-05-07.md` (Batched traversal benchmarks)
- `docs/performance.md` (Current runtime bottleneck profile)
- `docs/performance.md` (Current runtime bottleneck profile)
docs/research/strategy-memory-location.md
+1 -1
View File
@@ -46,7 +46,7 @@ sampling-mode branch. The `info_state` is computed by `_policy(state,
player, ...)` for the *current acting player*, which is the right thing in
both conventions.
`configs/deep_cfr/default.yaml` sets:
`legacy/deep-cfr/configs/default.yaml` sets:
```yaml
store_strategy_on_traverser_nodes: true
experiments/option_b_interleaved_traversal/README.md
+2 -2
View File
@@ -1,6 +1,6 @@
# Option B Interleaved Traversal Prototype
Experiment-only prototype for `docs/plans/option_b_interleaved_traversal.md`.
Experiment-only prototype for `docs/plans/archive/option_b_interleaved_traversal.md`.
It does not wire into the trainer and does not replace the production Cython
recursive traversal path.
@@ -37,7 +37,7 @@ uv run python experiments/option_b_interleaved_traversal/prototype_interleaved.p
--output experiments/option_b_interleaved_traversal/results_cuda.json
```
2026-05-07 results, `configs/deep_cfr/default.yaml`, RTX 3090 host:
2026-05-07 results, `legacy/deep-cfr/configs/default.yaml`, RTX 3090 host:
| Device | Mode | total s | forward s | scheduler s | batch mean | batch max | speedup |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
experiments/option_b_interleaved_traversal/prototype_interleaved.py
+1 -1
View File
@@ -650,7 +650,7 @@ def _build_proto_config(cfg: Any, max_depth: int | None, max_nodes: int | None)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", default="configs/deep_cfr/default.yaml")
parser.add_argument("--config", default="legacy/deep-cfr/configs/default.yaml")
parser.add_argument("--device", default="cpu")
parser.add_argument("--traversals", type=int, default=64)
parser.add_argument("--interleave-width", type=int, default=32)
experiments/traversal_policy_boundary/README.md
+1 -1
View File
@@ -30,7 +30,7 @@ uv run python experiments/traversal_policy_boundary/bench_policy_boundary.py \
--output experiments/traversal_policy_boundary/results_cuda.json
```
2026-05-07 results, `configs/deep_cfr/default.yaml`, RTX 3090 host:
2026-05-07 results, `legacy/deep-cfr/configs/default.yaml`, RTX 3090 host:
| Device | Component | Median us/call | p95 us/call |
| --- | --- | ---: | ---: |
experiments/traversal_policy_boundary/bench_policy_boundary.py
+1 -1
View File
@@ -321,7 +321,7 @@ def _print_table(result: dict[str, Any]) -> None:
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", default="configs/deep_cfr/default.yaml")
parser.add_argument("--config", default="legacy/deep-cfr/configs/default.yaml")
parser.add_argument("--device", default="cpu")
parser.add_argument("--traversals", type=int, default=32)
parser.add_argument("--runs", type=int, default=5)
legacy/deep-cfr/README.md
+4
View File
@@ -0,0 +1,4 @@
# Deep CFR archive
These YAML files are retained only to reproduce historical Deep CFR work. The
supported training stack is JAX PPO; start with `configs/jax_ppo/` instead.
configs/deep_cfr/default.yaml → legacy/deep-cfr/configs/default.yaml
View File
configs/deep_cfr/default_server.yaml → legacy/deep-cfr/configs/default_server.yaml
View File
configs/deep_cfr/smoke.yaml → legacy/deep-cfr/configs/smoke.yaml
View File
legacy/ismcts/README.md
+4
View File
@@ -0,0 +1,4 @@
# ISMCTS archive
These YAML files are retained only to reproduce historical ISMCTS work. The
supported training stack is JAX PPO; start with `configs/jax_ppo/` instead.
legacy/ismcts/autonomous_cycle_eval.sh
Executable
+31
View File
@@ -0,0 +1,31 @@
#!/bin/bash
# Autonomous cycle eval helper.
# Usage: ./autonomous_cycle_eval.sh <run-prefix> [extra eval args...]
# Finds latest run matching prefix, runs eval --ckpt latest.pt with 30 games,
# and reports: timeouts, natural-end wins per opponent.
set -euo pipefail
PREFIX="${1:-}"
shift || true
if [ -z "$PREFIX" ]; then
echo "usage: $0 <run-prefix> [extra eval args...]"
exit 1
fi
RUN=$(ls -td runs/*${PREFIX}* 2>/dev/null | head -1)
if [ -z "$RUN" ]; then
echo "no run matching ${PREFIX}" >&2
exit 1
fi
CKPT="$RUN/latest.pt"
if [ ! -f "$CKPT" ]; then
echo "no checkpoint at $CKPT" >&2
exit 1
fi
echo "=== eval $CKPT ==="
uv run python -m coolrl_lost_cities.games.classic.ismcts.cli \
eval --ckpt "$CKPT" --games 30 --verbose "$@" 2>&1
configs/ismcts/default.yaml → legacy/ismcts/configs/default.yaml
+19 -12
View File
@@ -1,8 +1,8 @@
run:
experiment_name: ismcts-default
max_iterations: 100
max_iterations: 500
seed: 1
device: auto
device: cuda
rules:
n_colors: 5
n_ranks: 9
@@ -17,15 +17,20 @@ encoding:
slot_aware_playability: true
network:
kind: mlp
hidden_size: 512
num_layers: 3
hidden_size: 768
num_layers: 4
activation: relu
mcts:
n_simulations: 50
c_puct: 1.5
c_puct: 5.0
max_depth: 200
parallel_simulations: 8
virtual_loss_value: 1.0
parallel_simulations: 64
virtual_loss_value: 5.0
eval_n_simulations: 16
rollout_policy: heuristic_balanced
use_rollout_value: false
root_dirichlet_alpha: 0.3
root_dirichlet_epsilon: 0.4
temperature:
training: 1.0
eval: 0.0
@@ -36,15 +41,17 @@ training:
replay_capacity: 100000
interleave_games: 8
interleave_max_batch: 64
num_workers: 8
worker_device: cuda
optimization:
learning_rate: 0.0003
grad_clip: 5.0
checkpoint:
save_every: 10
save_every: 20
save_latest: true
evaluation:
eval_every: 10
games: 20
eval_every: 5
games: 5
opponents: [random, discard-only, heuristic-cautious]
max_steps: 10000
num_workers: 1
max_steps: 500
num_workers: 8
configs/ismcts/mini.yaml → legacy/ismcts/configs/mini.yaml
View File
pyproject.toml
+10 -5
View File
@@ -1,7 +1,7 @@
[project]
name = "coolrl-lost-cities"
version = "0.1.0"
description = "Focused Lost Cities classic game extraction"
description = "JAX PPO training and browser play for Lost Cities"
readme = "README.md"
authors = [
{ name = "정시원", email = "sebastianrcnt@gmail.com" }
@@ -13,11 +13,16 @@ dependencies = [
"pydantic>=2.7",
"PyYAML>=6.0",
"torch>=2.3.0",
"jax>=0.10.2",
"chex>=0.1.92",
"flax>=0.12.7",
"optax>=0.2.8",
"orbax-checkpoint>=0.12.1",
]
[project.optional-dependencies]
gui = [
"pygame>=2.6.1",
"pygame-ce>=2.5.3",
"pygame-gui>=0.6.14",
]
wandb = [
@@ -28,8 +33,8 @@ wandb = [
lost-cities-classic = "coolrl_lost_cities.games.classic:main"
lost-cities-eval = "coolrl_lost_cities.games.classic.evaluation:main"
lost-cities-classic-gui = "coolrl_lost_cities.games.classic.pygame_pvp:main"
lost-cities-deep-cfr = "coolrl_lost_cities.games.classic.deep_cfr.cli:main"
lost-cities-ismcts = "coolrl_lost_cities.games.classic.ismcts.cli:main"
lost-cities-play = "coolrl_lost_cities.games.classic.pygame_table:main"
lost-cities-jax-ppo = "lost_cities_jax.ppo_cli:main"
[dependency-groups]
dev = [
@@ -45,7 +50,7 @@ build-backend = "setuptools.build_meta"
[tool.setuptools.packages.find]
where = ["src"]
include = ["coolrl_lost_cities*"]
include = ["coolrl_lost_cities*", "lost_cities_jax*"]
[tool.setuptools.package-data]
"coolrl_lost_cities.games.classic" = [
reference/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Reference implementations used by tests."""
reference/lost_cities_ref.py
+289
View File
@@ -0,0 +1,289 @@
"""Independent pure-Python Lost Cities rules reference.
This module intentionally uses ordinary Python containers rather than the JAX
state representation. It is deterministic under an explicit ``deck_order`` and
the shared flat action encoding.
"""
from __future__ import annotations
from bisect import insort
from dataclasses import dataclass
from random import Random
N_PLAYERS = 2
N_COLORS = 5
CARDS_PER_COLOR = 12
N_CARDS = N_COLORS * CARDS_PER_COLOR
HAND_SIZE = 8
INITIAL_DEAL = N_PLAYERS * HAND_SIZE
MAX_STEPS = 400
N_ACTIONS = HAND_SIZE * 2 * 6
LOC_DECK = 0
LOC_P0_HAND = 1
LOC_P1_HAND = 2
LOC_P0_BOARD = 3
LOC_P1_BOARD = 4
LOC_DISCARD = 5
NO_CARD = -1
PLAY = 0
DISCARD = 1
DRAW_DECK = 0
@dataclass
class RefState:
deck_order: list[int]
draw_ptr: int
card_loc: list[int]
hand_public: list[bool]
hands: list[list[int]]
board: list[list[list[int]]]
col_top: list[list[int]]
piles: list[list[int]]
to_move: int
just_discarded: int
step_count: int
done: bool
def reset(seed: int | None = None) -> RefState:
rng = Random(seed)
order = list(range(N_CARDS))
rng.shuffle(order)
return reset_from_order(order)
def reset_from_order(deck_order: list[int] | tuple[int, ...]) -> RefState:
order = [int(card) for card in deck_order]
card_loc = [LOC_DECK] * N_CARDS
p0_hand = sorted(order[:HAND_SIZE])
p1_hand = sorted(order[HAND_SIZE:INITIAL_DEAL])
for card in p0_hand:
card_loc[card] = LOC_P0_HAND
for card in p1_hand:
card_loc[card] = LOC_P1_HAND
return RefState(
deck_order=order,
draw_ptr=INITIAL_DEAL,
card_loc=card_loc,
hand_public=[False] * N_CARDS,
hands=[p0_hand, p1_hand],
board=[[[] for _ in range(N_COLORS)] for _ in range(N_PLAYERS)],
col_top=[[0 for _ in range(N_COLORS)] for _ in range(N_PLAYERS)],
piles=[[] for _ in range(N_COLORS)],
to_move=0,
just_discarded=NO_CARD,
step_count=0,
done=False,
)
def clone_state(state: RefState) -> RefState:
return RefState(
deck_order=list(state.deck_order),
draw_ptr=state.draw_ptr,
card_loc=list(state.card_loc),
hand_public=list(state.hand_public),
hands=[list(hand) for hand in state.hands],
board=[[list(col) for col in player] for player in state.board],
col_top=[list(player) for player in state.col_top],
piles=[list(pile) for pile in state.piles],
to_move=state.to_move,
just_discarded=state.just_discarded,
step_count=state.step_count,
done=state.done,
)
def decode_action(action: int) -> tuple[int, int, int]:
action = int(action)
hand_slot = action // 12
rem = action % 12
place_type = rem // 6
draw_source = rem % 6
return hand_slot, place_type, draw_source
def hand_cards(state: RefState, player: int | None = None) -> list[int]:
if player is None:
player = state.to_move
return list(state.hands[player])
def legal_action_mask(state: RefState) -> list[bool]:
bits = legal_action_bits(state)
return [bool(bits & (1 << action)) for action in range(N_ACTIONS)]
def legal_action_bits(state: RefState) -> int:
bits = 0
if state.done:
return bits
player = state.to_move
hand = state.hands[player]
draw_bits = 1
for color, pile in enumerate(state.piles):
if pile and pile[-1] != state.just_discarded:
draw_bits |= 1 << (color + 1)
for hand_slot, card in enumerate(hand[:HAND_SIZE]):
base = hand_slot * 12
if can_play_card(state, player, card):
bits |= draw_bits << base
color = card_color(card)
discard_bits = draw_bits & ~(1 << (color + 1))
bits |= discard_bits << (base + 6)
return bits
def nth_legal_action(bits: int, index: int) -> int:
"""Return the ``index``-th set action bit from low to high."""
remaining = int(index)
action = 0
while bits:
if bits & 1:
if remaining == 0:
return action
remaining -= 1
action += 1
bits >>= 1
raise IndexError(index)
def step(
state: RefState, action: int, *, validate: bool = True
) -> tuple[RefState, list[float], bool]:
if state.done or action < 0 or action >= N_ACTIONS:
return clone_state(state), [0.0, 0.0], state.done
if validate and not legal_action_mask(state)[action]:
return clone_state(state), [0.0, 0.0], state.done
next_state = clone_state(state)
_, reward, done = step_in_place(next_state, action, validate=False)
return next_state, reward, done
def step_in_place(
state: RefState, action: int, *, validate: bool = True
) -> tuple[RefState, list[float], bool]:
if state.done or action < 0 or action >= N_ACTIONS:
return state, [0.0, 0.0], state.done
if validate and not legal_action_mask(state)[action]:
return state, [0.0, 0.0], state.done
player = state.to_move
hand_slot, place_type, draw_source = decode_action(action)
hand = state.hands[player]
card = hand.pop(hand_slot)
color = card_color(card)
state.hand_public[card] = False
if place_type == PLAY:
state.board[player][color].append(card)
if not is_handshake(card):
state.col_top[player][color] = rank(card)
state.card_loc[card] = LOC_P0_BOARD + player
else:
state.piles[color].append(card)
state.card_loc[card] = LOC_DISCARD
if draw_source == DRAW_DECK:
drawn = state.deck_order[state.draw_ptr]
state.draw_ptr += 1
public = False
else:
src = draw_source - 1
drawn = state.piles[src].pop()
public = True
insort(hand, drawn)
state.card_loc[drawn] = LOC_P0_HAND + player
state.hand_public[drawn] = public
state.just_discarded = NO_CARD
state.step_count += 1
state.done = (draw_source == DRAW_DECK and state.draw_ptr >= N_CARDS) or (
state.step_count >= MAX_STEPS
)
state.to_move = 1 - player
reward = board_score(state) if state.done else [0.0, 0.0]
return state, reward, state.done
def can_play_card(state: RefState, player: int, card: int) -> bool:
top_rank = state.col_top[player][card_color(card)]
if is_handshake(card):
return top_rank == 0
return rank(card) > top_rank
def board_score(state: RefState) -> list[float]:
return [float(sum(score_column(column) for column in player)) for player in state.board]
def score(state: RefState) -> list[float]:
return board_score(state)
def score_column(column: list[int]) -> int:
if not column:
return 0
handshakes = sum(1 for card in column if is_handshake(card))
rank_sum = sum(rank(card) for card in column if not is_handshake(card))
value = (rank_sum - 20) * (1 + handshakes)
if len(column) >= 8:
value += 20
return value
def card_color(card: int) -> int:
return int(card) // CARDS_PER_COLOR
def card_slot(card: int) -> int:
return int(card) % CARDS_PER_COLOR
def is_handshake(card: int) -> bool:
return card_slot(card) < 3
def rank(card: int) -> int:
return card_slot(card) - 1
def _can_draw_after_place(state: RefState, card: int, place_type: int, draw_source: int) -> bool:
if draw_source == DRAW_DECK:
return True
src = draw_source - 1
same_discard_pile = place_type == DISCARD and card_color(card) == src
after_len = len(state.piles[src]) + int(same_discard_pile)
if after_len == 0:
return False
after_top = card if same_discard_pile else state.piles[src][-1]
just_discarded = card if place_type == DISCARD else state.just_discarded
return after_top != just_discarded
__all__ = [
"N_ACTIONS",
"RefState",
"board_score",
"clone_state",
"decode_action",
"hand_cards",
"legal_action_bits",
"legal_action_mask",
"nth_legal_action",
"reset",
"reset_from_order",
"score",
"step",
"step_in_place",
]
scripts/ablate_match.py
+148
View File
@@ -0,0 +1,148 @@
#!/usr/bin/env python3
"""What is each piece of the match stack worth?
Train one variant per switch at identical compute, then play each against the full
stack in duplicate matches -- same three deals, same coins, both seats -- so deal
luck cancels and only the policy difference is left.
The variants do not share an observation shape (dropping the privileged critic or
the match features changes the input dims), so this carries its own head-to-head
rather than reusing match_eval's, which assumes one architecture.
"""
from __future__ import annotations
import json
from pathlib import Path
import jax
import jax.numpy as jnp
import numpy as np
from lost_cities_jax.match import (
match_legal_action_mask,
match_reset_from,
match_score,
match_step,
)
from lost_cities_jax.match_eval import MATCH_SCAN_STEPS, _wilson, match_bank
from lost_cities_jax.match_ppo import (
Ablation,
MatchActorCritic,
_seat_views,
create_match_train_state,
match_train,
)
from lost_cities_jax.ppo import load_config, mask_logits, restore_checkpoint
VARIANTS = [
Ablation(privileged_critic=False),
Ablation(both_seats=False),
Ablation(match_obs=False),
]
MATCHES = 4096
FULL_CHECKPOINT = Path("runs/jax-ppo-match/2026-07-15_030152_match-linear/latest")
def _load(cfg, checkpoint: Path, ablation: Ablation):
state = create_match_train_state(cfg, jax.random.PRNGKey(0), ablation)
return restore_checkpoint(checkpoint, state).params
def _existing_run(label: str) -> Path | None:
runs = sorted(Path("runs/jax-ppo-match").glob(f"*ablate-{label}"))
for run in reversed(runs):
if (run / "latest").exists():
return run
return None
def duel(cfg, params_a, ablation_a, params_b, ablation_b, *, matches: int) -> dict:
"""A vs B over duplicate matches. Each side sees the world its own way."""
decks, coins = match_bank(20260719, matches)
model = MatchActorCritic(cfg.network.hidden_size, cfg.network.num_layers)
seats = jnp.arange(2, dtype=jnp.int32)
@jax.jit
def run(env, a_seat):
def body(carry, _):
env, _unused = carry
to_move = env.round.to_move.astype(jnp.int32)
mask = jax.vmap(match_legal_action_mask)(env)
def act(params, ablation):
obs, critic = _seat_views(env, seats, ablation)
idx = to_move[None, :, None]
seat_obs = jnp.take_along_axis(obs, idx, axis=0)[0]
seat_critic = jnp.take_along_axis(critic, idx, axis=0)[0]
logits, _ = model.apply(params, seat_obs, seat_critic)
return jnp.argmax(mask_logits(logits, mask), axis=-1).astype(jnp.int32)
action = jnp.where(
to_move == a_seat, act(params_a, ablation_a), act(params_b, ablation_b)
)
env, _, _ = jax.vmap(match_step, in_axes=(0, 0))(env, action)
return (env, _unused), None
(env, _), _ = jax.lax.scan(body, (env, jnp.int32(0)), xs=None, length=MATCH_SCAN_STEPS)
return env
leads = []
for seat in (0, 1):
env = jax.vmap(match_reset_from)(decks, coins)
final = run(env, jnp.full((matches,), seat, dtype=jnp.int32))
totals = np.asarray(jax.vmap(match_score)(final))
leads.append(totals[:, seat] - totals[:, 1 - seat])
lead = np.concatenate(leads)
games = float(lead.size)
wins = float((lead > 0).sum())
low, high = _wilson(wins, games)
return {
"matches": games,
"a_win_rate": wins / games,
"wilson_low": low,
"wilson_high": high,
"a_mean_lead": float(lead.mean()),
}
def main() -> None:
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
full = _load(cfg, FULL_CHECKPOINT, Ablation())
rows = []
for ablation in VARIANTS:
label = ablation.label()
run_dir = _existing_run(label)
if run_dir is None:
print(f"\n===== training {label} =====", flush=True)
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
cfg.run.experiment_name = f"ablate-{label}"
run_dir = match_train(cfg, ablation=ablation)
else:
print(f"\n===== reusing {run_dir} =====", flush=True)
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
params = _load(cfg, run_dir / "latest", ablation)
# The ablated net plays seat A, the full stack answers. Below 0.5 means
# the piece we removed was carrying weight.
result = duel(cfg, params, ablation, full, Ablation(), matches=MATCHES)
rows.append({"ablation": label, **result})
print(f" {label}: win rate vs full = {result['a_win_rate']:.4f}", flush=True)
print("\n\n==================== ablation ====================")
print(f"{'removed':<26}{'win rate vs full':>18}{'95% CI':>22}{'mean lead':>12}")
print("-" * 78)
for row in rows:
ci = f"[{row['wilson_low']:.3f}, {row['wilson_high']:.3f}]"
print(
f"{row['ablation']:<26}{row['a_win_rate']:>18.4f}{ci:>22}{row['a_mean_lead']:>+12.1f}"
)
print("\nbelow 0.5 = the removed piece was carrying weight")
Path("runs/jax-ppo-match/ablation.json").write_text(json.dumps(rows, indent=2, sort_keys=True))
if __name__ == "__main__":
main()
scripts/bench_inference_backend.py
+2 -2
View File
@@ -37,12 +37,12 @@ def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Benchmark Deep CFR traversal inference backends.")
parser.add_argument(
"--config-local",
default="configs/deep_cfr/default.yaml",
default="legacy/deep-cfr/configs/default.yaml",
help="Config for the local traversal inference backend.",
)
parser.add_argument(
"--config-server",
default="configs/deep_cfr/default_server.yaml",
default="legacy/deep-cfr/configs/default_server.yaml",
help="Config for the server traversal inference backend.",
)
parser.add_argument("--iterations", type=int, default=10)
scripts/diminishing_returns.py
+518
View File
@@ -0,0 +1,518 @@
from __future__ import annotations
import json
import time
from dataclasses import dataclass
from itertools import pairwise
from pathlib import Path
from typing import Any
import matplotlib.pyplot as plt
from lost_cities_jax.ppo import (
evaluate_checkpoint_match,
evaluate_checkpoint_vs_static,
load_config,
)
DATE = "2026-07-05"
ARTIFACT_DIR = Path("/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05")
REPORT_PATH = Path(f"docs/reports/diminishing-returns-{DATE}.md")
SUMMARY_PATH = Path(f"docs/reports/diminishing-returns-{DATE}-summary.jsonl")
GAMES_PER_SEAT = 2000
TOTAL_DUPLICATE_GAMES = GAMES_PER_SEAT * 2
@dataclass(frozen=True)
class CheckpointSpec:
name: str
label: str
config: str
checkpoint: str
stage_index: int
CHECKPOINTS = [
CheckpointSpec(
name="ladder_v2_gate3",
label="ladder v2 gate3",
config="configs/jax_ppo/ladder-v2-expert.yaml",
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest",
stage_index=0,
),
CheckpointSpec(
name="league_v1_update_250",
label="league v1 update 250",
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json",
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250",
stage_index=1,
),
CheckpointSpec(
name="league_v1_update_500",
label="league v1 update 500",
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json",
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500",
stage_index=2,
),
CheckpointSpec(
name="repair_c01_update_500",
label="repair c01 update 500",
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json",
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500",
stage_index=3,
),
]
EXPLOITER_ROWS = [
{
"phase": "ladder_v2",
"target": "ladder_v2_gate3",
"protocol": "short_random_shaping",
"budget": "same-budget phase-1 exploiter",
"warmstart": "random",
"win_rate": 0.83135,
"mean_score_diff": 54.7721,
"source": "docs/reports/ladder-v2-2026-07-05-summary.jsonl",
},
{
"phase": "league_v1",
"target": "league_v1_update_250",
"protocol": "league_cycle_exploiter",
"budget": "league v1 cycle exploiter",
"warmstart": "random",
"win_rate": 0.50225,
"mean_score_diff": 0.6818,
"source": "docs/reports/league-v1-2026-07-05-summary.jsonl",
},
{
"phase": "league_v1",
"target": "league_v1_update_500",
"protocol": "league_cycle_exploiter",
"budget": "same exploiter vs final",
"warmstart": "random",
"win_rate": 0.47095,
"mean_score_diff": -2.63775,
"source": "docs/reports/league-v1-2026-07-05-summary.jsonl",
},
{
"phase": "gates_1_2_original",
"target": "league_v1_update_500",
"protocol": "long_random_shaping",
"budget": "1200 updates, shaping anneal",
"warmstart": "random",
"win_rate": 0.587,
"mean_score_diff": 10.95125,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
{
"phase": "gates_1_2_original",
"target": "league_v1_update_500",
"protocol": "warmstart_gate3_no_shaping",
"budget": "900 updates, shaping 0",
"warmstart": "ladder_v2_gate3",
"win_rate": 0.590,
"mean_score_diff": 11.38075,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
{
"phase": "gates_1_2_original",
"target": "league_v1_update_500",
"protocol": "replay_exploiter_no_shaping",
"budget": "900 updates, shaping 0",
"warmstart": "league_v1_cycle_1_exploiter",
"win_rate": 0.582,
"mean_score_diff": 10.42525,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
{
"phase": "repair_c01",
"target": "repair_c01_update_500",
"protocol": "long_random_shaping",
"budget": "1200 updates, shaping anneal",
"warmstart": "random",
"win_rate": 0.539,
"mean_score_diff": 6.27925,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
{
"phase": "repair_c01",
"target": "repair_c01_update_500",
"protocol": "warmstart_gate3_no_shaping",
"budget": "900 updates, shaping 0",
"warmstart": "ladder_v2_gate3",
"win_rate": 0.549,
"mean_score_diff": 7.0285,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
{
"phase": "repair_c01",
"target": "repair_c01_update_500",
"protocol": "replay_exploiter_no_shaping",
"budget": "900 updates, shaping 0",
"warmstart": "league_v1_cycle_1_exploiter",
"win_rate": 0.53925,
"mean_score_diff": 6.10225,
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
},
]
def main() -> None:
started = time.perf_counter()
ARTIFACT_DIR.mkdir(parents=True, exist_ok=True)
REPORT_PATH.parent.mkdir(parents=True, exist_ok=True)
h2h_rows = run_h2h()
expert_rows = run_expert_anchor()
exploiter_rows = add_exploiter_deltas(EXPLOITER_ROWS)
plot_paths = write_plots(h2h_rows, expert_rows, exploiter_rows)
report = build_report(h2h_rows, expert_rows, exploiter_rows, plot_paths)
REPORT_PATH.write_text(report, encoding="utf-8")
write_summary(h2h_rows, expert_rows, exploiter_rows, time.perf_counter() - started)
def run_h2h() -> list[dict[str, Any]]:
rows = []
for previous, current in pairwise(CHECKPOINTS):
output = ARTIFACT_DIR / f"h2h_{current.name}_vs_{previous.name}.json"
result = evaluate_checkpoint_match(
load_config(current.config),
current.checkpoint,
load_config(previous.config),
previous.checkpoint,
games=GAMES_PER_SEAT,
duplicate=True,
output=output,
)
rows.append(
{
"event": "h2h_adjacent",
"previous": previous.name,
"current": current.name,
"previous_label": previous.label,
"current_label": current.label,
"json": str(output),
**summarize_match(result),
}
)
return rows
def run_expert_anchor() -> list[dict[str, Any]]:
rows = []
for spec in CHECKPOINTS:
output = ARTIFACT_DIR / f"expert_anchor_{spec.name}.json"
result = evaluate_checkpoint_vs_static(
load_config(spec.config),
spec.checkpoint,
"heuristic_expert",
games=GAMES_PER_SEAT,
duplicate=True,
output=output,
)
rows.append(
{
"event": "expert_anchor",
"checkpoint": spec.name,
"label": spec.label,
"stage_index": spec.stage_index,
"json": str(output),
**summarize_match(result),
}
)
return rows
def summarize_match(result: dict[str, Any]) -> dict[str, Any]:
return {
"games": result["games"],
"win_rate": result["win_rate"],
"wilson_low": result["wilson_low"],
"wilson_high": result["wilson_high"],
"mean_score_diff": result["mean_score_diff"],
"score_diff_ci95_low": result["score_diff_ci95_low"],
"score_diff_ci95_high": result["score_diff_ci95_high"],
"score_diff_std": result["score_diff_std"],
"mean_game_length": result["mean_game_length"],
"max_steps_rate": result["max_steps_rate"],
"opened_colors_per_game": result["opened_colors_per_game"],
"play_action_rate": result["play_action_rate"],
"positive_expeditions_per_game": result["positive_expeditions_per_game"],
}
def add_exploiter_deltas(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
enriched = [dict(row) for row in rows]
by_protocol: dict[str, list[dict[str, Any]]] = {}
for row in enriched:
by_protocol.setdefault(row["protocol"], []).append(row)
for protocol_rows in by_protocol.values():
previous = None
for row in protocol_rows:
row["delta_from_previous_same_protocol"] = (
None if previous is None else row["win_rate"] - previous["win_rate"]
)
previous = row
return enriched
def write_plots(
h2h_rows: list[dict[str, Any]],
expert_rows: list[dict[str, Any]],
exploiter_rows: list[dict[str, Any]],
) -> dict[str, str]:
paths = {
"h2h": REPORT_PATH.parent / f"diminishing-returns-{DATE}-h2h.png",
"expert": REPORT_PATH.parent / f"diminishing-returns-{DATE}-expert.png",
"exploiter": REPORT_PATH.parent / f"diminishing-returns-{DATE}-exploiter.png",
}
plt.figure(figsize=(7.5, 4.2))
x = list(range(len(h2h_rows)))
means = [row["mean_score_diff"] for row in h2h_rows]
lows = [row["score_diff_ci95_low"] for row in h2h_rows]
highs = [row["score_diff_ci95_high"] for row in h2h_rows]
plt.errorbar(
x,
means,
yerr=[
[m - lo for m, lo in zip(means, lows, strict=True)],
[hi - m for m, hi in zip(means, highs, strict=True)],
],
fmt="o-",
capsize=4,
)
plt.axhline(0, color="black", linewidth=1)
plt.xticks(x, [row["current"] for row in h2h_rows], rotation=20, ha="right")
plt.ylabel("Mean score diff vs previous")
plt.title("Adjacent checkpoint gains")
plt.tight_layout()
plt.savefig(paths["h2h"], dpi=160)
plt.close()
plt.figure(figsize=(7.5, 4.2))
x = [row["stage_index"] for row in expert_rows]
means = [row["mean_score_diff"] for row in expert_rows]
lows = [row["score_diff_ci95_low"] for row in expert_rows]
highs = [row["score_diff_ci95_high"] for row in expert_rows]
plt.errorbar(
x,
means,
yerr=[
[m - lo for m, lo in zip(means, lows, strict=True)],
[hi - m for m, hi in zip(means, highs, strict=True)],
],
fmt="o-",
capsize=4,
)
plt.axhline(0, color="black", linewidth=1)
plt.xticks(x, [row["checkpoint"] for row in expert_rows], rotation=20, ha="right")
plt.ylabel("Mean score diff vs heuristic_expert")
plt.title("External anchor trajectory")
plt.tight_layout()
plt.savefig(paths["expert"], dpi=160)
plt.close()
plt.figure(figsize=(8.2, 4.4))
protocols = sorted({row["protocol"] for row in exploiter_rows})
for protocol in protocols:
series = [row for row in exploiter_rows if row["protocol"] == protocol]
plt.plot(
[row["target"] for row in series],
[row["win_rate"] for row in series],
"o-",
label=protocol,
)
plt.axhline(0.55, color="black", linewidth=1, linestyle="--", label="0.55 gate")
plt.xticks(rotation=20, ha="right")
plt.ylabel("Exploiter win rate")
plt.title("Exploitability by protocol")
plt.legend(fontsize=8)
plt.tight_layout()
plt.savefig(paths["exploiter"], dpi=160)
plt.close()
return {key: str(path) for key, path in paths.items()}
def build_report(
h2h_rows: list[dict[str, Any]],
expert_rows: list[dict[str, Any]],
exploiter_rows: list[dict[str, Any]],
plot_paths: dict[str, str],
) -> str:
expert_recent = expert_rows[-1]["mean_score_diff"] - expert_rows[-2]["mean_score_diff"]
expert_recent_ci_crosses_zero = (
expert_rows[-1]["score_diff_ci95_low"] <= expert_rows[-2]["score_diff_ci95_high"]
and expert_rows[-2]["score_diff_ci95_low"] <= expert_rows[-1]["score_diff_ci95_high"]
)
same_protocol_repairs = [
row
for row in exploiter_rows
if row["phase"] == "repair_c01" and row["delta_from_previous_same_protocol"] is not None
]
repair_deltas = [row["delta_from_previous_same_protocol"] for row in same_protocol_repairs]
mean_repair_drop = -sum(repair_deltas) / len(repair_deltas)
expected = estimate_expected_improvement(h2h_rows, expert_recent, mean_repair_drop)
judgment = (
f"추가 학습 1사이클(~1.5시간 GPU)의 기대 개선은 인접 H2H 최근 이득 "
f"{h2h_rows[-1]['mean_score_diff']:+.2f}점, expert 앵커 최근 변화 "
f"{expert_recent:+.2f}점, 동일 프로토콜 exploiter 평균 피탈률 감소 "
f"{mean_repair_drop:.3f}에 따라 대략 {expected} 수준으로 추정되며, "
f"수확체감 구간에 진입했다. 근거는 최근 H2H 이득이 첫 리그 전이보다 작고, "
f"expert 앵커 성능이 v1 update 250 이후 하락/회복을 반복하며, 보수 후 "
f"exploiter 최악값이 0.549로 게이트 바로 아래에 머문다는 세 측정의 일치다."
)
if not expert_recent_ci_crosses_zero:
judgment += " 단, repair가 expert 앵커를 유의하게 회복시킨 점은 별도 긍정 신호다."
lines = [
f"# Diminishing Returns Diagnostic - {DATE}",
"",
"신규 학습 없이 기존 체크포인트와 평가 롤아웃만 사용했다. 모든 duplicate 평가는 "
f"셔플 뱅크 seed 20260704, 2,000쌍({TOTAL_DUPLICATE_GAMES} games) 기준이다.",
"",
f"Raw artifacts: `{ARTIFACT_DIR}`",
"",
"## Checkpoints",
"",
"| Order | Name | Config | Checkpoint |",
"| ---: | --- | --- | --- |",
]
for spec in CHECKPOINTS:
lines.append(
f"| {spec.stage_index} | `{spec.name}` | `{spec.config}` | `{spec.checkpoint}` |"
)
lines.extend(
[
"",
"## 1. Adjacent Head-to-Head",
"",
f"![Adjacent H2H]({Path(plot_paths['h2h']).name})",
"",
"| Later | Earlier | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |",
"| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |",
]
)
for row in h2h_rows:
lines.append(match_table_row(row, row["current"], row["previous"]))
lines.extend(
[
"",
"## 2. External Anchor Trajectory",
"",
f"![Expert anchor]({Path(plot_paths['expert']).name})",
"",
"| Checkpoint | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |",
"| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
]
)
for row in expert_rows:
lines.append(anchor_table_row(row))
lines.extend(
[
"",
f"Recent expert-anchor slope: `{expert_recent:+.4f}` points "
f"({expert_rows[-2]['checkpoint']} -> {expert_rows[-1]['checkpoint']}).",
"",
"## 3. Exploitability Trajectory",
"",
f"![Exploitability]({Path(plot_paths['exploiter']).name})",
"",
"프로토콜이 다른 피탈률은 한 곡선에 섞지 않았다. `Delta`는 같은 프로토콜 안에서만 계산했다.",
"",
"| Phase | Target | Protocol | Budget | Warm start | Win rate | Delta same protocol | Mean diff | Source |",
"| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |",
]
)
for row in exploiter_rows:
delta = row["delta_from_previous_same_protocol"]
delta_text = "n/a" if delta is None else f"{delta:+.5f}"
source_name = Path(row["source"]).name
lines.append(
f"| `{row['phase']}` | `{row['target']}` | `{row['protocol']}` | "
f"{row['budget']} | `{row['warmstart']}` | {row['win_rate']:.5f} | "
f"{delta_text} | {row['mean_score_diff']:+.4f} | [{source_name}]({source_name}) |"
)
lines.extend(
[
"",
"## Judgment",
"",
judgment,
"",
"## Notes",
"",
"- H2H와 expert 앵커 평가는 이번 작업에서 새로 실행했다.",
"- Exploiter 표는 지금까지 생성된 리포트/JSONL의 기존 측정값만 재정리했다.",
"- 이 작업에서는 신규 학습, 봇 수정, 체크포인트 수정이 없었다.",
]
)
return "\n".join(lines) + "\n"
def match_table_row(row: dict[str, Any], left: str, right: str) -> str:
return (
f"| `{left}` | `{right}` | {row['win_rate']:.4f} | "
f"[{row['wilson_low']:.4f}, {row['wilson_high']:.4f}] | "
f"{row['mean_score_diff']:+.4f} | "
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] | "
f"{row['opened_colors_per_game']:.4f} | {row['max_steps_rate']:.4f} |"
)
def anchor_table_row(row: dict[str, Any]) -> str:
return (
f"| `{row['checkpoint']}` | {row['win_rate']:.4f} | "
f"[{row['wilson_low']:.4f}, {row['wilson_high']:.4f}] | "
f"{row['mean_score_diff']:+.4f} | "
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] | "
f"{row['opened_colors_per_game']:.4f} | {row['max_steps_rate']:.4f} |"
)
def estimate_expected_improvement(
h2h_rows: list[dict[str, Any]],
expert_recent: float,
mean_repair_drop: float,
) -> str:
latest_h2h = h2h_rows[-1]["mean_score_diff"]
previous_h2h = h2h_rows[-2]["mean_score_diff"]
conservative_h2h = max(0.0, min(latest_h2h, previous_h2h))
expert_component = max(0.0, expert_recent)
return (
f"+{conservative_h2h:.1f}~+{max(conservative_h2h, expert_component):.1f}점 "
f"또는 피탈률 -{mean_repair_drop:.3f} 내외"
)
def write_summary(
h2h_rows: list[dict[str, Any]],
expert_rows: list[dict[str, Any]],
exploiter_rows: list[dict[str, Any]],
elapsed_seconds: float,
) -> None:
rows = []
rows.extend(h2h_rows)
rows.extend(expert_rows)
rows.extend({"event": "exploiter_trajectory", **row} for row in exploiter_rows)
rows.append(
{
"event": "diminishing_returns_complete",
"elapsed_seconds": elapsed_seconds,
"artifact_dir": str(ARTIFACT_DIR),
"report": str(REPORT_PATH),
}
)
SUMMARY_PATH.write_text(
"\n".join(json.dumps(row, ensure_ascii=False, sort_keys=True) for row in rows) + "\n",
encoding="utf-8",
)
if __name__ == "__main__":
main()
scripts/export_jax_ppo_onnx.py
+117
View File
@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""Export the actor head of a JAX PPO Orbax checkpoint to ONNX."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import jax.numpy as jnp
import numpy as np
from lost_cities_jax.human_play import infer_config_path, load_agent
from lost_cities_jax.ppo import load_config
from lost_cities_jax.types import N_ACTIONS, OBS_DIM
def build_argparser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--checkpoint", type=Path, required=True)
parser.add_argument("--config", type=Path)
parser.add_argument("--output", type=Path, required=True)
return parser
def export_model(checkpoint: Path, config: Path | None, output: Path) -> None:
try:
import onnx
from onnx import TensorProto, helper, numpy_helper
from onnx.reference import ReferenceEvaluator
except ImportError as exc:
raise SystemExit("onnx is required; run with `uv run --with onnx ...`") from exc
config_path = infer_config_path(checkpoint) if config is None else config
cfg = load_config(config_path)
params, flax_model = load_agent(cfg, checkpoint)
dense = params["params"]
nodes = []
initializers = []
previous = "obs"
for index in range(cfg.network.num_layers):
layer = dense[f"Dense_{index}"]
weight_name = f"dense_{index}.weight"
bias_name = f"dense_{index}.bias"
linear_name = f"dense_{index}.linear"
output_name = f"dense_{index}.relu"
initializers.extend(
[
numpy_helper.from_array(np.asarray(layer["kernel"], dtype=np.float32), weight_name),
numpy_helper.from_array(np.asarray(layer["bias"], dtype=np.float32), bias_name),
]
)
nodes.append(helper.make_node("Gemm", [previous, weight_name, bias_name], [linear_name]))
nodes.append(helper.make_node("Relu", [linear_name], [output_name]))
previous = output_name
actor = dense[f"Dense_{cfg.network.num_layers}"]
initializers.extend(
[
numpy_helper.from_array(np.asarray(actor["kernel"], dtype=np.float32), "actor.weight"),
numpy_helper.from_array(np.asarray(actor["bias"], dtype=np.float32), "actor.bias"),
]
)
nodes.append(helper.make_node("Gemm", [previous, "actor.weight", "actor.bias"], ["logits"]))
graph = helper.make_graph(
nodes,
"coolrl-lost-cities-jax-ppo-actor",
[helper.make_tensor_value_info("obs", TensorProto.FLOAT, [None, OBS_DIM])],
[helper.make_tensor_value_info("logits", TensorProto.FLOAT, [None, N_ACTIONS])],
initializer=initializers,
)
model = helper.make_model(
graph,
producer_name="coolrl-lost-cities",
opset_imports=[helper.make_opsetid("", 17)],
)
model.ir_version = 8
onnx.checker.check_model(model)
sample = np.random.default_rng(20260713).normal(size=(3, OBS_DIM)).astype(np.float32)
expected_logits, _ = flax_model.apply(params, jnp.asarray(sample))
actual_logits = ReferenceEvaluator(model).run(None, {"obs": sample})[0]
np.testing.assert_allclose(actual_logits, np.asarray(expected_logits), rtol=2e-5, atol=2e-5)
np.testing.assert_array_equal(
np.argmax(actual_logits, axis=1), np.argmax(np.asarray(expected_logits), axis=1)
)
output.parent.mkdir(parents=True, exist_ok=True)
onnx.save(model, output)
model_bytes = output.read_bytes()
manifest = {
"format": "coolrl-lost-cities-jax-ppo-onnx-v1",
"model_file": output.name,
"model_size_bytes": len(model_bytes),
"model_sha256": hashlib.sha256(model_bytes).hexdigest(),
"source_checkpoint": checkpoint.name,
"source_config": config_path.name,
"observation_size": OBS_DIM,
"action_size": N_ACTIONS,
"hidden_size": cfg.network.hidden_size,
"num_layers": cfg.network.num_layers,
"dtype": "float32",
"validation_max_abs_error": float(
np.max(np.abs(actual_logits - np.asarray(expected_logits)))
),
}
output.with_suffix(".json").write_text(json.dumps(manifest, indent=2) + "\n")
print(f"exported {output} ({output.stat().st_size:,} bytes)")
def main() -> None:
args = build_argparser().parse_args()
export_model(args.checkpoint, args.config, args.output)
if __name__ == "__main__":
main()
scripts/final_cycles.py
+503
View File
@@ -0,0 +1,503 @@
from __future__ import annotations
import json
import shutil
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import yaml
from lost_cities_jax.gates import ExploiterSpec
from lost_cities_jax.league import run_league
from lost_cities_jax.ppo import (
evaluate_checkpoint_match,
evaluate_checkpoint_vs_static,
load_config,
train_against_checkpoint,
)
DATE = "2026-07-05"
ROOT = Path("/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05")
REPORT_PATH = Path(f"docs/reports/final-cycles-and-human-play-{DATE}.md")
SUMMARY_PATH = Path(f"docs/reports/final-cycles-and-human-play-{DATE}-summary.jsonl")
START_CONFIG = (
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/"
"gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json"
)
START_CHECKPOINT = (
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/"
"gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/"
"cycle_01_update_000500"
)
LEAGUE_TEMPLATE = "configs/jax_ppo/league-v1.yaml"
GAMES = 2000
PASS_THRESHOLD = 0.52
GUARD_CI_LOW = 0.0
GUARD_MAX_STEPS = 0.02
@dataclass(frozen=True)
class Target:
name: str
config: str
checkpoint: str
EXPLOITERS = [
ExploiterSpec(
name="long_random",
config="configs/jax_ppo/gates-1-2-exploiter-long-random.yaml",
notes="random init + shaping anneal, 1200 updates",
),
ExploiterSpec(
name="warmstart_gate3",
config="configs/jax_ppo/gates-1-2-exploiter-warmstart.yaml",
resume=(
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/"
"2026-07-05_013223_jax-ppo-ladder-v2-expert/latest"
),
notes="ladder v2 gate-3 warm start, shaping disabled, 900 updates",
),
ExploiterSpec(
name="replay_exploiter",
config="configs/jax_ppo/gates-1-2-exploiter-replay.yaml",
resume=(
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/"
"2026-07-05_052325_jax-ppo-league-v1/exploiters/"
"2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest"
),
notes="league v1 cycle-1 exploiter warm start, shaping disabled, 900 updates",
),
]
def main() -> None:
started = time.perf_counter()
ROOT.mkdir(parents=True, exist_ok=True)
REPORT_PATH.parent.mkdir(parents=True, exist_ok=True)
SUMMARY_PATH.write_text("", encoding="utf-8")
rows: list[dict[str, Any]] = []
current = Target("repair_c01_update_500", START_CONFIG, START_CHECKPOINT)
best = current
worst_protocol = "warmstart_gate3"
extra_pool: list[dict[str, Any]] = []
stop_reason = "max_cycles_exhausted"
for cycle in range(1, 3):
previous = current
league_config = write_league_config(cycle, previous, worst_protocol, extra_pool)
league_dir = run_league(league_config)
league_rows = read_jsonl(league_dir / "league_summary.jsonl")
completion = latest_row(league_rows, "league_complete")
if completion is None:
stop_reason = "league_missing_completion"
rows.append({"event": "final_cycle_error", "cycle": cycle, "reason": stop_reason})
break
current = Target(
f"final_cycle_{cycle:02d}",
str(league_dir / "main_ppo_config.json"),
completion["final_checkpoint"],
)
cycle_row = {
"event": "final_cycle_league",
"cycle": cycle,
"worst_protocol_used": worst_protocol,
"league_config": str(league_config),
"league_run_dir": str(league_dir),
"target_config": current.config,
"target_checkpoint": current.checkpoint,
}
rows.append(cycle_row)
append_jsonl(SUMMARY_PATH, cycle_row)
exploiter_member = exploiter_member_from_league(league_rows, cycle)
if exploiter_member is not None:
extra_pool.append(exploiter_member)
battery = run_battery(cycle, current)
for row in battery:
rows.append(row)
append_jsonl(SUMMARY_PATH, row)
judgment = battery_judgment(cycle, battery)
rows.append(judgment)
append_jsonl(SUMMARY_PATH, judgment)
guard = evaluate_guard(cycle, current)
rows.append(guard)
append_jsonl(SUMMARY_PATH, guard)
h2h = evaluate_h2h(cycle, current, previous)
rows.append(h2h)
append_jsonl(SUMMARY_PATH, h2h)
best = current
if not guard["passed"]:
stop_reason = "expert_guard_failed"
current = previous
best = previous
rows.append(
{
"event": "final_cycle_rollback",
"cycle": cycle,
"rolled_back_to": previous.name,
"rolled_back_checkpoint": previous.checkpoint,
"failed_checkpoint": guard["target_checkpoint"],
}
)
append_jsonl(SUMMARY_PATH, rows[-1])
break
if judgment["worst_win_rate"] <= PASS_THRESHOLD:
stop_reason = "success_exploiter_threshold"
break
if h2h["score_diff_ci95_low"] <= 0.0 <= h2h["score_diff_ci95_high"]:
stop_reason = "h2h_stagnation"
break
worst_protocol = judgment["worst_exploiter"]
final_candidate = fix_final_candidate(best)
final_row = {
"event": "final_cycles_complete",
"stop_reason": stop_reason,
"final_candidate": str(final_candidate),
"final_config": best.config,
"source_checkpoint": best.checkpoint,
"elapsed_seconds": time.perf_counter() - started,
}
rows.append(final_row)
append_jsonl(SUMMARY_PATH, final_row)
write_report(rows, final_row)
def write_league_config(
cycle: int, target: Target, worst_protocol: str, extra_pool: list[dict[str, Any]]
) -> Path:
data = yaml.safe_load(Path(LEAGUE_TEMPLATE).read_text(encoding="utf-8")) or {}
data["base_config"] = target.config
data["warm_start_checkpoint"] = target.checkpoint
run = data.setdefault("run", {})
run["experiment_name"] = f"jax-ppo-final-cycle-c{cycle:02d}"
run["artifact_root"] = str(ROOT / "league")
run["seed"] = int(run.get("seed", 20260705)) + 100 + cycle
league = data.setdefault("league", {})
league["cycles"] = 1
league["league_updates_per_cycle"] = 500
league["snapshot_interval_updates"] = 500
league["success_exploiter_win_rate"] = PASS_THRESHOLD
evaluation = data.setdefault("evaluation", {})
evaluation["games"] = GAMES
evaluation["batch_games"] = 8192
exploiter = data.setdefault("exploiter", {})
spec = exploiter_spec(worst_protocol)
exploiter["config"] = spec.config
exploiter["updates"] = load_config(spec.config).run.total_updates
if spec.resume:
exploiter["resume"] = spec.resume
else:
exploiter.pop("resume", None)
guards = data.setdefault("guards", {})
guards["expert_ci_low"] = GUARD_CI_LOW
guards["max_steps_rate"] = GUARD_MAX_STEPS
cycle_dir = ROOT / f"cycle_{cycle:02d}"
tracking = data.setdefault("tracking", {})
tracking["tracked_summary_path"] = str(cycle_dir / "league_summary.jsonl")
tracking["report_path"] = str(cycle_dir / "league_report.md")
anchors = list(data.get("anchors", []))
names = {item.get("name") for item in anchors}
for member in extra_pool:
if member["name"] not in names:
anchors.append(member)
names.add(member["name"])
data["anchors"] = anchors
path = cycle_dir / "league_config.yaml"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
return path
def evaluate_guard(cycle: int, target: Target) -> dict[str, Any]:
output = ROOT / f"cycle_{cycle:02d}" / "guard_vs_heuristic_expert.json"
result = evaluate_checkpoint_vs_static(
load_config(target.config),
target.checkpoint,
"heuristic_expert",
games=GAMES,
duplicate=True,
output=output,
)
return {
"event": "final_cycle_guard_expert",
"cycle": cycle,
"target": target.name,
"target_config": target.config,
"target_checkpoint": target.checkpoint,
"json": str(output),
"passed": result["score_diff_ci95_low"] > GUARD_CI_LOW
and result["max_steps_rate"] <= GUARD_MAX_STEPS,
**summary(result),
}
def evaluate_h2h(cycle: int, target: Target, previous: Target) -> dict[str, Any]:
output = ROOT / f"cycle_{cycle:02d}" / f"h2h_{target.name}_vs_{previous.name}.json"
result = evaluate_checkpoint_match(
load_config(target.config),
target.checkpoint,
load_config(previous.config),
previous.checkpoint,
games=GAMES,
duplicate=True,
output=output,
)
return {
"event": "final_cycle_h2h",
"cycle": cycle,
"current": target.name,
"previous": previous.name,
"current_checkpoint": target.checkpoint,
"previous_checkpoint": previous.checkpoint,
"json": str(output),
**summary(result),
}
def run_battery(cycle: int, target: Target) -> list[dict[str, Any]]:
rows = []
for spec in EXPLOITERS:
target_cfg = load_config(target.config)
exploiter_cfg = load_config(spec.config)
exploiter_cfg.run.artifact_root = str(ROOT / f"cycle_{cycle:02d}" / "battery")
exploiter_cfg.run.experiment_name = f"final-c{cycle:02d}-{spec.name}-exploiter"
train_dir = train_against_checkpoint(
exploiter_cfg,
target_cfg,
target.checkpoint,
resume=spec.resume,
)
output = train_dir / f"eval_vs_{target.name}_duplicate.json"
result = evaluate_checkpoint_match(
exploiter_cfg,
train_dir / "latest",
target_cfg,
target.checkpoint,
games=GAMES,
duplicate=True,
output=output,
)
rows.append(
{
"event": "final_cycle_exploiter",
"cycle": cycle,
"exploiter": spec.name,
"target": target.name,
"target_config": target.config,
"target_checkpoint": target.checkpoint,
"train_dir": str(train_dir),
"checkpoint": str(train_dir / "latest"),
"resume": spec.resume,
"notes": spec.notes,
"json": str(output),
**summary(result),
}
)
return rows
def battery_judgment(cycle: int, rows: list[dict[str, Any]]) -> dict[str, Any]:
worst = max(rows, key=lambda row: row["win_rate"])
return {
"event": "final_cycle_battery_judgment",
"cycle": cycle,
"worst_exploiter": worst["exploiter"],
"worst_win_rate": worst["win_rate"],
"threshold": PASS_THRESHOLD,
"passed": worst["win_rate"] <= PASS_THRESHOLD,
}
def exploiter_member_from_league(rows: list[dict[str, Any]], cycle: int) -> dict[str, Any] | None:
row = latest_row(rows, "exploiter_eval")
if row is None:
return None
checkpoint = Path(row["exploiter_checkpoint"])
return {
"name": f"final_cycle_exploiter_c{cycle:02d}",
"kind": "checkpoint",
"anchor": False,
"stalling": False,
"exploiter": True,
"config": str(checkpoint.parent / "config.json"),
"checkpoint": str(checkpoint),
"recent_win_rate": 1.0 - float(row["win_rate"]),
"created_cycle": cycle,
"created_update": 0,
}
def fix_final_candidate(target: Target) -> Path:
destination = ROOT / "final_candidate"
if destination.exists():
shutil.rmtree(destination)
shutil.copytree(target.checkpoint, destination)
(ROOT / "final_candidate_config.txt").write_text(target.config + "\n", encoding="utf-8")
shutil.copy2(target.config, ROOT / "main_ppo_config.json")
return destination
def write_report(rows: list[dict[str, Any]], final: dict[str, Any]) -> None:
guards = [row for row in rows if row.get("event") == "final_cycle_guard_expert"]
h2h = [row for row in rows if row.get("event") == "final_cycle_h2h"]
exploiters = [row for row in rows if row.get("event") == "final_cycle_exploiter"]
judgments = [row for row in rows if row.get("event") == "final_cycle_battery_judgment"]
lines = [
f"# Final Cycles and Human Play - {DATE}",
"",
"## Part A - Closing Reinforcement Cycles",
"",
f"Raw artifacts: `{ROOT}`",
f"Stop reason: `{final['stop_reason']}`",
f"Final candidate: `{final['final_candidate']}`",
f"Final candidate config: `{final['final_config']}`",
"",
"### Expert Guard",
"",
"| Cycle | Passed | Win rate | Mean diff | CI low | Opened colors | Max-step |",
"| ---: | ---: | ---: | ---: | ---: | ---: | ---: |",
]
for row in guards:
lines.append(match_row(row, leading=[str(row["cycle"]), str(row["passed"])]))
lines.extend(
[
"",
"### Adjacent H2H",
"",
"| Cycle | Current | Previous | Win rate | Mean diff | Score CI |",
"| ---: | --- | --- | ---: | ---: | ---: |",
]
)
for row in h2h:
lines.append(
f"| {row['cycle']} | `{row['current']}` | `{row['previous']}` | "
f"{row['win_rate']:.4f} | {row['mean_score_diff']:+.4f} | "
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] |"
)
lines.extend(
[
"",
"### Strengthened Exploiter Battery",
"",
"| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |",
"| ---: | --- | ---: | ---: | ---: | ---: | ---: |",
]
)
for row in exploiters:
lines.append(match_row(row, leading=[str(row["cycle"]), f"`{row['exploiter']}`"]))
lines.extend(
[
"",
"### Battery Judgment",
"",
"| Cycle | Worst exploiter | Worst win rate | Threshold | Passed |",
"| ---: | --- | ---: | ---: | ---: |",
]
)
for row in judgments:
lines.append(
f"| {row['cycle']} | `{row['worst_exploiter']}` | {row['worst_win_rate']:.4f} | "
f"{row['threshold']:.4f} | {row['passed']} |"
)
lines.extend(human_play_usage())
REPORT_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")
def human_play_usage() -> list[str]:
return [
"",
"## Part B - Human Play Interface",
"",
"Start a single game:",
"",
"```bash",
"uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \\",
" --checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \\",
" --seat 0",
"```",
"",
"Start a duplicate set with one shared shuffle and swapped seats:",
"",
"```bash",
"uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \\",
" --checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \\",
" --seat 0 --duplicate",
"```",
"",
"Summarize logged human games:",
"",
"```bash",
"uv run lost-cities-jax-ppo human-play summarize \\",
" --log-dir /mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/",
"```",
"",
"Move syntax: `play R7 draw deck`, `discard G3 draw Y`, or `play RHS draw deck`.",
"The renderer shows only the human hand, both boards, all public discard piles, deck count, and current board score differential. Opponent hand and deck order are not rendered.",
"Every game is appended to `/mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/games.jsonl` with deck seed/index, full action list, AI top-3 policy actions/probabilities, value outputs, scoring breakdown, and optional human comment.",
]
def match_row(row: dict[str, Any], leading: list[str]) -> str:
return (
"| " + " | ".join(leading) + f" | {row['win_rate']:.4f} | {row['mean_score_diff']:+.4f} | "
f"{row['score_diff_ci95_low']:+.4f} | {row['opened_colors_per_game']:.4f} | "
f"{row['max_steps_rate']:.4f} |"
)
def summary(result: dict[str, Any]) -> dict[str, Any]:
keys = [
"games",
"wins",
"losses",
"ties",
"win_rate",
"wilson_low",
"wilson_high",
"mean_score_diff",
"score_diff_ci95_low",
"score_diff_ci95_high",
"mean_game_length",
"max_steps_rate",
"opened_colors_per_game",
"play_action_rate",
"positive_expeditions_per_game",
]
return {key: result[key] for key in keys if key in result}
def exploiter_spec(name: str) -> ExploiterSpec:
for spec in EXPLOITERS:
if spec.name == name:
return spec
raise ValueError(f"unknown exploiter protocol: {name}")
def latest_row(rows: list[dict[str, Any]], event: str) -> dict[str, Any] | None:
return next((row for row in reversed(rows) if row.get("event") == event), None)
def read_jsonl(path: Path) -> list[dict[str, Any]]:
if not path.exists():
return []
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line]
def append_jsonl(path: Path, row: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(row, sort_keys=True) + "\n")
if __name__ == "__main__":
main()
scripts/generate_web_parity_fixture.py
+64
View File
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""Generate deterministic JAX states for TypeScript engine parity tests."""
from __future__ import annotations
import json
from pathlib import Path
import jax.numpy as jnp
import numpy as np
from lost_cities_jax.engine import legal_action_mask, reset_from_order, step
from lost_cities_jax.obs import observation
OUTPUT = Path(__file__).resolve().parents[1] / "web" / "src" / "game" / "parity-fixture.json"
def state_json(state) -> dict:
return {
"deckOrder": np.asarray(state.deck_order).astype(int).tolist(),
"drawPtr": int(state.draw_ptr),
"cardLoc": np.asarray(state.card_loc).astype(int).tolist(),
"handPublic": np.asarray(state.hand_public).astype(bool).tolist(),
"colTop": np.asarray(state.col_top).astype(int).tolist(),
"colHandshakes": np.asarray(state.col_hs).astype(int).tolist(),
"colLength": np.asarray(state.col_len).astype(int).tolist(),
"piles": [
np.asarray(state.pile[color, : int(state.pile_len[color])]).astype(int).tolist()
for color in range(5)
],
"toMove": int(state.to_move),
"stepCount": int(state.step_count),
"done": bool(state.done),
}
def main() -> None:
rng = np.random.default_rng(20260713)
order = rng.permutation(60).astype(np.int8)
state = reset_from_order(jnp.asarray(order))
rows = []
for index in range(24):
mask = np.asarray(legal_action_mask(state), dtype=bool)
rows.append(
{
"index": index,
"state": state_json(state),
"legalMask": mask.tolist(),
"observationP0": np.asarray(observation(state, jnp.int32(0))).tolist(),
"observationP1": np.asarray(observation(state, jnp.int32(1))).tolist(),
}
)
legal = np.flatnonzero(mask)
action = int(legal[(index * 17 + 3) % len(legal)])
rows[-1]["action"] = action
state, _, _ = step(state, jnp.int32(action))
if bool(state.done):
break
OUTPUT.write_text(json.dumps({"format": "jax-web-parity-v1", "rows": rows}) + "\n")
print(f"wrote {OUTPUT} ({len(rows)} states)")
if __name__ == "__main__":
main()
scripts/librarian-ignore.txt
+1 -8
View File
@@ -7,14 +7,7 @@
#
# Format examples:
# scripts/foo.sh # exact path
# configs/deep_cfr/model-*.yaml # glob
# configs/jax_ppo/model-*.yaml # glob
# docs/research/*.md # whole directory
#
# Comments after `#` are stripped per line. Blank lines ignored.
# Future config + script described in docs/plans/model_size_experiment.md
configs/deep_cfr/model-size-*.yaml
scripts/run_model_size_experiment.sh
# Future config described in docs/plans/torch_compile.md
configs/deep_cfr/default_compile.yaml
scripts/measure_exploitability.py
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Freeze each policy, train an exploiter against it on the same budget, compare.
Winning the head-to-head says a policy is strong on average. It does not say the
policy is hard to beat. This does: whatever the exploiter reaches is a habit the
frozen policy could not defend.
"""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import jax
import numpy as np
from lost_cities_jax.exploit import (
match_frozen_policy,
single_round_frozen_policy,
train_exploiter,
)
from lost_cities_jax.match import match_reset_from, match_score
from lost_cities_jax.match_eval import MATCH_SCAN_STEPS, _wilson, match_bank
from lost_cities_jax.match_ppo import Ablation, create_match_train_state
from lost_cities_jax.ppo import create_train_state, load_config, restore_checkpoint
OURS = Path("runs/jax-ppo-match/2026-07-15_031529_match-scaled/latest")
LEAGUE = Path(
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/latest"
)
MATCHES = 4096
def _final_score(cfg, exploiter_params, frozen, matches: int) -> dict:
"""Play the trained exploiter against the frozen policy, both seats."""
import jax.numpy as jnp
from lost_cities_jax.exploit import match_frozen_policy as _mk
from lost_cities_jax.match import match_step
attacker = _mk(cfg, exploiter_params, Ablation())
decks, coins = match_bank(20260722, matches)
@jax.jit
def run(env, a_seat):
def body(carry, _):
env, _u = carry
action = jnp.where(
env.round.to_move.astype(jnp.int32) == a_seat,
attacker(env, a_seat),
frozen(env, 1 - a_seat),
)
env, _, _ = jax.vmap(match_step, in_axes=(0, 0))(env, action)
return (env, _u), None
(env, _), _ = jax.lax.scan(body, (env, jnp.int32(0)), xs=None, length=MATCH_SCAN_STEPS)
return env
leads = []
for seat in (0, 1):
env = jax.vmap(match_reset_from)(decks, coins)
final = run(env, jnp.full((matches,), seat, dtype=jnp.int32))
totals = np.asarray(jax.vmap(match_score)(final))
leads.append(totals[:, seat] - totals[:, 1 - seat])
lead = np.concatenate(leads)
games = float(lead.size)
wins = float((lead > 0).sum())
low, high = _wilson(wins, games)
return {
"exploiter_win_rate": wins / games,
"wilson_low": low,
"wilson_high": high,
"exploiter_mean_lead": float(lead.mean()),
"matches": games,
}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--updates", type=int, default=250, help="exploiter training updates")
parser.add_argument("--batch-games", type=int, default=1024)
parser.add_argument("--tag", default="", help="suffix for the run dir, to keep runs apart")
args = parser.parse_args()
match_cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
old_cfg = load_config("configs/jax_ppo/balanced.yaml")
ours = restore_checkpoint(
OURS, create_match_train_state(match_cfg, jax.random.PRNGKey(0), Ablation())
).params
league = restore_checkpoint(LEAGUE, create_train_state(old_cfg, jax.random.PRNGKey(0))).params
targets = {
"ours (match stack, 131M)": match_frozen_policy(match_cfg, ours, Ablation()),
"league (web-deployed)": single_round_frozen_policy(old_cfg, league),
}
rows = []
for name, frozen in targets.items():
print(f"\n===== training an exploiter against: {name} =====", flush=True)
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
cfg.ppo.batch_games = args.batch_games
cfg.run.total_updates = args.updates
cfg.run.log_every = max(1, args.updates // 10)
slug = name.split()[0] + args.tag
state = train_exploiter(cfg, frozen, Path(f"runs/jax-ppo-match/exploit-{slug}"))
result = _final_score(cfg, state.params, frozen, MATCHES)
rows.append({"target": name, **result})
print(f" exploiter reached {result['exploiter_win_rate']:.4f} vs {name}", flush=True)
print("\n\n============ exploitability (same exploiter budget) ============")
print(f"{'frozen policy':<28}{'exploiter win rate':>20}{'95% CI':>22}")
print("-" * 72)
for row in rows:
ci = f"[{row['wilson_low']:.3f}, {row['wilson_high']:.3f}]"
print(f"{row['target']:<28}{row['exploiter_win_rate']:>20.4f}{ci:>22}")
print("\nhigher = the frozen policy had more to farm. 0.5 = nothing found.")
Path(f"runs/jax-ppo-match/exploitability{args.tag}.json").write_text(json.dumps(rows, indent=2))
if __name__ == "__main__":
main()
scripts/profile_gpu_forward.py
+2 -2
View File
@@ -1,7 +1,7 @@
"""Profile GPU forward-pass throughput for the Deep CFR trainer network.
Builds the same DeepCFRMLP that ``DeepCFRTrainer.__init__`` constructs from
``configs/deep_cfr/default.yaml``, then measures average forward-pass time on
``legacy/deep-cfr/configs/default.yaml``, then measures average forward-pass time on
CUDA across a sweep of batch sizes. The goal is to decide whether batched
traversal inference (Optimization Priorities #5) is worth implementing.
"""
@@ -19,7 +19,7 @@ from coolrl_lost_cities.games.classic.deep_cfr.config import load_config
from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP
REPO_ROOT = Path(__file__).resolve().parent.parent
CONFIG_PATH = REPO_ROOT / "configs" / "deep_cfr" / "default.yaml"
CONFIG_PATH = REPO_ROOT / "legacy" / "deep-cfr" / "configs" / "default.yaml"
BATCH_SIZES = [1, 4, 16, 64, 256, 1024]
WARMUP_ITERS = 10
scripts/render_jax_ppo_transcripts.py
+334
View File
@@ -0,0 +1,334 @@
"""Render human-readable JAX PPO Lost Cities transcripts."""
from __future__ import annotations
import argparse
import json
from collections import Counter
from pathlib import Path
from typing import NamedTuple
import jax
import jax.numpy as jnp
import numpy as np
from lost_cities_jax.engine import (
board_score,
current_hand_sorted,
decode_action,
reset_from_order,
step,
)
from lost_cities_jax.opponents import policy_by_name
from lost_cities_jax.ppo import checkpoint_policy, load_config, make_shuffle_bank
from lost_cities_jax.types import (
CARDS_PER_COLOR,
DISCARD,
DRAW_DECK,
LOC_P0_BOARD,
MAX_STEPS,
N_CARDS,
N_COLORS,
PLAY,
State,
)
COLOR_NAMES = ["red", "green", "white", "blue", "yellow"]
STEP_JIT = jax.jit(step)
class PolicySpec(NamedTuple):
label: str
fn: object
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", required=True)
parser.add_argument("--checkpoint", required=True)
parser.add_argument("--opponent", default="heuristic_cautious")
parser.add_argument("--shuffle-bank-seed", type=int, default=20260704)
parser.add_argument("--pairs", type=int, default=10)
parser.add_argument("--output", required=True)
args = parser.parse_args()
cfg = load_config(args.config)
agent = PolicySpec("gate3_checkpoint", jax.jit(checkpoint_policy(cfg, args.checkpoint)))
opponent = PolicySpec(args.opponent, jax.jit(policy_by_name(args.opponent)))
orders = make_shuffle_bank(args.shuffle_bank_seed, args.pairs)
sections = []
summaries = []
for idx, order in enumerate(orders):
sections.append(f"\n\n## Pair {idx:02d} / learner seat 0\n")
text, summary = render_game(order, [agent, opponent], agent_seat=0, game_id=idx * 2)
sections.append(text)
summaries.append(summary)
sections.append(f"\n\n## Pair {idx:02d} / learner seat 1\n")
text, summary = render_game(order, [opponent, agent], agent_seat=1, game_id=idx * 2 + 1)
sections.append(text)
summaries.append(summary)
output = render_summary(summaries, args) + "".join(sections)
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
Path(args.output).write_text(output, encoding="utf-8")
def render_game(
deck_order: list[int], policies: list[PolicySpec], *, agent_seat: int, game_id: int
) -> tuple[str, dict]:
state = reset_from_order(jnp.asarray(deck_order, dtype=jnp.int8))
lines = [f"Game {game_id}: P0={policies[0].label}, P1={policies[1].label}"]
handshake_events = []
cautious_openings = []
cautious_useful_discards = []
while not bool(state.done):
player = int(state.to_move)
policy = policies[player]
key = jax.random.PRNGKey(game_id * 1000 + int(state.step_count))
hand = [int(x) for x in current_hand_sorted(state, player)]
action = int(policy.fn(state, jnp.asarray(player, dtype=jnp.int32), key))
hand_slot, place_type, draw_source = [int(x) for x in decode_action(jnp.asarray(action))]
card = hand[hand_slot]
drawn = peek_drawn_card(state, place_type, draw_source)
before_col_len = int(state.col_len[player, card_color(card)])
before_hand = [card_label(x) for x in hand if x < N_CARDS]
before_scores = [int(x) for x in board_score(state)]
if policy.label == "heuristic_cautious":
if place_type == PLAY and before_col_len == 0:
cautious_openings.append(
{
"ply": int(state.step_count),
"seat": player,
"card": card_label(card),
"rank": card_rank(card),
}
)
if place_type == DISCARD and can_play_card(state, 1 - player, card):
cautious_useful_discards.append(
{
"ply": int(state.step_count),
"seat": player,
"card": card_label(card),
"opponent": 1 - player,
}
)
if place_type == PLAY and is_handshake(card):
handshake_events.append(
{
"ply": int(state.step_count),
"seat": player,
"policy": policy.label,
"card": card_label(card),
"hand": before_hand,
}
)
next_state, _, _ = STEP_JIT(state, jnp.asarray(action, dtype=jnp.int32))
place_text = "PLAY" if place_type == PLAY else "DISCARD"
target = "expedition" if place_type == PLAY else "discard"
draw_text = "deck" if draw_source == DRAW_DECK else f"{COLOR_NAMES[draw_source - 1]} pile"
lines.append(
f"Ply {int(state.step_count):03d} P{player} {policy.label}: "
f"{place_text} {card_label(card)} to {target}; "
f"DRAW {draw_text} -> {card_label(drawn)}; "
f"score_before={before_scores}"
)
lines.extend(render_board(next_state))
state = next_state
final_scores = [int(x) for x in board_score(state)]
lines.append(f"Final score: P0={final_scores[0]} P1={final_scores[1]}")
lines.extend(render_final_breakdown(state))
agent_score = final_scores[agent_seat]
opponent_score = final_scores[1 - agent_seat]
summary = {
"game_id": game_id,
"agent_seat": agent_seat,
"agent_score_diff": agent_score - opponent_score,
"agent_opened_colors": int(jnp.sum(state.col_len[agent_seat] > 0)),
"length": int(state.step_count),
"max_steps": int(state.step_count) >= MAX_STEPS,
"handshake_events": handshake_events,
"cautious_openings": cautious_openings,
"cautious_useful_discards": cautious_useful_discards,
}
return "\n".join(lines) + "\n", summary
def render_summary(summaries: list[dict], args: argparse.Namespace) -> str:
diffs = np.asarray([item["agent_score_diff"] for item in summaries], dtype=np.float64)
opened = np.asarray([item["agent_opened_colors"] for item in summaries], dtype=np.float64)
lengths = np.asarray([item["length"] for item in summaries], dtype=np.float64)
handshakes = [event for item in summaries for event in item["handshake_events"]]
cautious_openings = [event for item in summaries for event in item["cautious_openings"]]
useful_discards = [event for item in summaries for event in item["cautious_useful_discards"]]
opening_ranks = Counter(str(event["rank"]) for event in cautious_openings)
low_openings = [
event for event in cautious_openings if event["rank"] not in ("HS",) and event["rank"] < 7
]
lines = [
"# Gate-3 Transcript Dump",
"",
f"config: `{args.config}`",
f"checkpoint: `{args.checkpoint}`",
f"opponent: `{args.opponent}`",
f"duplicate pairs: `{args.pairs}`",
"",
"## Summary",
"",
f"games: {len(summaries)}",
f"agent_score_diff_mean: {float(np.mean(diffs)):.3f}",
f"agent_score_diff_min_max: {int(np.min(diffs))} / {int(np.max(diffs))}",
f"agent_opened_colors_mean: {float(np.mean(opened)):.3f}",
f"game_length_mean: {float(np.mean(lengths)):.3f}",
f"game_length_min_p50_p95_max: {int(np.min(lengths))} / "
f"{float(np.quantile(lengths, 0.50)):.1f} / "
f"{float(np.quantile(lengths, 0.95)):.1f} / {int(np.max(lengths))}",
f"max_steps_rate: {float(np.mean([item['max_steps'] for item in summaries])):.3f}",
f"handshake_play_events: {len(handshakes)}",
f"cautious_openings: {len(cautious_openings)}",
f"cautious_opening_rank_counts: {json.dumps(dict(sorted(opening_ranks.items())))}",
f"cautious_low_openings_lt7: {len(low_openings)}",
f"cautious_discards_immediately_playable_by_opponent: {len(useful_discards)}",
"",
"## Handshake Play Contexts",
"",
]
if handshakes:
for event in handshakes[:80]:
lines.append(
f"- game_event ply={event['ply']} seat=P{event['seat']} "
f"policy={event['policy']} card={event['card']} hand={event['hand']}"
)
if len(handshakes) > 80:
lines.append(f"- ... {len(handshakes) - 80} more")
else:
lines.append("- none")
lines.extend(["", "## Cautious Opening Audit", ""])
if cautious_openings:
for event in cautious_openings[:80]:
lines.append(
f"- ply={event['ply']} seat=P{event['seat']} card={event['card']} "
f"rank={event['rank']}"
)
if len(cautious_openings) > 80:
lines.append(f"- ... {len(cautious_openings) - 80} more")
else:
lines.append("- none")
lines.extend(["", "## Cautious Useful Discard Audit", ""])
if useful_discards:
for event in useful_discards[:80]:
lines.append(
f"- ply={event['ply']} seat=P{event['seat']} card={event['card']} "
f"opponent=P{event['opponent']}"
)
if len(useful_discards) > 80:
lines.append(f"- ... {len(useful_discards) - 80} more")
else:
lines.append("- none")
lines.append("")
return "\n".join(lines)
def render_board(state: State) -> list[str]:
scores = [int(x) for x in board_score(state)]
lines = [f" Board scores: P0={scores[0]} P1={scores[1]}"]
for player in range(2):
parts = []
for color in range(N_COLORS):
cards = board_cards(state, player, color)
score = color_score(cards)
parts.append(f"{COLOR_NAMES[color]}={format_cards(cards)}({score:+d})")
lines.append(f" P{player}: " + " | ".join(parts))
return lines
def render_final_breakdown(state: State) -> list[str]:
lines = ["Final color breakdown:"]
for player in range(2):
parts = []
for color in range(N_COLORS):
cards = board_cards(state, player, color)
parts.append(f"{COLOR_NAMES[color]} {format_cards(cards)} => {color_score(cards):+d}")
lines.append(f" P{player}: " + "; ".join(parts))
return lines
def board_cards(state: State, player: int, color: int) -> list[int]:
loc = np.asarray(state.card_loc)
board_loc = LOC_P0_BOARD + player
start = color * CARDS_PER_COLOR
cards = [card for card in range(start, start + CARDS_PER_COLOR) if int(loc[card]) == board_loc]
return sorted(
cards, key=lambda card: (0 if is_handshake(card) else 1, card_rank_value(card), card)
)
def color_score(cards: list[int]) -> int:
if not cards:
return 0
handshakes = sum(1 for card in cards if is_handshake(card))
ranks = [card_rank_value(card) for card in cards if not is_handshake(card)]
score = (sum(ranks) - 20) * (1 + handshakes)
if len(cards) >= 8:
score += 20
return score
def peek_drawn_card(state: State, place_type: int, draw_source: int) -> int:
del place_type
if draw_source == DRAW_DECK:
return int(state.deck_order[int(state.draw_ptr)])
color = draw_source - 1
length = int(state.pile_len[color])
return int(state.pile[color, length - 1])
def can_play_card(state: State, player: int, card: int) -> bool:
top = int(state.col_top[player, card_color(card)])
if is_handshake(card):
return top == 0
return card_rank_value(card) > top
def format_cards(cards: list[int]) -> str:
if not cards:
return "[]"
return "[" + ",".join(card_label(card) for card in cards) + "]"
def card_label(card: int) -> str:
if card < 0 or card >= N_CARDS:
return "none"
color = COLOR_NAMES[card_color(card)][0].upper()
if is_handshake(card):
return f"{color}HS{card % CARDS_PER_COLOR + 1}"
return f"{color}{card_rank_value(card)}"
def card_color(card: int) -> int:
return card // CARDS_PER_COLOR
def is_handshake(card: int) -> bool:
return card % CARDS_PER_COLOR < 3
def card_rank(card: int) -> int | str:
return "HS" if is_handshake(card) else card_rank_value(card)
def card_rank_value(card: int) -> int:
return card % CARDS_PER_COLOR - 1
if __name__ == "__main__":
main()
scripts/train_match.py
+62
View File
@@ -0,0 +1,62 @@
#!/usr/bin/env python3
"""Train the classic three-round agent by self-play, then measure it."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import jax
from lost_cities_jax.match_eval import carry_probe, match_evaluate
from lost_cities_jax.match_ppo import create_match_train_state, match_train
from lost_cities_jax.ppo import load_config, restore_checkpoint
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", default="configs/jax_ppo/match-selfplay.yaml")
parser.add_argument("--eval-only", type=Path, default=None)
parser.add_argument("--matches", type=int, default=1024)
parser.add_argument("--set", action="append", default=[])
args = parser.parse_args()
cfg = load_config(args.config)
for override in args.set:
path, _, raw = override.partition("=")
section, _, field = path.partition(".")
target = getattr(cfg, section)
current = getattr(target, field)
value = type(current)(raw) if not isinstance(current, bool) else raw == "true"
setattr(target, field, value)
if args.eval_only is None:
run_dir = match_train(cfg)
checkpoint = run_dir / "latest"
else:
checkpoint = args.eval_only
state = create_match_train_state(cfg, jax.random.PRNGKey(0))
state = restore_checkpoint(Path(checkpoint), state)
result = match_evaluate(cfg, state.params, matches=args.matches)
print("\n== duplicate match eval (self-play) ==")
print(json.dumps(result, indent=2, sort_keys=True))
print("\n== carry probe: does round-three play react to the deficit? ==")
rows = carry_probe(cfg, state.params, matches=args.matches // 2)
header = f"{'carry':>7}{'win_rate':>10}{'opened':>9}{'wagers':>9}{'deck_race':>11}{'plies':>8}"
print(header)
for row in rows:
print(
f"{row['carry']:>7}{row['win_rate']:>10.3f}{row['opened_colors']:>9.2f}"
f"{row['wagers_played']:>9.2f}{row['deck_race_rate']:>11.3f}{row['mean_plies']:>8.0f}"
)
Path(checkpoint).parent.joinpath("carry_probe.json").write_text(
json.dumps({"eval": result, "probe": rows}, indent=2, sort_keys=True)
)
if __name__ == "__main__":
main()
src/coolrl_lost_cities/games/classic/ismcts/cli.py
+37
View File
@@ -79,6 +79,19 @@ def train_command(args: argparse.Namespace) -> None:
device=config.run.device,
tracker=tracker,
)
if args.resume_from:
import torch
ckpt = torch.load(args.resume_from, map_location=trainer.device, weights_only=False)
trainer.network.load_state_dict(ckpt["network"])
if "optimizer" in ckpt:
trainer.optimizer.load_state_dict(ckpt["optimizer"])
print(
f"[resume] loaded network + optimizer from {args.resume_from} "
f"(prior iteration={ckpt.get('iteration', '?')}); "
f"new run starts at iteration 1 with current config",
flush=True,
)
try:
trainer.train()
finally:
@@ -111,7 +124,31 @@ def main(argv: list[str] | None = None) -> None:
train.add_argument("--wandb-job-type")
train.add_argument("--wandb-tag", action="append", default=[])
train.add_argument("--wandb-notes")
train.add_argument(
"--resume-from",
default=None,
help="Path to a .pt checkpoint to warm-start network + optimizer state.",
)
train.set_defaults(func=train_command)
from .eval_checkpoint import add_eval_args, run_eval
eval_cmd = subparsers.add_parser(
"eval",
help="Evaluate a saved checkpoint vs heuristic bots in parallel.",
)
add_eval_args(eval_cmd)
eval_cmd.set_defaults(func=lambda a: run_eval(a))
from .pretrain import add_pretrain_args, run_pretrain
pretrain_cmd = subparsers.add_parser(
"pretrain",
help="Behavior-clone a heuristic bot into the AlphaZero network as a warm start.",
)
add_pretrain_args(pretrain_cmd)
pretrain_cmd.set_defaults(func=lambda a: run_pretrain(a))
args = parser.parse_args(argv)
args.func(args)
src/coolrl_lost_cities/games/classic/ismcts/config.py
+45
View File
@@ -30,6 +30,20 @@ class MctsConfig(StrictModel):
virtual_loss_value: float = 1.0
eval_with_mcts: bool = True
eval_n_simulations: int = 0
root_dirichlet_alpha: float = 0.0
root_dirichlet_epsilon: float = 0.0
# Divisor applied to Q values inside PUCT to bring them onto roughly the
# same scale as the exploration bonus. With value_scale=100 score units,
# raw Q can swing ±100 while c_puct * prior * sqrt(N) is ~1-10, so a single
# bad backup permanently kills an action. Setting q_scale=100 normalizes Q
# to ~[-1, 1] (consistent with AlphaZero's convention).
q_scale: float = 100.0
# Opponent-aware search: when set, the search tree treats the opponent
# seat as a fixed external policy (heuristic bot) instead of expanding it
# with the network's priors/value. Used during mixed-opponent self-play
# so root visit distributions reflect the *actual* opponent the trainee
# faces. Bot name is taken from training.mixed_opponent_bot.
opponent_aware_search: bool = False
@field_validator("n_simulations", "max_depth", "parallel_simulations")
@classmethod
@@ -60,6 +74,37 @@ class TrainingConfig(StrictModel):
interleave_max_batch: int = 64
num_workers: int = 1
worker_device: str = "cpu"
# Multiplier on the value-head MSE loss (already normalized by value_scale**2).
# Default 1.0 keeps current behavior; raising it (e.g. 50-100) makes the value
# head learn faster relative to policy loss. Useful when value_prediction_error
# is large but loss/value is tiny because of the normalization.
value_loss_weight: float = 1.0
# Optional KL anchor to a reference (e.g. behavior-cloned) policy. The
# reference network is loaded once at trainer start and frozen; on every
# gradient step we add `kl_anchor_beta * KL(current || reference)` to the
# loss. Anchors self-play training to the pretrained policy and prevents
# catastrophic forgetting / drift to weak self-play equilibria.
kl_anchor_ckpt: str | None = None
kl_anchor_beta: float = 0.0
# Mirror-descent target mixing for policy loss. Alternative to kl_anchor;
# blends MCTS visit distribution with the reference (BC) policy in log
# space, then trains the network to match. pi_target = softmax(
# alpha * log(pi_mcts) + (1 - alpha) * log(pi_ref)
# ). Anneal alpha from low (rely on BC) to high (rely on MCTS) over
# training. Requires kl_anchor_ckpt to be set as the reference source.
md_target_ref_ckpt: str | None = None
md_target_alpha_start: float = 0.3
md_target_alpha_end: float = 0.8
md_target_alpha_iters: int = 500
# Mixed-opponent self-play: a fraction of games per iteration are played
# against a fixed external bot instead of the current network. Only the
# trainee's decisions are stored as policy targets; opponent moves are
# taken by `mixed_opponent_bot.act(state)`. Combined with
# mcts.opponent_aware_search, the MCTS tree models the opponent as that
# same bot so root-visit distributions reflect the real opponent.
# Set fraction=0 to disable (pure self-play).
mixed_opponent_fraction: float = 0.0
mixed_opponent_bot: str = "heuristic-balanced"
@field_validator(
"games_per_iter",
src/coolrl_lost_cities/games/classic/ismcts/eval_checkpoint.py
+333
View File
@@ -0,0 +1,333 @@
"""Standalone parallel evaluation of an ISMCTS checkpoint vs heuristic bots.
Reuses the same MCTS / bot stack as the training-loop eval, but does not
interfere with a running training process. Useful for comparing a snapshot
against opponents that are not in `evaluation.opponents` (e.g. heuristic-balanced,
the rollout policy) and for running many more games than per-iter eval typically
allows.
Available through the archived ISMCTS CLI module (see ``cli.py``).
"""
from __future__ import annotations
import argparse
import multiprocessing as mp
import os
import random
import sys
import time
from concurrent.futures import ProcessPoolExecutor, as_completed
from dataclasses import dataclass, field
from pathlib import Path
import torch
@dataclass(frozen=True)
class _WorkerJob:
ckpt_path: str
opponent: str
game_indices: tuple[int, ...]
seed: int
device: str
worker_index: int
verbose: bool
n_sims_override: int = 0 # 0 means use checkpoint's eval_n_simulations
@dataclass
class _GameResult:
game_index: int
policy_player: int
score_diff: float
turns: int
policy_turns: int
play_actions: int
timed_out: bool
@dataclass
class _WorkerResult:
worker_index: int
opponent: str
games: list[_GameResult] = field(default_factory=list)
elapsed: float = 0.0
def _run_games(job: _WorkerJob) -> _WorkerResult:
# Inside-worker imports + thread-cap to avoid CPU oversubscription
from coolrl_lost_cities.games.classic.bots.registry import build_bot
from coolrl_lost_cities.games.classic.deep_cfr.encoding import input_dim
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig
from coolrl_lost_cities.games.classic.ismcts.config import IsMctsConfig
from coolrl_lost_cities.games.classic.ismcts.mcts import IsMctsSearcher
from coolrl_lost_cities.games.classic.ismcts.network import AlphaZeroNet
os.environ.setdefault("OMP_NUM_THREADS", "1")
os.environ.setdefault("MKL_NUM_THREADS", "1")
torch.set_num_threads(1)
started = time.perf_counter()
print(
f" [worker {job.worker_index}] start ({len(job.game_indices)} games "
f"vs {job.opponent}, device={job.device})",
flush=True,
)
ckpt = torch.load(job.ckpt_path, map_location="cpu", weights_only=False)
cfg = IsMctsConfig.model_validate(ckpt["config"])
game_config = LostCitiesConfig(**ckpt["game_config"])
probe = GameState.new_game(game_config, seed=cfg.run.seed)
dim = input_dim(probe, cfg.encoding)
device = torch.device(job.device)
net = AlphaZeroNet.from_config(dim, probe.action_size, cfg).to(device)
net.load_state_dict(ckpt["network"])
net.eval()
eval_mcts_cfg = cfg.mcts.model_copy()
if job.n_sims_override > 0:
eval_mcts_cfg = eval_mcts_cfg.model_copy(update={"n_simulations": job.n_sims_override})
elif cfg.mcts.eval_n_simulations > 0:
eval_mcts_cfg = eval_mcts_cfg.model_copy(
update={"n_simulations": cfg.mcts.eval_n_simulations}
)
rng = random.Random(job.seed + job.worker_index * 7919)
result = _WorkerResult(worker_index=job.worker_index, opponent=job.opponent)
max_steps = 500
for game_index in job.game_indices:
game_started = time.perf_counter()
policy_player = game_index % 2
opps = [
build_bot(job.opponent, seed=job.seed + game_index),
build_bot(job.opponent, seed=job.seed + game_index + 1),
]
state = GameState.new_game(game_config, seed=job.seed + game_index)
turns = 0
policy_turns = 0
play_actions = 0
timed_out = False
while True:
if state.terminal:
break
if turns >= max_steps:
timed_out = True
break
current = int(state.current_player)
if current == policy_player:
searcher = IsMctsSearcher(
net,
eval_mcts_cfg,
device=device,
encoding=cfg.encoding,
rng=random.Random(rng.randrange(2**31)),
)
visits = searcher.search(state, current)
unified = (
max(visits, key=visits.get) if visits else state.unified_legal_actions()[0]
)
if state.phase == "card":
policy_turns += 1
if unified % 2 == 0:
play_actions += 1
state.apply_unified_action(unified)
else:
state.apply_action(opps[current].act(state))
turns += 1
diff = float(state.score_diff(policy_player))
gr = _GameResult(
game_index=game_index,
policy_player=policy_player,
score_diff=diff,
turns=turns,
policy_turns=policy_turns,
play_actions=play_actions,
timed_out=timed_out,
)
result.games.append(gr)
if job.verbose:
elapsed_g = time.perf_counter() - game_started
pa = play_actions / policy_turns if policy_turns else 0.0
print(
f" [worker {job.worker_index}] game {game_index:3d} "
f"as P{policy_player} | turns={turns:3d} "
f"score={diff:+6.1f} PA={pa:.2f}"
f"{' TIMEOUT' if timed_out else ''} "
f"({elapsed_g:.1f}s)",
flush=True,
)
result.elapsed = time.perf_counter() - started
won = sum(1 for g in result.games if g.score_diff > 0)
print(
f" [worker {job.worker_index}] done {len(result.games)} games "
f"vs {job.opponent} in {result.elapsed:.1f}s "
f"(W={won}/{len(result.games)})",
flush=True,
)
return result
def _split_games(n_games: int, n_workers: int) -> list[tuple[int, ...]]:
n_workers = max(1, min(n_workers, n_games))
base = n_games // n_workers
rem = n_games % n_workers
out: list[tuple[int, ...]] = []
cursor = 0
for i in range(n_workers):
count = base + (1 if i < rem else 0)
out.append(tuple(range(cursor, cursor + count)))
cursor += count
return out
def _summarize(games: list[_GameResult]) -> dict[str, float]:
n = len(games)
if n == 0:
return {}
wins = sum(1 for g in games if g.score_diff > 0)
losses = sum(1 for g in games if g.score_diff < 0)
draws = sum(1 for g in games if g.score_diff == 0)
timeouts = sum(1 for g in games if g.timed_out)
score_diffs = [g.score_diff for g in games]
avg = sum(score_diffs) / n
var = sum((d - avg) ** 2 for d in score_diffs) / n if n > 1 else 0.0
std = var**0.5
total_policy_turns = sum(g.policy_turns for g in games)
total_play_actions = sum(g.play_actions for g in games)
pa = total_play_actions / total_policy_turns if total_policy_turns else 0.0
avg_turns = sum(g.turns for g in games) / n
z = 1.96
# Wilson CI on win rate (half-width only; report as wr ± half)
wr = wins / n
denom = 1 + z * z / n
half = z / denom * ((wr * (1 - wr) / n + z * z / (4 * n * n)) ** 0.5)
score_ci = z * std / (n**0.5) if n > 1 else 0.0
return {
"games": n,
"wins": wins,
"losses": losses,
"draws": draws,
"timeouts": timeouts,
"win_rate": wr,
"win_rate_ci_half": half,
"avg_score_diff": avg,
"score_std": std,
"score_ci_half": score_ci,
"play_action_rate": pa,
"avg_turns": avg_turns,
}
def _default_ckpt() -> Path:
"""Find latest 'ismcts-overnight*' run's latest.pt, or fallback to newest run."""
candidates = sorted(Path("runs").glob("*ismcts-overnight*"))
if candidates:
return candidates[-1] / "latest.pt"
candidates = sorted(Path("runs").iterdir())
if not candidates:
raise SystemExit("no runs/ directory entries")
return candidates[-1] / "latest.pt"
def add_eval_args(parser: argparse.ArgumentParser) -> None:
parser.add_argument(
"--ckpt",
default=None,
help="Path to checkpoint .pt. Default: latest overnight run latest.pt.",
)
parser.add_argument(
"--opponents",
nargs="+",
default=["heuristic-balanced", "heuristic-aggressive", "heuristic-cautious"],
help="Opponent bot names from registry.",
)
parser.add_argument("--games", type=int, default=50, help="Games per opponent (default: 50).")
parser.add_argument(
"--device",
choices=("cpu", "cuda"),
default="cpu",
help="Worker device (default: cpu; cuda may compete with running training).",
)
parser.add_argument(
"--num-workers", type=int, default=8, help="Parallel worker count (default: 8)."
)
parser.add_argument("--seed", type=int, default=99999)
parser.add_argument(
"--verbose",
action="store_true",
help="Print per-game result lines (turns/score/PA) in addition to per-worker summaries.",
)
parser.add_argument(
"--n-sims",
type=int,
default=0,
help="Override n_simulations at eval time (0 = use checkpoint's eval_n_simulations).",
)
def run_eval(args: argparse.Namespace) -> None:
ckpt_path = Path(args.ckpt) if args.ckpt else _default_ckpt()
if not ckpt_path.exists():
print(f"checkpoint not found: {ckpt_path}", file=sys.stderr)
raise SystemExit(1)
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)
iteration = ckpt.get("iteration", "?")
print(f"checkpoint : {ckpt_path}")
print(f"iteration : {iteration}")
print(f"device : {args.device}")
print(f"workers : {args.num_workers}")
print(f"games/opp : {args.games}")
print(f"opponents : {args.opponents}")
print(f"seed : {args.seed}")
if args.verbose:
print("verbose : True (per-game logging)")
print()
ctx = mp.get_context("spawn")
started_all = time.perf_counter()
for opponent in args.opponents:
opp_started = time.perf_counter()
slices = _split_games(args.games, args.num_workers)
jobs = [
_WorkerJob(
ckpt_path=str(ckpt_path),
opponent=opponent,
game_indices=tuple(slices[i]),
seed=args.seed,
device=args.device,
worker_index=i,
verbose=args.verbose,
n_sims_override=int(args.n_sims),
)
for i in range(len(slices))
]
all_games: list[_GameResult] = []
with ProcessPoolExecutor(max_workers=len(jobs), mp_context=ctx) as ex:
futures = [ex.submit(_run_games, j) for j in jobs]
for f in as_completed(futures):
res = f.result()
all_games.extend(res.games)
elapsed = time.perf_counter() - opp_started
summary = _summarize(all_games)
n = int(summary["games"])
print()
print(
f"vs {opponent:22s} | W={summary['wins']}/{n} "
f"({summary['win_rate']:.2f} ± {summary['win_rate_ci_half']:.2f}) "
f"| S={summary['avg_score_diff']:+6.1f} ± {summary['score_ci_half']:5.1f} "
f"(σ={summary['score_std']:.1f}) | PA={summary['play_action_rate']:.2f} "
f"| turns={summary['avg_turns']:.0f} "
f"| timeouts={summary['timeouts']} "
f"| elapsed={elapsed:.1f}s"
)
print()
total = time.perf_counter() - started_all
print(f"total elapsed: {total:.1f}s")
src/coolrl_lost_cities/games/classic/ismcts/interleaved_self_play.py
+54 -1
View File
@@ -6,6 +6,7 @@ from dataclasses import dataclass, field
import numpy as np
import torch
from coolrl_lost_cities.games.classic.bots.registry import build_bot
from coolrl_lost_cities.games.classic.deep_cfr.encoding import encode_info_state
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig
@@ -34,6 +35,11 @@ class _GameContext:
game_index: int
decisions: list[_PendingDecision] = field(default_factory=list)
steps: int = 0
# Mixed-opponent setup: when traverser_seat is not None, only that seat
# uses MCTS+network; the other seat is played by `opponent_bot`. None for
# pure self-play games (both seats use MCTS).
traverser_seat: int | None = None
opponent_bot: object | None = None
@dataclass
@@ -61,15 +67,28 @@ def play_self_play_iteration(
active: list[_GameContext] = []
started = 0
target_games = training_config.games_per_iter
mixed_fraction = float(training_config.mixed_opponent_fraction)
def fill_active() -> None:
nonlocal started
while len(active) < training_config.interleave_games and started < target_games:
traverser_seat: int | None = None
opponent_bot = None
if mixed_fraction > 0.0 and rng.random() < mixed_fraction:
# Alternate trainee seat so MCTS sees both first- and
# second-player perspectives equally.
traverser_seat = started % 2
opponent_bot = build_bot(
training_config.mixed_opponent_bot,
seed=rng.randrange(2**31),
)
active.append(
_GameContext(
state=GameState.new_game(game_config, seed=rng.randrange(2**31)),
rng=random.Random(rng.randrange(2**31)),
game_index=started,
traverser_seat=traverser_seat,
opponent_bot=opponent_bot,
)
)
started += 1
@@ -83,6 +102,19 @@ def play_self_play_iteration(
completed.append(_finalize_context(context))
continue
player = int(context.state.current_player)
# Mixed-opponent: if it's the opponent's turn in a mixed game,
# let the heuristic bot move directly (no MCTS, no sample).
if (
context.traverser_seat is not None
and context.opponent_bot is not None
and player != context.traverser_seat
):
phase_action = context.opponent_bot.act(context.state)
unified = context.state.to_unified_action(phase_action)
context.state.apply_unified_action(unified)
context.steps += 1
still_active.append(context)
continue
searcher = IsMctsSearcher(
network,
mcts_config,
@@ -90,6 +122,18 @@ def play_self_play_iteration(
encoding=encoding,
rng=random.Random(context.rng.randrange(2**31)),
)
# Pass opponent bot into the searcher for opponent-aware
# determinization (search models the real opponent the trainee
# faces, not a self-play mirror).
if (
mcts_config.opponent_aware_search
and context.traverser_seat is not None
and context.opponent_bot is not None
):
searcher.set_opponent_bot(
context.opponent_bot,
traverser_seat=context.traverser_seat,
)
jobs.append(
_SearchJob(
context=context,
@@ -181,6 +225,7 @@ def _evaluate_global_batch(
item.legal_actions,
priors_by_id[id(item)],
values_by_id[id(item)],
not item.path, # is_root for Dirichlet noise
)
job.searcher._backup(item.path, value, item.leaf_player)
@@ -222,7 +267,15 @@ def _finish_decision(
def _finalize_context(context: _GameContext) -> list[ReplaySample]:
final_diff0 = float(context.state.score_diff(0))
# If the game did not terminate naturally (hit max_steps), the score
# reflects an incomplete game — typically a "stall" outcome where both
# sides have under-developed expeditions. Treating that as a real win
# for either player creates a degenerate stall-and-pray learning
# signal. Zero it out so the trajectory is neutral.
if not context.state.terminal:
final_diff0 = 0.0
else:
final_diff0 = float(context.state.score_diff(0))
samples: list[ReplaySample] = []
for decision in context.decisions:
value = final_diff0 if decision.player == 0 else -final_diff0
src/coolrl_lost_cities/games/classic/ismcts/mcts.py
+25 -5
View File
@@ -89,6 +89,13 @@ class IsMctsSearcher:
self._rollout_bot = (
HeuristicBot() if config.rollout_policy == "heuristic_balanced" else None
)
# Opponent-aware search: see mcts.pyx for the rationale.
self._opponent_bot: object | None = None
self._traverser_seat: int = -1
def set_opponent_bot(self, bot: object, *, traverser_seat: int) -> None:
self._opponent_bot = bot
self._traverser_seat = int(traverser_seat)
def search(
self,
@@ -133,34 +140,45 @@ class IsMctsSearcher:
# Cache the info-set key for the current node so we don't recompute it
# after applying an action (the child's key becomes the next iter's key).
cached_key: bytes | None = None
opponent_aware = self._opponent_bot is not None
trav_seat = self._traverser_seat
while True:
player = int(state.current_player)
if state.terminal or depth >= self.config.max_depth:
leaf_seat = trav_seat if opponent_aware else player
return PendingSimulation(
path=path,
leaf_state=state,
leaf_node=None,
leaf_player=player,
leaf_player=leaf_seat,
info_state=None,
legal_mask=None,
legal_actions=[],
terminal_value=float(state.score_diff(player)),
terminal_value=float(state.score_diff(leaf_seat)),
)
if opponent_aware and player != trav_seat:
phase_action = self._opponent_bot.act(state)
unified = state.to_unified_action(phase_action)
state.apply_unified_action(unified)
cached_key = None
depth += 1
continue
key = cached_key if cached_key is not None else canonical_info_set_key(state, player)
node = self.tree.get_or_create(key, player=player, terminal=state.terminal)
if not node.is_expanded():
legal_actions = state.unified_legal_actions()
if not legal_actions:
node.terminal = True
leaf_seat = trav_seat if opponent_aware else player
return PendingSimulation(
path=path,
leaf_state=state,
leaf_node=node,
leaf_player=player,
leaf_player=leaf_seat,
info_state=None,
legal_mask=None,
legal_actions=[],
terminal_value=float(state.score_diff(player)),
terminal_value=float(state.score_diff(leaf_seat)),
)
return PendingSimulation(
path=path,
@@ -263,6 +281,7 @@ class IsMctsSearcher:
sqrt_total = math.sqrt(max(1, total_visits))
best_score = -float("inf")
best_action = legal_actions[0]
q_scale = float(getattr(self.config, "q_scale", 100.0)) or 1.0
for action in legal_actions:
n = node.visits.get(action, 0)
virtual = node.virtual_visits.get(action, 0)
@@ -274,7 +293,8 @@ class IsMctsSearcher:
q_eff = (
node.value_sum.get(action, 0.0) - virtual * self.config.virtual_loss_value
) / n_eff
score = q_eff + self.config.c_puct * prior * sqrt_total / (1 + n_eff)
# Normalize Q to match exploration-bonus scale; see mcts.pyx for details.
score = q_eff / q_scale + self.config.c_puct * prior * sqrt_total / (1 + n_eff)
if score > best_score:
best_score = score
best_action = action
src/coolrl_lost_cities/games/classic/ismcts/mcts.pyx
+79 -6
View File
@@ -296,6 +296,12 @@ cdef class IsMctsSearcher:
cdef public MctsTree tree
cdef HeuristicBot _rollout_bot
cdef int action_size
# Opponent-aware search: when set, the search treats one seat as a fixed
# external policy (heuristic bot). Opponent moves are applied directly
# without entering the tree, and all values are taken from the
# traverser's perspective. None for standard symmetric self-play search.
cdef public object _opponent_bot
cdef public int _traverser_seat
def __init__(
self,
@@ -318,6 +324,12 @@ cdef class IsMctsSearcher:
self._rollout_bot = (
<HeuristicBot>PyHeuristicBot() if config.rollout_policy == "heuristic_balanced" else None
)
self._opponent_bot = None
self._traverser_seat = -1
def set_opponent_bot(self, object bot, *, int traverser_seat):
self._opponent_bot = bot
self._traverser_seat = traverser_seat
cdef inline int _from_unified_action_c(self, GameState state, int action_id) noexcept:
cdef int card_action_size = 2 * state.hand_size
@@ -338,12 +350,16 @@ cdef class IsMctsSearcher:
)
cdef int sims = int(n_sims or self.config.n_simulations)
cdef int completed = 0
cdef int batch_size
cdef list pending
cdef list legal
cdef int action
cdef dict result
while completed < sims:
pending = self.prepare_simulation_batch(state, traverser, 1)
batch_size = min(int(self.config.parallel_simulations), sims - completed)
if batch_size <= 0:
batch_size = 1
pending = self.prepare_simulation_batch(state, traverser, batch_size)
if not pending:
break
self.evaluate_and_backup(pending)
@@ -386,19 +402,41 @@ cdef class IsMctsSearcher:
cdef int actions[MAX_ACTIONS]
cdef int action_count
cdef int i
cdef bint opponent_aware = self._opponent_bot is not None
cdef int leaf_player_seat
cdef int trav_seat = self._traverser_seat
cdef object phase_action
cdef int unified_action
while True:
player = state.current_player
if state.terminal or depth >= int(self.config.max_depth):
if opponent_aware:
leaf_player_seat = trav_seat
else:
leaf_player_seat = player
return PendingSimulation(
path=path,
leaf_state=state,
leaf_node=None,
leaf_player=player,
leaf_player=leaf_player_seat,
info_state=None,
legal_mask=None,
legal_actions=[],
terminal_value=float(state.total_scores[player] - state.total_scores[1 - player]),
terminal_value=float(
state.total_scores[leaf_player_seat]
- state.total_scores[1 - leaf_player_seat]
),
)
# Opponent-aware: if it's the opponent's turn, let the heuristic
# bot move directly instead of expanding the tree.
if opponent_aware and player != trav_seat:
phase_action = self._opponent_bot.act(state)
unified_action = state.to_unified_action(phase_action)
local_action = self._from_unified_action_c(state, unified_action)
state._push_action_c(local_action)
cached_key = None
depth += 1
continue
if cached_key is None:
key = canonical_info_set_key(state, player)
else:
@@ -409,15 +447,22 @@ cdef class IsMctsSearcher:
legal_actions = [actions[i] for i in range(action_count)]
if not legal_actions:
node.terminal = True
if opponent_aware:
leaf_player_seat = trav_seat
else:
leaf_player_seat = player
return PendingSimulation(
path=path,
leaf_state=state,
leaf_node=node,
leaf_player=player,
leaf_player=leaf_player_seat,
info_state=None,
legal_mask=None,
legal_actions=[],
terminal_value=float(state.total_scores[player] - state.total_scores[1 - player]),
terminal_value=float(
state.total_scores[leaf_player_seat]
- state.total_scores[1 - leaf_player_seat]
),
)
return PendingSimulation(
path=path,
@@ -489,6 +534,7 @@ cdef class IsMctsSearcher:
item.legal_actions,
priors_by_id[id(item)],
values_by_id[id(item)],
not item.path, # is_root: empty path means leaf == root
)
self._backup(item.path, value, item.leaf_player)
@@ -500,14 +546,32 @@ cdef class IsMctsSearcher:
list legal_actions,
object probs,
double network_value,
bint is_root=False,
):
cdef int action
cdef int i
cdef int n_legal
cdef double alpha
cdef double epsilon
cdef object rollout_value
cdef object noise
cdef object np_rng
legal_actions = self._unified_legal_actions_list_c(state)
if not legal_actions:
node.terminal = True
return float(state.total_scores[player] - state.total_scores[1 - player])
node.expanded = True
# Dirichlet noise at root (AlphaZero pattern: force exploration of low-prior actions)
alpha = float(self.config.root_dirichlet_alpha)
epsilon = float(self.config.root_dirichlet_epsilon)
if is_root and alpha > 0.0 and epsilon > 0.0:
n_legal = len(legal_actions)
# Use numpy with seed from self.rng for reproducibility under fixed seeds
np_rng = np.random.default_rng(self.rng.randrange(2**31))
noise = np_rng.dirichlet([alpha] * n_legal)
for i in range(n_legal):
action = legal_actions[i]
probs[action] = (1.0 - epsilon) * float(probs[action]) + epsilon * float(noise[i])
for action in legal_actions:
(<_ArrayMap>node.priors).set_float(action, float(probs[action]))
if not (<_ArrayMap>node.visits).has(action):
@@ -530,13 +594,17 @@ cdef class IsMctsSearcher:
cdef double sqrt_total
cdef double prior
cdef double q_eff
cdef double q_normalized
cdef double score
cdef double best_score = -float("inf")
cdef int best_action = int(legal_actions[0])
cdef double q_scale = float(getattr(self.config, "q_scale", 100.0))
cdef _ArrayMap visits = <_ArrayMap>node.visits
cdef _ArrayMap virtual_visits = <_ArrayMap>node.virtual_visits
cdef _ArrayMap priors = <_ArrayMap>node.priors
cdef _ArrayMap value_sum = <_ArrayMap>node.value_sum
if q_scale <= 0.0:
q_scale = 1.0
for action in legal_actions:
total_visits += visits.get_int(action, 0) + virtual_visits.get_int(action, 0)
sqrt_total = math.sqrt(max(1, total_visits))
@@ -552,7 +620,12 @@ cdef class IsMctsSearcher:
value_sum.get_float(action, 0.0)
- virtual * float(self.config.virtual_loss_value)
) / n_eff
score = q_eff + float(self.config.c_puct) * prior * sqrt_total / (1 + n_eff)
# Normalize Q to roughly [-1, 1] so the exploration bonus
# (c_puct * prior * sqrt(N) / (1+n)) competes on the right scale.
# Without this, raw score-units Q (±100) dominates and a single
# noisy backup kills exploration of low-prior actions.
q_normalized = q_eff / q_scale
score = q_normalized + float(self.config.c_puct) * prior * sqrt_total / (1 + n_eff)
if score > best_score:
best_score = score
best_action = action
src/coolrl_lost_cities/games/classic/ismcts/pretrain.py
+277
View File
@@ -0,0 +1,277 @@
"""Behavior cloning warm-start for the SO-ISMCTS network.
Generates games from a fixed heuristic policy vs itself, then trains the
AlphaZero-style network's policy + value heads in supervised fashion:
- policy loss: cross-entropy between network logits and the heuristic's chosen
action (one-hot target, masked to legal actions).
- value loss: MSE on the final game score diff from each decision-maker's
perspective, normalized by value_scale (same convention as the trainer).
The resulting checkpoint can be passed to the archived ISMCTS trainer to start
self-play with a heuristic-level prior instead of a random init. This addresses
the self-play "weak-equilibrium" problem: starting from random, MCTS visit
distributions converge to a mutually mediocre policy that has near-zero win
rate against the heuristic. Warm-starting at heuristic level gives self-play a
meaningful baseline to improve from.
"""
from __future__ import annotations
import argparse
import random
import time
from pathlib import Path
import numpy as np
import torch
import torch.nn.functional as F
from torch import nn
from coolrl_lost_cities.games.classic.bots.registry import build_bot
from coolrl_lost_cities.games.classic.deep_cfr.encoding import encode_info_state, input_dim
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig
from .config import IsMctsConfig, load_config
from .network import AlphaZeroNet
def _collect_samples(
game_config: LostCitiesConfig,
encoding,
n_games: int,
bot_name: str = "heuristic-balanced",
seed: int = 0,
max_turns: int = 500,
) -> list[tuple[np.ndarray, np.ndarray, int, float]]:
"""Roll out n_games of bot vs bot, returning per-decision samples.
Each sample: (info_state, legal_mask, action_idx, value).
value is the player-perspective score diff at game end.
"""
samples: list[tuple[np.ndarray, np.ndarray, int, float]] = []
for game_idx in range(n_games):
bots = [
build_bot(bot_name, seed=seed + game_idx * 2),
build_bot(bot_name, seed=seed + game_idx * 2 + 1),
]
state = GameState.new_game(game_config, seed=seed + game_idx)
decisions: list[tuple[np.ndarray, np.ndarray, int, int]] = []
turns = 0
while not state.terminal and turns < max_turns:
player = int(state.current_player)
info_state = encode_info_state(state, player, encoding)
legal_mask = np.asarray(state.unified_legal_mask(), dtype=bool)
phase_action = bots[player].act(state)
unified = state.to_unified_action(phase_action)
decisions.append((info_state, legal_mask, int(unified), player))
state.apply_unified_action(unified)
turns += 1
final_diff0 = float(state.score_diff(0))
for info, mask, act, player in decisions:
value = final_diff0 if player == 0 else -final_diff0
samples.append((info, mask, act, value))
return samples
def _train_supervised(
network: AlphaZeroNet,
samples: list,
device: torch.device,
*,
epochs: int,
batch_size: int,
lr: float,
weight_decay: float,
grad_clip: float,
value_loss_weight: float,
) -> None:
network.train()
optimizer = torch.optim.AdamW(network.parameters(), lr=lr, weight_decay=max(weight_decay, 1e-4))
rng = random.Random(0)
indices = list(range(len(samples)))
v_scale = float(network.value_scale)
for epoch in range(1, epochs + 1):
rng.shuffle(indices)
n_batches = (len(indices) + batch_size - 1) // batch_size
epoch_pl = 0.0
epoch_vl = 0.0
epoch_acc = 0.0
epoch_n = 0
for b in range(n_batches):
batch_idx = indices[b * batch_size : (b + 1) * batch_size]
infos = np.stack([samples[i][0] for i in batch_idx])
masks = np.stack([samples[i][1] for i in batch_idx])
actions = np.array([samples[i][2] for i in batch_idx], dtype=np.int64)
values = np.array([samples[i][3] for i in batch_idx], dtype=np.float32)
info_t = torch.as_tensor(infos, dtype=torch.float32, device=device)
mask_t = torch.as_tensor(masks, dtype=torch.bool, device=device)
action_t = torch.as_tensor(actions, device=device)
value_t = torch.as_tensor(values, device=device)
logits, value_pred = network(info_t, mask_t)
policy_loss = F.cross_entropy(logits, action_t)
value_loss = F.mse_loss(value_pred / v_scale, value_t / v_scale)
loss = policy_loss + value_loss_weight * value_loss
optimizer.zero_grad(set_to_none=True)
loss.backward()
if grad_clip > 0:
nn.utils.clip_grad_norm_(network.parameters(), grad_clip)
optimizer.step()
with torch.no_grad():
preds = logits.argmax(dim=-1)
acc = (preds == action_t).float().mean().item()
epoch_pl += float(policy_loss.item()) * len(batch_idx)
epoch_vl += float(value_loss.item()) * len(batch_idx)
epoch_acc += acc * len(batch_idx)
epoch_n += len(batch_idx)
print(
f" epoch {epoch:3d}: policy_loss={epoch_pl / epoch_n:.4f} "
f"value_loss={epoch_vl / epoch_n:.4f} "
f"top1_match={epoch_acc / epoch_n:.3f}",
flush=True,
)
network.eval()
return optimizer
def add_pretrain_args(parser: argparse.ArgumentParser) -> None:
parser.add_argument(
"--config", default=None, help="ISMCTS config YAML (controls network shape + rules)."
)
parser.add_argument(
"--set",
action="append",
default=[],
dest="config_overrides",
metavar="PATH=VALUE",
)
parser.add_argument(
"--bot",
default="heuristic-balanced",
help="Bot to clone (default: heuristic-balanced).",
)
parser.add_argument(
"--games", type=int, default=2000, help="Number of bot-vs-bot games to roll out."
)
parser.add_argument("--epochs", type=int, default=10, help="Supervised training epochs.")
parser.add_argument("--batch-size", type=int, default=256)
parser.add_argument("--lr", type=float, default=3.0e-4)
parser.add_argument("--weight-decay", type=float, default=1.0e-4)
parser.add_argument("--grad-clip", type=float, default=5.0)
parser.add_argument(
"--value-loss-weight",
type=float,
default=50.0,
help="Multiplier on value MSE (raw_MSE / value_scale^2). Default 50 to "
"make value loss magnitude comparable to policy CE.",
)
parser.add_argument(
"--out",
default="runs/pretrain/heuristic_clone.pt",
help="Output checkpoint path (compatible with --resume-from).",
)
parser.add_argument("--device", default="cuda")
parser.add_argument("--seed", type=int, default=12345)
def _apply_config_overrides(config: IsMctsConfig, assignments: list[str]) -> IsMctsConfig:
import yaml
def deep_update(base: dict, patch: dict) -> None:
for k, v in patch.items():
if isinstance(v, dict) and isinstance(base.get(k), dict):
deep_update(base[k], v)
else:
base[k] = v
overrides: dict = {}
for assignment in assignments:
if "=" not in assignment:
raise ValueError(f"override must be PATH=VALUE: {assignment}")
path, raw_value = assignment.split("=", 1)
value = yaml.safe_load(raw_value)
cursor = overrides
keys = path.split(".")
for k in keys[:-1]:
cursor = cursor.setdefault(k, {})
cursor[keys[-1]] = value
data = config.model_dump(mode="python")
deep_update(data, overrides)
return IsMctsConfig.model_validate(data)
def run_pretrain(args: argparse.Namespace) -> None:
config = load_config(args.config) if args.config else IsMctsConfig()
config = _apply_config_overrides(config, args.config_overrides)
game_config = config.rules.to_lost_cities_config(seed=config.run.seed)
device = torch.device(args.device)
probe = GameState.new_game(game_config, seed=config.run.seed)
in_dim = input_dim(probe, config.encoding)
network = AlphaZeroNet.from_config(in_dim, probe.action_size, config).to(device)
print(
f"network: input_dim={in_dim} action_size={probe.action_size} "
f"hidden={config.network.hidden_size} layers={config.network.num_layers}",
flush=True,
)
print(f"rolling out {args.games} games of {args.bot} vs {args.bot}...", flush=True)
t0 = time.perf_counter()
samples = _collect_samples(
game_config,
config.encoding,
n_games=args.games,
bot_name=args.bot,
seed=args.seed,
)
rollout_secs = time.perf_counter() - t0
print(
f"collected {len(samples)} decisions from {args.games} games "
f"in {rollout_secs:.1f}s ({len(samples) / args.games:.1f} decisions/game)",
flush=True,
)
print(
f"training: {args.epochs} epochs, batch={args.batch_size}, lr={args.lr}, "
f"value_weight={args.value_loss_weight}",
flush=True,
)
t1 = time.perf_counter()
optimizer = _train_supervised(
network,
samples,
device,
epochs=args.epochs,
batch_size=args.batch_size,
lr=args.lr,
weight_decay=args.weight_decay,
grad_clip=args.grad_clip,
value_loss_weight=args.value_loss_weight,
)
train_secs = time.perf_counter() - t1
print(f"training done in {train_secs:.1f}s", flush=True)
out_path = Path(args.out)
out_path.parent.mkdir(parents=True, exist_ok=True)
payload = {
"config": config.to_dict(),
"game_config": game_config.to_snapshot(),
"iteration": 0,
"network": network.state_dict(),
"optimizer": optimizer.state_dict(),
"metrics": {
"pretrain/games": args.games,
"pretrain/samples": len(samples),
"pretrain/epochs": args.epochs,
"pretrain/bot": args.bot,
},
}
torch.save(payload, out_path)
print(f"saved pretrained checkpoint to {out_path}", flush=True)
src/coolrl_lost_cities/games/classic/ismcts/trainer.py
+100 -2
View File
@@ -83,6 +83,51 @@ class IsMctsTrainer:
self.metrics_path = self.run_dir / "metrics.jsonl"
self.rng = random.Random(config.run.seed)
# Optional KL anchor: load a frozen reference network whose policy we
# use to regularize updates (KL(current || reference) added to loss).
# Prevents drift from a BC pretrained baseline during self-play.
self.kl_anchor_ref: AlphaZeroNet | None = None
self.kl_anchor_beta = float(config.training.kl_anchor_beta)
if config.training.kl_anchor_ckpt and self.kl_anchor_beta > 0.0:
ref_path = Path(config.training.kl_anchor_ckpt)
if not ref_path.exists():
raise FileNotFoundError(f"kl_anchor_ckpt not found: {ref_path}")
ref_payload = torch.load(ref_path, map_location=self.device, weights_only=False)
self.kl_anchor_ref = AlphaZeroNet.from_config(
self.input_dim, self.action_size, config
).to(self.device)
self.kl_anchor_ref.load_state_dict(ref_payload["network"])
self.kl_anchor_ref.eval()
for p in self.kl_anchor_ref.parameters():
p.requires_grad = False
print(
f"[trainer] KL anchor: ref={ref_path} beta={self.kl_anchor_beta}",
flush=True,
)
# Optional mirror-descent reference policy: blend MCTS visit dist
# with this reference in log space before computing CE loss.
self.md_target_ref: AlphaZeroNet | None = None
if config.training.md_target_ref_ckpt:
md_ref_path = Path(config.training.md_target_ref_ckpt)
if not md_ref_path.exists():
raise FileNotFoundError(f"md_target_ref_ckpt not found: {md_ref_path}")
md_payload = torch.load(md_ref_path, map_location=self.device, weights_only=False)
self.md_target_ref = AlphaZeroNet.from_config(
self.input_dim, self.action_size, config
).to(self.device)
self.md_target_ref.load_state_dict(md_payload["network"])
self.md_target_ref.eval()
for p in self.md_target_ref.parameters():
p.requires_grad = False
print(
f"[trainer] mirror-descent ref={md_ref_path} "
f"alpha {config.training.md_target_alpha_start} -> "
f"{config.training.md_target_alpha_end} over "
f"{config.training.md_target_alpha_iters} iters",
flush=True,
)
self._current_md_alpha = float(config.training.md_target_alpha_start)
def _resolve_device(self, device: torch.device | str) -> torch.device:
token = str(device)
if token == "auto":
@@ -116,6 +161,15 @@ class IsMctsTrainer:
return metrics
def run_iteration(self, iteration: int) -> IterationMetrics:
# Update mirror-descent alpha schedule (linear from start to end over alpha_iters).
if self.md_target_ref is not None:
cfg_t = self.config.training
n = max(1, int(cfg_t.md_target_alpha_iters))
frac = min(1.0, float(iteration) / float(n))
self._current_md_alpha = float(
cfg_t.md_target_alpha_start
+ (cfg_t.md_target_alpha_end - cfg_t.md_target_alpha_start) * frac
)
print(
f"[iter {iteration}] self-play start (workers={self.config.training.num_workers})",
flush=True,
@@ -258,10 +312,45 @@ class IsMctsTrainer:
)
logits, value_pred = self.network(info, legal)
log_probs = torch.log_softmax(logits, dim=-1)
policy_loss = -(pi * log_probs).sum(dim=-1).mean()
# Optional mirror-descent target: blend MCTS visit distribution with
# the frozen reference (BC) policy in log space, then train CE to that
# blended target. This is the standard regularized policy improvement
# operator: pi_target = softmax(alpha * log(pi_mcts) + (1-alpha) * log(pi_ref)).
if self.md_target_ref is not None:
with torch.no_grad():
ref_logits, _ref_value = self.md_target_ref(info, legal)
ref_log_probs = torch.log_softmax(ref_logits, dim=-1)
alpha = float(self._current_md_alpha)
# Clamp pi to avoid log(0); MCTS visit dist already has only legal
# actions positive, so this affects illegal actions which the mask
# in the network forward already zeroed out via -inf logits.
log_pi = torch.log(pi.clamp_min(1.0e-12))
mixed = alpha * log_pi + (1.0 - alpha) * ref_log_probs
mixed = mixed.masked_fill(~legal, torch.finfo(mixed.dtype).min)
pi_target = torch.softmax(mixed, dim=-1)
policy_loss = -(pi_target * log_probs).sum(dim=-1).mean()
else:
policy_loss = -(pi * log_probs).sum(dim=-1).mean()
v_scale = float(self.network.value_scale)
value_loss = nn.functional.mse_loss(value_pred / v_scale, value_target / v_scale)
loss = policy_loss + value_loss
value_weight = float(self.config.training.value_loss_weight)
loss = policy_loss + value_weight * value_loss
# KL anchor: KL(current || reference) over legal actions only.
# Encourages current policy to stay close to the reference (BC) policy.
kl_anchor_loss = 0.0
if self.kl_anchor_ref is not None and self.kl_anchor_beta > 0.0:
with torch.no_grad():
ref_logits, _ref_value = self.kl_anchor_ref(info, legal)
ref_log_probs = torch.log_softmax(ref_logits, dim=-1)
# KL(current || ref) = sum_a p_cur(a) * (log p_cur(a) - log p_ref(a))
cur_probs = log_probs.exp()
legal_f = legal.float()
kl_per_action = cur_probs * (log_probs - ref_log_probs) * legal_f
kl = kl_per_action.sum(dim=-1).mean()
loss = loss + self.kl_anchor_beta * kl
kl_anchor_loss = float(kl.item())
self.optimizer.zero_grad(set_to_none=True)
loss.backward()
if self.config.optimization.grad_clip > 0:
@@ -270,6 +359,9 @@ class IsMctsTrainer:
self.config.optimization.grad_clip,
)
self.optimizer.step()
# Stash auxiliary loss for the IterationMetrics path; we return only
# the three primary scalars for backward compatibility.
self._last_kl_anchor_loss = kl_anchor_loss
return float(policy_loss.item()), float(value_loss.item()), float(loss.item())
def _evaluate(self, iteration: int) -> dict[str, float | int]:
@@ -364,9 +456,15 @@ class IsMctsTrainer:
with torch.inference_mode():
_logits, value_pred = self.network(info, legal)
value_error = nn.functional.mse_loss(value_pred, target)
value_rmse = float(value_error.item()) ** 0.5
target_abs_mean = float(target.abs().mean().item())
target_std = float(target.std().item()) if target.numel() > 1 else 0.0
return {
"mcts/avg_visit_entropy": float(np.mean(entropies)) if entropies else 0.0,
"mcts/value_prediction_error": float(value_error.item()),
"mcts/value_rmse": value_rmse,
"mcts/v_target_abs_mean": target_abs_mean,
"mcts/v_target_std": target_std,
"mcts/policy_mcts_kl": float(np.mean(policy_kls)) if policy_kls else 0.0,
}
src/coolrl_lost_cities/games/classic/jax_ppo_policy.py
+196
View File
@@ -0,0 +1,196 @@
"""Adapter from the classic two-phase GUI to a JAX PPO checkpoint."""
from __future__ import annotations
import logging
from collections.abc import Mapping, Sequence
from pathlib import Path
from typing import Any
import jax.numpy as jnp
import numpy as np
from lost_cities_jax.human_play import (
evaluate_agent_policy,
infer_config_path,
load_agent,
)
from lost_cities_jax.ppo import load_config
from lost_cities_jax.types import (
LOC_DECK,
LOC_DISCARD,
LOC_P0_BOARD,
LOC_P0_HAND,
MAX_PILE_SIZE,
N_CARDS,
N_COLORS,
NO_CARD,
State,
)
from .game import Card, GameState
from .snapshots import Snapshot, snapshot_from_state
def _jax_card_id(card: Card, handshake_copy: int = 0) -> int:
slot = handshake_copy if card.is_handshake else card.numeric_value(2) + 1
return card.color * 12 + slot
PublicHandCounts = Sequence[Mapping[tuple[int, int], int]]
def snapshot_to_jax_state(
snapshot: Snapshot,
public_hand_counts: PublicHandCounts | None = None,
) -> tuple[State, list[int]]:
"""Convert a classic card-phase snapshot into the equivalent JAX state.
The returned list maps each JAX sorted hand slot back to the classic hand
slot. Handshake copies are interchangeable to both rules engines and the
policy observation, so they are assigned stable per-color identities here.
"""
if (
snapshot.config.n_colors != N_COLORS
or snapshot.config.n_ranks != 9
or snapshot.config.n_handshakes != 3
or snapshot.config.hand_size != 8
or snapshot.config.min_rank != 2
):
raise ValueError("JAX PPO checkpoints require the classic 5-color, 60-card rules")
if snapshot.phase != "card":
raise ValueError("JAX PPO policy conversion must start during the card phase")
handshake_next = [0] * N_COLORS
def allocate(card: Card) -> int:
if not card.is_handshake:
return _jax_card_id(card)
copy = handshake_next[card.color]
if copy >= 3:
raise ValueError(f"too many handshake copies for color {card.color}")
handshake_next[card.color] += 1
return _jax_card_id(card, copy)
card_loc = np.full(N_CARDS, LOC_DECK, dtype=np.int8)
hand_public = np.zeros(N_CARDS, dtype=bool)
zone_ids: list[int] = []
hand_pairs: list[tuple[int, int]] = []
deck_ids = [allocate(card) for card in snapshot.deck]
zone_ids.extend(deck_ids)
for player, hand in enumerate(snapshot.hands):
public_remaining = dict(public_hand_counts[player]) if public_hand_counts else {}
for classic_slot, card in enumerate(hand):
card_id = allocate(card)
card_loc[card_id] = LOC_P0_HAND + player
zone_ids.append(card_id)
face = (card.color, 0 if card.is_handshake else card.numeric_value(2))
if public_remaining.get(face, 0) > 0:
hand_public[card_id] = True
public_remaining[face] -= 1
if player == snapshot.current_player:
hand_pairs.append((card_id, classic_slot))
col_top = np.zeros((2, N_COLORS), dtype=np.int8)
col_hs = np.zeros((2, N_COLORS), dtype=np.int8)
col_len = np.zeros((2, N_COLORS), dtype=np.int8)
for player, expeditions in enumerate(snapshot.expeditions):
for color, expedition in enumerate(expeditions):
for card in expedition:
card_id = allocate(card)
card_loc[card_id] = LOC_P0_BOARD + player
zone_ids.append(card_id)
col_len[player, color] += 1
if card.is_handshake:
col_hs[player, color] += 1
else:
col_top[player, color] = card.numeric_value(2)
pile = np.full((N_COLORS, MAX_PILE_SIZE), NO_CARD, dtype=np.int8)
pile_len = np.zeros(N_COLORS, dtype=np.int8)
for color, discard in enumerate(snapshot.discards):
for index, card in enumerate(discard):
card_id = allocate(card)
card_loc[card_id] = LOC_DISCARD
pile[color, index] = card_id
pile_len[color] += 1
zone_ids.append(card_id)
if len(zone_ids) != N_CARDS or len(set(zone_ids)) != N_CARDS:
raise ValueError("classic snapshot does not contain one complete 60-card deck")
# Classic draws from the end of its deck list. JAX draws forward from
# draw_ptr, so its remaining suffix is the reversed classic deck.
remaining = list(reversed(deck_ids))
deck_id_set = set(deck_ids)
used = [card_id for card_id in range(N_CARDS) if card_id not in deck_id_set]
deck_order = np.asarray([*used, *remaining], dtype=np.int8)
draw_ptr = len(used)
hand_slot_map = [slot for _, slot in sorted(hand_pairs)]
state = State(
deck_order=jnp.asarray(deck_order),
draw_ptr=jnp.asarray(draw_ptr, dtype=jnp.int32),
card_loc=jnp.asarray(card_loc),
hand_public=jnp.asarray(hand_public),
col_top=jnp.asarray(col_top),
col_hs=jnp.asarray(col_hs),
col_len=jnp.asarray(col_len),
pile=jnp.asarray(pile),
pile_len=jnp.asarray(pile_len),
to_move=jnp.asarray(snapshot.current_player, dtype=jnp.int8),
just_discarded=jnp.asarray(NO_CARD, dtype=jnp.int8),
step_count=jnp.asarray(snapshot.turn_count, dtype=jnp.int32),
done=jnp.asarray(snapshot.terminal),
)
return state, hand_slot_map
class JaxPPOPolicy:
"""Expose an atomic-action JAX policy through the classic policy protocol."""
def __init__(self, checkpoint: str | Path, *, config: str | Path | None = None):
# The classic GUI enables root DEBUG logging for its own interaction
# trace. Keep JAX/Orbax internals from flooding the launcher terminal.
for logger_name in ("jax", "absl", "orbax"):
logging.getLogger(logger_name).setLevel(logging.WARNING)
cfg_path = infer_config_path(checkpoint) if config is None else Path(config)
self.cfg = load_config(cfg_path)
self.params, self.model = load_agent(self.cfg, checkpoint)
self.checkpoint = Path(checkpoint)
self.pending_draw: int | None = None
self.last_evaluation: Any | None = None
def act(
self,
obs_or_state: Any,
public_hand_counts: PublicHandCounts | None = None,
) -> int:
if not isinstance(obs_or_state, GameState):
raise TypeError("JAX PPO GUI policy requires a GameState")
if obs_or_state.phase == "draw":
if self.pending_draw is None:
raise RuntimeError("JAX PPO draw requested without a pending atomic action")
draw = self.pending_draw
self.pending_draw = None
return draw
snapshot = snapshot_from_state(obs_or_state)
jax_state, hand_slot_map = snapshot_to_jax_state(snapshot, public_hand_counts)
result = evaluate_agent_policy(
self.cfg,
self.params,
self.model,
jax_state,
snapshot.current_player,
)
self.last_evaluation = result
jax_hand_slot = result.action // 12
remainder = result.action % 12
place = remainder // 6
self.pending_draw = remainder % 6
return 2 * hand_slot_map[jax_hand_slot] + place
__all__ = ["JaxPPOPolicy", "PublicHandCounts", "snapshot_to_jax_state"]

Some files were not shown because too many files have changed in this diff Show More