The question was what any of this actually improved over the training method
that existed. Duplicate matches, 8192 a piece, same three deals and coins from
both seats:
- At 39.3M learner actions the match stack beats the Phase 0a gate agent
(0.5842) which had 411M -- 10.5x the data. Sample efficiency is the headline.
- At 39.3M it *loses* to the league policy (0.3142). That is a budget gap, not a
strength gap: league had 122.6M plus a league/exploiter structure.
- Scaled to a matched budget (131M vs league's 122.6M) it wins: 0.6094
(CI 0.599-0.620), +22.0 points.
So: same compute, stronger agent, measured on the actual game.
Caveat kept honest in the plan -- league was trained with exploiters, and we have
measured average strength, not exploitability. "Harder to exploit" is not shown.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
The user asked why we were not simply maximising the three-round total, and they
were right. Each ply now pays the points by which it moved the running match
difference; at gamma=1 that telescopes to the final total, so the objective is
exactly the rulebook's, handed out every ply instead of once 150 plies later.
Three measurements killed the tanh design:
- Rounds are independent (corr(m1,m2)=0.004, corr(m1+m2,m3)=0.05), so a reward
linear in the total decomposes the match into three independent rounds and
carry enters the objective nowhere. The only coupling, the start-player rule,
is worth +0.73 +/- 0.84 points -- indistinguishable from zero.
- Risk attitude, the one thing tanh buys, is worthless here. A policy made to
gamble when it trails by 20 entering round three *loses* to a greedy clone over
6144 duplicate matches (0.482); gambling only at -40 breaks even (0.498). A
marginal wager buys about +1.7 sigma for -2 to -3 expected points. Ceiling on
the whole carry-conditioning idea: under one win-rate point.
- Head to head over 10,000 duplicate matches at equal compute, the linear reward
*beats* tanh(total/12): 0.5859 (CI 0.576-0.596), +20.3 points. Dropping it is
not merely free, it is better -- not because of risk, but because tanh hands a
~150-ply match one saturated +/-1 and leaves all credit assignment to the critic.
The flat carry probe was not exploration collapse: sampled play still opens 5.00
expeditions, entropy settles at 1.36 nats (3.9 effective actions), and the critic
reads carry cleanly (round-three values run -0.87 to +0.86, monotone). The signal
was there; there was nothing to buy with it.
Criterion 1 (a monotone carry response) comes off the gate accordingly -- the
optimal response barely exists in this game. carry stays in the observation: it
costs nothing and the start-player rule keys off it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck-
race rate, so the stalling that static opponents induced is gone. Duplicate match
eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both
seats, deal luck cancelling exactly.
Success criterion 1 does not pass. The carry probe is close to flat: expeditions
opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use
does move monotonically across all six carry levels, and in the right direction
(behind -> more multipliers), but the spread is 0.31 wagers.
Two diagnoses, one of which was mine and wrong:
- Residual potential shaping was NOT the cause. Annealing it fully to zero left
the probe just as flat.
- terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over
any realistic round margin, and maximising E[tanh] on a linear stretch is just
maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears
where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12
widens the wager spread 0.19 -> 0.31, which is the mechanism showing up.
The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is
a saturated dead zone with no gradient and the policy has learned nothing there.
The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to
be set from the scale rather than fixed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
The cap item existed so a three-round match would fit inside a 400-step scan.
Phase 0a's truncation bootstrap removed that constraint -- scan length is a free
parameter now -- and the same phase showed the cap is the thing that turns
stalled games into losses: 20.6% of cap-hit games lost, 99.4% of all losses. A
120-ply cap would put *more* games into the wall and make "freeze the round
while ahead" easier, so the round cap stays at 400 and max_steps_rate stays a
watched metric.
Also records what the plan never listed: ppo.py, gates.py and league.py all take
a single-round State and still have to be wired to MatchState. That is the
largest remaining piece of Phase 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
At equal compute (250 updates either way, 5.8x the samples) the new loop beats
the baseline against heuristic_expert: 0.4525 -> 0.5071 win rate, -4.28 ->
-0.56 mean score diff. Expert ends rounds at their natural length, so that is
a clean read on card play.
It appears to *lose* ground against heuristic_balanced (0.9866 -> 0.9292), but
that is entirely the MAX_STEPS=400 cap. Drawing from a discard pile does not
deplete the deck, so a round can be stalled indefinitely; against a weak
opponent the extra turns are worth points, and with 5.8x the samples the agent
learns the exploit harder (greedy rounds run 225 plies). Splitting 2048 games:
naturally-ended games lose 0.08%, cap-hit games lose 20.62%, and 99.4% of all
losses land in cap-hit games. The board freezes mid-expedition and the -20s
stand.
Self-play does not have this failure mode -- league runs converge to 53.6-ply
rounds and never hit the cap -- which is the regime the 3-round work targets.
Noted in the plan: static heuristic anchors are unusable as gates, and the
per-round cap needs care in Phase 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
rollout_steps was pinned to MAX_STEPS (400) while a round actually runs
~50 plies, and finished envs were only reset between updates. 83% of every
rollout was spent stepping already-done envs to produce masked-out zeros.
Measured at rollout_steps=400: active steps go 17.3% -> 100%, i.e. 5.8x the
learner actions per update for the same compute.
Resetting in-scan exposes three things that were previously benign:
- compute_gae bootstrapped truncated episodes from zero. That was safe only
because every episode used to terminate inside the scan; now episodes cross
the boundary, so thread V(s_T) through.
- rollout_metrics read final_env and summed rewards along the scan axis, both
of which assume one episode per slot. With several episodes per slot that
silently produces garbage, so aggregate at done boundaries instead.
- league assignments were redrawn only between updates, which would pin a slot
to one seat/opponent across every episode in a scan. Redraw them on reset.
Also anneal potential shaping against learner actions rather than padded scan
steps: the old accounting counted the dead steps, so a 5M-step anneal expired
within two updates of 250. Any earlier evidence that shaping does not help was
gathered with it effectively off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
coolguyandClaude Opus 4.8