The cap item existed so a three-round match would fit inside a 400-step scan.
Phase 0a's truncation bootstrap removed that constraint -- scan length is a free
parameter now -- and the same phase showed the cap is the thing that turns
stalled games into losses: 20.6% of cap-hit games lost, 99.4% of all losses. A
120-ply cap would put *more* games into the wall and make "freeze the round
while ahead" easier, so the round cap stays at 400 and max_steps_rate stays a
watched metric.
Also records what the plan never listed: ppo.py, gates.py and league.py all take
a single-round State and still have to be wired to MatchState. That is the
largest remaining piece of Phase 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
At equal compute (250 updates either way, 5.8x the samples) the new loop beats
the baseline against heuristic_expert: 0.4525 -> 0.5071 win rate, -4.28 ->
-0.56 mean score diff. Expert ends rounds at their natural length, so that is
a clean read on card play.
It appears to *lose* ground against heuristic_balanced (0.9866 -> 0.9292), but
that is entirely the MAX_STEPS=400 cap. Drawing from a discard pile does not
deplete the deck, so a round can be stalled indefinitely; against a weak
opponent the extra turns are worth points, and with 5.8x the samples the agent
learns the exploit harder (greedy rounds run 225 plies). Splitting 2048 games:
naturally-ended games lose 0.08%, cap-hit games lose 20.62%, and 99.4% of all
losses land in cap-hit games. The board freezes mid-expedition and the -20s
stand.
Self-play does not have this failure mode -- league runs converge to 53.6-ply
rounds and never hit the cap -- which is the regime the 3-round work targets.
Noted in the plan: static heuristic anchors are unusable as gates, and the
per-round cap needs care in Phase 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
rollout_steps was pinned to MAX_STEPS (400) while a round actually runs
~50 plies, and finished envs were only reset between updates. 83% of every
rollout was spent stepping already-done envs to produce masked-out zeros.
Measured at rollout_steps=400: active steps go 17.3% -> 100%, i.e. 5.8x the
learner actions per update for the same compute.
Resetting in-scan exposes three things that were previously benign:
- compute_gae bootstrapped truncated episodes from zero. That was safe only
because every episode used to terminate inside the scan; now episodes cross
the boundary, so thread V(s_T) through.
- rollout_metrics read final_env and summed rewards along the scan axis, both
of which assume one episode per slot. With several episodes per slot that
silently produces garbage, so aggregate at done boundaries instead.
- league assignments were redrawn only between updates, which would pin a slot
to one seat/opponent across every episode in a scan. Redraw them on reset.
Also anneal potential shaping against learner actions rather than padded scan
steps: the old accounting counted the dead steps, so a 5M-step anneal expired
within two updates of 250. Any earlier evidence that shaping does not help was
gathered with it effectively off.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
coolguyandClaude Opus 4.8