The user asked why we were not simply maximising the three-round total, and they
were right. Each ply now pays the points by which it moved the running match
difference; at gamma=1 that telescopes to the final total, so the objective is
exactly the rulebook's, handed out every ply instead of once 150 plies later.
Three measurements killed the tanh design:
- Rounds are independent (corr(m1,m2)=0.004, corr(m1+m2,m3)=0.05), so a reward
linear in the total decomposes the match into three independent rounds and
carry enters the objective nowhere. The only coupling, the start-player rule,
is worth +0.73 +/- 0.84 points -- indistinguishable from zero.
- Risk attitude, the one thing tanh buys, is worthless here. A policy made to
gamble when it trails by 20 entering round three *loses* to a greedy clone over
6144 duplicate matches (0.482); gambling only at -40 breaks even (0.498). A
marginal wager buys about +1.7 sigma for -2 to -3 expected points. Ceiling on
the whole carry-conditioning idea: under one win-rate point.
- Head to head over 10,000 duplicate matches at equal compute, the linear reward
*beats* tanh(total/12): 0.5859 (CI 0.576-0.596), +20.3 points. Dropping it is
not merely free, it is better -- not because of risk, but because tanh hands a
~150-ply match one saturated +/-1 and leaves all credit assignment to the critic.
The flat carry probe was not exploration collapse: sampled play still opens 5.00
expeditions, entropy settles at 1.36 nats (3.9 effective actions), and the critic
reads carry cleanly (round-three values run -0.87 to +0.86, monotone). The signal
was there; there was nothing to buy with it.
Criterion 1 (a monotone carry response) comes off the gate accordingly -- the
optimal response barely exists in this game. carry stays in the observation: it
costs nothing and the start-player rule keys off it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
coolguyandClaude Opus 4.8
8860f62030