Run the classic self-play trainer and record what the carry probe says

Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck-
race rate, so the stalling that static opponents induced is gone. Duplicate match
eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both
seats, deal luck cancelling exactly.

Success criterion 1 does not pass. The carry probe is close to flat: expeditions
opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use
does move monotonically across all six carry levels, and in the right direction
(behind -> more multipliers), but the spread is 0.31 wagers.

Two diagnoses, one of which was mine and wrong:

- Residual potential shaping was NOT the cause. Annealing it fully to zero left
  the probe just as flat.
- terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over
  any realistic round margin, and maximising E[tanh] on a linear stretch is just
  maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears
  where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12
  widens the wager spread 0.19 -> 0.31, which is the mechanism showing up.

The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is
a saturated dead zone with no gradient and the policy has learned nothing there.
The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to
be set from the scale rather than fixed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh

configs/jax_ppo/match-selfplay.yaml

@@ -0,0 +1,50 @@
+# Classic Lost Cities: three rounds, self-play.
+#
+# rollout_steps is deliberately shorter than a match. Early self-play stalls
+# rounds badly (untrained matches run ~900 plies), and the env carries across
+# updates, so a long match simply spans several rollouts -- the GAE truncation
+# bootstrap keeps that unbiased. Sizing the scan to the worst-case match instead
+# would blow up the rollout tensors, which are already doubled by training both
+# seats and widened by the critic's 681-dim privileged view.
+run:
+  experiment_name: match-selfplay
+  seed: 20260715
+  learner_seat: 0
+  total_updates: 300
+  log_every: 10
+  checkpoint_every: 50
+  artifact_root: runs/jax-ppo-match
+
+opponent:
+  name: discard_only # unused: self-play
+
+network:
+  hidden_size: 512
+  num_layers: 3
+
+ppo:
+  batch_games: 512
+  rollout_steps: 256
+  gamma: 1.0
+  gae_lambda: 0.97 # a match is ~3x a round; 0.95 reaches too little of it
+  clip_epsilon: 0.2
+  entropy_coef: 0.01
+  value_coef: 0.5
+  max_grad_norm: 0.5
+  learning_rate: 0.0003
+  epochs: 4
+  minibatches: 32
+
+reward:
+  terminal_scale: 50.0
+  # Counted in learner actions now, not padded scan steps. Held slightly above
+  # zero: the match-terminal signal alone is one bounded number per ~160 plies.
+  potential_shaping_initial: 1.0
+  potential_shaping_final: 0.05
+  potential_shaping_anneal_steps: 20000000
+
+evaluation:
+  games: 2000
+  duplicate: true
+  shuffle_bank_seed: 20260715
+  batch_games: 512