coolguyandClaude Opus 4.8
f170fcdcfd
Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck- race rate, so the stalling that static opponents induced is gone. Duplicate match eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both seats, deal luck cancelling exactly. Success criterion 1 does not pass. The carry probe is close to flat: expeditions opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use does move monotonically across all six carry levels, and in the right direction (behind -> more multipliers), but the spread is 0.31 wagers. Two diagnoses, one of which was mine and wrong: - Residual potential shaping was NOT the cause. Annealing it fully to zero left the probe just as flat. - terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over any realistic round margin, and maximising E[tanh] on a linear stretch is just maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12 widens the wager spread 0.19 -> 0.31, which is the mechanism showing up. The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is a saturated dead zone with no gradient and the policy has learned nothing there. The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to be set from the scale rather than fixed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh
51 lines
1.4 KiB
YAML
51 lines
1.4 KiB
YAML
# Classic Lost Cities: three rounds, self-play.
|
|
#
|
|
# rollout_steps is deliberately shorter than a match. Early self-play stalls
|
|
# rounds badly (untrained matches run ~900 plies), and the env carries across
|
|
# updates, so a long match simply spans several rollouts -- the GAE truncation
|
|
# bootstrap keeps that unbiased. Sizing the scan to the worst-case match instead
|
|
# would blow up the rollout tensors, which are already doubled by training both
|
|
# seats and widened by the critic's 681-dim privileged view.
|
|
run:
|
|
experiment_name: match-selfplay
|
|
seed: 20260715
|
|
learner_seat: 0
|
|
total_updates: 300
|
|
log_every: 10
|
|
checkpoint_every: 50
|
|
artifact_root: runs/jax-ppo-match
|
|
|
|
opponent:
|
|
name: discard_only # unused: self-play
|
|
|
|
network:
|
|
hidden_size: 512
|
|
num_layers: 3
|
|
|
|
ppo:
|
|
batch_games: 512
|
|
rollout_steps: 256
|
|
gamma: 1.0
|
|
gae_lambda: 0.97 # a match is ~3x a round; 0.95 reaches too little of it
|
|
clip_epsilon: 0.2
|
|
entropy_coef: 0.01
|
|
value_coef: 0.5
|
|
max_grad_norm: 0.5
|
|
learning_rate: 0.0003
|
|
epochs: 4
|
|
minibatches: 32
|
|
|
|
reward:
|
|
terminal_scale: 50.0
|
|
# Counted in learner actions now, not padded scan steps. Held slightly above
|
|
# zero: the match-terminal signal alone is one bounded number per ~160 plies.
|
|
potential_shaping_initial: 1.0
|
|
potential_shaping_final: 0.05
|
|
potential_shaping_anneal_steps: 20000000
|
|
|
|
evaluation:
|
|
games: 2000
|
|
duplicate: true
|
|
shuffle_bank_seed: 20260715
|
|
batch_games: 512
|