Add Dirichlet root noise + standalone eval CLI, fix self-play stall trap
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw from discard pile to extend deck) until max_steps timeout, then squeak by on opponents' negative scores. All eval wins were from timeouts; agent never won a naturally-terminating game. Self-play reinforced this because timeout games still got a positive value target. Fixes (no algorithm change, all MCTS hyperparameters or signal shaping): - Dirichlet noise at root prior (AlphaZero standard, was missing): mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion mixes prior with Dirichlet(α). Callers in interleaved_self_play and the internal evaluate_and_backup pass `not item.path`. - Default config strengthens exploration on the 50-sim batched search: c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25. - Self-play timeout signal zeroed: `_finalize_context` sets v_target=0 if context.state is not terminal. Stops the network from learning "stall = positive value". New standalone evaluator: - `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a checkpoint, runs N games per opponent across a parallel pool, reports win/score with 95% CIs plus per-game logging via --verbose. Defaults cover heuristic-balanced/aggressive/cautious (rollout policy isn't in the training-eval opponent list, so this is the natural way to compare the trained policy against its rollout target). Tests (19) still pass; .so rebuilt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
+16
-10
@@ -1,8 +1,8 @@
|
||||
run:
|
||||
experiment_name: ismcts-default
|
||||
max_iterations: 100
|
||||
max_iterations: 500
|
||||
seed: 1
|
||||
device: auto
|
||||
device: cuda
|
||||
rules:
|
||||
n_colors: 5
|
||||
n_ranks: 9
|
||||
@@ -22,10 +22,14 @@ network:
|
||||
activation: relu
|
||||
mcts:
|
||||
n_simulations: 50
|
||||
c_puct: 1.5
|
||||
c_puct: 3.0
|
||||
max_depth: 200
|
||||
parallel_simulations: 8
|
||||
virtual_loss_value: 1.0
|
||||
parallel_simulations: 64
|
||||
virtual_loss_value: 5.0
|
||||
eval_n_simulations: 16
|
||||
rollout_policy: heuristic_balanced
|
||||
root_dirichlet_alpha: 0.3
|
||||
root_dirichlet_epsilon: 0.25
|
||||
temperature:
|
||||
training: 1.0
|
||||
eval: 0.0
|
||||
@@ -36,15 +40,17 @@ training:
|
||||
replay_capacity: 100000
|
||||
interleave_games: 8
|
||||
interleave_max_batch: 64
|
||||
num_workers: 8
|
||||
worker_device: cuda
|
||||
optimization:
|
||||
learning_rate: 0.0003
|
||||
grad_clip: 5.0
|
||||
checkpoint:
|
||||
save_every: 10
|
||||
save_every: 20
|
||||
save_latest: true
|
||||
evaluation:
|
||||
eval_every: 10
|
||||
games: 20
|
||||
eval_every: 5
|
||||
games: 5
|
||||
opponents: [random, discard-only, heuristic-cautious]
|
||||
max_steps: 10000
|
||||
num_workers: 1
|
||||
max_steps: 500
|
||||
num_workers: 8
|
||||
|
||||
Reference in New Issue
Block a user