coolguyandClaude Opus 4.7
f289997c1c
Trap diagnosis: agent learned to stall (avoid opening expeditions, draw from discard pile to extend deck) until max_steps timeout, then squeak by on opponents' negative scores. All eval wins were from timeouts; agent never won a naturally-terminating game. Self-play reinforced this because timeout games still got a positive value target. Fixes (no algorithm change, all MCTS hyperparameters or signal shaping): - Dirichlet noise at root prior (AlphaZero standard, was missing): mcts.pyx `_expand_with_prior` takes `is_root` flag; root expansion mixes prior with Dirichlet(α). Callers in interleaved_self_play and the internal evaluate_and_backup pass `not item.path`. - Default config strengthens exploration on the 50-sim batched search: c_puct 1.5 -> 3.0, virtual_loss_value 1.0 -> 5.0, plus new root_dirichlet_alpha=0.3 / root_dirichlet_epsilon=0.25. - Self-play timeout signal zeroed: `_finalize_context` sets v_target=0 if context.state is not terminal. Stops the network from learning "stall = positive value". New standalone evaluator: - `lost-cities-ismcts eval` subcommand (eval_checkpoint.py): loads a checkpoint, runs N games per opponent across a parallel pool, reports win/score with 95% CIs plus per-game logging via --verbose. Defaults cover heuristic-balanced/aggressive/cautious (rollout policy isn't in the training-eval opponent list, so this is the natural way to compare the trained policy against its rollout target). Tests (19) still pass; .so rebuilt. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
57 lines
1.0 KiB
YAML
57 lines
1.0 KiB
YAML
run:
|
|
experiment_name: ismcts-default
|
|
max_iterations: 500
|
|
seed: 1
|
|
device: cuda
|
|
rules:
|
|
n_colors: 5
|
|
n_ranks: 9
|
|
min_rank: 2
|
|
n_handshakes: 3
|
|
hand_size: 8
|
|
expedition_penalty: -20
|
|
bonus_threshold: 8
|
|
bonus_amount: 20
|
|
encoding:
|
|
derived_playability: true
|
|
slot_aware_playability: true
|
|
network:
|
|
kind: mlp
|
|
hidden_size: 512
|
|
num_layers: 3
|
|
activation: relu
|
|
mcts:
|
|
n_simulations: 50
|
|
c_puct: 3.0
|
|
max_depth: 200
|
|
parallel_simulations: 64
|
|
virtual_loss_value: 5.0
|
|
eval_n_simulations: 16
|
|
rollout_policy: heuristic_balanced
|
|
root_dirichlet_alpha: 0.3
|
|
root_dirichlet_epsilon: 0.25
|
|
temperature:
|
|
training: 1.0
|
|
eval: 0.0
|
|
training:
|
|
games_per_iter: 10
|
|
gradient_steps_per_iter: 10
|
|
batch_size: 128
|
|
replay_capacity: 100000
|
|
interleave_games: 8
|
|
interleave_max_batch: 64
|
|
num_workers: 8
|
|
worker_device: cuda
|
|
optimization:
|
|
learning_rate: 0.0003
|
|
grad_clip: 5.0
|
|
checkpoint:
|
|
save_every: 20
|
|
save_latest: true
|
|
evaluation:
|
|
eval_every: 5
|
|
games: 5
|
|
opponents: [random, discard-only, heuristic-cautious]
|
|
max_steps: 500
|
|
num_workers: 8
|