coolguy
8b7ed66ffd
Codex deep diagnosis surfaced two real issues in our MCTS pipeline: 1. mcts.pyx::search() hardcoded prepare_simulation_batch(state, traverser, 1) instead of respecting MctsConfig.parallel_simulations. Standalone evals (eval_checkpoint, evaluate_with_mcts sequential path, eval_worker) all use this entry point, so all eval-time MCTS was running 1 sim per batch regardless of the configured 64. Training was unaffected because it goes through interleaved_self_play._run_search_jobs which respects the config. Fix uses min(config.parallel_simulations, sims - completed). 2. use_rollout_value defaulted to True (config.py) but was never set in the YAML. With this, _expand_with_prior returns the heuristic rollout value and discards network_value, so the network value head is trained from final game scores but its outputs are never fed back into MCTS backups. This explains why mcts/value_prediction_error stays high despite training -- learning the value head produces no behavioral change because MCTS never reads it. Now setting use_rollout_value=false in default.yaml so the network value head closes the loop. Combined with the existing Dirichlet root noise + heuristic rollout removal, this should give the network's value learning actual leverage on action selection. Also: updated test_search_visit_counts_match_with_parallel_simulations to test the correct invariant (legal-action set match + total visit count near n_sims) rather than literal visit-count equality, which was only true under the previous bug. Tests: 19/19 passing.
58 lines
1.1 KiB
YAML
58 lines
1.1 KiB
YAML
run:
|
|
experiment_name: ismcts-default
|
|
max_iterations: 500
|
|
seed: 1
|
|
device: cuda
|
|
rules:
|
|
n_colors: 5
|
|
n_ranks: 9
|
|
min_rank: 2
|
|
n_handshakes: 3
|
|
hand_size: 8
|
|
expedition_penalty: -20
|
|
bonus_threshold: 8
|
|
bonus_amount: 20
|
|
encoding:
|
|
derived_playability: true
|
|
slot_aware_playability: true
|
|
network:
|
|
kind: mlp
|
|
hidden_size: 512
|
|
num_layers: 3
|
|
activation: relu
|
|
mcts:
|
|
n_simulations: 50
|
|
c_puct: 5.0
|
|
max_depth: 200
|
|
parallel_simulations: 64
|
|
virtual_loss_value: 5.0
|
|
eval_n_simulations: 16
|
|
rollout_policy: heuristic_balanced
|
|
use_rollout_value: false
|
|
root_dirichlet_alpha: 0.3
|
|
root_dirichlet_epsilon: 0.4
|
|
temperature:
|
|
training: 1.0
|
|
eval: 0.0
|
|
training:
|
|
games_per_iter: 10
|
|
gradient_steps_per_iter: 10
|
|
batch_size: 128
|
|
replay_capacity: 100000
|
|
interleave_games: 8
|
|
interleave_max_batch: 64
|
|
num_workers: 8
|
|
worker_device: cuda
|
|
optimization:
|
|
learning_rate: 0.0003
|
|
grad_clip: 5.0
|
|
checkpoint:
|
|
save_every: 20
|
|
save_latest: true
|
|
evaluation:
|
|
eval_every: 5
|
|
games: 5
|
|
opponents: [random, discard-only, heuristic-cautious]
|
|
max_steps: 500
|
|
num_workers: 8
|