coolguy
0d35341bbe
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score avg -85 to -97 vs three heuristic opponents. Bigger network alone did not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without rollout) but agent still loses every natural-end game. Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not break the loss pattern. Score average actually slightly worse than 512x3 baseline. So the bottleneck is not network capacity. Going to the long-run experiment: AlphaZero-correct setup with the network value loop closed. use_rollout_value=false means leaf Q comes from network value head. Training signal: network value learns from actual game outcomes; MCTS uses those values to pick actions; better actions produce better outcomes; cycle closes. 100 iter previously gave essentially the same result as rollout=true (comparing c3 to trapfix baseline). Both are too early in the AlphaZero training curve. Standard AlphaZero papers train 1000s of iterations. Going long: 1000 iter with the current config. Self-play ~9s/iter without rollout, total wall ~150 min for the train phase. Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval and generate analyze.py plots to visualize trajectory. Network kept at 768x4 since bigger capacity does not actively hurt.
58 lines
1.1 KiB
YAML
58 lines
1.1 KiB
YAML
run:
|
|
experiment_name: ismcts-default
|
|
max_iterations: 500
|
|
seed: 1
|
|
device: cuda
|
|
rules:
|
|
n_colors: 5
|
|
n_ranks: 9
|
|
min_rank: 2
|
|
n_handshakes: 3
|
|
hand_size: 8
|
|
expedition_penalty: -20
|
|
bonus_threshold: 8
|
|
bonus_amount: 20
|
|
encoding:
|
|
derived_playability: true
|
|
slot_aware_playability: true
|
|
network:
|
|
kind: mlp
|
|
hidden_size: 768
|
|
num_layers: 4
|
|
activation: relu
|
|
mcts:
|
|
n_simulations: 50
|
|
c_puct: 5.0
|
|
max_depth: 200
|
|
parallel_simulations: 64
|
|
virtual_loss_value: 5.0
|
|
eval_n_simulations: 16
|
|
rollout_policy: heuristic_balanced
|
|
use_rollout_value: false
|
|
root_dirichlet_alpha: 0.3
|
|
root_dirichlet_epsilon: 0.4
|
|
temperature:
|
|
training: 1.0
|
|
eval: 0.0
|
|
training:
|
|
games_per_iter: 10
|
|
gradient_steps_per_iter: 10
|
|
batch_size: 128
|
|
replay_capacity: 100000
|
|
interleave_games: 8
|
|
interleave_max_batch: 64
|
|
num_workers: 8
|
|
worker_device: cuda
|
|
optimization:
|
|
learning_rate: 0.0003
|
|
grad_clip: 5.0
|
|
checkpoint:
|
|
save_every: 20
|
|
save_latest: true
|
|
evaluation:
|
|
eval_every: 5
|
|
games: 5
|
|
opponents: [random, discard-only, heuristic-cautious]
|
|
max_steps: 500
|
|
num_workers: 8
|