Cycle 5 setup: long run with use_rollout_value=false + 768x4 network
C4 (768x4 + rollout=True, 150 iter) result: 0/300 natural wins, score avg -85 to -97 vs three heuristic opponents. Bigger network alone did not produce wins; PA shifted up to 0.23-0.25 (similar to c3 without rollout) but agent still loses every natural-end game. Capacity hypothesis rejected: 2.5x more params (~2M vs ~800k) did not break the loss pattern. Score average actually slightly worse than 512x3 baseline. So the bottleneck is not network capacity. Going to the long-run experiment: AlphaZero-correct setup with the network value loop closed. use_rollout_value=false means leaf Q comes from network value head. Training signal: network value learns from actual game outcomes; MCTS uses those values to pick actions; better actions produce better outcomes; cycle closes. 100 iter previously gave essentially the same result as rollout=true (comparing c3 to trapfix baseline). Both are too early in the AlphaZero training curve. Standard AlphaZero papers train 1000s of iterations. Going long: 1000 iter with the current config. Self-play ~9s/iter without rollout, total wall ~150 min for the train phase. Plotting strategy: at iter 200, 500, 1000, run 100-game standalone eval and generate analyze.py plots to visualize trajectory. Network kept at 768x4 since bigger capacity does not actively hurt.
This commit is contained in:
@@ -28,7 +28,7 @@ mcts:
|
||||
virtual_loss_value: 5.0
|
||||
eval_n_simulations: 16
|
||||
rollout_policy: heuristic_balanced
|
||||
use_rollout_value: true
|
||||
use_rollout_value: false
|
||||
root_dirichlet_alpha: 0.3
|
||||
root_dirichlet_epsilon: 0.4
|
||||
temperature:
|
||||
|
||||
Reference in New Issue
Block a user