Add diagnostic value-head metrics: rmse, target stats

Codex flagged that mcts/value_prediction_error mathematically reconciles
with loss/value (MSE / value_scale^2 = 0.05) but the latter looks healthy
while the former says the value head is far off. To make this clearer in
W&B, expose:

- mcts/value_rmse — sqrt(MSE), in raw score units (interpretable)
- mcts/v_target_abs_mean — magnitude of |v_target|, indicates if game
  outcomes are very lopsided (always negative for a losing agent)
- mcts/v_target_std — spread, low std means targets are saturated to
  one end (e.g., always -100ish)

These let us see whether value head is failing because targets are
unlearnable variance, or just hard-to-predict, or because of saturation
at the value_scale=100 tanh boundary.

Tests: 19/19 passing.
This commit is contained in:
2026-05-11 06:43:50 +09:00
parent 8b7ed66ffd
commit 200129d16d
@@ -365,9 +365,15 @@ class IsMctsTrainer:
with torch.inference_mode(): with torch.inference_mode():
_logits, value_pred = self.network(info, legal) _logits, value_pred = self.network(info, legal)
value_error = nn.functional.mse_loss(value_pred, target) value_error = nn.functional.mse_loss(value_pred, target)
value_rmse = float(value_error.item()) ** 0.5
target_abs_mean = float(target.abs().mean().item())
target_std = float(target.std().item()) if target.numel() > 1 else 0.0
return { return {
"mcts/avg_visit_entropy": float(np.mean(entropies)) if entropies else 0.0, "mcts/avg_visit_entropy": float(np.mean(entropies)) if entropies else 0.0,
"mcts/value_prediction_error": float(value_error.item()), "mcts/value_prediction_error": float(value_error.item()),
"mcts/value_rmse": value_rmse,
"mcts/v_target_abs_mean": target_abs_mean,
"mcts/v_target_std": target_std,
"mcts/policy_mcts_kl": float(np.mean(policy_kls)) if policy_kls else 0.0, "mcts/policy_mcts_kl": float(np.mean(policy_kls)) if policy_kls else 0.0,
} }