Honour all_negative_fallback in interleaved scheduler; sync default.yaml
The interleaved traversal scheduler's _regret_matching was hard-coded to spread fallback policy uniformly across legal actions, regardless of the configured regret_matching.all_negative_fallback. default.yaml has shipped with all_negative_fallback: argmax_tiebreak since618d5f8based on the 20-iter audit + 1000-iter empirical comparison in docs/archive/deep-cfr-regret-fallback-audit-2026-05-07.md, but the default scheduler was switched to interleaved in09bbe7c, after which the configured fallback mode silently no-op'd. _regret_matching now takes fallback_mode and concentrates policy mass on the lowest-index tied action when "argmax_tiebreak". Tiebreak is deterministic; the Cython recursive traverser randomises ties using its per-traverser RNG, which the batched policy does not have. Behaviour matches the spirit of the recursive path (concentrate on best, do not dilute uniformly). Plumbed through BatchedPolicy, InterleavedTraversalConfig, run_interleaved_traversal_batch, trainer.py, workers.py, and the analyze_first_open_targets.py caller. Two unit tests added. Also bumps default.yaml outcome_sampling_epsilon 0.2 -> 0.05. The 200-iter sweep in docs/plans/deep-cfr-selectivity.md section 1 showed 0.05 produced the best short-run safe_heuristic_strict score diff (-40.01 vs -57.87 for 0.20). Recent experiments already used 0.05; the default now matches actual experimental practice. Neither change targets the diagnosed selection-bias bottleneck. They align config intent with scheduler behaviour and make the default config reproduce known-best knob settings out of the box. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1455,3 +1455,42 @@ def test_first_open_prior_zero_alpha_is_noop() -> None:
|
||||
target = np.zeros(legal_mask.shape[0], dtype=np.float32)
|
||||
_apply_first_open_prior(target, state, player, legal_mask, sampled_action=0, alpha=0.0)
|
||||
assert np.all(target == 0.0)
|
||||
|
||||
|
||||
def test_interleaved_regret_matching_argmax_tiebreak_concentrates_on_best() -> None:
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.interleaved_traversal import _regret_matching
|
||||
|
||||
advantages = np.array([-1.0, -0.5, -0.5, -2.0, -0.5], dtype=np.float32)
|
||||
legal_mask = np.array([True, True, True, True, True])
|
||||
|
||||
uniform_policy, fallback_u, _, _ = _regret_matching(
|
||||
advantages, legal_mask, epsilon=1.0e-8, fallback_mode="uniform"
|
||||
)
|
||||
argmax_policy, fallback_a, tie_size, _ = _regret_matching(
|
||||
advantages, legal_mask, epsilon=1.0e-8, fallback_mode="argmax_tiebreak"
|
||||
)
|
||||
|
||||
assert fallback_u is True
|
||||
assert fallback_a is True
|
||||
assert tie_size == 3
|
||||
assert np.allclose(uniform_policy, np.full(5, 0.2, dtype=np.float32))
|
||||
expected_argmax = np.zeros(5, dtype=np.float32)
|
||||
expected_argmax[1] = 1.0
|
||||
assert np.allclose(argmax_policy, expected_argmax)
|
||||
|
||||
|
||||
def test_interleaved_regret_matching_no_fallback_unchanged_by_mode() -> None:
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.interleaved_traversal import _regret_matching
|
||||
|
||||
advantages = np.array([1.0, 3.0, 0.0, 2.0], dtype=np.float32)
|
||||
legal_mask = np.array([True, True, True, True])
|
||||
policy_uniform, fallback_u, _, _ = _regret_matching(
|
||||
advantages, legal_mask, epsilon=1.0e-8, fallback_mode="uniform"
|
||||
)
|
||||
policy_argmax, fallback_a, _, _ = _regret_matching(
|
||||
advantages, legal_mask, epsilon=1.0e-8, fallback_mode="argmax_tiebreak"
|
||||
)
|
||||
assert fallback_u is False
|
||||
assert fallback_a is False
|
||||
assert np.allclose(policy_uniform, policy_argmax)
|
||||
assert np.allclose(policy_uniform.sum(), 1.0)
|
||||
|
||||
Reference in New Issue
Block a user