coolguyandClaude Opus 4.7
004b913a7b
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
70 lines
1.9 KiB
Python
70 lines
1.9 KiB
Python
from coolrl_lost_cities.games.classic import (
|
|
LostCitiesConfig,
|
|
build_bot,
|
|
make_policy_factory,
|
|
play_game_for_evaluation,
|
|
play_match,
|
|
)
|
|
from coolrl_lost_cities.games.classic.evaluation import MATCH_EVAL_RECORD_TYPE, main
|
|
|
|
|
|
def test_play_game_for_evaluation_finishes_small_match() -> None:
|
|
config = LostCitiesConfig(n_colors=3, n_ranks=5, n_handshakes=1, hand_size=5)
|
|
|
|
state, result = play_game_for_evaluation(
|
|
build_bot("random", seed=1),
|
|
build_bot("discard-only", seed=2),
|
|
config,
|
|
seed=3,
|
|
max_steps=200,
|
|
)
|
|
|
|
assert state.terminal is True
|
|
assert result.timed_out is False
|
|
assert result.steps > 0
|
|
assert result.score_diff0 == result.score0 - result.score1
|
|
|
|
|
|
def test_play_match_alternates_seats_and_reports_rates() -> None:
|
|
config = LostCitiesConfig(n_colors=3, n_ranks=5, n_handshakes=1, hand_size=5)
|
|
|
|
result = play_match(
|
|
make_policy_factory("random"),
|
|
make_policy_factory("discard-only"),
|
|
config,
|
|
games=4,
|
|
seed=10,
|
|
max_steps=200,
|
|
)
|
|
|
|
assert result.games == 4
|
|
assert result.wins0 + result.wins1 + result.draws == 4
|
|
assert result.avg_game_length > 0.0
|
|
assert result.games_per_second > 0.0
|
|
assert result.steps_per_second > 0.0
|
|
|
|
|
|
def test_evaluation_cli_smoke_json(capsys) -> None:
|
|
main(
|
|
[
|
|
"--bot0",
|
|
"random",
|
|
"--bot1",
|
|
"discard-only",
|
|
"--games",
|
|
"2",
|
|
"--seed",
|
|
"20",
|
|
"--json",
|
|
]
|
|
)
|
|
|
|
captured = capsys.readouterr()
|
|
assert f'"type": "{MATCH_EVAL_RECORD_TYPE}"' in captured.out
|
|
assert '"bots": {' in captured.out
|
|
assert '"settings": {' in captured.out
|
|
assert '"result": {' in captured.out
|
|
assert '"timing": {' in captured.out
|
|
assert '"games": 2' in captured.out
|
|
assert '"win_rate0"' in captured.out
|