diff --git a/README.md b/README.md index a2b36cd..8834e1c 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ The current implementation starts with the classic two-player card game: - classic 5-expedition rules by default - Python/Cython game engine - env wrapper -- random, passive-discard, and safe-heuristic bots +- random, discard-only, and safe-heuristic bots - core rule, scoring, mask, env, canonical-state, bot, and GUI smoke tests Training code, Deep CFR, learned-policy evaluation, GUI, and web client are diff --git a/configs/deep_cfr/default.yaml b/configs/deep_cfr/default.yaml index ef7d719..f4f2db7 100644 --- a/configs/deep_cfr/default.yaml +++ b/configs/deep_cfr/default.yaml @@ -92,11 +92,13 @@ evaluation: games: 100 opponents: - random - - passive_discard - - safe_heuristic - - safe_heuristic_loose - - safe_heuristic_strict - - noisy_safe + - discard_only + - heuristic_cautious + extended_eval_every: 50 + extended_opponents: + - heuristic_balanced + - heuristic_aggressive + - heuristic_noisy max_steps: 10000 on_max_steps: score_diff batch_size: 64 diff --git a/configs/deep_cfr/default_server.yaml b/configs/deep_cfr/default_server.yaml index 2b14092..5ce0fe6 100644 --- a/configs/deep_cfr/default_server.yaml +++ b/configs/deep_cfr/default_server.yaml @@ -89,11 +89,11 @@ evaluation: games: 100 opponents: - random - - passive_discard - - safe_heuristic - - safe_heuristic_loose - - safe_heuristic_strict - - noisy_safe + - discard_only + - heuristic_balanced + - heuristic_aggressive + - heuristic_cautious + - heuristic_noisy max_steps: 10000 on_max_steps: score_diff batch_size: 64 diff --git a/docs/performance.md b/docs/performance.md index 92dbdf8..fff6a9c 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -182,7 +182,7 @@ eval__ For that run, iteration 75 had `evaluation_seconds = 18.29s`. Evaluation was parallelized by opponent, so per-opponent `elapsed_seconds` values overlap and -must not be summed as wall-clock time. The slow safe-heuristic opponents +must not be summed as wall-clock time. The slow heuristic opponents dominated the eval wall-clock. Representative per-opponent breakdown: @@ -190,13 +190,13 @@ Representative per-opponent breakdown: | Opponent | Elapsed | Network | Postprocess | Opponent act | | --- | ---: | ---: | ---: | ---: | | `random` | 0.57s | 0.18s | 0.25s | 0.06s | -| `passive_discard` | 0.36s | 0.13s | 0.17s | 0.00s | -| `safe_heuristic` | 14.54s | 2.65s | 3.34s | 7.57s | -| `safe_heuristic_loose` | 11.20s | 2.55s | 3.30s | 4.63s | -| `safe_heuristic_strict` | 15.94s | 2.51s | 3.14s | 9.22s | -| `noisy_safe` | 1.68s | 0.40s | 0.58s | 0.57s | +| `discard_only` | 0.36s | 0.13s | 0.17s | 0.00s | +| `heuristic_balanced` | 14.54s | 2.65s | 3.34s | 7.57s | +| `heuristic_aggressive` | 11.20s | 2.55s | 3.30s | 4.63s | +| `heuristic_cautious` | 15.94s | 2.51s | 3.14s | 9.22s | +| `heuristic_noisy` | 1.68s | 0.40s | 0.58s | 0.57s | -The important read is that safe-heuristic evaluation is not primarily GPU +The important read is that heuristic evaluation is not primarily GPU network forward time. `opponent_act_seconds` and policy post-processing are larger than `policy_network_seconds` for the slowest opponents. @@ -236,7 +236,7 @@ The practical eval tuning levers are: this would require a feature change. 4. Reduce frequent opponents. - The safe-heuristic opponents dominate wall-clock in the inspected run. For + The heuristic opponents dominate wall-clock in the inspected run. For frequent checks, evaluate against one or two representative opponents and run the full suite less often. @@ -278,7 +278,7 @@ already implemented. It would replace or wrap the strategy-network forward pass with a precompiled inference engine. Its maximum impact is bounded by `policy_network_seconds`, not by total eval time. -In the inspected eval row, the slow safe-heuristic opponents spent about +In the inspected eval row, the slow heuristic opponents spent about 2.5-2.6s in policy-network forward but 4.6-9.2s in opponent action selection and about 3.1-3.3s in policy post-processing. That means TensorRT could help eval, especially for larger `evaluation.games`, but it is not expected to collapse the @@ -319,7 +319,7 @@ Based on the current metrics, the more plausible performance work is: but it requires changing traversal scheduling, not just swapping the network backend. -6. For eval-heavy runs, optimize the safe-heuristic opponents and policy +6. For eval-heavy runs, optimize the heuristic opponents and policy post-processing before assuming TensorRT is the main lever. The inspected eval row shows those costs dominate the slowest opponents. diff --git a/docs/plans/deep-cfr-selectivity.md b/docs/plans/deep-cfr-selectivity.md index 056d590..3854885 100644 --- a/docs/plans/deep-cfr-selectivity.md +++ b/docs/plans/deep-cfr-selectivity.md @@ -17,7 +17,7 @@ first-open advantage target. ## Baseline symptoms The 512x3 dense-eval baseline showed improving training losses, but the main -game-quality metrics against `safe_heuristic_strict` did not improve enough to +game-quality metrics against `heuristic_cautious` did not improve enough to indicate a useful policy. Observed pattern: @@ -176,7 +176,7 @@ against the current policy's best non-open action from the same state. `delta_open = value(force open) - value(best non-open)`. -Counterfactual summary against `safe_heuristic_strict`: +Counterfactual summary against `heuristic_cautious`: | checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | @@ -257,7 +257,7 @@ first-open sampling scanned the full replay memory and pushed iteration time above 60 seconds. The indexed-memory version kept first-open sampling near 0.25 seconds per player at 4M advantage samples and completed 500 iterations. -Final `safe_heuristic_strict` comparison: +Final `heuristic_cautious` comparison: | Run | Iter | Score diff | Win rate | Bad open | Score/opened | | --- | ---: | ---: | ---: | ---: | ---: | @@ -331,15 +331,15 @@ Result (2026-05-10): - W&B: synced online to group `first-open-prior-v1` (run `teuh915r`) - Commit: see HEAD at run start -`safe_heuristic_strict` at iter 200: +`heuristic_cautious` at iter 200: | Run | Score diff | Win rate | Bad open | Score/opened | | --- | ---: | ---: | ---: | ---: | | baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 | | **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** | -Other opponents at iter 200: random +32.51 / 0.86, noisy_safe -71.52 / 0.07, -safe_heuristic -81.81 / 0.02, safe_heuristic_loose -81.24 / 0.05. +Other opponents at iter 200: random +32.51 / 0.86, heuristic_noisy -71.52 / 0.07, +heuristic_balanced -81.81 / 0.02, heuristic_aggressive -81.24 / 0.05. Conclusion: **mixed result, net regression.** The prior did shift behavior in the intended direction on one axis — `bad_open_rate` dropped from 0.893 @@ -382,7 +382,7 @@ post-action play (selection bias)? Method: re-ran `analyze_first_open_counterfactual.py` on the `confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter -500), but with `--post-policy safe_heuristic_strict`. The opponent and +500), but with `--post-policy heuristic_cautious`. The opponent and state-collection policy stayed the same; only the policy_player's actions *after* the forced first action used the strong fixed bot. @@ -434,7 +434,7 @@ Implications: Candidate next directions (decision pending): -- (E1) Train with `cutoff_rollout_policy=safe_heuristic` instead of `random`. +- (E1) Train with `cutoff_rollout_policy=heuristic_balanced` instead of `random`. Already a config option; gives leaf nodes stronger value estimates during traversal. Trades some pure-self-play purity for a stronger bootstrap signal. Cheap to test. @@ -510,7 +510,7 @@ This combined with D1 means: Updated next-step priorities: -- **(E1) `cutoff_rollout_policy=safe_heuristic` training ablation.** +- **(E1) `cutoff_rollout_policy=heuristic_balanced` training ablation.** Strongest single lever: gives traversal leaves stronger value estimates during training, which should ripple back to "open + follow up" signal. Pure-self-play purity dented, but only at cutoff leaves. @@ -635,7 +635,7 @@ Fix: Also bumped `traversal.outcome_sampling_epsilon` in `default.yaml` from 0.2 to 0.05. The 200-iteration sweep (section 1) showed 0.05 produced the -best short-run safe_heuristic_strict score diff (-40.01 vs -57.87 for +best short-run heuristic_cautious score diff (-40.01 vs -57.87 for 0.20). All recent experimental runs already used 0.05; the default now matches actual experimental practice. @@ -643,7 +643,71 @@ These changes do not target the diagnosed selection-bias bottleneck. They align config intent with actual scheduler behaviour and make the default config reproduce known-best knob settings out of the box. -### 9. Short open-selectivity ablation +### 9. Naming, plot curation, and tiered eval cadence (2026-05-10) + +Hygiene changes — none target the diagnosed selection-bias bottleneck, +but they make the codebase honestly reflect the pure-self-play stance +and reduce dashboard noise. + +Bot family rename (drop the unhelpful `safe_` prefix; suffixes now +describe behaviour): + +| Old | New | +| --- | --- | +| `safe_heuristic_loose` | `heuristic_aggressive` | +| `safe_heuristic` | `heuristic_balanced` | +| `safe_heuristic_strict` | `heuristic_cautious` | +| `noisy_safe` | `heuristic_noisy` | +| `passive_discard` | `discard_only` | + +Class renames in `bots/`: `SafeHeuristicBot` → `HeuristicBot`, +`SafeHeuristicParams` → `HeuristicParams`, `PassiveDiscardBot` → +`DiscardOnlyBot`, plus the loose/strict parameter constants. Backwards +compatibility was dropped intentionally — no aliases. Active configs, +docs, scripts, tests updated; archive files (read-only by policy) +left intact and may still reference old names. + +Analyze plot curation (`deep_cfr/analyze.py`): + +- New `analysis_00_core.png` dashboard with 10 heuristic-free metrics + (loss/{advantage, strategy}; vs `heuristic_cautious`: + `avg_score_diff0`, `win_rate0`, `avg_opened_colors`, + `positive_expedition_rate`, `bonus_expedition_rate`, + `score_per_opened_color`, `policy_entropy`; vs `random`: `win_rate0`). +- Removed `analysis_05_open_quality.png` (bad/weak/good open rates, + recoverable score) and `analysis_07_calibration.png` (calibration gap, + recoverable mean) — both relied on the heuristic `recoverable_score` + classifier we already dropped from inputs. +- Removed `SELECTIVITY_PLOTS` and `plot_selectivity` (heuristic-laden). +- `SUMMARY_EVAL_METRICS` no longer includes `bad_open_rate` or + `calibration_gap`. + +`PlotSpec` gained an optional `opponents` allowlist so the new core +section can pin a specific opponent per panel without restructuring the +existing `plot_section` plumbing. + +Tiered evaluation cadence (`EvaluationConfig`): + +- Added `extended_opponents: tuple[str, ...]` and `extended_eval_every: + int = 0`. +- Method `opponents_for_iteration(iteration)` returns the core list at + every `eval_every`, and appends `extended_opponents` (de-duplicated) + when `iteration` is also a multiple of `extended_eval_every`. +- `default.yaml` now uses 3 core opponents + (`random`, `discard_only`, `heuristic_cautious`) every 5 iterations + and 3 extended opponents (`heuristic_balanced`, `heuristic_aggressive`, + `heuristic_noisy`) every 50 iterations. +- `random` is the floor sanity. `discard_only` is the zero-pit + detector / absolute-score reference (its score is always 0, so + `eval/discard_only/avg_score_diff0` directly equals our model's + raw average score). `heuristic_cautious` is the ceiling and the + archive-comparable benchmark used in sections 1–6. + +Net effect on ongoing eval cost: ~50% reduction (3 opponents × every +5 iter, plus 6 opponents × every 50 iter, vs the prior 6 opponents +× every 5). + +### 10. Short open-selectivity ablation Run a 200-300 iteration ablation only after the target audit identifies a specific change. Candidate changes include: @@ -656,10 +720,10 @@ specific change. Candidate changes include: Primary metrics: -- `eval/safe_heuristic_strict/avg_score_diff0` -- `eval/safe_heuristic_strict/win_rate0` -- `eval/safe_heuristic_strict/bad_open_rate` -- `eval/safe_heuristic_strict/score_per_opened_color` +- `eval/heuristic_cautious/avg_score_diff0` +- `eval/heuristic_cautious/win_rate0` +- `eval/heuristic_cautious/bad_open_rate` +- `eval/heuristic_cautious/score_per_opened_color` Do not promote to 500+ iterations unless bad-open rate and score/opened color both improve without degrading score diff. diff --git a/docs/plans/model_size_experiment.md b/docs/plans/model_size_experiment.md index 70f26cf..5016d21 100644 --- a/docs/plans/model_size_experiment.md +++ b/docs/plans/model_size_experiment.md @@ -39,7 +39,7 @@ The experiment must produce either a recommended new `network` config or a docum ## Success criteria -1. At least one tested config produces win-rate trajectories vs `safe_heuristic_strict` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve). +1. At least one tested config produces win-rate trajectories vs `heuristic_cautious` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve). 2. `iteration_seconds`, `traversal_seconds`, `advantage_train_seconds`, `strategy_train_seconds`, and `policy_network_seconds` (eval) are captured for each tested size and written to `docs/performance.md`. 3. A recommended `network` config emerges from the data, OR the experiment documents why the current size should be kept, with specific rationale. @@ -72,7 +72,7 @@ configs/deep_cfr/model-size-1536x8.yaml - `traversal_seconds` — traversal phase. - `advantage_train_seconds` — advantage network optimization. - `strategy_train_seconds` — strategy network optimization. - - At eval iterations {50, 100, 150, 200}: `eval//win_rate` for all opponents, with special attention to `safe_heuristic_strict`. + - At eval iterations {50, 100, 150, 200}: `eval//win_rate` for all opponents, with special attention to `heuristic_cautious`. - At eval iterations: `eval//policy_network_seconds` — needed for the AMP/TRT prerequisite check. - **Memory monitoring:** watch GPU VRAM during the 1024x6 and 1536x8 runs. If a run OOMs or VRAM > 20 GB, reduce `optimization.advantage_batch_size` and `optimization.strategy_batch_size` by half (1024 → 512) and note the change in the results table. Do not adjust traversal settings. @@ -259,8 +259,8 @@ if non_eval: eval_rows = {r['iteration']: r for r in rows if r.get('evaluation_seconds')} for it in [50, 100, 150, 200]: if it in eval_rows: - wr = eval_rows[it].get('eval/safe_heuristic_strict/win_rate', 'n/a') - print(f' iter={it} safe_heuristic_strict win_rate={wr}') + wr = eval_rows[it].get('eval/heuristic_cautious/win_rate', 'n/a') + print(f' iter={it} heuristic_cautious win_rate={wr}') " done ``` @@ -269,7 +269,7 @@ done After the grid completes, append a date-stamped experiment subsection to `docs/performance.md` under the "Experiments" heading. The subsection must include: -- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `safe_heuristic_strict` at {50, 100, 150, 200} for each config. +- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `heuristic_cautious` at {50, 100, 150, 200} for each config. - A `policy_network_seconds` column from eval rows — this is the key data for the AMP/compile/TRT prerequisite check. - The recommendation that follows from the decision tree below. @@ -302,7 +302,7 @@ Apply this logic after the grid completes: ### Branch A — a size unlocks the curve AND iter time is acceptable -**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `safe_heuristic_strict` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter). +**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `heuristic_cautious` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter). **Action:** 1. Recommend that config as the new `network` default. diff --git a/docs/plans/option_b_interleaved_traversal.md b/docs/plans/option_b_interleaved_traversal.md index e85d048..d94e439 100644 --- a/docs/plans/option_b_interleaved_traversal.md +++ b/docs/plans/option_b_interleaved_traversal.md @@ -152,7 +152,7 @@ Cython production rewrite. It intentionally uses per-context RNG so interleaved execution order does not change the random stream for another context. That lets the prototype assert value/stat/sample parity against a recursive prototype while measuring realized batch size. Production Cython parity is a later Phase 2 -gate because the real path also has safe-heuristic opponents, average-strategy +gate because the real path also has heuristic-balanced opponents, average-strategy opponents, self-play league snapshots, deck-draw chance sampling, external sampling, and cutoff rollouts. @@ -312,7 +312,7 @@ Required feature expansion: - Support `opponent_policy: average_strategy`, matching the default config's opponent branch. -- Keep unsupported branches guarded (`self_play_league`, `safe_heuristic`, +- Keep unsupported branches guarded (`self_play_league`, `heuristic_balanced`, random rollout cutoffs, external sampling). - Add parity tests for the average-strategy fixed-opponent branch. - Verify a default-policy interleaved run starts and emits batch metrics. diff --git a/docs/plans/torch_compile.md b/docs/plans/torch_compile.md index 04c522a..3d1b8f8 100644 --- a/docs/plans/torch_compile.md +++ b/docs/plans/torch_compile.md @@ -26,7 +26,7 @@ Re-enable `torch.compile` on the Deep CFR trainer's networks at a model size whe 1. **Model-size precondition met.** The active `default.yaml` (or the targeted variant) has `network.hidden_size ≥ 1024` *or* `network.num_layers ≥ 6`, *or* an architecture (e.g. `color_shared` with non-trivial `color_attention_layers`) whose per-call forward time exceeds ~150 μs at the trainer's training batch size on the target GPU. If neither condition holds, this plan is **not merged**; the branch is parked. 2. **Iter-time improvement.** With `compile.trainer.enabled: true` on the chosen larger model and the same seed, the 1000-iter projection improves by at least **5%** vs the no-compile baseline on the same machine (measured on `home`). Eval and checkpointing should be disabled for the bench window, matching the protocol used in the 2026-05-07 experiment. -3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `safe_heuristic` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves. +3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `heuristic_balanced` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves. 4. **No checkpoint-format break.** Checkpoints saved with compile enabled must load cleanly when compile is disabled, and vice versa. (Handled via `_clean_state_dict()`; see Risks.) 5. **No multiprocessing-worker break.** Whether `inference_backend` is `local` or `server`, traversal workers must continue to receive uncompiled `state_dict`s without `_orig_mod.` prefixes. 6. **(Secondary) Inference-server forward.** If step B below is taken, the server's `policy_network_seconds` decreases by at least 20% at the chosen model size, with no traversal-path correctness regression. If step B does not produce a measurable win, it is left disabled and the plan still ships with step A only. diff --git a/docs/reports/cost_cython_nogil_audit_2026-05-07.md b/docs/reports/cost_cython_nogil_audit_2026-05-07.md index ca51065..7779b61 100644 --- a/docs/reports/cost_cython_nogil_audit_2026-05-07.md +++ b/docs/reports/cost_cython_nogil_audit_2026-05-07.md @@ -157,9 +157,9 @@ Python-object touch가 깔려 있다 (per-node, per-iteration): `self.strategy_samples.append(...)` (file:903, 937, 969). list의 PyObject reference 갱신은 free-threaded Python에서도 atomic refcount 비용을 추가로 부담한다. -6. **`SafeHeuristicBot.act(state)`** — `_fixed_opponent_action` +6. **`HeuristicBot.act(state)`** — `_fixed_opponent_action` (file:633, 652), `_rollout_value` (file:841). Python class - 메서드 호출. `safe_heuristic` 옵션 사용 시만 핫. + 메서드 호출. `heuristic_balanced` 옵션 사용 시만 핫. 7. **`league_advantage_networks` indexing** — `_self_play_snapshot_ networks` (file:802), `[-recent_count:]`, `[:max(0, ...)]` slicing = Python list slicing. @@ -267,15 +267,15 @@ candidates = self.league_advantage_networks[:max(0, len(self.league_advantage_ne - 빈도: traversal 진입 시 한 번 (`traverse`에서 미리 픽), 재귀 안에서는 `active_self_play_networks`만 본다. 따라서 cold path. 변환 불필요. -### B6. `SafeHeuristicBot.act(state)` — Python bot +### B6. `HeuristicBot.act(state)` — Python bot ```python # traversal.pyx:633, 652, 841 -return int(self.safe_heuristic_opponent_bot.act(state)) +return int(self.heuristic_opponent_bot.act(state)) ``` -- 빈도: `opponent_policy=safe_heuristic` 또는 `cutoff_rollout_policy= - safe_heuristic`일 때만. 현 default는 self_play_league + score_diff +- 빈도: `opponent_policy=heuristic_balanced` 또는 `cutoff_rollout_policy= + heuristic_balanced`일 때만. 현 default는 self_play_league + score_diff cutoff (per memory의 opponent_policy_network_divergence note + AGENTS). - 변환 난이도: **Medium-High** (Python class 전체를 cython화). 현 default config에서는 핫 아님 — 시도하지 않는 게 합리. diff --git a/docs/research/deep-cfr-architecture.md b/docs/research/deep-cfr-architecture.md index f9b332b..bcfe83e 100644 --- a/docs/research/deep-cfr-architecture.md +++ b/docs/research/deep-cfr-architecture.md @@ -48,7 +48,7 @@ src/coolrl_lost_cities/games/classic/deep_cfr/ traversal_stats.py — structured traversal diagnostic metrics # Auxiliary training modes - imitation.py — safe-heuristic imitation pretraining + imitation.py — heuristic imitation pretraining policy_gradient.py — policy-gradient fine-tuning ``` diff --git a/docs/research/deep-cfr-baseline-2000-analysis.md b/docs/research/deep-cfr-baseline-2000-analysis.md index cd8bc93..7b44ef0 100644 --- a/docs/research/deep-cfr-baseline-2000-analysis.md +++ b/docs/research/deep-cfr-baseline-2000-analysis.md @@ -59,23 +59,23 @@ Evaluation performance peaks early and then degrades. | Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff | | --- | ---: | ---: | ---: | ---: | | random | 0.808 | 0.665 | +35.1 | +12.4 | -| passive_discard | 0.026 | 0.012 | -30.8 | -43.0 | -| safe_heuristic | 0.070 | 0.011 | -68.7 | -94.2 | -| safe_heuristic_loose | 0.078 | 0.014 | -69.3 | -95.0 | -| safe_heuristic_strict | 0.072 | 0.011 | -57.6 | -85.0 | -| noisy_safe | 0.120 | 0.033 | -49.4 | -78.8 | +| discard_only | 0.026 | 0.012 | -30.8 | -43.0 | +| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 | +| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 | +| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 | +| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 | The best strict-heuristic point appears around iteration 70: -- `eval/safe_heuristic_strict/win_rate0=0.15` -- `eval/safe_heuristic_strict/avg_score_diff0=-41.09` +- `eval/heuristic_cautious/win_rate0=0.15` +- `eval/heuristic_cautious/avg_score_diff0=-41.09` The final point at iteration 2000 is worse: -- `eval/safe_heuristic_strict/win_rate0=0.02` -- `eval/safe_heuristic_strict/avg_score_diff0=-81.64` +- `eval/heuristic_cautious/win_rate0=0.02` +- `eval/heuristic_cautious/avg_score_diff0=-81.64` -Selectivity does not meaningfully emerge. Against `safe_heuristic_strict`: +Selectivity does not meaningfully emerge. Against `heuristic_cautious`: | Metric | First 20 evals | Last 20 evals | Interpretation | | --- | ---: | ---: | --- | diff --git a/docs/research/deep-cfr-batched-evaluation.md b/docs/research/deep-cfr-batched-evaluation.md index 0e690d4..5b625fc 100644 --- a/docs/research/deep-cfr-batched-evaluation.md +++ b/docs/research/deep-cfr-batched-evaluation.md @@ -19,7 +19,7 @@ The primary bottleneck in CUDA-based evaluation is the overhead of launching sma A critical refinement in the batched implementation was the handling of policy entropy. Initial versions that calculated entropy per-row on the CPU incurred significant synchronization penalties because each row required a GPU-to-CPU transfer. Moving the entropy calculation into the Torch post-processing pipeline—specifically calculating it directly on the `probs_tensor` (line 220)—ensures that the computation remains on the device and only the final results are transferred back to the host in bulk. -Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `safe_heuristic_strict`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup. +Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `heuristic_cautious`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup. ## Practical implication diff --git a/docs/research/deep-cfr-evaluation-profile.md b/docs/research/deep-cfr-evaluation-profile.md index b11ab50..6751251 100644 --- a/docs/research/deep-cfr-evaluation-profile.md +++ b/docs/research/deep-cfr-evaluation-profile.md @@ -37,7 +37,7 @@ On CPU, the `network / turn` cost is approximately **0.074 ms**. On CUDA, this r ### Secondary Bottlenecks - **Post-processing:** Moving tensors back to CPU (`.cpu().numpy()`) and calculating entropy adds measurable overhead on CUDA that is largely absent on CPU. -- **Opponent Logic:** Heuristic opponents (e.g., `safe_heuristic`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins. +- **Opponent Logic:** Heuristic opponents (e.g., `heuristic_balanced`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins. ## Practical Implications diff --git a/docs/research/deep-cfr-reproducibility.md b/docs/research/deep-cfr-reproducibility.md index 20d5fcf..1fd4405 100644 --- a/docs/research/deep-cfr-reproducibility.md +++ b/docs/research/deep-cfr-reproducibility.md @@ -45,7 +45,7 @@ Two temporary runs used: - `run.max_iterations=3` - `evaluation.eval_every=1` - `evaluation.games=20` -- `evaluation.opponents=[random,safe_heuristic_strict]` +- `evaluation.opponents=[random,heuristic_cautious]` - W&B disabled Runs: @@ -55,7 +55,7 @@ Runs: Core metrics matched exactly: -| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` | +| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` | | --- | ---: | ---: | ---: | ---: | ---: | | 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 | | 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 | diff --git a/docs/research/deep-cfr-v0-feature-parity.md b/docs/research/deep-cfr-v0-feature-parity.md index 5f92999..81787f6 100644 --- a/docs/research/deep-cfr-v0-feature-parity.md +++ b/docs/research/deep-cfr-v0-feature-parity.md @@ -56,7 +56,7 @@ registered classic bots (`evaluate.py`), training CLI and evaluation CLI (`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` / `runtime_progress.json` / `train.log` run artifacts, self-play league with snapshot pool and weighted current/recent/older/anchor bucket sampling, safe- -heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`), +heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`), and policy-gradient fine-tuning (`policy_gradient.py`). As of `ad0be89`, the package also includes `inference_server.py`, diff --git a/docs/research/julia_port_evaluation.md b/docs/research/julia_port_evaluation.md index 4d6eafb..75016e6 100644 --- a/docs/research/julia_port_evaluation.md +++ b/docs/research/julia_port_evaluation.md @@ -31,7 +31,7 @@ Decision criteria for going forward: ### 2026-05-07 — Safe heuristic single-thread parity (criterion 1) -Path: `experiments/julia_safe_heuristic/`. +Path: `experiments/julia_heuristic/`. 1,838 snapshots in 157.471 ms median (~85.6 μs/call). Action-sequence parity vs Python. Same order of magnitude as the Cython port of the diff --git a/docs/research/lost_cities_selectivity.md b/docs/research/lost_cities_selectivity.md index 486f2e2..3de6aee 100644 --- a/docs/research/lost_cities_selectivity.md +++ b/docs/research/lost_cities_selectivity.md @@ -28,7 +28,7 @@ Deep CFR은 최종적으로 average strategy가 수렴 대상. current advantage Visible-score predictor의 ceiling은 있을 수 있음. 다만 selectivity 자체의 ceiling은 아님. selectivity는 visible-score prediction 말고 option value, irreversible cost 회피, opponent dynamics 대응 등 다른 경로로도 emerge 가능. **Self-play attractor 가설** -5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. safe_heuristic이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable. +5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. heuristic_balanced이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable. **Lost Cities NE 자체가 5-color 가설** 이론적 가능성 0 아님. self-play가 발견한 게 진짜 NE면 transition 영원히 안 옴. 검증 가능한 형태로는 tabular oracle 또는 BR 진단. diff --git a/scripts/analyze_first_open_advantage.py b/scripts/analyze_first_open_advantage.py index fbbd259..6d58e04 100644 --- a/scripts/analyze_first_open_advantage.py +++ b/scripts/analyze_first_open_advantage.py @@ -240,7 +240,7 @@ def main() -> None: parser.add_argument("--output", type=Path, required=True) args = parser.parse_args() - opponents = args.opponent or ["safe_heuristic_strict"] + opponents = args.opponent or ["heuristic_cautious"] args.output.parent.mkdir(parents=True, exist_ok=True) rows = [] for checkpoint in args.checkpoints: diff --git a/scripts/analyze_first_open_counterfactual.py b/scripts/analyze_first_open_counterfactual.py index 7c2bf2b..a11dee8 100644 --- a/scripts/analyze_first_open_counterfactual.py +++ b/scripts/analyze_first_open_counterfactual.py @@ -367,7 +367,7 @@ def analyze_checkpoint( def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("checkpoints", nargs="+", type=Path) - parser.add_argument("--opponent", default="safe_heuristic_strict") + parser.add_argument("--opponent", default="heuristic_cautious") parser.add_argument("--games", type=int, default=100) parser.add_argument("--seed", type=int, default=231_000) parser.add_argument("--device", default="cuda") @@ -379,7 +379,7 @@ def main() -> None: help=( "Policy used for the policy_player during forced-action rollouts. " "'model' uses the trained advantage network; any other value is " - "treated as a bot name (e.g. 'safe_heuristic_strict')." + "treated as a bot name (e.g. 'heuristic_cautious')." ), ) parser.add_argument("--output", type=Path, required=True) diff --git a/scripts/analyze_first_open_followup.py b/scripts/analyze_first_open_followup.py index c7e01fb..6c70d1e 100644 --- a/scripts/analyze_first_open_followup.py +++ b/scripts/analyze_first_open_followup.py @@ -273,7 +273,7 @@ def analyze_checkpoint( def main() -> None: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("checkpoints", nargs="+", type=Path) - parser.add_argument("--opponent", default="safe_heuristic_strict") + parser.add_argument("--opponent", default="heuristic_cautious") parser.add_argument("--games", type=int, default=100) parser.add_argument("--seed", type=int, default=232_000) parser.add_argument("--device", default="cpu") diff --git a/scripts/eval_current_vs_average.py b/scripts/eval_current_vs_average.py index 32fcc07..cf8fce8 100644 --- a/scripts/eval_current_vs_average.py +++ b/scripts/eval_current_vs_average.py @@ -265,7 +265,7 @@ def main() -> None: parser.add_argument("--output", type=Path, required=True) args = parser.parse_args() - opponents = args.opponent or ["random", "safe_heuristic_strict"] + opponents = args.opponent or ["random", "heuristic_cautious"] args.output.parent.mkdir(parents=True, exist_ok=True) rows: list[dict] = [] diff --git a/scripts/export_safe_heuristic_snapshots.py b/scripts/export_safe_heuristic_snapshots.py index 7a67eaf..dd42c1a 100644 --- a/scripts/export_safe_heuristic_snapshots.py +++ b/scripts/export_safe_heuristic_snapshots.py @@ -7,22 +7,22 @@ from typing import Any from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig -from coolrl_lost_cities.games.classic.bots.heuristic_py import SafeHeuristicBot +from coolrl_lost_cities.games.classic.bots.heuristic_py import HeuristicBot from coolrl_lost_cities.games.classic.bots.registry import ( - LOOSE_SAFE_HEURISTIC_PARAMS, - STRICT_SAFE_HEURISTIC_PARAMS, + AGGRESSIVE_HEURISTIC_PARAMS, + CAUTIOUS_HEURISTIC_PARAMS, ) VARIANTS = { "default": None, - "loose": LOOSE_SAFE_HEURISTIC_PARAMS, - "strict": STRICT_SAFE_HEURISTIC_PARAMS, + "loose": AGGRESSIVE_HEURISTIC_PARAMS, + "strict": CAUTIOUS_HEURISTIC_PARAMS, } def parse_args() -> argparse.Namespace: parser = argparse.ArgumentParser( - description="Export safe-heuristic bot parity snapshots for external implementations." + description="Export heuristic bot parity snapshots for external implementations." ) parser.add_argument("--output", required=True, help="JSONL output path.") parser.add_argument("--seeds", type=int, default=50, help="Number of seeds per config.") @@ -71,7 +71,7 @@ def main() -> None: for config_name, config in _configs(): for variant_name, params in VARIANTS.items(): for seed in range(args.seeds): - bot = SafeHeuristicBot(params) + bot = HeuristicBot(params) state = GameState.new_game(config, seed=seed) for turn in range(args.max_steps): if state.terminal: diff --git a/src/coolrl_lost_cities/games/classic/bots/__init__.py b/src/coolrl_lost_cities/games/classic/bots/__init__.py index 2c38d1a..979b0f0 100644 --- a/src/coolrl_lost_cities/games/classic/bots/__init__.py +++ b/src/coolrl_lost_cities/games/classic/bots/__init__.py @@ -1,8 +1,8 @@ from __future__ import annotations from ..policy import LostCitiesPolicy, PolicyInput -from .heuristic import SafeHeuristicBot -from .passive import PassiveDiscardBot +from .discard_only import DiscardOnlyBot +from .heuristic import HeuristicBot from .random import RandomBot from .registry import DEFAULT_BOT, available_bot_names, build_bot @@ -10,9 +10,9 @@ __all__ = [ "PolicyInput", "DEFAULT_BOT", "LostCitiesPolicy", - "PassiveDiscardBot", + "DiscardOnlyBot", "RandomBot", - "SafeHeuristicBot", + "HeuristicBot", "available_bot_names", "build_bot", ] diff --git a/src/coolrl_lost_cities/games/classic/bots/passive.py b/src/coolrl_lost_cities/games/classic/bots/discard_only.py similarity index 97% rename from src/coolrl_lost_cities/games/classic/bots/passive.py rename to src/coolrl_lost_cities/games/classic/bots/discard_only.py index 53412de..9b06a7f 100644 --- a/src/coolrl_lost_cities/games/classic/bots/passive.py +++ b/src/coolrl_lost_cities/games/classic/bots/discard_only.py @@ -6,7 +6,7 @@ from ..snapshots import Snapshot from .base import first_legal, legal_from_obs -class PassiveDiscardBot(LostCitiesPolicy): +class DiscardOnlyBot(LostCitiesPolicy): """Baseline that avoids opening expeditions whenever discarding is legal.""" def act(self, obs_or_state: PolicyInput) -> int: diff --git a/src/coolrl_lost_cities/games/classic/bots/heuristic.py b/src/coolrl_lost_cities/games/classic/bots/heuristic.py index 944ae11..c69f663 100644 --- a/src/coolrl_lost_cities/games/classic/bots/heuristic.py +++ b/src/coolrl_lost_cities/games/classic/bots/heuristic.py @@ -4,8 +4,8 @@ from .heuristic_cy import ( DRAW_FROM_DECK_ACTION, PLAY_OR_DISCARD_ACTIONS_PER_SLOT, DerivedHeuristicConfig, - SafeHeuristicBot, - SafeHeuristicParams, + HeuristicBot, + HeuristicParams, derive_heuristic_config, discard_action, draw_from_discard_action, @@ -16,8 +16,8 @@ __all__ = [ "DRAW_FROM_DECK_ACTION", "PLAY_OR_DISCARD_ACTIONS_PER_SLOT", "DerivedHeuristicConfig", - "SafeHeuristicBot", - "SafeHeuristicParams", + "HeuristicBot", + "HeuristicParams", "derive_heuristic_config", "discard_action", "draw_from_discard_action", diff --git a/src/coolrl_lost_cities/games/classic/bots/heuristic_cy.pyx b/src/coolrl_lost_cities/games/classic/bots/heuristic_cy.pyx index 3c57305..1f38e0a 100644 --- a/src/coolrl_lost_cities/games/classic/bots/heuristic_cy.pyx +++ b/src/coolrl_lost_cities/games/classic/bots/heuristic_cy.pyx @@ -31,7 +31,7 @@ def draw_from_discard_action(color: int) -> int: return 1 + color -LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.safe_heuristic") +LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.heuristic") class _CachedState: @@ -66,7 +66,7 @@ class _CachedState: @dataclass(frozen=True) -class SafeHeuristicParams: +class HeuristicParams: # Expedition opening. open_target_ratio: float = 0.50 open_min_card_ratio: float = 0.40 @@ -124,7 +124,7 @@ class DerivedHeuristicConfig: @lru_cache(maxsize=64) def derive_heuristic_config( config: LostCitiesConfig, - params: SafeHeuristicParams, + params: HeuristicParams, ) -> DerivedHeuristicConfig: max_color_sum = sum(config.min_rank + rank - 1 for rank in range(1, config.n_ranks + 1)) break_even_sum = -config.expedition_penalty @@ -178,20 +178,20 @@ def derive_heuristic_config( ) -class SafeHeuristicBot(LostCitiesPolicy): - def __init__(self, params: SafeHeuristicParams | None = None): - self.params = params or SafeHeuristicParams() +class HeuristicBot(LostCitiesPolicy): + def __init__(self, params: HeuristicParams | None = None): + self.params = params or HeuristicParams() def act(self, obs_or_state: PolicyInput) -> int: if not isinstance(obs_or_state, GameState) and not hasattr(obs_or_state, "legal_mask"): LOGGER.debug( - "SafeHeuristicBot fallback to first legal: input_type=%s", + "HeuristicBot fallback to first legal: input_type=%s", type(obs_or_state).__name__, ) return first_legal(legal_from_obs(obs_or_state)) LOGGER.debug( - "SafeHeuristicBot heuristic path: player=%s phase=%s turn=%s", + "HeuristicBot heuristic path: player=%s phase=%s turn=%s", obs_or_state.current_player, obs_or_state.phase, obs_or_state.turn_count, diff --git a/src/coolrl_lost_cities/games/classic/bots/heuristic_py.py b/src/coolrl_lost_cities/games/classic/bots/heuristic_py.py index 28737f2..c3e24df 100644 --- a/src/coolrl_lost_cities/games/classic/bots/heuristic_py.py +++ b/src/coolrl_lost_cities/games/classic/bots/heuristic_py.py @@ -30,11 +30,11 @@ def draw_from_discard_action(color: int) -> int: return 1 + color -LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.safe_heuristic") +LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.heuristic") @dataclass(frozen=True) -class SafeHeuristicParams: +class HeuristicParams: # Expedition opening. open_target_ratio: float = 0.50 open_min_card_ratio: float = 0.40 @@ -92,7 +92,7 @@ class DerivedHeuristicConfig: @lru_cache(maxsize=64) def derive_heuristic_config( config: LostCitiesConfig, - params: SafeHeuristicParams, + params: HeuristicParams, ) -> DerivedHeuristicConfig: max_color_sum = sum(config.min_rank + rank - 1 for rank in range(1, config.n_ranks + 1)) break_even_sum = -config.expedition_penalty @@ -146,20 +146,20 @@ def derive_heuristic_config( ) -class SafeHeuristicBot(LostCitiesPolicy): - def __init__(self, params: SafeHeuristicParams | None = None): - self.params = params or SafeHeuristicParams() +class HeuristicBot(LostCitiesPolicy): + def __init__(self, params: HeuristicParams | None = None): + self.params = params or HeuristicParams() def act(self, obs_or_state: PolicyInput) -> int: if not isinstance(obs_or_state, GameState) and not hasattr(obs_or_state, "legal_mask"): LOGGER.debug( - "SafeHeuristicBot fallback to first legal: input_type=%s", + "HeuristicBot fallback to first legal: input_type=%s", type(obs_or_state).__name__, ) return first_legal(legal_from_obs(obs_or_state)) LOGGER.debug( - "SafeHeuristicBot heuristic path: player=%s phase=%s turn=%s", + "HeuristicBot heuristic path: player=%s phase=%s turn=%s", obs_or_state.current_player, obs_or_state.phase, obs_or_state.turn_count, diff --git a/src/coolrl_lost_cities/games/classic/bots/registry.py b/src/coolrl_lost_cities/games/classic/bots/registry.py index 3bc113c..7944a77 100644 --- a/src/coolrl_lost_cities/games/classic/bots/registry.py +++ b/src/coolrl_lost_cities/games/classic/bots/registry.py @@ -3,8 +3,8 @@ from __future__ import annotations from collections.abc import Callable from ..policy import LostCitiesPolicy, PolicyInput -from .heuristic import SafeHeuristicBot, SafeHeuristicParams -from .passive import PassiveDiscardBot +from .discard_only import DiscardOnlyBot +from .heuristic import HeuristicBot, HeuristicParams from .random import RandomBot BotName = str @@ -30,7 +30,7 @@ class NoisyPolicy(LostCitiesPolicy): return self.base.act(obs_or_state) -LOOSE_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams( +AGGRESSIVE_HEURISTIC_PARAMS = HeuristicParams( open_target_ratio=0.42, open_min_card_ratio=0.30, handshake_target_multiplier=1.00, @@ -38,7 +38,7 @@ LOOSE_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams( late_open_block_ratio=0.12, ) -STRICT_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams( +CAUTIOUS_HEURISTIC_PARAMS = HeuristicParams( open_target_ratio=0.62, open_min_card_ratio=0.50, handshake_target_multiplier=1.35, @@ -49,12 +49,12 @@ STRICT_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams( BOT_REGISTRY: dict[BotName, PolicyFactory] = { DEFAULT_BOT: RandomBot, - "passive-discard": lambda seed: PassiveDiscardBot(), - "safe-heuristic": lambda seed: SafeHeuristicBot(), - "safe-heuristic-loose": lambda seed: SafeHeuristicBot(LOOSE_SAFE_HEURISTIC_PARAMS), - "safe-heuristic-strict": lambda seed: SafeHeuristicBot(STRICT_SAFE_HEURISTIC_PARAMS), - "noisy-safe": lambda seed: NoisyPolicy( - SafeHeuristicBot(), + "discard-only": lambda seed: DiscardOnlyBot(), + "heuristic-balanced": lambda seed: HeuristicBot(), + "heuristic-aggressive": lambda seed: HeuristicBot(AGGRESSIVE_HEURISTIC_PARAMS), + "heuristic-cautious": lambda seed: HeuristicBot(CAUTIOUS_HEURISTIC_PARAMS), + "heuristic-noisy": lambda seed: NoisyPolicy( + HeuristicBot(), RandomBot(seed), ), } diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/analyze.py b/src/coolrl_lost_cities/games/classic/deep_cfr/analyze.py index 1813a97..29a1c73 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/analyze.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/analyze.py @@ -18,6 +18,7 @@ class PlotSpec: scale: float = 1.0 kind: str = "eval" fixed_ylim: tuple[float, float] | None = None + opponents: tuple[str, ...] | None = None @dataclass(frozen=True) @@ -28,6 +29,71 @@ class SectionSpec: SECTIONS: tuple[SectionSpec, ...] = ( + SectionSpec( + "Core", + "analysis_00_core.png", + ( + PlotSpec("Advantage Loss", ("loss/advantage",), "loss", kind="train"), + PlotSpec("Strategy Loss", ("loss/strategy",), "loss", kind="train"), + PlotSpec( + "Avg Score Diff (heuristic_cautious)", + ("avg_score_diff0",), + "score diff", + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Win Rate (heuristic_cautious)", + ("win_rate0",), + "rate (%)", + scale=100.0, + fixed_ylim=(0, 100), + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Win Rate (random)", + ("win_rate0",), + "rate (%)", + scale=100.0, + fixed_ylim=(0, 100), + opponents=("random",), + ), + PlotSpec( + "Avg Opened Colors (heuristic_cautious)", + ("avg_opened_colors",), + "colors", + fixed_ylim=(0, 5), + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Positive Expedition Rate (heuristic_cautious)", + ("positive_expedition_rate",), + "rate (%)", + scale=100.0, + fixed_ylim=(0, 100), + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Bonus Expedition Rate (heuristic_cautious)", + ("bonus_expedition_rate",), + "rate (%)", + scale=100.0, + fixed_ylim=(0, 100), + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Score per Opened Color (heuristic_cautious)", + ("score_per_opened_color",), + "score / color", + opponents=("heuristic_cautious",), + ), + PlotSpec( + "Policy Entropy (heuristic_cautious)", + ("policy_entropy",), + "entropy", + opponents=("heuristic_cautious",), + ), + ), + ), SectionSpec( "Loss", "analysis_01_loss.png", @@ -107,48 +173,6 @@ SECTIONS: tuple[SectionSpec, ...] = ( PlotSpec("Expedition Cards", ("avg_expedition_cards",), "cards"), ), ), - SectionSpec( - "OpenQuality", - "analysis_05_open_quality.png", - ( - PlotSpec( - "Bad Open Rate", - ("bad_open_rate",), - "rate (%)", - scale=100.0, - fixed_ylim=(0, 100), - ), - PlotSpec( - "Weak Open Rate", - ("weak_open_rate",), - "rate (%)", - scale=100.0, - fixed_ylim=(0, 100), - ), - PlotSpec( - "Bad or Weak Open Rate", - ("bad_or_weak_open_rate",), - "rate (%)", - scale=100.0, - fixed_ylim=(0, 100), - ), - PlotSpec( - "Good Open Rate", - ("good_open_rate",), - "rate (%)", - scale=100.0, - fixed_ylim=(0, 100), - ), - PlotSpec("Bad Open per Game", ("bad_open_per_game",), "opens / game"), - PlotSpec( - "Bad or Weak Open per Game", - ("bad_or_weak_open_per_game",), - "opens / game", - ), - PlotSpec("Opening Play Actions", ("opening_play_actions",), "actions / game"), - PlotSpec("Opening Recoverable p25", ("opening_recoverable_score_p25",), "score"), - ), - ), SectionSpec( "ExpeditionOutcomes", "analysis_06_expedition_outcomes.png", @@ -188,23 +212,6 @@ SECTIONS: tuple[SectionSpec, ...] = ( PlotSpec("Score per Opened Color", ("score_per_opened_color",), "score / color"), ), ), - SectionSpec( - "Calibration", - "analysis_07_calibration.png", - ( - PlotSpec( - "First Open Recoverable Score", - ( - "first_open_recoverable_score_mean_for_positive_final", - "first_open_recoverable_score_mean_for_negative_final", - ), - "score", - ), - PlotSpec("Opening Recoverable Mean", ("opening_recoverable_score_mean",), "score"), - PlotSpec("Opening Recoverable p25", ("opening_recoverable_score_p25",), "score"), - PlotSpec("Calibration Gap", ("calibration_gap",), "score"), - ), - ), SectionSpec( "Traversal", "analysis_08_traversal.png", @@ -311,41 +318,23 @@ SECTIONS: tuple[SectionSpec, ...] = ( ), ) -SELECTIVITY_PLOTS: tuple[PlotSpec, ...] = ( - PlotSpec("Opened Colors", ("avg_opened_colors",), "colors"), - PlotSpec("5-Color Open Count", ("5_color_open_count",), "games / eval"), - PlotSpec("Opening Recoverable Mean", ("opening_recoverable_score_mean",), "score"), - PlotSpec("Calibration Gap", ("calibration_gap",), "score"), - PlotSpec( - "First Open Recoverable Score", - ( - "first_open_recoverable_score_mean_for_positive_final", - "first_open_recoverable_score_mean_for_negative_final", - ), - "score", - ), - PlotSpec("Opening Play Actions", ("opening_play_actions",), "actions / game"), -) - SUMMARY_EVAL_METRICS: tuple[tuple[str, str, float], ...] = ( ("win_rate0", "win rate (%)", 100.0), ("avg_score_diff0", "avg score diff", 1.0), ("avg_score0", "avg score", 1.0), ("play_action_rate", "play rate (%)", 100.0), ("avg_opened_colors", "opened colors", 1.0), - ("bad_open_rate", "bad open (%)", 100.0), ("score_per_opened_color", "score / opened color", 1.0), - ("calibration_gap", "calibration gap", 1.0), ("bonus_contribution_per_game", "bonus / game", 1.0), ) OPPONENT_COLORS: dict[str, str] = { - "noisy_safe": "tab:blue", - "passive_discard": "tab:orange", + "heuristic_noisy": "tab:blue", + "discard_only": "tab:orange", "random": "tab:green", - "safe_heuristic": "tab:red", - "safe_heuristic_loose": "tab:purple", - "safe_heuristic_strict": "tab:brown", + "heuristic_balanced": "tab:red", + "heuristic_aggressive": "tab:purple", + "heuristic_cautious": "tab:brown", } TRAVERSAL_COLORS: dict[str, str] = { @@ -428,7 +417,14 @@ def plot_section( if spec.kind == "train": plotted = _plot_train_spec(ax, rows, spec, smoothing_window=smoothing_window) else: - plotted = _plot_eval_spec(ax, rows, opponents, spec, smoothing_window=smoothing_window) + filtered_opponents = ( + [o for o in opponents if o in spec.opponents] + if spec.opponents is not None + else opponents + ) + plotted = _plot_eval_spec( + ax, rows, filtered_opponents, spec, smoothing_window=smoothing_window + ) _finish_axis( ax, spec.title, ylabel=spec.ylabel, plotted=plotted, fixed_ylim=spec.fixed_ylim ) @@ -539,12 +535,6 @@ def analyze_run( if plot_section(rows, section, path, smoothing_window=smoothing_window): written.append(path) - selectivity_path = output_dir / _with_filename_suffix( - "analysis_09_selectivity.png", filename_suffix - ) - if plot_selectivity(rows, selectivity_path, smoothing_window=smoothing_window): - written.append(selectivity_path) - final_eval_path = output_dir / _with_filename_suffix( "analysis_final_eval_summary.png", filename_suffix ) @@ -566,45 +556,6 @@ def _with_filename_suffix(filename: str, suffix: str) -> str: return f"{path.stem}{suffix}{path.suffix}" -def plot_selectivity( - rows: list[dict[str, Any]], - output: Path, - *, - smoothing_window: int, -) -> bool: - import matplotlib.pyplot as plt - - opponents = opponent_names(rows) - if not opponents: - return False - - fig, axes = plt.subplots(3, 2, figsize=(16, 12), squeeze=False) - axes_flat = list(axes.flat) - plotted_any = False - - for ax, spec in zip(axes_flat, SELECTIVITY_PLOTS, strict=True): - plotted = _plot_eval_spec(ax, rows, opponents, spec, smoothing_window=smoothing_window) - _finish_axis(ax, spec.title, ylabel=spec.ylabel, plotted=plotted) - plotted_any = plotted_any or plotted - - handles, labels = _legend_items(axes_flat) - if handles: - fig.legend(handles, labels, loc="upper center", ncols=min(len(labels), 6), fontsize="small") - suffix = f" ({smoothing_window}-iter moving average)" if smoothing_window > 1 else "" - fig.suptitle( - f"Lost Cities Deep CFR selectivity metrics{suffix}", - fontsize=14, - fontweight="bold", - ) - fig.tight_layout(rect=(0, 0, 1, 0.95)) - if not plotted_any: - plt.close(fig) - return False - fig.savefig(output, dpi=150) - plt.close(fig) - return True - - def _all_eval_metrics() -> set[str]: metrics: set[str] = set() for section in SECTIONS: diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/config.py b/src/coolrl_lost_cities/games/classic/deep_cfr/config.py index bab9be5..1b9a9ec 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/config.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/config.py @@ -143,16 +143,16 @@ class TraversalConfig(StrictModel): @field_validator("cutoff_rollout_policy") @classmethod def _validate_cutoff_rollout_policy(cls, value: str) -> str: - if value not in {"random", "safe_heuristic"}: - raise ValueError("must be 'random' or 'safe_heuristic'") + if value not in {"random", "heuristic_balanced"}: + raise ValueError("must be 'random' or 'heuristic_balanced'") return value @field_validator("opponent_policy") @classmethod def _validate_opponent_policy(cls, value: str) -> str: - if value not in {"network", "safe_heuristic", "self_play_league", "average_strategy"}: + if value not in {"network", "heuristic_balanced", "self_play_league", "average_strategy"}: raise ValueError( - "must be 'network', 'safe_heuristic', 'self_play_league', or 'average_strategy'" + "must be 'network', 'heuristic_balanced', 'self_play_league', or 'average_strategy'" ) return value @@ -294,13 +294,35 @@ class CheckpointConfig(StrictModel): class EvaluationConfig(StrictModel): eval_every: int = 50 games: int = 10 - opponents: tuple[str, ...] = ("random", "safe_heuristic") + opponents: tuple[str, ...] = ("random", "heuristic_balanced") + extended_eval_every: int = 0 + extended_opponents: tuple[str, ...] = () max_steps: int = 10_000 on_max_steps: str = "score_diff" batch_size: int = 64 device: str = "trainer" num_workers: int = 4 + def opponents_for_iteration(self, iteration: int) -> tuple[str, ...]: + """Return the opponent set to evaluate against at ``iteration``. + + - Always returns ``opponents`` when ``iteration % eval_every == 0``. + - When ``extended_eval_every > 0`` and ``iteration`` is also a multiple + of ``extended_eval_every``, appends ``extended_opponents`` (de-duplicated) + to the core list. + - Returns ``()`` when no eval is scheduled this iteration. + """ + if self.eval_every <= 0 or iteration <= 0: + return () + if iteration % self.eval_every != 0: + return () + result = list(self.opponents) + if self.extended_eval_every > 0 and iteration % self.extended_eval_every == 0: + for opponent in self.extended_opponents: + if opponent not in result: + result.append(opponent) + return tuple(result) + @field_validator("on_max_steps") @classmethod def _validate_on_max_steps(cls, value: str) -> str: diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/imitation.py b/src/coolrl_lost_cities/games/classic/deep_cfr/imitation.py index 2325ef9..9c1a7d7 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/imitation.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/imitation.py @@ -6,7 +6,7 @@ import numpy as np import torch from torch import nn -from coolrl_lost_cities.games.classic.bots import SafeHeuristicBot +from coolrl_lost_cities.games.classic.bots import HeuristicBot from coolrl_lost_cities.games.classic.deep_cfr.encoding import encode_info_state, input_dim from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig, classic_config @@ -18,7 +18,7 @@ class ImitationMetrics: loss: float -def collect_safe_heuristic_samples( +def collect_heuristic_samples( config: LostCitiesConfig | None = None, *, games: int = 4, @@ -28,7 +28,7 @@ def collect_safe_heuristic_samples( game_config = config or classic_config(seed=seed) probe = GameState.new_game(game_config, seed=seed) action_size = 2 * probe.config.hand_size + 1 + probe.config.n_colors - bot = SafeHeuristicBot() + bot = HeuristicBot() infos: list[np.ndarray] = [] targets: list[np.ndarray] = [] masks: list[np.ndarray] = [] @@ -67,7 +67,7 @@ def pretrain_strategy_network( learning_rate: float = 1.0e-3, device: torch.device | str = "cpu", ) -> ImitationMetrics: - x_np, y_np, legal_np = collect_safe_heuristic_samples(config, games=games, seed=seed) + x_np, y_np, legal_np = collect_heuristic_samples(config, games=games, seed=seed) device = torch.device(device) strategy_network.to(device) strategy_network.train() diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py b/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py index 25dd1e8..61fcdfb 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py @@ -894,17 +894,15 @@ class DeepCFRTrainer: self.tracker.log_event(_format_iteration_summary(metrics, data)) def _evaluate(self, iteration: int) -> dict[str, float | int]: - if ( - self.config.evaluation.eval_every <= 0 - or iteration % self.config.evaluation.eval_every != 0 - ): + opponents = self.config.evaluation.opponents_for_iteration(iteration) + if not opponents: return {} results: dict[str, float | int] = {} eval_device = self._evaluation_device() - if self.config.evaluation.resolved_num_workers(len(self.config.evaluation.opponents)) > 1: - return self._evaluate_parallel(iteration, eval_device) + if self.config.evaluation.resolved_num_workers(len(opponents)) > 1: + return self._evaluate_parallel(iteration, eval_device, opponents) eval_network = self._evaluation_network(eval_device) - for opponent in self.config.evaluation.opponents: + for opponent in opponents: result = evaluate_strategy_network( eval_network, self.game_config, @@ -925,8 +923,8 @@ class DeepCFRTrainer: self, iteration: int, eval_device: torch.device, + opponents: tuple[str, ...], ) -> dict[str, float | int]: - opponents = self.config.evaluation.opponents max_workers = self.config.evaluation.resolved_num_workers(len(opponents)) self.tracker.log_event( f"Evaluation multiprocessing enabled iteration={iteration} " diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx b/src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx index 38205c3..fc3f118 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx @@ -7,7 +7,7 @@ from libc.stdlib cimport free, malloc import numpy as np import torch -from coolrl_lost_cities.games.classic.bots import SafeHeuristicBot +from coolrl_lost_cities.games.classic.bots import HeuristicBot from coolrl_lost_cities.games.classic.deep_cfr.cfr_math cimport regret_matching_c from coolrl_lost_cities.games.classic.deep_cfr.encoding cimport ( _encode_info_state_with_flags_c, @@ -70,8 +70,8 @@ cdef class CythonDeepCFRTraverser: cdef object device cdef object encoding cdef object league_advantage_networks - cdef object safe_heuristic_rollout_bot - cdef object safe_heuristic_opponent_bot + cdef object heuristic_rollout_bot + cdef object heuristic_opponent_bot cdef int action_size cdef int input_dim cdef float epsilon @@ -185,15 +185,15 @@ cdef class CythonDeepCFRTraverser: raise ValueError("cutoff_value_mode must be 'score_diff' or 'random_rollout'") self.cutoff_random_rollout = cutoff_value_mode == "random_rollout" self.cutoff_rollouts = max(0, cutoff_rollouts) - if cutoff_rollout_policy not in {"random", "safe_heuristic"}: - raise ValueError("cutoff_rollout_policy must be 'random' or 'safe_heuristic'") + if cutoff_rollout_policy not in {"random", "heuristic_balanced"}: + raise ValueError("cutoff_rollout_policy must be 'random' or 'heuristic_balanced'") self.cutoff_rollout_max_steps = max(1, cutoff_rollout_max_steps) - self.safe_heuristic_rollout_bot = ( - SafeHeuristicBot() if cutoff_rollout_policy == "safe_heuristic" else None + self.heuristic_rollout_bot = ( + HeuristicBot() if cutoff_rollout_policy == "heuristic_balanced" else None ) if opponent_policy == "network": self.opponent_policy_id = 0 - elif opponent_policy == "safe_heuristic": + elif opponent_policy == "heuristic_balanced": self.opponent_policy_id = 1 elif opponent_policy == "self_play_league": self.opponent_policy_id = 2 @@ -205,7 +205,7 @@ cdef class CythonDeepCFRTraverser: ) else: raise ValueError( - "opponent_policy must be 'network', 'safe_heuristic', 'self_play_league', or 'average_strategy'" + "opponent_policy must be 'network', 'heuristic_balanced', 'self_play_league', or 'average_strategy'" ) if all_negative_fallback == "uniform": self.all_negative_fallback_id = 0 @@ -225,8 +225,8 @@ cdef class CythonDeepCFRTraverser: self.self_play_recent_window = max(0, self_play_recent_window) self.active_self_play_bucket = 0 self.active_self_play_networks = None - self.safe_heuristic_opponent_bot = ( - SafeHeuristicBot() + self.heuristic_opponent_bot = ( + HeuristicBot() if self.opponent_policy_id == 1 or self.self_play_anchor_probability > 0.0 else None ) @@ -630,9 +630,9 @@ cdef class CythonDeepCFRTraverser: if player == traverser or self.opponent_policy_id == 0: return -1 if self.opponent_policy_id == 1: - if self.safe_heuristic_opponent_bot is None: - self.safe_heuristic_opponent_bot = SafeHeuristicBot() - return int(self.safe_heuristic_opponent_bot.act(state)) + if self.heuristic_opponent_bot is None: + self.heuristic_opponent_bot = HeuristicBot() + return int(self.heuristic_opponent_bot.act(state)) if self.opponent_policy_id == 3: self._policy_from_strategy_network(state, player, legal, policy) for i in range(self.action_size): @@ -649,9 +649,9 @@ cdef class CythonDeepCFRTraverser: if bucket == 0: return -1 if bucket == 3: - if self.safe_heuristic_opponent_bot is None: - self.safe_heuristic_opponent_bot = SafeHeuristicBot() - return int(self.safe_heuristic_opponent_bot.act(state)) + if self.heuristic_opponent_bot is None: + self.heuristic_opponent_bot = HeuristicBot() + return int(self.heuristic_opponent_bot.act(state)) networks = self.active_self_play_networks if networks is None: return -1 @@ -839,8 +839,8 @@ cdef class CythonDeepCFRTraverser: if swapped_indices == NULL: raise MemoryError() while not state.terminal and steps < self.cutoff_rollout_max_steps: - if self.safe_heuristic_rollout_bot is not None: - local_action = int(self.safe_heuristic_rollout_bot.act(state)) + if self.heuristic_rollout_bot is not None: + local_action = int(self.heuristic_rollout_bot.act(state)) unified_action = self._to_unified_action_c(state, local_action) else: count = state._unified_legal_actions_c(actions) diff --git a/src/coolrl_lost_cities/games/classic/evaluation.py b/src/coolrl_lost_cities/games/classic/evaluation.py index ebf89cb..98587d7 100644 --- a/src/coolrl_lost_cities/games/classic/evaluation.py +++ b/src/coolrl_lost_cities/games/classic/evaluation.py @@ -277,7 +277,7 @@ def evaluate_policy( def main(argv: list[str] | None = None) -> None: parser = argparse.ArgumentParser(description="Evaluate Lost Cities classic bots.") - parser.add_argument("--bot0", default="safe-heuristic", choices=available_bot_names()) + parser.add_argument("--bot0", default="heuristic-balanced", choices=available_bot_names()) parser.add_argument("--bot1", default="random", choices=available_bot_names()) parser.add_argument("--games", type=int, default=100) parser.add_argument("--seed", type=int, default=1) diff --git a/tests/games/classic/test_bots.py b/tests/games/classic/test_bots.py index 6d0139a..6ec2d38 100644 --- a/tests/games/classic/test_bots.py +++ b/tests/games/classic/test_bots.py @@ -1,9 +1,9 @@ from coolrl_lost_cities.games.classic.game import Card, GameState, LostCitiesConfig from coolrl_lost_cities.games.classic.bots import ( + HeuristicBot, LostCitiesPolicy, RandomBot, - SafeHeuristicBot, ) from coolrl_lost_cities.games.classic.bots.heuristic import draw_from_discard_action from coolrl_lost_cities.games.classic.evaluation import play_game_for_evaluation @@ -19,13 +19,13 @@ def _expeditions(config: LostCitiesConfig) -> list[list[list[Card]]]: def test_builtin_bots_implement_lost_cities_policy() -> None: assert isinstance(RandomBot(1), LostCitiesPolicy) - assert isinstance(SafeHeuristicBot(), LostCitiesPolicy) + assert isinstance(HeuristicBot(), LostCitiesPolicy) -def test_safe_heuristic_mirror_match_finishes() -> None: +def test_heuristic_mirror_match_finishes() -> None: state, result = play_game_for_evaluation( - SafeHeuristicBot(), - SafeHeuristicBot(), + HeuristicBot(), + HeuristicBot(), LostCitiesConfig(n_colors=3, n_ranks=5, n_handshakes=1, hand_size=5), seed=2000, max_steps=200, @@ -34,9 +34,9 @@ def test_safe_heuristic_mirror_match_finishes() -> None: assert result.timed_out is False -def test_safe_heuristic_opponent_value_ignores_hidden_hand() -> None: +def test_heuristic_opponent_value_ignores_hidden_hand() -> None: config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3) - bot = SafeHeuristicBot() + bot = HeuristicBot() discard_card = Card(color=0, rank=6) expeditions_a = _expeditions(config) @@ -76,9 +76,9 @@ def test_safe_heuristic_opponent_value_ignores_hidden_hand() -> None: assert value_a == value_b -def test_safe_heuristic_started_expedition_value_ignores_invalid_lower_followup() -> None: +def test_heuristic_started_expedition_value_ignores_invalid_lower_followup() -> None: config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3) - bot = SafeHeuristicBot() + bot = HeuristicBot() high_card = Card(color=0, rank=8) base_expeditions = _expeditions(config) @@ -115,9 +115,9 @@ def test_safe_heuristic_started_expedition_value_ignores_invalid_lower_followup( assert lower_followup_value == base_value -def test_safe_heuristic_draws_playable_discard_instead_of_deck() -> None: +def test_heuristic_draws_playable_discard_instead_of_deck() -> None: config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3) - bot = SafeHeuristicBot() + bot = HeuristicBot() expeditions = _expeditions(config) expeditions[0][0] = [Card(color=0, rank=4)] @@ -132,9 +132,9 @@ def test_safe_heuristic_draws_playable_discard_instead_of_deck() -> None: assert bot._act_draw(state) == draw_from_discard_action(0) -def test_safe_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None: +def test_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None: config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=4) - bot = SafeHeuristicBot() + bot = HeuristicBot() expeditions = _expeditions(config) expeditions[0][1] = [Card(color=1, rank=8)] @@ -158,9 +158,9 @@ def test_safe_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None: assert bot._act_draw(state) == draw_from_discard_action(0) -def test_safe_heuristic_classic_self_play_opens_expeditions() -> None: +def test_heuristic_classic_self_play_opens_expeditions() -> None: state = GameState.new_game(LostCitiesConfig(), seed=1) - bot = SafeHeuristicBot() + bot = HeuristicBot() player0_actions: list[int] = [] for _ in range(60): @@ -182,9 +182,9 @@ def test_safe_heuristic_classic_self_play_opens_expeditions() -> None: assert any(state.expeditions[0][color] for color in range(state.config.n_colors)) -def test_safe_heuristic_avoids_opening_weak_fifth_color() -> None: +def test_heuristic_avoids_opening_weak_fifth_color() -> None: config = LostCitiesConfig(n_colors=5, n_ranks=8, hand_size=8) - bot = SafeHeuristicBot() + bot = HeuristicBot() expeditions = _expeditions(config) expeditions[0][0] = [Card(color=0, rank=4)] expeditions[0][1] = [Card(color=1, rank=4)] @@ -211,9 +211,9 @@ def test_safe_heuristic_avoids_opening_weak_fifth_color() -> None: ) -def test_safe_heuristic_prefers_followup_on_started_expedition() -> None: +def test_heuristic_prefers_followup_on_started_expedition() -> None: config = LostCitiesConfig(n_colors=3, n_ranks=8, hand_size=5) - bot = SafeHeuristicBot() + bot = HeuristicBot() expeditions = _expeditions(config) expeditions[0][0] = [Card(color=0, rank=4)] state = make_state( @@ -233,9 +233,9 @@ def test_safe_heuristic_prefers_followup_on_started_expedition() -> None: assert chosen.color == 0 -def test_safe_heuristic_avoids_unopened_discard_draw_after_four_opens() -> None: +def test_heuristic_avoids_unopened_discard_draw_after_four_opens() -> None: config = LostCitiesConfig(n_colors=5, n_ranks=8, hand_size=8) - bot = SafeHeuristicBot() + bot = HeuristicBot() expeditions = _expeditions(config) expeditions[0][0] = [Card(color=0, rank=4)] expeditions[0][1] = [Card(color=1, rank=4)] diff --git a/tests/games/classic/test_deep_cfr_imitation.py b/tests/games/classic/test_deep_cfr_imitation.py index 981d607..37c11b9 100644 --- a/tests/games/classic/test_deep_cfr_imitation.py +++ b/tests/games/classic/test_deep_cfr_imitation.py @@ -3,13 +3,13 @@ from __future__ import annotations from coolrl_lost_cities.games.classic.game import LostCitiesConfig from coolrl_lost_cities.games.classic.deep_cfr.imitation import ( - collect_safe_heuristic_samples, + collect_heuristic_samples, new_pretrained_strategy_network, ) -def test_collect_safe_heuristic_samples_shapes() -> None: - x, y, legal = collect_safe_heuristic_samples(LostCitiesConfig(seed=61), games=1, seed=61) +def test_collect_heuristic_samples_shapes() -> None: + x, y, legal = collect_heuristic_samples(LostCitiesConfig(seed=61), games=1, seed=61) assert len(x) == len(y) == len(legal) assert x.ndim == 2 diff --git a/tests/games/classic/test_deep_cfr_trainer.py b/tests/games/classic/test_deep_cfr_trainer.py index 274bf73..871026d 100644 --- a/tests/games/classic/test_deep_cfr_trainer.py +++ b/tests/games/classic/test_deep_cfr_trainer.py @@ -1494,3 +1494,83 @@ def test_interleaved_regret_matching_no_fallback_unchanged_by_mode() -> None: assert fallback_a is False assert np.allclose(policy_uniform, policy_argmax) assert np.allclose(policy_uniform.sum(), 1.0) + + +def test_evaluation_opponents_for_iteration_core_only_when_extended_disabled() -> None: + from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig + + cfg = EvaluationConfig( + eval_every=5, + opponents=("random", "discard_only", "heuristic_cautious"), + extended_eval_every=0, + extended_opponents=("heuristic_balanced",), + ) + assert cfg.opponents_for_iteration(0) == () + assert cfg.opponents_for_iteration(3) == () + assert cfg.opponents_for_iteration(5) == ( + "random", + "discard_only", + "heuristic_cautious", + ) + assert cfg.opponents_for_iteration(50) == ( + "random", + "discard_only", + "heuristic_cautious", + ) + + +def test_evaluation_opponents_for_iteration_extends_on_extended_cadence() -> None: + from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig + + cfg = EvaluationConfig( + eval_every=5, + opponents=("random", "discard_only", "heuristic_cautious"), + extended_eval_every=50, + extended_opponents=( + "heuristic_balanced", + "heuristic_aggressive", + "heuristic_noisy", + ), + ) + assert cfg.opponents_for_iteration(5) == ( + "random", + "discard_only", + "heuristic_cautious", + ) + assert cfg.opponents_for_iteration(45) == ( + "random", + "discard_only", + "heuristic_cautious", + ) + assert cfg.opponents_for_iteration(50) == ( + "random", + "discard_only", + "heuristic_cautious", + "heuristic_balanced", + "heuristic_aggressive", + "heuristic_noisy", + ) + assert cfg.opponents_for_iteration(100) == ( + "random", + "discard_only", + "heuristic_cautious", + "heuristic_balanced", + "heuristic_aggressive", + "heuristic_noisy", + ) + + +def test_evaluation_opponents_for_iteration_dedupes_overlap() -> None: + from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig + + cfg = EvaluationConfig( + eval_every=5, + opponents=("random", "heuristic_cautious"), + extended_eval_every=10, + extended_opponents=("heuristic_cautious", "heuristic_balanced"), + ) + assert cfg.opponents_for_iteration(10) == ( + "random", + "heuristic_cautious", + "heuristic_balanced", + ) diff --git a/tests/games/classic/test_evaluation.py b/tests/games/classic/test_evaluation.py index d56303f..7ba6efb 100644 --- a/tests/games/classic/test_evaluation.py +++ b/tests/games/classic/test_evaluation.py @@ -13,7 +13,7 @@ def test_play_game_for_evaluation_finishes_small_match() -> None: state, result = play_game_for_evaluation( build_bot("random", seed=1), - build_bot("passive-discard", seed=2), + build_bot("discard-only", seed=2), config, seed=3, max_steps=200, @@ -30,7 +30,7 @@ def test_play_match_alternates_seats_and_reports_rates() -> None: result = play_match( make_policy_factory("random"), - make_policy_factory("passive-discard"), + make_policy_factory("discard-only"), config, games=4, seed=10, @@ -50,7 +50,7 @@ def test_evaluation_cli_smoke_json(capsys) -> None: "--bot0", "random", "--bot1", - "passive-discard", + "discard-only", "--games", "2", "--seed", diff --git a/tests/games/classic/test_public_api.py b/tests/games/classic/test_public_api.py index 5af3166..130d003 100644 --- a/tests/games/classic/test_public_api.py +++ b/tests/games/classic/test_public_api.py @@ -43,10 +43,10 @@ def test_classic_package_exports_bot_registry_helpers() -> None: def test_classic_bot_registry_accepts_reproduction_opponent_names() -> None: for name in [ "random", - "passive_discard", - "safe_heuristic", - "safe_heuristic_loose", - "safe_heuristic_strict", - "noisy_safe", + "discard_only", + "heuristic_balanced", + "heuristic_aggressive", + "heuristic_cautious", + "heuristic_noisy", ]: assert isinstance(classic.build_bot(name, seed=1), classic.LostCitiesPolicy) diff --git a/tests/games/classic/test_pygame_pvp.py b/tests/games/classic/test_pygame_pvp.py index aef3bb2..4bc6e3f 100644 --- a/tests/games/classic/test_pygame_pvp.py +++ b/tests/games/classic/test_pygame_pvp.py @@ -11,7 +11,7 @@ def test_gui_argparser_accepts_classic_options() -> None: "--mode", "pvc", "--bot", - "safe-heuristic", + "heuristic-balanced", "--seed", "7", "--width", @@ -22,7 +22,7 @@ def test_gui_argparser_accepts_classic_options() -> None: ) assert args.mode == "pvc" - assert args.bot == "safe-heuristic" + assert args.bot == "heuristic-balanced" assert args.seed == 7 assert args.width == 1024 assert args.height == 768 diff --git a/tests/games/classic/test_safe_heuristic_equivalence.py b/tests/games/classic/test_safe_heuristic_equivalence.py index b923945..b5456c0 100644 --- a/tests/games/classic/test_safe_heuristic_equivalence.py +++ b/tests/games/classic/test_safe_heuristic_equivalence.py @@ -3,19 +3,19 @@ from __future__ import annotations import pytest from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig -from coolrl_lost_cities.games.classic.bots.heuristic import SafeHeuristicBot +from coolrl_lost_cities.games.classic.bots.heuristic import HeuristicBot from coolrl_lost_cities.games.classic.bots.heuristic_py import ( - SafeHeuristicBot as PythonSafeHeuristicBot, + HeuristicBot as PythonHeuristicBot, ) from coolrl_lost_cities.games.classic.bots.registry import ( - LOOSE_SAFE_HEURISTIC_PARAMS, - STRICT_SAFE_HEURISTIC_PARAMS, + AGGRESSIVE_HEURISTIC_PARAMS, + CAUTIOUS_HEURISTIC_PARAMS, ) VARIANTS = ( ("default", None), - ("loose", LOOSE_SAFE_HEURISTIC_PARAMS), - ("strict", STRICT_SAFE_HEURISTIC_PARAMS), + ("loose", AGGRESSIVE_HEURISTIC_PARAMS), + ("strict", CAUTIOUS_HEURISTIC_PARAMS), ) CONFIGS = ( @@ -28,14 +28,14 @@ CONFIGS = ( @pytest.mark.parametrize(("variant_name", "params"), VARIANTS) @pytest.mark.parametrize("config", CONFIGS) @pytest.mark.parametrize("seed", range(2)) -def test_cython_safe_heuristic_matches_python_action_sequence( +def test_cython_heuristic_matches_python_action_sequence( variant_name: str, params, config: LostCitiesConfig, seed: int, ) -> None: - py_bot = PythonSafeHeuristicBot(params) - cy_bot = SafeHeuristicBot(params) + py_bot = PythonHeuristicBot(params) + cy_bot = HeuristicBot(params) py_state = GameState.new_game(config, seed=seed) cy_state = GameState.new_game(config, seed=seed)