Rename bot family, curate analyze plots, tier evaluation cadence
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -7,7 +7,7 @@ The current implementation starts with the classic two-player card game:
|
||||
- classic 5-expedition rules by default
|
||||
- Python/Cython game engine
|
||||
- env wrapper
|
||||
- random, passive-discard, and safe-heuristic bots
|
||||
- random, discard-only, and safe-heuristic bots
|
||||
- core rule, scoring, mask, env, canonical-state, bot, and GUI smoke tests
|
||||
|
||||
Training code, Deep CFR, learned-policy evaluation, GUI, and web client are
|
||||
|
||||
@@ -92,11 +92,13 @@ evaluation:
|
||||
games: 100
|
||||
opponents:
|
||||
- random
|
||||
- passive_discard
|
||||
- safe_heuristic
|
||||
- safe_heuristic_loose
|
||||
- safe_heuristic_strict
|
||||
- noisy_safe
|
||||
- discard_only
|
||||
- heuristic_cautious
|
||||
extended_eval_every: 50
|
||||
extended_opponents:
|
||||
- heuristic_balanced
|
||||
- heuristic_aggressive
|
||||
- heuristic_noisy
|
||||
max_steps: 10000
|
||||
on_max_steps: score_diff
|
||||
batch_size: 64
|
||||
|
||||
@@ -89,11 +89,11 @@ evaluation:
|
||||
games: 100
|
||||
opponents:
|
||||
- random
|
||||
- passive_discard
|
||||
- safe_heuristic
|
||||
- safe_heuristic_loose
|
||||
- safe_heuristic_strict
|
||||
- noisy_safe
|
||||
- discard_only
|
||||
- heuristic_balanced
|
||||
- heuristic_aggressive
|
||||
- heuristic_cautious
|
||||
- heuristic_noisy
|
||||
max_steps: 10000
|
||||
on_max_steps: score_diff
|
||||
batch_size: 64
|
||||
|
||||
+10
-10
@@ -182,7 +182,7 @@ eval_<opponent>_<metric>
|
||||
|
||||
For that run, iteration 75 had `evaluation_seconds = 18.29s`. Evaluation was
|
||||
parallelized by opponent, so per-opponent `elapsed_seconds` values overlap and
|
||||
must not be summed as wall-clock time. The slow safe-heuristic opponents
|
||||
must not be summed as wall-clock time. The slow heuristic opponents
|
||||
dominated the eval wall-clock.
|
||||
|
||||
Representative per-opponent breakdown:
|
||||
@@ -190,13 +190,13 @@ Representative per-opponent breakdown:
|
||||
| Opponent | Elapsed | Network | Postprocess | Opponent act |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| `random` | 0.57s | 0.18s | 0.25s | 0.06s |
|
||||
| `passive_discard` | 0.36s | 0.13s | 0.17s | 0.00s |
|
||||
| `safe_heuristic` | 14.54s | 2.65s | 3.34s | 7.57s |
|
||||
| `safe_heuristic_loose` | 11.20s | 2.55s | 3.30s | 4.63s |
|
||||
| `safe_heuristic_strict` | 15.94s | 2.51s | 3.14s | 9.22s |
|
||||
| `noisy_safe` | 1.68s | 0.40s | 0.58s | 0.57s |
|
||||
| `discard_only` | 0.36s | 0.13s | 0.17s | 0.00s |
|
||||
| `heuristic_balanced` | 14.54s | 2.65s | 3.34s | 7.57s |
|
||||
| `heuristic_aggressive` | 11.20s | 2.55s | 3.30s | 4.63s |
|
||||
| `heuristic_cautious` | 15.94s | 2.51s | 3.14s | 9.22s |
|
||||
| `heuristic_noisy` | 1.68s | 0.40s | 0.58s | 0.57s |
|
||||
|
||||
The important read is that safe-heuristic evaluation is not primarily GPU
|
||||
The important read is that heuristic evaluation is not primarily GPU
|
||||
network forward time. `opponent_act_seconds` and policy post-processing are
|
||||
larger than `policy_network_seconds` for the slowest opponents.
|
||||
|
||||
@@ -236,7 +236,7 @@ The practical eval tuning levers are:
|
||||
this would require a feature change.
|
||||
|
||||
4. Reduce frequent opponents.
|
||||
The safe-heuristic opponents dominate wall-clock in the inspected run. For
|
||||
The heuristic opponents dominate wall-clock in the inspected run. For
|
||||
frequent checks, evaluate against one or two representative opponents and run
|
||||
the full suite less often.
|
||||
|
||||
@@ -278,7 +278,7 @@ already implemented. It would replace or wrap the strategy-network forward pass
|
||||
with a precompiled inference engine. Its maximum impact is bounded by
|
||||
`policy_network_seconds`, not by total eval time.
|
||||
|
||||
In the inspected eval row, the slow safe-heuristic opponents spent about
|
||||
In the inspected eval row, the slow heuristic opponents spent about
|
||||
2.5-2.6s in policy-network forward but 4.6-9.2s in opponent action selection and
|
||||
about 3.1-3.3s in policy post-processing. That means TensorRT could help eval,
|
||||
especially for larger `evaluation.games`, but it is not expected to collapse the
|
||||
@@ -319,7 +319,7 @@ Based on the current metrics, the more plausible performance work is:
|
||||
but it requires changing traversal scheduling, not just swapping the network
|
||||
backend.
|
||||
|
||||
6. For eval-heavy runs, optimize the safe-heuristic opponents and policy
|
||||
6. For eval-heavy runs, optimize the heuristic opponents and policy
|
||||
post-processing before assuming TensorRT is the main lever.
|
||||
The inspected eval row shows those costs dominate the slowest opponents.
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ first-open advantage target.
|
||||
## Baseline symptoms
|
||||
|
||||
The 512x3 dense-eval baseline showed improving training losses, but the main
|
||||
game-quality metrics against `safe_heuristic_strict` did not improve enough to
|
||||
game-quality metrics against `heuristic_cautious` did not improve enough to
|
||||
indicate a useful policy.
|
||||
|
||||
Observed pattern:
|
||||
@@ -176,7 +176,7 @@ against the current policy's best non-open action from the same state.
|
||||
|
||||
`delta_open = value(force open) - value(best non-open)`.
|
||||
|
||||
Counterfactual summary against `safe_heuristic_strict`:
|
||||
Counterfactual summary against `heuristic_cautious`:
|
||||
|
||||
| checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -257,7 +257,7 @@ first-open sampling scanned the full replay memory and pushed iteration time
|
||||
above 60 seconds. The indexed-memory version kept first-open sampling near
|
||||
0.25 seconds per player at 4M advantage samples and completed 500 iterations.
|
||||
|
||||
Final `safe_heuristic_strict` comparison:
|
||||
Final `heuristic_cautious` comparison:
|
||||
|
||||
| Run | Iter | Score diff | Win rate | Bad open | Score/opened |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
@@ -331,15 +331,15 @@ Result (2026-05-10):
|
||||
- W&B: synced online to group `first-open-prior-v1` (run `teuh915r`)
|
||||
- Commit: see HEAD at run start
|
||||
|
||||
`safe_heuristic_strict` at iter 200:
|
||||
`heuristic_cautious` at iter 200:
|
||||
|
||||
| Run | Score diff | Win rate | Bad open | Score/opened |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 |
|
||||
| **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** |
|
||||
|
||||
Other opponents at iter 200: random +32.51 / 0.86, noisy_safe -71.52 / 0.07,
|
||||
safe_heuristic -81.81 / 0.02, safe_heuristic_loose -81.24 / 0.05.
|
||||
Other opponents at iter 200: random +32.51 / 0.86, heuristic_noisy -71.52 / 0.07,
|
||||
heuristic_balanced -81.81 / 0.02, heuristic_aggressive -81.24 / 0.05.
|
||||
|
||||
Conclusion: **mixed result, net regression.** The prior did shift behavior
|
||||
in the intended direction on one axis — `bad_open_rate` dropped from 0.893
|
||||
@@ -382,7 +382,7 @@ post-action play (selection bias)?
|
||||
|
||||
Method: re-ran `analyze_first_open_counterfactual.py` on the
|
||||
`confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter
|
||||
500), but with `--post-policy safe_heuristic_strict`. The opponent and
|
||||
500), but with `--post-policy heuristic_cautious`. The opponent and
|
||||
state-collection policy stayed the same; only the policy_player's actions
|
||||
*after* the forced first action used the strong fixed bot.
|
||||
|
||||
@@ -434,7 +434,7 @@ Implications:
|
||||
|
||||
Candidate next directions (decision pending):
|
||||
|
||||
- (E1) Train with `cutoff_rollout_policy=safe_heuristic` instead of `random`.
|
||||
- (E1) Train with `cutoff_rollout_policy=heuristic_balanced` instead of `random`.
|
||||
Already a config option; gives leaf nodes stronger value estimates
|
||||
during traversal. Trades some pure-self-play purity for a stronger
|
||||
bootstrap signal. Cheap to test.
|
||||
@@ -510,7 +510,7 @@ This combined with D1 means:
|
||||
|
||||
Updated next-step priorities:
|
||||
|
||||
- **(E1) `cutoff_rollout_policy=safe_heuristic` training ablation.**
|
||||
- **(E1) `cutoff_rollout_policy=heuristic_balanced` training ablation.**
|
||||
Strongest single lever: gives traversal leaves stronger value estimates
|
||||
during training, which should ripple back to "open + follow up" signal.
|
||||
Pure-self-play purity dented, but only at cutoff leaves.
|
||||
@@ -635,7 +635,7 @@ Fix:
|
||||
|
||||
Also bumped `traversal.outcome_sampling_epsilon` in `default.yaml` from
|
||||
0.2 to 0.05. The 200-iteration sweep (section 1) showed 0.05 produced the
|
||||
best short-run safe_heuristic_strict score diff (-40.01 vs -57.87 for
|
||||
best short-run heuristic_cautious score diff (-40.01 vs -57.87 for
|
||||
0.20). All recent experimental runs already used 0.05; the default now
|
||||
matches actual experimental practice.
|
||||
|
||||
@@ -643,7 +643,71 @@ These changes do not target the diagnosed selection-bias bottleneck. They
|
||||
align config intent with actual scheduler behaviour and make the default
|
||||
config reproduce known-best knob settings out of the box.
|
||||
|
||||
### 9. Short open-selectivity ablation
|
||||
### 9. Naming, plot curation, and tiered eval cadence (2026-05-10)
|
||||
|
||||
Hygiene changes — none target the diagnosed selection-bias bottleneck,
|
||||
but they make the codebase honestly reflect the pure-self-play stance
|
||||
and reduce dashboard noise.
|
||||
|
||||
Bot family rename (drop the unhelpful `safe_` prefix; suffixes now
|
||||
describe behaviour):
|
||||
|
||||
| Old | New |
|
||||
| --- | --- |
|
||||
| `safe_heuristic_loose` | `heuristic_aggressive` |
|
||||
| `safe_heuristic` | `heuristic_balanced` |
|
||||
| `safe_heuristic_strict` | `heuristic_cautious` |
|
||||
| `noisy_safe` | `heuristic_noisy` |
|
||||
| `passive_discard` | `discard_only` |
|
||||
|
||||
Class renames in `bots/`: `SafeHeuristicBot` → `HeuristicBot`,
|
||||
`SafeHeuristicParams` → `HeuristicParams`, `PassiveDiscardBot` →
|
||||
`DiscardOnlyBot`, plus the loose/strict parameter constants. Backwards
|
||||
compatibility was dropped intentionally — no aliases. Active configs,
|
||||
docs, scripts, tests updated; archive files (read-only by policy)
|
||||
left intact and may still reference old names.
|
||||
|
||||
Analyze plot curation (`deep_cfr/analyze.py`):
|
||||
|
||||
- New `analysis_00_core.png` dashboard with 10 heuristic-free metrics
|
||||
(loss/{advantage, strategy}; vs `heuristic_cautious`:
|
||||
`avg_score_diff0`, `win_rate0`, `avg_opened_colors`,
|
||||
`positive_expedition_rate`, `bonus_expedition_rate`,
|
||||
`score_per_opened_color`, `policy_entropy`; vs `random`: `win_rate0`).
|
||||
- Removed `analysis_05_open_quality.png` (bad/weak/good open rates,
|
||||
recoverable score) and `analysis_07_calibration.png` (calibration gap,
|
||||
recoverable mean) — both relied on the heuristic `recoverable_score`
|
||||
classifier we already dropped from inputs.
|
||||
- Removed `SELECTIVITY_PLOTS` and `plot_selectivity` (heuristic-laden).
|
||||
- `SUMMARY_EVAL_METRICS` no longer includes `bad_open_rate` or
|
||||
`calibration_gap`.
|
||||
|
||||
`PlotSpec` gained an optional `opponents` allowlist so the new core
|
||||
section can pin a specific opponent per panel without restructuring the
|
||||
existing `plot_section` plumbing.
|
||||
|
||||
Tiered evaluation cadence (`EvaluationConfig`):
|
||||
|
||||
- Added `extended_opponents: tuple[str, ...]` and `extended_eval_every:
|
||||
int = 0`.
|
||||
- Method `opponents_for_iteration(iteration)` returns the core list at
|
||||
every `eval_every`, and appends `extended_opponents` (de-duplicated)
|
||||
when `iteration` is also a multiple of `extended_eval_every`.
|
||||
- `default.yaml` now uses 3 core opponents
|
||||
(`random`, `discard_only`, `heuristic_cautious`) every 5 iterations
|
||||
and 3 extended opponents (`heuristic_balanced`, `heuristic_aggressive`,
|
||||
`heuristic_noisy`) every 50 iterations.
|
||||
- `random` is the floor sanity. `discard_only` is the zero-pit
|
||||
detector / absolute-score reference (its score is always 0, so
|
||||
`eval/discard_only/avg_score_diff0` directly equals our model's
|
||||
raw average score). `heuristic_cautious` is the ceiling and the
|
||||
archive-comparable benchmark used in sections 1–6.
|
||||
|
||||
Net effect on ongoing eval cost: ~50% reduction (3 opponents × every
|
||||
5 iter, plus 6 opponents × every 50 iter, vs the prior 6 opponents
|
||||
× every 5).
|
||||
|
||||
### 10. Short open-selectivity ablation
|
||||
|
||||
Run a 200-300 iteration ablation only after the target audit identifies a
|
||||
specific change. Candidate changes include:
|
||||
@@ -656,10 +720,10 @@ specific change. Candidate changes include:
|
||||
|
||||
Primary metrics:
|
||||
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0`
|
||||
- `eval/safe_heuristic_strict/win_rate0`
|
||||
- `eval/safe_heuristic_strict/bad_open_rate`
|
||||
- `eval/safe_heuristic_strict/score_per_opened_color`
|
||||
- `eval/heuristic_cautious/avg_score_diff0`
|
||||
- `eval/heuristic_cautious/win_rate0`
|
||||
- `eval/heuristic_cautious/bad_open_rate`
|
||||
- `eval/heuristic_cautious/score_per_opened_color`
|
||||
|
||||
Do not promote to 500+ iterations unless bad-open rate and score/opened color
|
||||
both improve without degrading score diff.
|
||||
|
||||
@@ -39,7 +39,7 @@ The experiment must produce either a recommended new `network` config or a docum
|
||||
|
||||
## Success criteria
|
||||
|
||||
1. At least one tested config produces win-rate trajectories vs `safe_heuristic_strict` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
|
||||
1. At least one tested config produces win-rate trajectories vs `heuristic_cautious` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
|
||||
2. `iteration_seconds`, `traversal_seconds`, `advantage_train_seconds`, `strategy_train_seconds`, and `policy_network_seconds` (eval) are captured for each tested size and written to `docs/performance.md`.
|
||||
3. A recommended `network` config emerges from the data, OR the experiment documents why the current size should be kept, with specific rationale.
|
||||
|
||||
@@ -72,7 +72,7 @@ configs/deep_cfr/model-size-1536x8.yaml
|
||||
- `traversal_seconds` — traversal phase.
|
||||
- `advantage_train_seconds` — advantage network optimization.
|
||||
- `strategy_train_seconds` — strategy network optimization.
|
||||
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `safe_heuristic_strict`.
|
||||
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `heuristic_cautious`.
|
||||
- At eval iterations: `eval/<opponent>/policy_network_seconds` — needed for the AMP/TRT prerequisite check.
|
||||
- **Memory monitoring:** watch GPU VRAM during the 1024x6 and 1536x8 runs. If a run OOMs or VRAM > 20 GB, reduce `optimization.advantage_batch_size` and `optimization.strategy_batch_size` by half (1024 → 512) and note the change in the results table. Do not adjust traversal settings.
|
||||
|
||||
@@ -259,8 +259,8 @@ if non_eval:
|
||||
eval_rows = {r['iteration']: r for r in rows if r.get('evaluation_seconds')}
|
||||
for it in [50, 100, 150, 200]:
|
||||
if it in eval_rows:
|
||||
wr = eval_rows[it].get('eval/safe_heuristic_strict/win_rate', 'n/a')
|
||||
print(f' iter={it} safe_heuristic_strict win_rate={wr}')
|
||||
wr = eval_rows[it].get('eval/heuristic_cautious/win_rate', 'n/a')
|
||||
print(f' iter={it} heuristic_cautious win_rate={wr}')
|
||||
"
|
||||
done
|
||||
```
|
||||
@@ -269,7 +269,7 @@ done
|
||||
|
||||
After the grid completes, append a date-stamped experiment subsection to `docs/performance.md` under the "Experiments" heading. The subsection must include:
|
||||
|
||||
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `safe_heuristic_strict` at {50, 100, 150, 200} for each config.
|
||||
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `heuristic_cautious` at {50, 100, 150, 200} for each config.
|
||||
- A `policy_network_seconds` column from eval rows — this is the key data for the AMP/compile/TRT prerequisite check.
|
||||
- The recommendation that follows from the decision tree below.
|
||||
|
||||
@@ -302,7 +302,7 @@ Apply this logic after the grid completes:
|
||||
|
||||
### Branch A — a size unlocks the curve AND iter time is acceptable
|
||||
|
||||
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `safe_heuristic_strict` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
|
||||
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `heuristic_cautious` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
|
||||
|
||||
**Action:**
|
||||
1. Recommend that config as the new `network` default.
|
||||
|
||||
@@ -152,7 +152,7 @@ Cython production rewrite. It intentionally uses per-context RNG so interleaved
|
||||
execution order does not change the random stream for another context. That
|
||||
lets the prototype assert value/stat/sample parity against a recursive prototype
|
||||
while measuring realized batch size. Production Cython parity is a later Phase 2
|
||||
gate because the real path also has safe-heuristic opponents, average-strategy
|
||||
gate because the real path also has heuristic-balanced opponents, average-strategy
|
||||
opponents, self-play league snapshots, deck-draw chance sampling, external
|
||||
sampling, and cutoff rollouts.
|
||||
|
||||
@@ -312,7 +312,7 @@ Required feature expansion:
|
||||
|
||||
- Support `opponent_policy: average_strategy`, matching the default config's
|
||||
opponent branch.
|
||||
- Keep unsupported branches guarded (`self_play_league`, `safe_heuristic`,
|
||||
- Keep unsupported branches guarded (`self_play_league`, `heuristic_balanced`,
|
||||
random rollout cutoffs, external sampling).
|
||||
- Add parity tests for the average-strategy fixed-opponent branch.
|
||||
- Verify a default-policy interleaved run starts and emits batch metrics.
|
||||
|
||||
@@ -26,7 +26,7 @@ Re-enable `torch.compile` on the Deep CFR trainer's networks at a model size whe
|
||||
|
||||
1. **Model-size precondition met.** The active `default.yaml` (or the targeted variant) has `network.hidden_size ≥ 1024` *or* `network.num_layers ≥ 6`, *or* an architecture (e.g. `color_shared` with non-trivial `color_attention_layers`) whose per-call forward time exceeds ~150 μs at the trainer's training batch size on the target GPU. If neither condition holds, this plan is **not merged**; the branch is parked.
|
||||
2. **Iter-time improvement.** With `compile.trainer.enabled: true` on the chosen larger model and the same seed, the 1000-iter projection improves by at least **5%** vs the no-compile baseline on the same machine (measured on `home`). Eval and checkpointing should be disabled for the bench window, matching the protocol used in the 2026-05-07 experiment.
|
||||
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `safe_heuristic` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
|
||||
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `heuristic_balanced` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
|
||||
4. **No checkpoint-format break.** Checkpoints saved with compile enabled must load cleanly when compile is disabled, and vice versa. (Handled via `_clean_state_dict()`; see Risks.)
|
||||
5. **No multiprocessing-worker break.** Whether `inference_backend` is `local` or `server`, traversal workers must continue to receive uncompiled `state_dict`s without `_orig_mod.` prefixes.
|
||||
6. **(Secondary) Inference-server forward.** If step B below is taken, the server's `policy_network_seconds` decreases by at least 20% at the chosen model size, with no traversal-path correctness regression. If step B does not produce a measurable win, it is left disabled and the plan still ships with step A only.
|
||||
|
||||
@@ -157,9 +157,9 @@ Python-object touch가 깔려 있다 (per-node, per-iteration):
|
||||
`self.strategy_samples.append(...)` (file:903, 937, 969). list의
|
||||
PyObject reference 갱신은 free-threaded Python에서도 atomic refcount
|
||||
비용을 추가로 부담한다.
|
||||
6. **`SafeHeuristicBot.act(state)`** — `_fixed_opponent_action`
|
||||
6. **`HeuristicBot.act(state)`** — `_fixed_opponent_action`
|
||||
(file:633, 652), `_rollout_value` (file:841). Python class
|
||||
메서드 호출. `safe_heuristic` 옵션 사용 시만 핫.
|
||||
메서드 호출. `heuristic_balanced` 옵션 사용 시만 핫.
|
||||
7. **`league_advantage_networks` indexing** — `_self_play_snapshot_
|
||||
networks` (file:802), `[-recent_count:]`, `[:max(0, ...)]` slicing
|
||||
= Python list slicing.
|
||||
@@ -267,15 +267,15 @@ candidates = self.league_advantage_networks[:max(0, len(self.league_advantage_ne
|
||||
- 빈도: traversal 진입 시 한 번 (`traverse`에서 미리 픽), 재귀 안에서는
|
||||
`active_self_play_networks`만 본다. 따라서 cold path. 변환 불필요.
|
||||
|
||||
### B6. `SafeHeuristicBot.act(state)` — Python bot
|
||||
### B6. `HeuristicBot.act(state)` — Python bot
|
||||
|
||||
```python
|
||||
# traversal.pyx:633, 652, 841
|
||||
return int(self.safe_heuristic_opponent_bot.act(state))
|
||||
return int(self.heuristic_opponent_bot.act(state))
|
||||
```
|
||||
|
||||
- 빈도: `opponent_policy=safe_heuristic` 또는 `cutoff_rollout_policy=
|
||||
safe_heuristic`일 때만. 현 default는 self_play_league + score_diff
|
||||
- 빈도: `opponent_policy=heuristic_balanced` 또는 `cutoff_rollout_policy=
|
||||
heuristic_balanced`일 때만. 현 default는 self_play_league + score_diff
|
||||
cutoff (per memory의 opponent_policy_network_divergence note + AGENTS).
|
||||
- 변환 난이도: **Medium-High** (Python class 전체를 cython화). 현 default
|
||||
config에서는 핫 아님 — 시도하지 않는 게 합리.
|
||||
|
||||
@@ -48,7 +48,7 @@ src/coolrl_lost_cities/games/classic/deep_cfr/
|
||||
traversal_stats.py — structured traversal diagnostic metrics
|
||||
|
||||
# Auxiliary training modes
|
||||
imitation.py — safe-heuristic imitation pretraining
|
||||
imitation.py — heuristic imitation pretraining
|
||||
policy_gradient.py — policy-gradient fine-tuning
|
||||
```
|
||||
|
||||
|
||||
@@ -59,23 +59,23 @@ Evaluation performance peaks early and then degrades.
|
||||
| Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| random | 0.808 | 0.665 | +35.1 | +12.4 |
|
||||
| passive_discard | 0.026 | 0.012 | -30.8 | -43.0 |
|
||||
| safe_heuristic | 0.070 | 0.011 | -68.7 | -94.2 |
|
||||
| safe_heuristic_loose | 0.078 | 0.014 | -69.3 | -95.0 |
|
||||
| safe_heuristic_strict | 0.072 | 0.011 | -57.6 | -85.0 |
|
||||
| noisy_safe | 0.120 | 0.033 | -49.4 | -78.8 |
|
||||
| discard_only | 0.026 | 0.012 | -30.8 | -43.0 |
|
||||
| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 |
|
||||
| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 |
|
||||
| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 |
|
||||
| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 |
|
||||
|
||||
The best strict-heuristic point appears around iteration 70:
|
||||
|
||||
- `eval/safe_heuristic_strict/win_rate0=0.15`
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0=-41.09`
|
||||
- `eval/heuristic_cautious/win_rate0=0.15`
|
||||
- `eval/heuristic_cautious/avg_score_diff0=-41.09`
|
||||
|
||||
The final point at iteration 2000 is worse:
|
||||
|
||||
- `eval/safe_heuristic_strict/win_rate0=0.02`
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0=-81.64`
|
||||
- `eval/heuristic_cautious/win_rate0=0.02`
|
||||
- `eval/heuristic_cautious/avg_score_diff0=-81.64`
|
||||
|
||||
Selectivity does not meaningfully emerge. Against `safe_heuristic_strict`:
|
||||
Selectivity does not meaningfully emerge. Against `heuristic_cautious`:
|
||||
|
||||
| Metric | First 20 evals | Last 20 evals | Interpretation |
|
||||
| --- | ---: | ---: | --- |
|
||||
|
||||
@@ -19,7 +19,7 @@ The primary bottleneck in CUDA-based evaluation is the overhead of launching sma
|
||||
|
||||
A critical refinement in the batched implementation was the handling of policy entropy. Initial versions that calculated entropy per-row on the CPU incurred significant synchronization penalties because each row required a GPU-to-CPU transfer. Moving the entropy calculation into the Torch post-processing pipeline—specifically calculating it directly on the `probs_tensor` (line 220)—ensures that the computation remains on the device and only the final results are transferred back to the host in bulk.
|
||||
|
||||
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `safe_heuristic_strict`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
|
||||
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `heuristic_cautious`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
|
||||
|
||||
## Practical implication
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ On CPU, the `network / turn` cost is approximately **0.074 ms**. On CUDA, this r
|
||||
|
||||
### Secondary Bottlenecks
|
||||
- **Post-processing:** Moving tensors back to CPU (`.cpu().numpy()`) and calculating entropy adds measurable overhead on CUDA that is largely absent on CPU.
|
||||
- **Opponent Logic:** Heuristic opponents (e.g., `safe_heuristic`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
|
||||
- **Opponent Logic:** Heuristic opponents (e.g., `heuristic_balanced`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
|
||||
|
||||
## Practical Implications
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ Two temporary runs used:
|
||||
- `run.max_iterations=3`
|
||||
- `evaluation.eval_every=1`
|
||||
- `evaluation.games=20`
|
||||
- `evaluation.opponents=[random,safe_heuristic_strict]`
|
||||
- `evaluation.opponents=[random,heuristic_cautious]`
|
||||
- W&B disabled
|
||||
|
||||
Runs:
|
||||
@@ -55,7 +55,7 @@ Runs:
|
||||
|
||||
Core metrics matched exactly:
|
||||
|
||||
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` |
|
||||
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
|
||||
| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
|
||||
|
||||
@@ -56,7 +56,7 @@ registered classic bots (`evaluate.py`), training CLI and evaluation CLI
|
||||
(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
|
||||
`runtime_progress.json` / `train.log` run artifacts, self-play league with
|
||||
snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
|
||||
heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`),
|
||||
heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`),
|
||||
and policy-gradient fine-tuning (`policy_gradient.py`).
|
||||
|
||||
As of `ad0be89`, the package also includes `inference_server.py`,
|
||||
|
||||
@@ -31,7 +31,7 @@ Decision criteria for going forward:
|
||||
|
||||
### 2026-05-07 — Safe heuristic single-thread parity (criterion 1)
|
||||
|
||||
Path: `experiments/julia_safe_heuristic/`.
|
||||
Path: `experiments/julia_heuristic/`.
|
||||
|
||||
1,838 snapshots in 157.471 ms median (~85.6 μs/call). Action-sequence
|
||||
parity vs Python. Same order of magnitude as the Cython port of the
|
||||
|
||||
@@ -28,7 +28,7 @@ Deep CFR은 최종적으로 average strategy가 수렴 대상. current advantage
|
||||
Visible-score predictor의 ceiling은 있을 수 있음. 다만 selectivity 자체의 ceiling은 아님. selectivity는 visible-score prediction 말고 option value, irreversible cost 회피, opponent dynamics 대응 등 다른 경로로도 emerge 가능.
|
||||
|
||||
**Self-play attractor 가설**
|
||||
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. safe_heuristic이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
|
||||
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. heuristic_balanced이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
|
||||
|
||||
**Lost Cities NE 자체가 5-color 가설**
|
||||
이론적 가능성 0 아님. self-play가 발견한 게 진짜 NE면 transition 영원히 안 옴. 검증 가능한 형태로는 tabular oracle 또는 BR 진단.
|
||||
|
||||
@@ -240,7 +240,7 @@ def main() -> None:
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
opponents = args.opponent or ["safe_heuristic_strict"]
|
||||
opponents = args.opponent or ["heuristic_cautious"]
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
rows = []
|
||||
for checkpoint in args.checkpoints:
|
||||
|
||||
@@ -367,7 +367,7 @@ def analyze_checkpoint(
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("checkpoints", nargs="+", type=Path)
|
||||
parser.add_argument("--opponent", default="safe_heuristic_strict")
|
||||
parser.add_argument("--opponent", default="heuristic_cautious")
|
||||
parser.add_argument("--games", type=int, default=100)
|
||||
parser.add_argument("--seed", type=int, default=231_000)
|
||||
parser.add_argument("--device", default="cuda")
|
||||
@@ -379,7 +379,7 @@ def main() -> None:
|
||||
help=(
|
||||
"Policy used for the policy_player during forced-action rollouts. "
|
||||
"'model' uses the trained advantage network; any other value is "
|
||||
"treated as a bot name (e.g. 'safe_heuristic_strict')."
|
||||
"treated as a bot name (e.g. 'heuristic_cautious')."
|
||||
),
|
||||
)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
|
||||
@@ -273,7 +273,7 @@ def analyze_checkpoint(
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("checkpoints", nargs="+", type=Path)
|
||||
parser.add_argument("--opponent", default="safe_heuristic_strict")
|
||||
parser.add_argument("--opponent", default="heuristic_cautious")
|
||||
parser.add_argument("--games", type=int, default=100)
|
||||
parser.add_argument("--seed", type=int, default=232_000)
|
||||
parser.add_argument("--device", default="cpu")
|
||||
|
||||
@@ -265,7 +265,7 @@ def main() -> None:
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
opponents = args.opponent or ["random", "safe_heuristic_strict"]
|
||||
opponents = args.opponent or ["random", "heuristic_cautious"]
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
rows: list[dict] = []
|
||||
|
||||
@@ -7,22 +7,22 @@ from typing import Any
|
||||
|
||||
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig
|
||||
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic_py import SafeHeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic_py import HeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots.registry import (
|
||||
LOOSE_SAFE_HEURISTIC_PARAMS,
|
||||
STRICT_SAFE_HEURISTIC_PARAMS,
|
||||
AGGRESSIVE_HEURISTIC_PARAMS,
|
||||
CAUTIOUS_HEURISTIC_PARAMS,
|
||||
)
|
||||
|
||||
VARIANTS = {
|
||||
"default": None,
|
||||
"loose": LOOSE_SAFE_HEURISTIC_PARAMS,
|
||||
"strict": STRICT_SAFE_HEURISTIC_PARAMS,
|
||||
"loose": AGGRESSIVE_HEURISTIC_PARAMS,
|
||||
"strict": CAUTIOUS_HEURISTIC_PARAMS,
|
||||
}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Export safe-heuristic bot parity snapshots for external implementations."
|
||||
description="Export heuristic bot parity snapshots for external implementations."
|
||||
)
|
||||
parser.add_argument("--output", required=True, help="JSONL output path.")
|
||||
parser.add_argument("--seeds", type=int, default=50, help="Number of seeds per config.")
|
||||
@@ -71,7 +71,7 @@ def main() -> None:
|
||||
for config_name, config in _configs():
|
||||
for variant_name, params in VARIANTS.items():
|
||||
for seed in range(args.seeds):
|
||||
bot = SafeHeuristicBot(params)
|
||||
bot = HeuristicBot(params)
|
||||
state = GameState.new_game(config, seed=seed)
|
||||
for turn in range(args.max_steps):
|
||||
if state.terminal:
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from ..policy import LostCitiesPolicy, PolicyInput
|
||||
from .heuristic import SafeHeuristicBot
|
||||
from .passive import PassiveDiscardBot
|
||||
from .discard_only import DiscardOnlyBot
|
||||
from .heuristic import HeuristicBot
|
||||
from .random import RandomBot
|
||||
from .registry import DEFAULT_BOT, available_bot_names, build_bot
|
||||
|
||||
@@ -10,9 +10,9 @@ __all__ = [
|
||||
"PolicyInput",
|
||||
"DEFAULT_BOT",
|
||||
"LostCitiesPolicy",
|
||||
"PassiveDiscardBot",
|
||||
"DiscardOnlyBot",
|
||||
"RandomBot",
|
||||
"SafeHeuristicBot",
|
||||
"HeuristicBot",
|
||||
"available_bot_names",
|
||||
"build_bot",
|
||||
]
|
||||
|
||||
+1
-1
@@ -6,7 +6,7 @@ from ..snapshots import Snapshot
|
||||
from .base import first_legal, legal_from_obs
|
||||
|
||||
|
||||
class PassiveDiscardBot(LostCitiesPolicy):
|
||||
class DiscardOnlyBot(LostCitiesPolicy):
|
||||
"""Baseline that avoids opening expeditions whenever discarding is legal."""
|
||||
|
||||
def act(self, obs_or_state: PolicyInput) -> int:
|
||||
@@ -4,8 +4,8 @@ from .heuristic_cy import (
|
||||
DRAW_FROM_DECK_ACTION,
|
||||
PLAY_OR_DISCARD_ACTIONS_PER_SLOT,
|
||||
DerivedHeuristicConfig,
|
||||
SafeHeuristicBot,
|
||||
SafeHeuristicParams,
|
||||
HeuristicBot,
|
||||
HeuristicParams,
|
||||
derive_heuristic_config,
|
||||
discard_action,
|
||||
draw_from_discard_action,
|
||||
@@ -16,8 +16,8 @@ __all__ = [
|
||||
"DRAW_FROM_DECK_ACTION",
|
||||
"PLAY_OR_DISCARD_ACTIONS_PER_SLOT",
|
||||
"DerivedHeuristicConfig",
|
||||
"SafeHeuristicBot",
|
||||
"SafeHeuristicParams",
|
||||
"HeuristicBot",
|
||||
"HeuristicParams",
|
||||
"derive_heuristic_config",
|
||||
"discard_action",
|
||||
"draw_from_discard_action",
|
||||
|
||||
@@ -31,7 +31,7 @@ def draw_from_discard_action(color: int) -> int:
|
||||
return 1 + color
|
||||
|
||||
|
||||
LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.safe_heuristic")
|
||||
LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.heuristic")
|
||||
|
||||
|
||||
class _CachedState:
|
||||
@@ -66,7 +66,7 @@ class _CachedState:
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SafeHeuristicParams:
|
||||
class HeuristicParams:
|
||||
# Expedition opening.
|
||||
open_target_ratio: float = 0.50
|
||||
open_min_card_ratio: float = 0.40
|
||||
@@ -124,7 +124,7 @@ class DerivedHeuristicConfig:
|
||||
@lru_cache(maxsize=64)
|
||||
def derive_heuristic_config(
|
||||
config: LostCitiesConfig,
|
||||
params: SafeHeuristicParams,
|
||||
params: HeuristicParams,
|
||||
) -> DerivedHeuristicConfig:
|
||||
max_color_sum = sum(config.min_rank + rank - 1 for rank in range(1, config.n_ranks + 1))
|
||||
break_even_sum = -config.expedition_penalty
|
||||
@@ -178,20 +178,20 @@ def derive_heuristic_config(
|
||||
)
|
||||
|
||||
|
||||
class SafeHeuristicBot(LostCitiesPolicy):
|
||||
def __init__(self, params: SafeHeuristicParams | None = None):
|
||||
self.params = params or SafeHeuristicParams()
|
||||
class HeuristicBot(LostCitiesPolicy):
|
||||
def __init__(self, params: HeuristicParams | None = None):
|
||||
self.params = params or HeuristicParams()
|
||||
|
||||
def act(self, obs_or_state: PolicyInput) -> int:
|
||||
if not isinstance(obs_or_state, GameState) and not hasattr(obs_or_state, "legal_mask"):
|
||||
LOGGER.debug(
|
||||
"SafeHeuristicBot fallback to first legal: input_type=%s",
|
||||
"HeuristicBot fallback to first legal: input_type=%s",
|
||||
type(obs_or_state).__name__,
|
||||
)
|
||||
return first_legal(legal_from_obs(obs_or_state))
|
||||
|
||||
LOGGER.debug(
|
||||
"SafeHeuristicBot heuristic path: player=%s phase=%s turn=%s",
|
||||
"HeuristicBot heuristic path: player=%s phase=%s turn=%s",
|
||||
obs_or_state.current_player,
|
||||
obs_or_state.phase,
|
||||
obs_or_state.turn_count,
|
||||
|
||||
@@ -30,11 +30,11 @@ def draw_from_discard_action(color: int) -> int:
|
||||
return 1 + color
|
||||
|
||||
|
||||
LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.safe_heuristic")
|
||||
LOGGER = logging.getLogger("coolrl_lost_cities.games.classic.bots.heuristic")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class SafeHeuristicParams:
|
||||
class HeuristicParams:
|
||||
# Expedition opening.
|
||||
open_target_ratio: float = 0.50
|
||||
open_min_card_ratio: float = 0.40
|
||||
@@ -92,7 +92,7 @@ class DerivedHeuristicConfig:
|
||||
@lru_cache(maxsize=64)
|
||||
def derive_heuristic_config(
|
||||
config: LostCitiesConfig,
|
||||
params: SafeHeuristicParams,
|
||||
params: HeuristicParams,
|
||||
) -> DerivedHeuristicConfig:
|
||||
max_color_sum = sum(config.min_rank + rank - 1 for rank in range(1, config.n_ranks + 1))
|
||||
break_even_sum = -config.expedition_penalty
|
||||
@@ -146,20 +146,20 @@ def derive_heuristic_config(
|
||||
)
|
||||
|
||||
|
||||
class SafeHeuristicBot(LostCitiesPolicy):
|
||||
def __init__(self, params: SafeHeuristicParams | None = None):
|
||||
self.params = params or SafeHeuristicParams()
|
||||
class HeuristicBot(LostCitiesPolicy):
|
||||
def __init__(self, params: HeuristicParams | None = None):
|
||||
self.params = params or HeuristicParams()
|
||||
|
||||
def act(self, obs_or_state: PolicyInput) -> int:
|
||||
if not isinstance(obs_or_state, GameState) and not hasattr(obs_or_state, "legal_mask"):
|
||||
LOGGER.debug(
|
||||
"SafeHeuristicBot fallback to first legal: input_type=%s",
|
||||
"HeuristicBot fallback to first legal: input_type=%s",
|
||||
type(obs_or_state).__name__,
|
||||
)
|
||||
return first_legal(legal_from_obs(obs_or_state))
|
||||
|
||||
LOGGER.debug(
|
||||
"SafeHeuristicBot heuristic path: player=%s phase=%s turn=%s",
|
||||
"HeuristicBot heuristic path: player=%s phase=%s turn=%s",
|
||||
obs_or_state.current_player,
|
||||
obs_or_state.phase,
|
||||
obs_or_state.turn_count,
|
||||
|
||||
@@ -3,8 +3,8 @@ from __future__ import annotations
|
||||
from collections.abc import Callable
|
||||
|
||||
from ..policy import LostCitiesPolicy, PolicyInput
|
||||
from .heuristic import SafeHeuristicBot, SafeHeuristicParams
|
||||
from .passive import PassiveDiscardBot
|
||||
from .discard_only import DiscardOnlyBot
|
||||
from .heuristic import HeuristicBot, HeuristicParams
|
||||
from .random import RandomBot
|
||||
|
||||
BotName = str
|
||||
@@ -30,7 +30,7 @@ class NoisyPolicy(LostCitiesPolicy):
|
||||
return self.base.act(obs_or_state)
|
||||
|
||||
|
||||
LOOSE_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams(
|
||||
AGGRESSIVE_HEURISTIC_PARAMS = HeuristicParams(
|
||||
open_target_ratio=0.42,
|
||||
open_min_card_ratio=0.30,
|
||||
handshake_target_multiplier=1.00,
|
||||
@@ -38,7 +38,7 @@ LOOSE_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams(
|
||||
late_open_block_ratio=0.12,
|
||||
)
|
||||
|
||||
STRICT_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams(
|
||||
CAUTIOUS_HEURISTIC_PARAMS = HeuristicParams(
|
||||
open_target_ratio=0.62,
|
||||
open_min_card_ratio=0.50,
|
||||
handshake_target_multiplier=1.35,
|
||||
@@ -49,12 +49,12 @@ STRICT_SAFE_HEURISTIC_PARAMS = SafeHeuristicParams(
|
||||
|
||||
BOT_REGISTRY: dict[BotName, PolicyFactory] = {
|
||||
DEFAULT_BOT: RandomBot,
|
||||
"passive-discard": lambda seed: PassiveDiscardBot(),
|
||||
"safe-heuristic": lambda seed: SafeHeuristicBot(),
|
||||
"safe-heuristic-loose": lambda seed: SafeHeuristicBot(LOOSE_SAFE_HEURISTIC_PARAMS),
|
||||
"safe-heuristic-strict": lambda seed: SafeHeuristicBot(STRICT_SAFE_HEURISTIC_PARAMS),
|
||||
"noisy-safe": lambda seed: NoisyPolicy(
|
||||
SafeHeuristicBot(),
|
||||
"discard-only": lambda seed: DiscardOnlyBot(),
|
||||
"heuristic-balanced": lambda seed: HeuristicBot(),
|
||||
"heuristic-aggressive": lambda seed: HeuristicBot(AGGRESSIVE_HEURISTIC_PARAMS),
|
||||
"heuristic-cautious": lambda seed: HeuristicBot(CAUTIOUS_HEURISTIC_PARAMS),
|
||||
"heuristic-noisy": lambda seed: NoisyPolicy(
|
||||
HeuristicBot(),
|
||||
RandomBot(seed),
|
||||
),
|
||||
}
|
||||
|
||||
@@ -18,6 +18,7 @@ class PlotSpec:
|
||||
scale: float = 1.0
|
||||
kind: str = "eval"
|
||||
fixed_ylim: tuple[float, float] | None = None
|
||||
opponents: tuple[str, ...] | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -28,6 +29,71 @@ class SectionSpec:
|
||||
|
||||
|
||||
SECTIONS: tuple[SectionSpec, ...] = (
|
||||
SectionSpec(
|
||||
"Core",
|
||||
"analysis_00_core.png",
|
||||
(
|
||||
PlotSpec("Advantage Loss", ("loss/advantage",), "loss", kind="train"),
|
||||
PlotSpec("Strategy Loss", ("loss/strategy",), "loss", kind="train"),
|
||||
PlotSpec(
|
||||
"Avg Score Diff (heuristic_cautious)",
|
||||
("avg_score_diff0",),
|
||||
"score diff",
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Win Rate (heuristic_cautious)",
|
||||
("win_rate0",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Win Rate (random)",
|
||||
("win_rate0",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
opponents=("random",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Avg Opened Colors (heuristic_cautious)",
|
||||
("avg_opened_colors",),
|
||||
"colors",
|
||||
fixed_ylim=(0, 5),
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Positive Expedition Rate (heuristic_cautious)",
|
||||
("positive_expedition_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Bonus Expedition Rate (heuristic_cautious)",
|
||||
("bonus_expedition_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Score per Opened Color (heuristic_cautious)",
|
||||
("score_per_opened_color",),
|
||||
"score / color",
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
PlotSpec(
|
||||
"Policy Entropy (heuristic_cautious)",
|
||||
("policy_entropy",),
|
||||
"entropy",
|
||||
opponents=("heuristic_cautious",),
|
||||
),
|
||||
),
|
||||
),
|
||||
SectionSpec(
|
||||
"Loss",
|
||||
"analysis_01_loss.png",
|
||||
@@ -107,48 +173,6 @@ SECTIONS: tuple[SectionSpec, ...] = (
|
||||
PlotSpec("Expedition Cards", ("avg_expedition_cards",), "cards"),
|
||||
),
|
||||
),
|
||||
SectionSpec(
|
||||
"OpenQuality",
|
||||
"analysis_05_open_quality.png",
|
||||
(
|
||||
PlotSpec(
|
||||
"Bad Open Rate",
|
||||
("bad_open_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
),
|
||||
PlotSpec(
|
||||
"Weak Open Rate",
|
||||
("weak_open_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
),
|
||||
PlotSpec(
|
||||
"Bad or Weak Open Rate",
|
||||
("bad_or_weak_open_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
),
|
||||
PlotSpec(
|
||||
"Good Open Rate",
|
||||
("good_open_rate",),
|
||||
"rate (%)",
|
||||
scale=100.0,
|
||||
fixed_ylim=(0, 100),
|
||||
),
|
||||
PlotSpec("Bad Open per Game", ("bad_open_per_game",), "opens / game"),
|
||||
PlotSpec(
|
||||
"Bad or Weak Open per Game",
|
||||
("bad_or_weak_open_per_game",),
|
||||
"opens / game",
|
||||
),
|
||||
PlotSpec("Opening Play Actions", ("opening_play_actions",), "actions / game"),
|
||||
PlotSpec("Opening Recoverable p25", ("opening_recoverable_score_p25",), "score"),
|
||||
),
|
||||
),
|
||||
SectionSpec(
|
||||
"ExpeditionOutcomes",
|
||||
"analysis_06_expedition_outcomes.png",
|
||||
@@ -188,23 +212,6 @@ SECTIONS: tuple[SectionSpec, ...] = (
|
||||
PlotSpec("Score per Opened Color", ("score_per_opened_color",), "score / color"),
|
||||
),
|
||||
),
|
||||
SectionSpec(
|
||||
"Calibration",
|
||||
"analysis_07_calibration.png",
|
||||
(
|
||||
PlotSpec(
|
||||
"First Open Recoverable Score",
|
||||
(
|
||||
"first_open_recoverable_score_mean_for_positive_final",
|
||||
"first_open_recoverable_score_mean_for_negative_final",
|
||||
),
|
||||
"score",
|
||||
),
|
||||
PlotSpec("Opening Recoverable Mean", ("opening_recoverable_score_mean",), "score"),
|
||||
PlotSpec("Opening Recoverable p25", ("opening_recoverable_score_p25",), "score"),
|
||||
PlotSpec("Calibration Gap", ("calibration_gap",), "score"),
|
||||
),
|
||||
),
|
||||
SectionSpec(
|
||||
"Traversal",
|
||||
"analysis_08_traversal.png",
|
||||
@@ -311,41 +318,23 @@ SECTIONS: tuple[SectionSpec, ...] = (
|
||||
),
|
||||
)
|
||||
|
||||
SELECTIVITY_PLOTS: tuple[PlotSpec, ...] = (
|
||||
PlotSpec("Opened Colors", ("avg_opened_colors",), "colors"),
|
||||
PlotSpec("5-Color Open Count", ("5_color_open_count",), "games / eval"),
|
||||
PlotSpec("Opening Recoverable Mean", ("opening_recoverable_score_mean",), "score"),
|
||||
PlotSpec("Calibration Gap", ("calibration_gap",), "score"),
|
||||
PlotSpec(
|
||||
"First Open Recoverable Score",
|
||||
(
|
||||
"first_open_recoverable_score_mean_for_positive_final",
|
||||
"first_open_recoverable_score_mean_for_negative_final",
|
||||
),
|
||||
"score",
|
||||
),
|
||||
PlotSpec("Opening Play Actions", ("opening_play_actions",), "actions / game"),
|
||||
)
|
||||
|
||||
SUMMARY_EVAL_METRICS: tuple[tuple[str, str, float], ...] = (
|
||||
("win_rate0", "win rate (%)", 100.0),
|
||||
("avg_score_diff0", "avg score diff", 1.0),
|
||||
("avg_score0", "avg score", 1.0),
|
||||
("play_action_rate", "play rate (%)", 100.0),
|
||||
("avg_opened_colors", "opened colors", 1.0),
|
||||
("bad_open_rate", "bad open (%)", 100.0),
|
||||
("score_per_opened_color", "score / opened color", 1.0),
|
||||
("calibration_gap", "calibration gap", 1.0),
|
||||
("bonus_contribution_per_game", "bonus / game", 1.0),
|
||||
)
|
||||
|
||||
OPPONENT_COLORS: dict[str, str] = {
|
||||
"noisy_safe": "tab:blue",
|
||||
"passive_discard": "tab:orange",
|
||||
"heuristic_noisy": "tab:blue",
|
||||
"discard_only": "tab:orange",
|
||||
"random": "tab:green",
|
||||
"safe_heuristic": "tab:red",
|
||||
"safe_heuristic_loose": "tab:purple",
|
||||
"safe_heuristic_strict": "tab:brown",
|
||||
"heuristic_balanced": "tab:red",
|
||||
"heuristic_aggressive": "tab:purple",
|
||||
"heuristic_cautious": "tab:brown",
|
||||
}
|
||||
|
||||
TRAVERSAL_COLORS: dict[str, str] = {
|
||||
@@ -428,7 +417,14 @@ def plot_section(
|
||||
if spec.kind == "train":
|
||||
plotted = _plot_train_spec(ax, rows, spec, smoothing_window=smoothing_window)
|
||||
else:
|
||||
plotted = _plot_eval_spec(ax, rows, opponents, spec, smoothing_window=smoothing_window)
|
||||
filtered_opponents = (
|
||||
[o for o in opponents if o in spec.opponents]
|
||||
if spec.opponents is not None
|
||||
else opponents
|
||||
)
|
||||
plotted = _plot_eval_spec(
|
||||
ax, rows, filtered_opponents, spec, smoothing_window=smoothing_window
|
||||
)
|
||||
_finish_axis(
|
||||
ax, spec.title, ylabel=spec.ylabel, plotted=plotted, fixed_ylim=spec.fixed_ylim
|
||||
)
|
||||
@@ -539,12 +535,6 @@ def analyze_run(
|
||||
if plot_section(rows, section, path, smoothing_window=smoothing_window):
|
||||
written.append(path)
|
||||
|
||||
selectivity_path = output_dir / _with_filename_suffix(
|
||||
"analysis_09_selectivity.png", filename_suffix
|
||||
)
|
||||
if plot_selectivity(rows, selectivity_path, smoothing_window=smoothing_window):
|
||||
written.append(selectivity_path)
|
||||
|
||||
final_eval_path = output_dir / _with_filename_suffix(
|
||||
"analysis_final_eval_summary.png", filename_suffix
|
||||
)
|
||||
@@ -566,45 +556,6 @@ def _with_filename_suffix(filename: str, suffix: str) -> str:
|
||||
return f"{path.stem}{suffix}{path.suffix}"
|
||||
|
||||
|
||||
def plot_selectivity(
|
||||
rows: list[dict[str, Any]],
|
||||
output: Path,
|
||||
*,
|
||||
smoothing_window: int,
|
||||
) -> bool:
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
opponents = opponent_names(rows)
|
||||
if not opponents:
|
||||
return False
|
||||
|
||||
fig, axes = plt.subplots(3, 2, figsize=(16, 12), squeeze=False)
|
||||
axes_flat = list(axes.flat)
|
||||
plotted_any = False
|
||||
|
||||
for ax, spec in zip(axes_flat, SELECTIVITY_PLOTS, strict=True):
|
||||
plotted = _plot_eval_spec(ax, rows, opponents, spec, smoothing_window=smoothing_window)
|
||||
_finish_axis(ax, spec.title, ylabel=spec.ylabel, plotted=plotted)
|
||||
plotted_any = plotted_any or plotted
|
||||
|
||||
handles, labels = _legend_items(axes_flat)
|
||||
if handles:
|
||||
fig.legend(handles, labels, loc="upper center", ncols=min(len(labels), 6), fontsize="small")
|
||||
suffix = f" ({smoothing_window}-iter moving average)" if smoothing_window > 1 else ""
|
||||
fig.suptitle(
|
||||
f"Lost Cities Deep CFR selectivity metrics{suffix}",
|
||||
fontsize=14,
|
||||
fontweight="bold",
|
||||
)
|
||||
fig.tight_layout(rect=(0, 0, 1, 0.95))
|
||||
if not plotted_any:
|
||||
plt.close(fig)
|
||||
return False
|
||||
fig.savefig(output, dpi=150)
|
||||
plt.close(fig)
|
||||
return True
|
||||
|
||||
|
||||
def _all_eval_metrics() -> set[str]:
|
||||
metrics: set[str] = set()
|
||||
for section in SECTIONS:
|
||||
|
||||
@@ -143,16 +143,16 @@ class TraversalConfig(StrictModel):
|
||||
@field_validator("cutoff_rollout_policy")
|
||||
@classmethod
|
||||
def _validate_cutoff_rollout_policy(cls, value: str) -> str:
|
||||
if value not in {"random", "safe_heuristic"}:
|
||||
raise ValueError("must be 'random' or 'safe_heuristic'")
|
||||
if value not in {"random", "heuristic_balanced"}:
|
||||
raise ValueError("must be 'random' or 'heuristic_balanced'")
|
||||
return value
|
||||
|
||||
@field_validator("opponent_policy")
|
||||
@classmethod
|
||||
def _validate_opponent_policy(cls, value: str) -> str:
|
||||
if value not in {"network", "safe_heuristic", "self_play_league", "average_strategy"}:
|
||||
if value not in {"network", "heuristic_balanced", "self_play_league", "average_strategy"}:
|
||||
raise ValueError(
|
||||
"must be 'network', 'safe_heuristic', 'self_play_league', or 'average_strategy'"
|
||||
"must be 'network', 'heuristic_balanced', 'self_play_league', or 'average_strategy'"
|
||||
)
|
||||
return value
|
||||
|
||||
@@ -294,13 +294,35 @@ class CheckpointConfig(StrictModel):
|
||||
class EvaluationConfig(StrictModel):
|
||||
eval_every: int = 50
|
||||
games: int = 10
|
||||
opponents: tuple[str, ...] = ("random", "safe_heuristic")
|
||||
opponents: tuple[str, ...] = ("random", "heuristic_balanced")
|
||||
extended_eval_every: int = 0
|
||||
extended_opponents: tuple[str, ...] = ()
|
||||
max_steps: int = 10_000
|
||||
on_max_steps: str = "score_diff"
|
||||
batch_size: int = 64
|
||||
device: str = "trainer"
|
||||
num_workers: int = 4
|
||||
|
||||
def opponents_for_iteration(self, iteration: int) -> tuple[str, ...]:
|
||||
"""Return the opponent set to evaluate against at ``iteration``.
|
||||
|
||||
- Always returns ``opponents`` when ``iteration % eval_every == 0``.
|
||||
- When ``extended_eval_every > 0`` and ``iteration`` is also a multiple
|
||||
of ``extended_eval_every``, appends ``extended_opponents`` (de-duplicated)
|
||||
to the core list.
|
||||
- Returns ``()`` when no eval is scheduled this iteration.
|
||||
"""
|
||||
if self.eval_every <= 0 or iteration <= 0:
|
||||
return ()
|
||||
if iteration % self.eval_every != 0:
|
||||
return ()
|
||||
result = list(self.opponents)
|
||||
if self.extended_eval_every > 0 and iteration % self.extended_eval_every == 0:
|
||||
for opponent in self.extended_opponents:
|
||||
if opponent not in result:
|
||||
result.append(opponent)
|
||||
return tuple(result)
|
||||
|
||||
@field_validator("on_max_steps")
|
||||
@classmethod
|
||||
def _validate_on_max_steps(cls, value: str) -> str:
|
||||
|
||||
@@ -6,7 +6,7 @@ import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
from coolrl_lost_cities.games.classic.bots import SafeHeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots import HeuristicBot
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.encoding import encode_info_state, input_dim
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP
|
||||
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig, classic_config
|
||||
@@ -18,7 +18,7 @@ class ImitationMetrics:
|
||||
loss: float
|
||||
|
||||
|
||||
def collect_safe_heuristic_samples(
|
||||
def collect_heuristic_samples(
|
||||
config: LostCitiesConfig | None = None,
|
||||
*,
|
||||
games: int = 4,
|
||||
@@ -28,7 +28,7 @@ def collect_safe_heuristic_samples(
|
||||
game_config = config or classic_config(seed=seed)
|
||||
probe = GameState.new_game(game_config, seed=seed)
|
||||
action_size = 2 * probe.config.hand_size + 1 + probe.config.n_colors
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
infos: list[np.ndarray] = []
|
||||
targets: list[np.ndarray] = []
|
||||
masks: list[np.ndarray] = []
|
||||
@@ -67,7 +67,7 @@ def pretrain_strategy_network(
|
||||
learning_rate: float = 1.0e-3,
|
||||
device: torch.device | str = "cpu",
|
||||
) -> ImitationMetrics:
|
||||
x_np, y_np, legal_np = collect_safe_heuristic_samples(config, games=games, seed=seed)
|
||||
x_np, y_np, legal_np = collect_heuristic_samples(config, games=games, seed=seed)
|
||||
device = torch.device(device)
|
||||
strategy_network.to(device)
|
||||
strategy_network.train()
|
||||
|
||||
@@ -894,17 +894,15 @@ class DeepCFRTrainer:
|
||||
self.tracker.log_event(_format_iteration_summary(metrics, data))
|
||||
|
||||
def _evaluate(self, iteration: int) -> dict[str, float | int]:
|
||||
if (
|
||||
self.config.evaluation.eval_every <= 0
|
||||
or iteration % self.config.evaluation.eval_every != 0
|
||||
):
|
||||
opponents = self.config.evaluation.opponents_for_iteration(iteration)
|
||||
if not opponents:
|
||||
return {}
|
||||
results: dict[str, float | int] = {}
|
||||
eval_device = self._evaluation_device()
|
||||
if self.config.evaluation.resolved_num_workers(len(self.config.evaluation.opponents)) > 1:
|
||||
return self._evaluate_parallel(iteration, eval_device)
|
||||
if self.config.evaluation.resolved_num_workers(len(opponents)) > 1:
|
||||
return self._evaluate_parallel(iteration, eval_device, opponents)
|
||||
eval_network = self._evaluation_network(eval_device)
|
||||
for opponent in self.config.evaluation.opponents:
|
||||
for opponent in opponents:
|
||||
result = evaluate_strategy_network(
|
||||
eval_network,
|
||||
self.game_config,
|
||||
@@ -925,8 +923,8 @@ class DeepCFRTrainer:
|
||||
self,
|
||||
iteration: int,
|
||||
eval_device: torch.device,
|
||||
opponents: tuple[str, ...],
|
||||
) -> dict[str, float | int]:
|
||||
opponents = self.config.evaluation.opponents
|
||||
max_workers = self.config.evaluation.resolved_num_workers(len(opponents))
|
||||
self.tracker.log_event(
|
||||
f"Evaluation multiprocessing enabled iteration={iteration} "
|
||||
|
||||
@@ -7,7 +7,7 @@ from libc.stdlib cimport free, malloc
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from coolrl_lost_cities.games.classic.bots import SafeHeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots import HeuristicBot
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.cfr_math cimport regret_matching_c
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.encoding cimport (
|
||||
_encode_info_state_with_flags_c,
|
||||
@@ -70,8 +70,8 @@ cdef class CythonDeepCFRTraverser:
|
||||
cdef object device
|
||||
cdef object encoding
|
||||
cdef object league_advantage_networks
|
||||
cdef object safe_heuristic_rollout_bot
|
||||
cdef object safe_heuristic_opponent_bot
|
||||
cdef object heuristic_rollout_bot
|
||||
cdef object heuristic_opponent_bot
|
||||
cdef int action_size
|
||||
cdef int input_dim
|
||||
cdef float epsilon
|
||||
@@ -185,15 +185,15 @@ cdef class CythonDeepCFRTraverser:
|
||||
raise ValueError("cutoff_value_mode must be 'score_diff' or 'random_rollout'")
|
||||
self.cutoff_random_rollout = cutoff_value_mode == "random_rollout"
|
||||
self.cutoff_rollouts = max(0, cutoff_rollouts)
|
||||
if cutoff_rollout_policy not in {"random", "safe_heuristic"}:
|
||||
raise ValueError("cutoff_rollout_policy must be 'random' or 'safe_heuristic'")
|
||||
if cutoff_rollout_policy not in {"random", "heuristic_balanced"}:
|
||||
raise ValueError("cutoff_rollout_policy must be 'random' or 'heuristic_balanced'")
|
||||
self.cutoff_rollout_max_steps = max(1, cutoff_rollout_max_steps)
|
||||
self.safe_heuristic_rollout_bot = (
|
||||
SafeHeuristicBot() if cutoff_rollout_policy == "safe_heuristic" else None
|
||||
self.heuristic_rollout_bot = (
|
||||
HeuristicBot() if cutoff_rollout_policy == "heuristic_balanced" else None
|
||||
)
|
||||
if opponent_policy == "network":
|
||||
self.opponent_policy_id = 0
|
||||
elif opponent_policy == "safe_heuristic":
|
||||
elif opponent_policy == "heuristic_balanced":
|
||||
self.opponent_policy_id = 1
|
||||
elif opponent_policy == "self_play_league":
|
||||
self.opponent_policy_id = 2
|
||||
@@ -205,7 +205,7 @@ cdef class CythonDeepCFRTraverser:
|
||||
)
|
||||
else:
|
||||
raise ValueError(
|
||||
"opponent_policy must be 'network', 'safe_heuristic', 'self_play_league', or 'average_strategy'"
|
||||
"opponent_policy must be 'network', 'heuristic_balanced', 'self_play_league', or 'average_strategy'"
|
||||
)
|
||||
if all_negative_fallback == "uniform":
|
||||
self.all_negative_fallback_id = 0
|
||||
@@ -225,8 +225,8 @@ cdef class CythonDeepCFRTraverser:
|
||||
self.self_play_recent_window = max(0, self_play_recent_window)
|
||||
self.active_self_play_bucket = 0
|
||||
self.active_self_play_networks = None
|
||||
self.safe_heuristic_opponent_bot = (
|
||||
SafeHeuristicBot()
|
||||
self.heuristic_opponent_bot = (
|
||||
HeuristicBot()
|
||||
if self.opponent_policy_id == 1 or self.self_play_anchor_probability > 0.0
|
||||
else None
|
||||
)
|
||||
@@ -630,9 +630,9 @@ cdef class CythonDeepCFRTraverser:
|
||||
if player == traverser or self.opponent_policy_id == 0:
|
||||
return -1
|
||||
if self.opponent_policy_id == 1:
|
||||
if self.safe_heuristic_opponent_bot is None:
|
||||
self.safe_heuristic_opponent_bot = SafeHeuristicBot()
|
||||
return int(self.safe_heuristic_opponent_bot.act(state))
|
||||
if self.heuristic_opponent_bot is None:
|
||||
self.heuristic_opponent_bot = HeuristicBot()
|
||||
return int(self.heuristic_opponent_bot.act(state))
|
||||
if self.opponent_policy_id == 3:
|
||||
self._policy_from_strategy_network(state, player, legal, policy)
|
||||
for i in range(self.action_size):
|
||||
@@ -649,9 +649,9 @@ cdef class CythonDeepCFRTraverser:
|
||||
if bucket == 0:
|
||||
return -1
|
||||
if bucket == 3:
|
||||
if self.safe_heuristic_opponent_bot is None:
|
||||
self.safe_heuristic_opponent_bot = SafeHeuristicBot()
|
||||
return int(self.safe_heuristic_opponent_bot.act(state))
|
||||
if self.heuristic_opponent_bot is None:
|
||||
self.heuristic_opponent_bot = HeuristicBot()
|
||||
return int(self.heuristic_opponent_bot.act(state))
|
||||
networks = self.active_self_play_networks
|
||||
if networks is None:
|
||||
return -1
|
||||
@@ -839,8 +839,8 @@ cdef class CythonDeepCFRTraverser:
|
||||
if swapped_indices == NULL:
|
||||
raise MemoryError()
|
||||
while not state.terminal and steps < self.cutoff_rollout_max_steps:
|
||||
if self.safe_heuristic_rollout_bot is not None:
|
||||
local_action = int(self.safe_heuristic_rollout_bot.act(state))
|
||||
if self.heuristic_rollout_bot is not None:
|
||||
local_action = int(self.heuristic_rollout_bot.act(state))
|
||||
unified_action = self._to_unified_action_c(state, local_action)
|
||||
else:
|
||||
count = state._unified_legal_actions_c(actions)
|
||||
|
||||
@@ -277,7 +277,7 @@ def evaluate_policy(
|
||||
|
||||
def main(argv: list[str] | None = None) -> None:
|
||||
parser = argparse.ArgumentParser(description="Evaluate Lost Cities classic bots.")
|
||||
parser.add_argument("--bot0", default="safe-heuristic", choices=available_bot_names())
|
||||
parser.add_argument("--bot0", default="heuristic-balanced", choices=available_bot_names())
|
||||
parser.add_argument("--bot1", default="random", choices=available_bot_names())
|
||||
parser.add_argument("--games", type=int, default=100)
|
||||
parser.add_argument("--seed", type=int, default=1)
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
from coolrl_lost_cities.games.classic.game import Card, GameState, LostCitiesConfig
|
||||
|
||||
from coolrl_lost_cities.games.classic.bots import (
|
||||
HeuristicBot,
|
||||
LostCitiesPolicy,
|
||||
RandomBot,
|
||||
SafeHeuristicBot,
|
||||
)
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic import draw_from_discard_action
|
||||
from coolrl_lost_cities.games.classic.evaluation import play_game_for_evaluation
|
||||
@@ -19,13 +19,13 @@ def _expeditions(config: LostCitiesConfig) -> list[list[list[Card]]]:
|
||||
|
||||
def test_builtin_bots_implement_lost_cities_policy() -> None:
|
||||
assert isinstance(RandomBot(1), LostCitiesPolicy)
|
||||
assert isinstance(SafeHeuristicBot(), LostCitiesPolicy)
|
||||
assert isinstance(HeuristicBot(), LostCitiesPolicy)
|
||||
|
||||
|
||||
def test_safe_heuristic_mirror_match_finishes() -> None:
|
||||
def test_heuristic_mirror_match_finishes() -> None:
|
||||
state, result = play_game_for_evaluation(
|
||||
SafeHeuristicBot(),
|
||||
SafeHeuristicBot(),
|
||||
HeuristicBot(),
|
||||
HeuristicBot(),
|
||||
LostCitiesConfig(n_colors=3, n_ranks=5, n_handshakes=1, hand_size=5),
|
||||
seed=2000,
|
||||
max_steps=200,
|
||||
@@ -34,9 +34,9 @@ def test_safe_heuristic_mirror_match_finishes() -> None:
|
||||
assert result.timed_out is False
|
||||
|
||||
|
||||
def test_safe_heuristic_opponent_value_ignores_hidden_hand() -> None:
|
||||
def test_heuristic_opponent_value_ignores_hidden_hand() -> None:
|
||||
config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
discard_card = Card(color=0, rank=6)
|
||||
|
||||
expeditions_a = _expeditions(config)
|
||||
@@ -76,9 +76,9 @@ def test_safe_heuristic_opponent_value_ignores_hidden_hand() -> None:
|
||||
assert value_a == value_b
|
||||
|
||||
|
||||
def test_safe_heuristic_started_expedition_value_ignores_invalid_lower_followup() -> None:
|
||||
def test_heuristic_started_expedition_value_ignores_invalid_lower_followup() -> None:
|
||||
config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
high_card = Card(color=0, rank=8)
|
||||
|
||||
base_expeditions = _expeditions(config)
|
||||
@@ -115,9 +115,9 @@ def test_safe_heuristic_started_expedition_value_ignores_invalid_lower_followup(
|
||||
assert lower_followup_value == base_value
|
||||
|
||||
|
||||
def test_safe_heuristic_draws_playable_discard_instead_of_deck() -> None:
|
||||
def test_heuristic_draws_playable_discard_instead_of_deck() -> None:
|
||||
config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=3)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
|
||||
expeditions = _expeditions(config)
|
||||
expeditions[0][0] = [Card(color=0, rank=4)]
|
||||
@@ -132,9 +132,9 @@ def test_safe_heuristic_draws_playable_discard_instead_of_deck() -> None:
|
||||
assert bot._act_draw(state) == draw_from_discard_action(0)
|
||||
|
||||
|
||||
def test_safe_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None:
|
||||
def test_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None:
|
||||
config = LostCitiesConfig(n_colors=2, n_ranks=8, hand_size=4)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
|
||||
expeditions = _expeditions(config)
|
||||
expeditions[0][1] = [Card(color=1, rank=8)]
|
||||
@@ -158,9 +158,9 @@ def test_safe_heuristic_can_draw_discard_to_deny_opponent_when_losing() -> None:
|
||||
assert bot._act_draw(state) == draw_from_discard_action(0)
|
||||
|
||||
|
||||
def test_safe_heuristic_classic_self_play_opens_expeditions() -> None:
|
||||
def test_heuristic_classic_self_play_opens_expeditions() -> None:
|
||||
state = GameState.new_game(LostCitiesConfig(), seed=1)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
player0_actions: list[int] = []
|
||||
|
||||
for _ in range(60):
|
||||
@@ -182,9 +182,9 @@ def test_safe_heuristic_classic_self_play_opens_expeditions() -> None:
|
||||
assert any(state.expeditions[0][color] for color in range(state.config.n_colors))
|
||||
|
||||
|
||||
def test_safe_heuristic_avoids_opening_weak_fifth_color() -> None:
|
||||
def test_heuristic_avoids_opening_weak_fifth_color() -> None:
|
||||
config = LostCitiesConfig(n_colors=5, n_ranks=8, hand_size=8)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
expeditions = _expeditions(config)
|
||||
expeditions[0][0] = [Card(color=0, rank=4)]
|
||||
expeditions[0][1] = [Card(color=1, rank=4)]
|
||||
@@ -211,9 +211,9 @@ def test_safe_heuristic_avoids_opening_weak_fifth_color() -> None:
|
||||
)
|
||||
|
||||
|
||||
def test_safe_heuristic_prefers_followup_on_started_expedition() -> None:
|
||||
def test_heuristic_prefers_followup_on_started_expedition() -> None:
|
||||
config = LostCitiesConfig(n_colors=3, n_ranks=8, hand_size=5)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
expeditions = _expeditions(config)
|
||||
expeditions[0][0] = [Card(color=0, rank=4)]
|
||||
state = make_state(
|
||||
@@ -233,9 +233,9 @@ def test_safe_heuristic_prefers_followup_on_started_expedition() -> None:
|
||||
assert chosen.color == 0
|
||||
|
||||
|
||||
def test_safe_heuristic_avoids_unopened_discard_draw_after_four_opens() -> None:
|
||||
def test_heuristic_avoids_unopened_discard_draw_after_four_opens() -> None:
|
||||
config = LostCitiesConfig(n_colors=5, n_ranks=8, hand_size=8)
|
||||
bot = SafeHeuristicBot()
|
||||
bot = HeuristicBot()
|
||||
expeditions = _expeditions(config)
|
||||
expeditions[0][0] = [Card(color=0, rank=4)]
|
||||
expeditions[0][1] = [Card(color=1, rank=4)]
|
||||
|
||||
@@ -3,13 +3,13 @@ from __future__ import annotations
|
||||
from coolrl_lost_cities.games.classic.game import LostCitiesConfig
|
||||
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.imitation import (
|
||||
collect_safe_heuristic_samples,
|
||||
collect_heuristic_samples,
|
||||
new_pretrained_strategy_network,
|
||||
)
|
||||
|
||||
|
||||
def test_collect_safe_heuristic_samples_shapes() -> None:
|
||||
x, y, legal = collect_safe_heuristic_samples(LostCitiesConfig(seed=61), games=1, seed=61)
|
||||
def test_collect_heuristic_samples_shapes() -> None:
|
||||
x, y, legal = collect_heuristic_samples(LostCitiesConfig(seed=61), games=1, seed=61)
|
||||
|
||||
assert len(x) == len(y) == len(legal)
|
||||
assert x.ndim == 2
|
||||
|
||||
@@ -1494,3 +1494,83 @@ def test_interleaved_regret_matching_no_fallback_unchanged_by_mode() -> None:
|
||||
assert fallback_a is False
|
||||
assert np.allclose(policy_uniform, policy_argmax)
|
||||
assert np.allclose(policy_uniform.sum(), 1.0)
|
||||
|
||||
|
||||
def test_evaluation_opponents_for_iteration_core_only_when_extended_disabled() -> None:
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig
|
||||
|
||||
cfg = EvaluationConfig(
|
||||
eval_every=5,
|
||||
opponents=("random", "discard_only", "heuristic_cautious"),
|
||||
extended_eval_every=0,
|
||||
extended_opponents=("heuristic_balanced",),
|
||||
)
|
||||
assert cfg.opponents_for_iteration(0) == ()
|
||||
assert cfg.opponents_for_iteration(3) == ()
|
||||
assert cfg.opponents_for_iteration(5) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
)
|
||||
assert cfg.opponents_for_iteration(50) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
)
|
||||
|
||||
|
||||
def test_evaluation_opponents_for_iteration_extends_on_extended_cadence() -> None:
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig
|
||||
|
||||
cfg = EvaluationConfig(
|
||||
eval_every=5,
|
||||
opponents=("random", "discard_only", "heuristic_cautious"),
|
||||
extended_eval_every=50,
|
||||
extended_opponents=(
|
||||
"heuristic_balanced",
|
||||
"heuristic_aggressive",
|
||||
"heuristic_noisy",
|
||||
),
|
||||
)
|
||||
assert cfg.opponents_for_iteration(5) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
)
|
||||
assert cfg.opponents_for_iteration(45) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
)
|
||||
assert cfg.opponents_for_iteration(50) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
"heuristic_balanced",
|
||||
"heuristic_aggressive",
|
||||
"heuristic_noisy",
|
||||
)
|
||||
assert cfg.opponents_for_iteration(100) == (
|
||||
"random",
|
||||
"discard_only",
|
||||
"heuristic_cautious",
|
||||
"heuristic_balanced",
|
||||
"heuristic_aggressive",
|
||||
"heuristic_noisy",
|
||||
)
|
||||
|
||||
|
||||
def test_evaluation_opponents_for_iteration_dedupes_overlap() -> None:
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.config import EvaluationConfig
|
||||
|
||||
cfg = EvaluationConfig(
|
||||
eval_every=5,
|
||||
opponents=("random", "heuristic_cautious"),
|
||||
extended_eval_every=10,
|
||||
extended_opponents=("heuristic_cautious", "heuristic_balanced"),
|
||||
)
|
||||
assert cfg.opponents_for_iteration(10) == (
|
||||
"random",
|
||||
"heuristic_cautious",
|
||||
"heuristic_balanced",
|
||||
)
|
||||
|
||||
@@ -13,7 +13,7 @@ def test_play_game_for_evaluation_finishes_small_match() -> None:
|
||||
|
||||
state, result = play_game_for_evaluation(
|
||||
build_bot("random", seed=1),
|
||||
build_bot("passive-discard", seed=2),
|
||||
build_bot("discard-only", seed=2),
|
||||
config,
|
||||
seed=3,
|
||||
max_steps=200,
|
||||
@@ -30,7 +30,7 @@ def test_play_match_alternates_seats_and_reports_rates() -> None:
|
||||
|
||||
result = play_match(
|
||||
make_policy_factory("random"),
|
||||
make_policy_factory("passive-discard"),
|
||||
make_policy_factory("discard-only"),
|
||||
config,
|
||||
games=4,
|
||||
seed=10,
|
||||
@@ -50,7 +50,7 @@ def test_evaluation_cli_smoke_json(capsys) -> None:
|
||||
"--bot0",
|
||||
"random",
|
||||
"--bot1",
|
||||
"passive-discard",
|
||||
"discard-only",
|
||||
"--games",
|
||||
"2",
|
||||
"--seed",
|
||||
|
||||
@@ -43,10 +43,10 @@ def test_classic_package_exports_bot_registry_helpers() -> None:
|
||||
def test_classic_bot_registry_accepts_reproduction_opponent_names() -> None:
|
||||
for name in [
|
||||
"random",
|
||||
"passive_discard",
|
||||
"safe_heuristic",
|
||||
"safe_heuristic_loose",
|
||||
"safe_heuristic_strict",
|
||||
"noisy_safe",
|
||||
"discard_only",
|
||||
"heuristic_balanced",
|
||||
"heuristic_aggressive",
|
||||
"heuristic_cautious",
|
||||
"heuristic_noisy",
|
||||
]:
|
||||
assert isinstance(classic.build_bot(name, seed=1), classic.LostCitiesPolicy)
|
||||
|
||||
@@ -11,7 +11,7 @@ def test_gui_argparser_accepts_classic_options() -> None:
|
||||
"--mode",
|
||||
"pvc",
|
||||
"--bot",
|
||||
"safe-heuristic",
|
||||
"heuristic-balanced",
|
||||
"--seed",
|
||||
"7",
|
||||
"--width",
|
||||
@@ -22,7 +22,7 @@ def test_gui_argparser_accepts_classic_options() -> None:
|
||||
)
|
||||
|
||||
assert args.mode == "pvc"
|
||||
assert args.bot == "safe-heuristic"
|
||||
assert args.bot == "heuristic-balanced"
|
||||
assert args.seed == 7
|
||||
assert args.width == 1024
|
||||
assert args.height == 768
|
||||
|
||||
@@ -3,19 +3,19 @@ from __future__ import annotations
|
||||
import pytest
|
||||
from coolrl_lost_cities.games.classic.game import GameState, LostCitiesConfig
|
||||
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic import SafeHeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic import HeuristicBot
|
||||
from coolrl_lost_cities.games.classic.bots.heuristic_py import (
|
||||
SafeHeuristicBot as PythonSafeHeuristicBot,
|
||||
HeuristicBot as PythonHeuristicBot,
|
||||
)
|
||||
from coolrl_lost_cities.games.classic.bots.registry import (
|
||||
LOOSE_SAFE_HEURISTIC_PARAMS,
|
||||
STRICT_SAFE_HEURISTIC_PARAMS,
|
||||
AGGRESSIVE_HEURISTIC_PARAMS,
|
||||
CAUTIOUS_HEURISTIC_PARAMS,
|
||||
)
|
||||
|
||||
VARIANTS = (
|
||||
("default", None),
|
||||
("loose", LOOSE_SAFE_HEURISTIC_PARAMS),
|
||||
("strict", STRICT_SAFE_HEURISTIC_PARAMS),
|
||||
("loose", AGGRESSIVE_HEURISTIC_PARAMS),
|
||||
("strict", CAUTIOUS_HEURISTIC_PARAMS),
|
||||
)
|
||||
|
||||
CONFIGS = (
|
||||
@@ -28,14 +28,14 @@ CONFIGS = (
|
||||
@pytest.mark.parametrize(("variant_name", "params"), VARIANTS)
|
||||
@pytest.mark.parametrize("config", CONFIGS)
|
||||
@pytest.mark.parametrize("seed", range(2))
|
||||
def test_cython_safe_heuristic_matches_python_action_sequence(
|
||||
def test_cython_heuristic_matches_python_action_sequence(
|
||||
variant_name: str,
|
||||
params,
|
||||
config: LostCitiesConfig,
|
||||
seed: int,
|
||||
) -> None:
|
||||
py_bot = PythonSafeHeuristicBot(params)
|
||||
cy_bot = SafeHeuristicBot(params)
|
||||
py_bot = PythonHeuristicBot(params)
|
||||
cy_bot = HeuristicBot(params)
|
||||
py_state = GameState.new_game(config, seed=seed)
|
||||
cy_state = GameState.new_game(config, seed=seed)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user