Rename bot family, curate analyze plots, tier evaluation cadence

Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.

Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic       -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe           -> heuristic_noisy
- passive_discard      -> discard_only

Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.

Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
  heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
  avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
  bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
  win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
  recoverable score) and analysis_07_calibration.png (calibration gap,
  recoverable mean) - both relied on the heuristic recoverable_score
  classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
  calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
  a specific opponent per panel without restructuring plot_section.

Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
  eval_every and appends extended_opponents (de-duplicated) when
  iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
  heuristic_cautious) every 5 iterations and 3 extended opponents
  (heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
  iterations. random is the floor sanity. discard_only is the
  zero-pit detector / absolute-score reference (its score is always 0,
  so eval/discard_only/avg_score_diff0 directly equals the model's raw
  average score). heuristic_cautious is the ceiling and the
  archive-comparable benchmark used in the prior diagnostic sections.

Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).

Documented in docs/plans/deep-cfr-selectivity.md section 9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-10 15:32:55 +09:00
co-authored by Claude Opus 4.7
parent 0457efdf29
commit 004b913a7b
41 changed files with 441 additions and 324 deletions
+1 -1
View File
@@ -48,7 +48,7 @@ src/coolrl_lost_cities/games/classic/deep_cfr/
traversal_stats.py — structured traversal diagnostic metrics
# Auxiliary training modes
imitation.py — safe-heuristic imitation pretraining
imitation.py — heuristic imitation pretraining
policy_gradient.py — policy-gradient fine-tuning
```
@@ -59,23 +59,23 @@ Evaluation performance peaks early and then degrades.
| Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff |
| --- | ---: | ---: | ---: | ---: |
| random | 0.808 | 0.665 | +35.1 | +12.4 |
| passive_discard | 0.026 | 0.012 | -30.8 | -43.0 |
| safe_heuristic | 0.070 | 0.011 | -68.7 | -94.2 |
| safe_heuristic_loose | 0.078 | 0.014 | -69.3 | -95.0 |
| safe_heuristic_strict | 0.072 | 0.011 | -57.6 | -85.0 |
| noisy_safe | 0.120 | 0.033 | -49.4 | -78.8 |
| discard_only | 0.026 | 0.012 | -30.8 | -43.0 |
| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 |
| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 |
| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 |
| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 |
The best strict-heuristic point appears around iteration 70:
- `eval/safe_heuristic_strict/win_rate0=0.15`
- `eval/safe_heuristic_strict/avg_score_diff0=-41.09`
- `eval/heuristic_cautious/win_rate0=0.15`
- `eval/heuristic_cautious/avg_score_diff0=-41.09`
The final point at iteration 2000 is worse:
- `eval/safe_heuristic_strict/win_rate0=0.02`
- `eval/safe_heuristic_strict/avg_score_diff0=-81.64`
- `eval/heuristic_cautious/win_rate0=0.02`
- `eval/heuristic_cautious/avg_score_diff0=-81.64`
Selectivity does not meaningfully emerge. Against `safe_heuristic_strict`:
Selectivity does not meaningfully emerge. Against `heuristic_cautious`:
| Metric | First 20 evals | Last 20 evals | Interpretation |
| --- | ---: | ---: | --- |
+1 -1
View File
@@ -19,7 +19,7 @@ The primary bottleneck in CUDA-based evaluation is the overhead of launching sma
A critical refinement in the batched implementation was the handling of policy entropy. Initial versions that calculated entropy per-row on the CPU incurred significant synchronization penalties because each row required a GPU-to-CPU transfer. Moving the entropy calculation into the Torch post-processing pipeline—specifically calculating it directly on the `probs_tensor` (line 220)—ensures that the computation remains on the device and only the final results are transferred back to the host in bulk.
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `safe_heuristic_strict`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `heuristic_cautious`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
## Practical implication
+1 -1
View File
@@ -37,7 +37,7 @@ On CPU, the `network / turn` cost is approximately **0.074 ms**. On CUDA, this r
### Secondary Bottlenecks
- **Post-processing:** Moving tensors back to CPU (`.cpu().numpy()`) and calculating entropy adds measurable overhead on CUDA that is largely absent on CPU.
- **Opponent Logic:** Heuristic opponents (e.g., `safe_heuristic`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
- **Opponent Logic:** Heuristic opponents (e.g., `heuristic_balanced`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
## Practical Implications
+2 -2
View File
@@ -45,7 +45,7 @@ Two temporary runs used:
- `run.max_iterations=3`
- `evaluation.eval_every=1`
- `evaluation.games=20`
- `evaluation.opponents=[random,safe_heuristic_strict]`
- `evaluation.opponents=[random,heuristic_cautious]`
- W&B disabled
Runs:
@@ -55,7 +55,7 @@ Runs:
Core metrics matched exactly:
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` |
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` |
| --- | ---: | ---: | ---: | ---: | ---: |
| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
+1 -1
View File
@@ -56,7 +56,7 @@ registered classic bots (`evaluate.py`), training CLI and evaluation CLI
(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
`runtime_progress.json` / `train.log` run artifacts, self-play league with
snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`),
heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`),
and policy-gradient fine-tuning (`policy_gradient.py`).
As of `ad0be89`, the package also includes `inference_server.py`,
+1 -1
View File
@@ -31,7 +31,7 @@ Decision criteria for going forward:
### 2026-05-07 — Safe heuristic single-thread parity (criterion 1)
Path: `experiments/julia_safe_heuristic/`.
Path: `experiments/julia_heuristic/`.
1,838 snapshots in 157.471 ms median (~85.6 μs/call). Action-sequence
parity vs Python. Same order of magnitude as the Cython port of the
+1 -1
View File
@@ -28,7 +28,7 @@ Deep CFR은 최종적으로 average strategy가 수렴 대상. current advantage
Visible-score predictor의 ceiling은 있을 수 있음. 다만 selectivity 자체의 ceiling은 아님. selectivity는 visible-score prediction 말고 option value, irreversible cost 회피, opponent dynamics 대응 등 다른 경로로도 emerge 가능.
**Self-play attractor 가설**
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. safe_heuristic이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. heuristic_balanced이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
**Lost Cities NE 자체가 5-color 가설**
이론적 가능성 0 아님. self-play가 발견한 게 진짜 NE면 transition 영원히 안 옴. 검증 가능한 형태로는 tabular oracle 또는 BR 진단.