Rename bot family, curate analyze plots, tier evaluation cadence
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.
Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe -> heuristic_noisy
- passive_discard -> discard_only
Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.
Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
recoverable score) and analysis_07_calibration.png (calibration gap,
recoverable mean) - both relied on the heuristic recoverable_score
classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
a specific opponent per panel without restructuring plot_section.
Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
eval_every and appends extended_opponents (de-duplicated) when
iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
heuristic_cautious) every 5 iterations and 3 extended opponents
(heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
iterations. random is the floor sanity. discard_only is the
zero-pit detector / absolute-score reference (its score is always 0,
so eval/discard_only/avg_score_diff0 directly equals the model's raw
average score). heuristic_cautious is the ceiling and the
archive-comparable benchmark used in the prior diagnostic sections.
Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).
Documented in docs/plans/deep-cfr-selectivity.md section 9.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -48,7 +48,7 @@ src/coolrl_lost_cities/games/classic/deep_cfr/
|
||||
traversal_stats.py — structured traversal diagnostic metrics
|
||||
|
||||
# Auxiliary training modes
|
||||
imitation.py — safe-heuristic imitation pretraining
|
||||
imitation.py — heuristic imitation pretraining
|
||||
policy_gradient.py — policy-gradient fine-tuning
|
||||
```
|
||||
|
||||
|
||||
@@ -59,23 +59,23 @@ Evaluation performance peaks early and then degrades.
|
||||
| Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| random | 0.808 | 0.665 | +35.1 | +12.4 |
|
||||
| passive_discard | 0.026 | 0.012 | -30.8 | -43.0 |
|
||||
| safe_heuristic | 0.070 | 0.011 | -68.7 | -94.2 |
|
||||
| safe_heuristic_loose | 0.078 | 0.014 | -69.3 | -95.0 |
|
||||
| safe_heuristic_strict | 0.072 | 0.011 | -57.6 | -85.0 |
|
||||
| noisy_safe | 0.120 | 0.033 | -49.4 | -78.8 |
|
||||
| discard_only | 0.026 | 0.012 | -30.8 | -43.0 |
|
||||
| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 |
|
||||
| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 |
|
||||
| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 |
|
||||
| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 |
|
||||
|
||||
The best strict-heuristic point appears around iteration 70:
|
||||
|
||||
- `eval/safe_heuristic_strict/win_rate0=0.15`
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0=-41.09`
|
||||
- `eval/heuristic_cautious/win_rate0=0.15`
|
||||
- `eval/heuristic_cautious/avg_score_diff0=-41.09`
|
||||
|
||||
The final point at iteration 2000 is worse:
|
||||
|
||||
- `eval/safe_heuristic_strict/win_rate0=0.02`
|
||||
- `eval/safe_heuristic_strict/avg_score_diff0=-81.64`
|
||||
- `eval/heuristic_cautious/win_rate0=0.02`
|
||||
- `eval/heuristic_cautious/avg_score_diff0=-81.64`
|
||||
|
||||
Selectivity does not meaningfully emerge. Against `safe_heuristic_strict`:
|
||||
Selectivity does not meaningfully emerge. Against `heuristic_cautious`:
|
||||
|
||||
| Metric | First 20 evals | Last 20 evals | Interpretation |
|
||||
| --- | ---: | ---: | --- |
|
||||
|
||||
@@ -19,7 +19,7 @@ The primary bottleneck in CUDA-based evaluation is the overhead of launching sma
|
||||
|
||||
A critical refinement in the batched implementation was the handling of policy entropy. Initial versions that calculated entropy per-row on the CPU incurred significant synchronization penalties because each row required a GPU-to-CPU transfer. Moving the entropy calculation into the Torch post-processing pipeline—specifically calculating it directly on the `probs_tensor` (line 220)—ensures that the computation remains on the device and only the final results are transferred back to the host in bulk.
|
||||
|
||||
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `safe_heuristic_strict`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
|
||||
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `heuristic_cautious`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
|
||||
|
||||
## Practical implication
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ On CPU, the `network / turn` cost is approximately **0.074 ms**. On CUDA, this r
|
||||
|
||||
### Secondary Bottlenecks
|
||||
- **Post-processing:** Moving tensors back to CPU (`.cpu().numpy()`) and calculating entropy adds measurable overhead on CUDA that is largely absent on CPU.
|
||||
- **Opponent Logic:** Heuristic opponents (e.g., `safe_heuristic`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
|
||||
- **Opponent Logic:** Heuristic opponents (e.g., `heuristic_balanced`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
|
||||
|
||||
## Practical Implications
|
||||
|
||||
|
||||
@@ -45,7 +45,7 @@ Two temporary runs used:
|
||||
- `run.max_iterations=3`
|
||||
- `evaluation.eval_every=1`
|
||||
- `evaluation.games=20`
|
||||
- `evaluation.opponents=[random,safe_heuristic_strict]`
|
||||
- `evaluation.opponents=[random,heuristic_cautious]`
|
||||
- W&B disabled
|
||||
|
||||
Runs:
|
||||
@@ -55,7 +55,7 @@ Runs:
|
||||
|
||||
Core metrics matched exactly:
|
||||
|
||||
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` |
|
||||
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
|
||||
| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
|
||||
|
||||
@@ -56,7 +56,7 @@ registered classic bots (`evaluate.py`), training CLI and evaluation CLI
|
||||
(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
|
||||
`runtime_progress.json` / `train.log` run artifacts, self-play league with
|
||||
snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
|
||||
heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`),
|
||||
heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`),
|
||||
and policy-gradient fine-tuning (`policy_gradient.py`).
|
||||
|
||||
As of `ad0be89`, the package also includes `inference_server.py`,
|
||||
|
||||
@@ -31,7 +31,7 @@ Decision criteria for going forward:
|
||||
|
||||
### 2026-05-07 — Safe heuristic single-thread parity (criterion 1)
|
||||
|
||||
Path: `experiments/julia_safe_heuristic/`.
|
||||
Path: `experiments/julia_heuristic/`.
|
||||
|
||||
1,838 snapshots in 157.471 ms median (~85.6 μs/call). Action-sequence
|
||||
parity vs Python. Same order of magnitude as the Cython port of the
|
||||
|
||||
@@ -28,7 +28,7 @@ Deep CFR은 최종적으로 average strategy가 수렴 대상. current advantage
|
||||
Visible-score predictor의 ceiling은 있을 수 있음. 다만 selectivity 자체의 ceiling은 아님. selectivity는 visible-score prediction 말고 option value, irreversible cost 회피, opponent dynamics 대응 등 다른 경로로도 emerge 가능.
|
||||
|
||||
**Self-play attractor 가설**
|
||||
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. safe_heuristic이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
|
||||
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. heuristic_balanced이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
|
||||
|
||||
**Lost Cities NE 자체가 5-color 가설**
|
||||
이론적 가능성 0 아님. self-play가 발견한 게 진짜 NE면 transition 영원히 안 옴. 검증 가능한 형태로는 tabular oracle 또는 BR 진단.
|
||||
|
||||
Reference in New Issue
Block a user