coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Rename bot family, curate analyze plots, tier evaluation cadence

Browse Source 
Three coordinated hygiene changes; none target the diagnosed
selection-bias bottleneck. They make the codebase honestly reflect the
pure-self-play stance and reduce dashboard noise.

Bot rename (drop the unhelpful safe_ prefix; suffixes describe behaviour):
- safe_heuristic_loose -> heuristic_aggressive
- safe_heuristic       -> heuristic_balanced
- safe_heuristic_strict -> heuristic_cautious
- noisy_safe           -> heuristic_noisy
- passive_discard      -> discard_only

Class renames in bots/: SafeHeuristicBot -> HeuristicBot,
SafeHeuristicParams -> HeuristicParams, PassiveDiscardBot -> DiscardOnlyBot,
plus loose/strict parameter constants. Backwards compatibility was dropped
intentionally per user instruction; no aliases. Active configs, docs,
scripts, tests updated. Archive directories (configs/archive,
docs/archive, runs/archive) left intact and may still reference old
names per their read-only policy. The src/.../bots/passive.py module was
renamed to discard_only.py via git mv.

Analyze plot curation (deep_cfr/analyze.py):
- Added analysis_00_core.png as the canonical daily dashboard with 10
  heuristic-free metrics (loss/{advantage,strategy}; vs heuristic_cautious:
  avg_score_diff0, win_rate0, avg_opened_colors, positive_expedition_rate,
  bonus_expedition_rate, score_per_opened_color, policy_entropy; vs random:
  win_rate0).
- Removed analysis_05_open_quality.png (bad/weak/good open rates,
  recoverable score) and analysis_07_calibration.png (calibration gap,
  recoverable mean) - both relied on the heuristic recoverable_score
  classifier already dropped from inputs.
- Removed SELECTIVITY_PLOTS and plot_selectivity (heuristic-laden).
- SUMMARY_EVAL_METRICS no longer includes bad_open_rate or
  calibration_gap.
- PlotSpec gained an opponents allowlist so the new core section can pin
  a specific opponent per panel without restructuring plot_section.

Tiered evaluation cadence (EvaluationConfig):
- Added extended_opponents and extended_eval_every (default 0 = disabled).
- opponents_for_iteration(iteration) returns the core list every
  eval_every and appends extended_opponents (de-duplicated) when
  iteration is also a multiple of extended_eval_every.
- default.yaml now uses 3 core opponents (random, discard_only,
  heuristic_cautious) every 5 iterations and 3 extended opponents
  (heuristic_balanced, heuristic_aggressive, heuristic_noisy) every 50
  iterations. random is the floor sanity. discard_only is the
  zero-pit detector / absolute-score reference (its score is always 0,
  so eval/discard_only/avg_score_diff0 directly equals the model's raw
  average score). heuristic_cautious is the ceiling and the
  archive-comparable benchmark used in the prior diagnostic sections.

Net eval cost reduction: roughly 50% (3 opponents x every 5 iter, plus
6 opponents x every 50 iter, vs the prior 6 x every 5).

Documented in docs/plans/deep-cfr-selectivity.md section 9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
coolguy
2026-05-10 15:32:55 +09:00
co-authored by Claude Opus 4.7
parent 0457efdf29
commit 004b913a7b
41 changed files with 441 additions and 324 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/performance.md
+10 -10
View File
@@ -182,7 +182,7 @@ eval_<opponent>_<metric>
For that run, iteration 75 had `evaluation_seconds = 18.29s`. Evaluation was
parallelized by opponent, so per-opponent `elapsed_seconds` values overlap and
must not be summed as wall-clock time. The slow safe-heuristic opponents
must not be summed as wall-clock time. The slow heuristic opponents
dominated the eval wall-clock.
Representative per-opponent breakdown:
@@ -190,13 +190,13 @@ Representative per-opponent breakdown:
| Opponent | Elapsed | Network | Postprocess | Opponent act |
| --- | ---: | ---: | ---: | ---: |
| `random` | 0.57s | 0.18s | 0.25s | 0.06s |
| `passive_discard` | 0.36s | 0.13s | 0.17s | 0.00s |
| `safe_heuristic` | 14.54s | 2.65s | 3.34s | 7.57s |
| `safe_heuristic_loose` | 11.20s | 2.55s | 3.30s | 4.63s |
| `safe_heuristic_strict` | 15.94s | 2.51s | 3.14s | 9.22s |
| `noisy_safe` | 1.68s | 0.40s | 0.58s | 0.57s |
| `discard_only` | 0.36s | 0.13s | 0.17s | 0.00s |
| `heuristic_balanced` | 14.54s | 2.65s | 3.34s | 7.57s |
| `heuristic_aggressive` | 11.20s | 2.55s | 3.30s | 4.63s |
| `heuristic_cautious` | 15.94s | 2.51s | 3.14s | 9.22s |
| `heuristic_noisy` | 1.68s | 0.40s | 0.58s | 0.57s |
The important read is that safe-heuristic evaluation is not primarily GPU
The important read is that heuristic evaluation is not primarily GPU
network forward time. `opponent_act_seconds` and policy post-processing are
larger than `policy_network_seconds` for the slowest opponents.
@@ -236,7 +236,7 @@ The practical eval tuning levers are:
this would require a feature change.
4. Reduce frequent opponents.
The safe-heuristic opponents dominate wall-clock in the inspected run. For
The heuristic opponents dominate wall-clock in the inspected run. For
frequent checks, evaluate against one or two representative opponents and run
the full suite less often.
@@ -278,7 +278,7 @@ already implemented. It would replace or wrap the strategy-network forward pass
with a precompiled inference engine. Its maximum impact is bounded by
`policy_network_seconds`, not by total eval time.
In the inspected eval row, the slow safe-heuristic opponents spent about
In the inspected eval row, the slow heuristic opponents spent about
2.5-2.6s in policy-network forward but 4.6-9.2s in opponent action selection and
about 3.1-3.3s in policy post-processing. That means TensorRT could help eval,
especially for larger `evaluation.games`, but it is not expected to collapse the
@@ -319,7 +319,7 @@ Based on the current metrics, the more plausible performance work is:
but it requires changing traversal scheduling, not just swapping the network
backend.
6. For eval-heavy runs, optimize the safe-heuristic opponents and policy
6. For eval-heavy runs, optimize the heuristic opponents and policy
post-processing before assuming TensorRT is the main lever.
The inspected eval row shows those costs dominate the slowest opponents.
docs/plans/deep-cfr-selectivity.md
+79 -15
View File
@@ -17,7 +17,7 @@ first-open advantage target.
## Baseline symptoms
The 512x3 dense-eval baseline showed improving training losses, but the main
game-quality metrics against `safe_heuristic_strict` did not improve enough to
game-quality metrics against `heuristic_cautious` did not improve enough to
indicate a useful policy.
Observed pattern:
@@ -176,7 +176,7 @@ against the current policy's best non-open action from the same state.
`delta_open = value(force open) - value(best non-open)`.
Counterfactual summary against `safe_heuristic_strict`:
Counterfactual summary against `heuristic_cautious`:
| checkpoint | bucket | candidates | delta mean | delta median | delta positive | policy prob | selected rate |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -257,7 +257,7 @@ first-open sampling scanned the full replay memory and pushed iteration time
above 60 seconds. The indexed-memory version kept first-open sampling near
0.25 seconds per player at 4M advantage samples and completed 500 iterations.
Final `safe_heuristic_strict` comparison:
Final `heuristic_cautious` comparison:
| Run | Iter | Score diff | Win rate | Bad open | Score/opened |
| --- | ---: | ---: | ---: | ---: | ---: |
@@ -331,15 +331,15 @@ Result (2026-05-10):
- W&B: synced online to group `first-open-prior-v1` (run `teuh915r`)
- Commit: see HEAD at run start
`safe_heuristic_strict` at iter 200:
`heuristic_cautious` at iter 200:
| Run | Score diff | Win rate | Bad open | Score/opened |
| --- | ---: | ---: | ---: | ---: |
| baseline `confirm-eps-005-zero` (iter 200) | -40.01 | 0.12 | 0.893 | -6.25 |
| **A1 prior α=5.0 (iter 200)** | **-67.54** | **0.02** | **0.796** | **-8.85** |
Other opponents at iter 200: random +32.51 / 0.86, noisy_safe -71.52 / 0.07,
safe_heuristic -81.81 / 0.02, safe_heuristic_loose -81.24 / 0.05.
Other opponents at iter 200: random +32.51 / 0.86, heuristic_noisy -71.52 / 0.07,
heuristic_balanced -81.81 / 0.02, heuristic_aggressive -81.24 / 0.05.
Conclusion: **mixed result, net regression.** The prior did shift behavior
in the intended direction on one axis — `bad_open_rate` dropped from 0.893
@@ -382,7 +382,7 @@ post-action play (selection bias)?
Method: re-ran `analyze_first_open_counterfactual.py` on the
`confirm-eps-005-zero-512x3-det-500` baseline checkpoints (iter 200, iter
500), but with `--post-policy safe_heuristic_strict`. The opponent and
500), but with `--post-policy heuristic_cautious`. The opponent and
state-collection policy stayed the same; only the policy_player's actions
*after* the forced first action used the strong fixed bot.
@@ -434,7 +434,7 @@ Implications:
Candidate next directions (decision pending):
- (E1) Train with `cutoff_rollout_policy=safe_heuristic` instead of `random`.
- (E1) Train with `cutoff_rollout_policy=heuristic_balanced` instead of `random`.
Already a config option; gives leaf nodes stronger value estimates
during traversal. Trades some pure-self-play purity for a stronger
bootstrap signal. Cheap to test.
@@ -510,7 +510,7 @@ This combined with D1 means:
Updated next-step priorities:
- **(E1) `cutoff_rollout_policy=safe_heuristic` training ablation.**
- **(E1) `cutoff_rollout_policy=heuristic_balanced` training ablation.**
Strongest single lever: gives traversal leaves stronger value estimates
during training, which should ripple back to "open + follow up" signal.
Pure-self-play purity dented, but only at cutoff leaves.
@@ -635,7 +635,7 @@ Fix:
Also bumped `traversal.outcome_sampling_epsilon` in `default.yaml` from
0.2 to 0.05. The 200-iteration sweep (section 1) showed 0.05 produced the
best short-run safe_heuristic_strict score diff (-40.01 vs -57.87 for
best short-run heuristic_cautious score diff (-40.01 vs -57.87 for
0.20). All recent experimental runs already used 0.05; the default now
matches actual experimental practice.
@@ -643,7 +643,71 @@ These changes do not target the diagnosed selection-bias bottleneck. They
align config intent with actual scheduler behaviour and make the default
config reproduce known-best knob settings out of the box.
### 9. Short open-selectivity ablation
### 9. Naming, plot curation, and tiered eval cadence (2026-05-10)
Hygiene changes — none target the diagnosed selection-bias bottleneck,
but they make the codebase honestly reflect the pure-self-play stance
and reduce dashboard noise.
Bot family rename (drop the unhelpful `safe_` prefix; suffixes now
describe behaviour):
| Old | New |
| --- | --- |
| `safe_heuristic_loose` | `heuristic_aggressive` |
| `safe_heuristic` | `heuristic_balanced` |
| `safe_heuristic_strict` | `heuristic_cautious` |
| `noisy_safe` | `heuristic_noisy` |
| `passive_discard` | `discard_only` |
Class renames in `bots/`: `SafeHeuristicBot` → `HeuristicBot`,
`SafeHeuristicParams` → `HeuristicParams`, `PassiveDiscardBot` →
`DiscardOnlyBot`, plus the loose/strict parameter constants. Backwards
compatibility was dropped intentionally — no aliases. Active configs,
docs, scripts, tests updated; archive files (read-only by policy)
left intact and may still reference old names.
Analyze plot curation (`deep_cfr/analyze.py`):
- New `analysis_00_core.png` dashboard with 10 heuristic-free metrics
(loss/{advantage, strategy}; vs `heuristic_cautious`:
`avg_score_diff0`, `win_rate0`, `avg_opened_colors`,
`positive_expedition_rate`, `bonus_expedition_rate`,
`score_per_opened_color`, `policy_entropy`; vs `random`: `win_rate0`).
- Removed `analysis_05_open_quality.png` (bad/weak/good open rates,
recoverable score) and `analysis_07_calibration.png` (calibration gap,
recoverable mean) — both relied on the heuristic `recoverable_score`
classifier we already dropped from inputs.
- Removed `SELECTIVITY_PLOTS` and `plot_selectivity` (heuristic-laden).
- `SUMMARY_EVAL_METRICS` no longer includes `bad_open_rate` or
`calibration_gap`.
`PlotSpec` gained an optional `opponents` allowlist so the new core
section can pin a specific opponent per panel without restructuring the
existing `plot_section` plumbing.
Tiered evaluation cadence (`EvaluationConfig`):
- Added `extended_opponents: tuple[str, ...]` and `extended_eval_every:
int = 0`.
- Method `opponents_for_iteration(iteration)` returns the core list at
every `eval_every`, and appends `extended_opponents` (de-duplicated)
when `iteration` is also a multiple of `extended_eval_every`.
- `default.yaml` now uses 3 core opponents
(`random`, `discard_only`, `heuristic_cautious`) every 5 iterations
and 3 extended opponents (`heuristic_balanced`, `heuristic_aggressive`,
`heuristic_noisy`) every 50 iterations.
- `random` is the floor sanity. `discard_only` is the zero-pit
detector / absolute-score reference (its score is always 0, so
`eval/discard_only/avg_score_diff0` directly equals our model's
raw average score). `heuristic_cautious` is the ceiling and the
archive-comparable benchmark used in sections 1–6.
Net effect on ongoing eval cost: ~50% reduction (3 opponents × every
5 iter, plus 6 opponents × every 50 iter, vs the prior 6 opponents
× every 5).
### 10. Short open-selectivity ablation
Run a 200-300 iteration ablation only after the target audit identifies a
specific change. Candidate changes include:
@@ -656,10 +720,10 @@ specific change. Candidate changes include:
Primary metrics:
- `eval/safe_heuristic_strict/avg_score_diff0`
- `eval/safe_heuristic_strict/win_rate0`
- `eval/safe_heuristic_strict/bad_open_rate`
- `eval/safe_heuristic_strict/score_per_opened_color`
- `eval/heuristic_cautious/avg_score_diff0`
- `eval/heuristic_cautious/win_rate0`
- `eval/heuristic_cautious/bad_open_rate`
- `eval/heuristic_cautious/score_per_opened_color`
Do not promote to 500+ iterations unless bad-open rate and score/opened color
both improve without degrading score diff.
docs/plans/model_size_experiment.md
+6 -6
View File
@@ -39,7 +39,7 @@ The experiment must produce either a recommended new `network` config or a docum
## Success criteria
1. At least one tested config produces win-rate trajectories vs `safe_heuristic_strict` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
1. At least one tested config produces win-rate trajectories vs `heuristic_cautious` that are **clearly outside seed noise** compared to the current baseline at iteration 200 — OR a clear documented null result (no size in the tested range improves the curve).
2. `iteration_seconds`, `traversal_seconds`, `advantage_train_seconds`, `strategy_train_seconds`, and `policy_network_seconds` (eval) are captured for each tested size and written to `docs/performance.md`.
3. A recommended `network` config emerges from the data, OR the experiment documents why the current size should be kept, with specific rationale.
@@ -72,7 +72,7 @@ configs/deep_cfr/model-size-1536x8.yaml
- `traversal_seconds` — traversal phase.
- `advantage_train_seconds` — advantage network optimization.
- `strategy_train_seconds` — strategy network optimization.
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `safe_heuristic_strict`.
- At eval iterations {50, 100, 150, 200}: `eval/<opponent>/win_rate` for all opponents, with special attention to `heuristic_cautious`.
- At eval iterations: `eval/<opponent>/policy_network_seconds` — needed for the AMP/TRT prerequisite check.
- **Memory monitoring:** watch GPU VRAM during the 1024x6 and 1536x8 runs. If a run OOMs or VRAM > 20 GB, reduce `optimization.advantage_batch_size` and `optimization.strategy_batch_size` by half (1024 → 512) and note the change in the results table. Do not adjust traversal settings.
@@ -259,8 +259,8 @@ if non_eval:
eval_rows = {r['iteration']: r for r in rows if r.get('evaluation_seconds')}
for it in [50, 100, 150, 200]:
if it in eval_rows:
wr = eval_rows[it].get('eval/safe_heuristic_strict/win_rate', 'n/a')
print(f' iter={it} safe_heuristic_strict win_rate={wr}')
wr = eval_rows[it].get('eval/heuristic_cautious/win_rate', 'n/a')
print(f' iter={it} heuristic_cautious win_rate={wr}')
"
done
```
@@ -269,7 +269,7 @@ done
After the grid completes, append a date-stamped experiment subsection to `docs/performance.md` under the "Experiments" heading. The subsection must include:
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `safe_heuristic_strict` at {50, 100, 150, 200} for each config.
- A results table with `iteration_seconds` mean (non-eval) and win-rate vs `heuristic_cautious` at {50, 100, 150, 200} for each config.
- A `policy_network_seconds` column from eval rows — this is the key data for the AMP/compile/TRT prerequisite check.
- The recommendation that follows from the decision tree below.
@@ -302,7 +302,7 @@ Apply this logic after the grid completes:
### Branch A — a size unlocks the curve AND iter time is acceptable
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `safe_heuristic_strict` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
**Condition:** at least one config at or above 768x4 shows win-rate trajectories vs `heuristic_cautious` that are clearly outside seed noise vs 512x3 baseline at iteration 200, AND `iteration_seconds` at that size is ≤ 3× the baseline (i.e., ≤ ~54s/iter).
**Action:**
1. Recommend that config as the new `network` default.
docs/plans/option_b_interleaved_traversal.md
+2 -2
View File
@@ -152,7 +152,7 @@ Cython production rewrite. It intentionally uses per-context RNG so interleaved
execution order does not change the random stream for another context. That
lets the prototype assert value/stat/sample parity against a recursive prototype
while measuring realized batch size. Production Cython parity is a later Phase 2
gate because the real path also has safe-heuristic opponents, average-strategy
gate because the real path also has heuristic-balanced opponents, average-strategy
opponents, self-play league snapshots, deck-draw chance sampling, external
sampling, and cutoff rollouts.
@@ -312,7 +312,7 @@ Required feature expansion:
- Support `opponent_policy: average_strategy`, matching the default config's
opponent branch.
- Keep unsupported branches guarded (`self_play_league`, `safe_heuristic`,
- Keep unsupported branches guarded (`self_play_league`, `heuristic_balanced`,
random rollout cutoffs, external sampling).
- Add parity tests for the average-strategy fixed-opponent branch.
- Verify a default-policy interleaved run starts and emits batch metrics.
docs/plans/torch_compile.md
+1 -1
View File
@@ -26,7 +26,7 @@ Re-enable `torch.compile` on the Deep CFR trainer's networks at a model size whe
1. **Model-size precondition met.** The active `default.yaml` (or the targeted variant) has `network.hidden_size ≥ 1024` *or* `network.num_layers ≥ 6`, *or* an architecture (e.g. `color_shared` with non-trivial `color_attention_layers`) whose per-call forward time exceeds ~150 μs at the trainer's training batch size on the target GPU. If neither condition holds, this plan is **not merged**; the branch is parked.
2. **Iter-time improvement.** With `compile.trainer.enabled: true` on the chosen larger model and the same seed, the 1000-iter projection improves by at least **5%** vs the no-compile baseline on the same machine (measured on `home`). Eval and checkpointing should be disabled for the bench window, matching the protocol used in the 2026-05-07 experiment.
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `safe_heuristic` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
3. **No learning-curve drift.** Over at least 100 iterations with `compile.trainer.enabled: true` vs `false` (same seed, same config), the eval win-rate trajectories against `random` and `heuristic_balanced` are within seed noise. If trajectories visibly diverge, the plan does not ship even if iter time improves.
4. **No checkpoint-format break.** Checkpoints saved with compile enabled must load cleanly when compile is disabled, and vice versa. (Handled via `_clean_state_dict()`; see Risks.)
5. **No multiprocessing-worker break.** Whether `inference_backend` is `local` or `server`, traversal workers must continue to receive uncompiled `state_dict`s without `_orig_mod.` prefixes.
6. **(Secondary) Inference-server forward.** If step B below is taken, the server's `policy_network_seconds` decreases by at least 20% at the chosen model size, with no traversal-path correctness regression. If step B does not produce a measurable win, it is left disabled and the plan still ships with step A only.
docs/reports/cost_cython_nogil_audit_2026-05-07.md
+6 -6
View File
@@ -157,9 +157,9 @@ Python-object touch가 깔려 있다 (per-node, per-iteration):
`self.strategy_samples.append(...)` (file:903, 937, 969). list의
PyObject reference 갱신은 free-threaded Python에서도 atomic refcount
비용을 추가로 부담한다.
6. **`SafeHeuristicBot.act(state)`** — `_fixed_opponent_action`
6. **`HeuristicBot.act(state)`** — `_fixed_opponent_action`
(file:633, 652), `_rollout_value` (file:841). Python class
메서드 호출. `safe_heuristic` 옵션 사용 시만 핫.
메서드 호출. `heuristic_balanced` 옵션 사용 시만 핫.
7. **`league_advantage_networks` indexing** — `_self_play_snapshot_
networks` (file:802), `[-recent_count:]`, `[:max(0, ...)]` slicing
= Python list slicing.
@@ -267,15 +267,15 @@ candidates = self.league_advantage_networks[:max(0, len(self.league_advantage_ne
- 빈도: traversal 진입 시 한 번 (`traverse`에서 미리 픽), 재귀 안에서는
`active_self_play_networks`만 본다. 따라서 cold path. 변환 불필요.
### B6. `SafeHeuristicBot.act(state)` — Python bot
### B6. `HeuristicBot.act(state)` — Python bot
```python
# traversal.pyx:633, 652, 841
return int(self.safe_heuristic_opponent_bot.act(state))
return int(self.heuristic_opponent_bot.act(state))
```
- 빈도: `opponent_policy=safe_heuristic` 또는 `cutoff_rollout_policy=
safe_heuristic`일 때만. 현 default는 self_play_league + score_diff
- 빈도: `opponent_policy=heuristic_balanced` 또는 `cutoff_rollout_policy=
heuristic_balanced`일 때만. 현 default는 self_play_league + score_diff
cutoff (per memory의 opponent_policy_network_divergence note + AGENTS).
- 변환 난이도: **Medium-High** (Python class 전체를 cython화). 현 default
config에서는 핫 아님 — 시도하지 않는 게 합리.
docs/research/deep-cfr-architecture.md
+1 -1
View File
@@ -48,7 +48,7 @@ src/coolrl_lost_cities/games/classic/deep_cfr/
traversal_stats.py — structured traversal diagnostic metrics
# Auxiliary training modes
imitation.py — safe-heuristic imitation pretraining
imitation.py — heuristic imitation pretraining
policy_gradient.py — policy-gradient fine-tuning
```
docs/research/deep-cfr-baseline-2000-analysis.md
+10 -10
View File
@@ -59,23 +59,23 @@ Evaluation performance peaks early and then degrades.
| Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff |
| --- | ---: | ---: | ---: | ---: |
| random | 0.808 | 0.665 | +35.1 | +12.4 |
| passive_discard | 0.026 | 0.012 | -30.8 | -43.0 |
| safe_heuristic | 0.070 | 0.011 | -68.7 | -94.2 |
| safe_heuristic_loose | 0.078 | 0.014 | -69.3 | -95.0 |
| safe_heuristic_strict | 0.072 | 0.011 | -57.6 | -85.0 |
| noisy_safe | 0.120 | 0.033 | -49.4 | -78.8 |
| discard_only | 0.026 | 0.012 | -30.8 | -43.0 |
| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 |
| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 |
| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 |
| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 |
The best strict-heuristic point appears around iteration 70:
- `eval/safe_heuristic_strict/win_rate0=0.15`
- `eval/safe_heuristic_strict/avg_score_diff0=-41.09`
- `eval/heuristic_cautious/win_rate0=0.15`
- `eval/heuristic_cautious/avg_score_diff0=-41.09`
The final point at iteration 2000 is worse:
- `eval/safe_heuristic_strict/win_rate0=0.02`
- `eval/safe_heuristic_strict/avg_score_diff0=-81.64`
- `eval/heuristic_cautious/win_rate0=0.02`
- `eval/heuristic_cautious/avg_score_diff0=-81.64`
Selectivity does not meaningfully emerge. Against `safe_heuristic_strict`:
Selectivity does not meaningfully emerge. Against `heuristic_cautious`:
| Metric | First 20 evals | Last 20 evals | Interpretation |
| --- | ---: | ---: | --- |
docs/research/deep-cfr-batched-evaluation.md
+1 -1
View File
@@ -19,7 +19,7 @@ The primary bottleneck in CUDA-based evaluation is the overhead of launching sma
A critical refinement in the batched implementation was the handling of policy entropy. Initial versions that calculated entropy per-row on the CPU incurred significant synchronization penalties because each row required a GPU-to-CPU transfer. Moving the entropy calculation into the Torch post-processing pipeline—specifically calculating it directly on the `probs_tensor` (line 220)—ensures that the computation remains on the device and only the final results are transferred back to the host in bulk.
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `safe_heuristic_strict`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
Once network inference is batched, the remaining bottleneck often shifts to the CPU-bound logic of heuristic opponents (e.g., `heuristic_cautious`). Parallelizing the evaluation across multiple workers allows the trainer to evaluate against multiple opponents simultaneously. For a single iteration profile, using 4 parallel workers reduced wall-clock evaluation time from 14.8 seconds to 6.4 seconds, achieving a ~2.3x speedup.
## Practical implication
docs/research/deep-cfr-evaluation-profile.md
+1 -1
View File
@@ -37,7 +37,7 @@ On CPU, the `network / turn` cost is approximately **0.074 ms**. On CUDA, this r
### Secondary Bottlenecks
- **Post-processing:** Moving tensors back to CPU (`.cpu().numpy()`) and calculating entropy adds measurable overhead on CUDA that is largely absent on CPU.
- **Opponent Logic:** Heuristic opponents (e.g., `safe_heuristic`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
- **Opponent Logic:** Heuristic opponents (e.g., `heuristic_balanced`) contribute significant `opponent_act_seconds` (up to 3.5s per eval iteration). Since this logic is pure Python/Cython and runs on the CPU, it does not benefit from GPU acceleration, further diluting any potential CUDA wins.
## Practical Implications
docs/research/deep-cfr-reproducibility.md
+2 -2
View File
@@ -45,7 +45,7 @@ Two temporary runs used:
- `run.max_iterations=3`
- `evaluation.eval_every=1`
- `evaluation.games=20`
- `evaluation.opponents=[random,safe_heuristic_strict]`
- `evaluation.opponents=[random,heuristic_cautious]`
- W&B disabled
Runs:
@@ -55,7 +55,7 @@ Runs:
Core metrics matched exactly:
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/safe_heuristic_strict/win_rate0` |
| Iteration | `traversal/nodes` | `memory/advantage` | `loss/advantage` | `eval/random/win_rate0` | `eval/heuristic_cautious/win_rate0` |
| --- | ---: | ---: | ---: | ---: | ---: |
| 1 | 170822 | 85096 | 803.9475702643394 | 0.75 | 0.10 |
| 2 | 206225 | 187987 | 821.3355012834072 | 0.50 | 0.05 |
docs/research/deep-cfr-v0-feature-parity.md
+1 -1
View File
@@ -56,7 +56,7 @@ registered classic bots (`evaluate.py`), training CLI and evaluation CLI
(`cli.py`), traversal benchmark CLI (`benchmark.py`), `metrics.jsonl` /
`runtime_progress.json` / `train.log` run artifacts, self-play league with
snapshot pool and weighted current/recent/older/anchor bucket sampling, safe-
heuristic anchor opponent, safe-heuristic imitation pretraining (`imitation.py`),
heuristic anchor opponent, heuristic imitation pretraining (`imitation.py`),
and policy-gradient fine-tuning (`policy_gradient.py`).
As of `ad0be89`, the package also includes `inference_server.py`,
docs/research/julia_port_evaluation.md
+1 -1
View File
@@ -31,7 +31,7 @@ Decision criteria for going forward:
### 2026-05-07 — Safe heuristic single-thread parity (criterion 1)
Path: `experiments/julia_safe_heuristic/`.
Path: `experiments/julia_heuristic/`.
1,838 snapshots in 157.471 ms median (~85.6 μs/call). Action-sequence
parity vs Python. Same order of magnitude as the Cython port of the
docs/research/lost_cities_selectivity.md
+1 -1
View File
@@ -28,7 +28,7 @@ Deep CFR은 최종적으로 average strategy가 수렴 대상. current advantage
Visible-score predictor의 ceiling은 있을 수 있음. 다만 selectivity 자체의 ceiling은 아님. selectivity는 visible-score prediction 말고 option value, irreversible cost 회피, opponent dynamics 대응 등 다른 경로로도 emerge 가능.
**Self-play attractor 가설**
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. safe_heuristic이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
5-color가 stable equilibrium. 한쪽이 selectivity 시도하면 즉시 손해. heuristic_balanced이 3.7에서 강하면 진짜 NE는 아닐 듯. 다만 self-play 안에서는 stable.
**Lost Cities NE 자체가 5-color 가설**
이론적 가능성 0 아님. self-play가 발견한 게 진짜 NE면 transition 영원히 안 옴. 검증 가능한 형태로는 tabular oracle 또는 BR 진단.