# Deep CFR 512x3 2000-Iteration Baseline Analysis

**Last verified:** 2026-05-08, commit `aa898ed`
**Source run:** `runs/2026-05-08_051124_baseline-512x3-2000-dense-eval`

## Question

Does the 512x3 2000-iteration baseline show useful learning signals and
Lost Cities selectivity, and what should be improved before treating it as the
canonical comparison anchor?

## Short Answer

The run shows real optimization progress in the advantage networks, but the
policy that is evaluated does not improve. The main failure mode is not simply
"opens too many colors." By late training, the model often opens slightly fewer
colors against strong heuristic opponents, but the expeditions it opens are
lower quality and become negative more often. This looks more like passive or
degenerate average-policy learning than successful selectivity.

Use this run as evidence for the next diagnostics, not as a final baseline for
model-size decisions. Before running another long model-size comparison, separate
current regret-matching policy quality from average-strategy-network quality and
audit the interleaved outcome-sampling target behavior.

## Run Context

The source run used the old pre-deterministic baseline settings:

- `network.hidden_size=512`, `network.num_layers=3`
- `run.seed=79`
- `run.max_iterations=2000`
- `evaluation.eval_every=10`
- `checkpoint.save_every=100`
- `traversal.num_workers=8`
- `traversal.scheduler=interleaved`
- `traversal.sampling_mode=outcome`
- `traversal.outcome_unsampled_regret=zero`
- `run.deterministic=false`

It is therefore not the canonical reproducibility baseline after the deterministic
traversal changes. Its value is diagnostic: it is long enough to show how the
current 512x3 setup behaves after memory warm-up and many strategy updates.

## Evidence

Advantage fitting improves, but strategy fitting does not produce a better
evaluated policy.

| Metric | Early | Late | Interpretation |
| --- | ---: | ---: | --- |
| `loss/advantage` mean | ~2403 at iters 1-100 | ~807 at iters 1501-2000 | advantage networks fit their targets better |
| `loss/strategy` mean | ~1.53 at iters 1-100 | ~1.61 at iters 1501-2000 | average policy training does not clearly improve |
| `memory/advantage` | full by iter 101-500 | full | both player buffers saturated at 2M each |
| `memory/strategy` | full by iter 101-500 | full | strategy buffer saturated at 2M |

Evaluation performance peaks early and then degrades.

| Opponent | First 20 evals `win_rate0` | Last 20 evals `win_rate0` | First 20 evals score diff | Last 20 evals score diff |
| --- | ---: | ---: | ---: | ---: |
| random | 0.808 | 0.665 | +35.1 | +12.4 |
| discard_only | 0.026 | 0.012 | -30.8 | -43.0 |
| heuristic_balanced | 0.070 | 0.011 | -68.7 | -94.2 |
| heuristic_aggressive | 0.078 | 0.014 | -69.3 | -95.0 |
| heuristic_cautious | 0.072 | 0.011 | -57.6 | -85.0 |
| heuristic_noisy | 0.120 | 0.033 | -49.4 | -78.8 |

The best strict-heuristic point appears around iteration 70:

- `eval/heuristic_cautious/win_rate0=0.15`
- `eval/heuristic_cautious/avg_score_diff0=-41.09`

The final point at iteration 2000 is worse:

- `eval/heuristic_cautious/win_rate0=0.02`
- `eval/heuristic_cautious/avg_score_diff0=-81.64`

Selectivity does not meaningfully emerge. Against `heuristic_cautious`:

| Metric | First 20 evals | Last 20 evals | Interpretation |
| --- | ---: | ---: | --- |
| `avg_opened_colors` | 4.31 | 3.92 | opens slightly fewer colors |
| `score_per_opened_color` | -8.09 | -15.41 | opened expeditions become worse |
| `positive_expedition_rate` | 0.142 | 0.046 | fewer opened expeditions end positive |
| `negative_expedition_rate` | 0.834 | 0.947 | almost every opened expedition is negative |
| `bad_open_rate` | 0.905 | 0.908 | first-open quality does not improve |

The open-quality metrics are generated by `src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py:124`
and classify first-open actions around `src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py:546`.
A "good" open has non-negative visible recoverable score; a "bad" open has no
bonus path and negative recoverable score.

Traversal gets deeper and slower late in training:

- `traversal/nodes` mean rises from ~119k at iters 1-100 to ~199k at iters
  1501-2000.
- `traversal/avg_endpoint_depth` rises from ~212 to ~354.
- `time/iteration_seconds` rises from ~12.0s to ~14.4s.
- late node-limit cutoffs appear, but remain small on average: about 1.0% of
  endpoints in iters 1501-2000.

This suggests the policy is changing game length and trajectory shape, but not
toward higher-quality scoring.

## Interpretation

The strongest signal is a current-vs-average policy split. Deep CFR trains
advantage networks to define a current regret-matching policy, but the evaluated
agent is the strategy network, which approximates the average policy. In this
run, advantage loss improves while strategy loss and eval quality do not. That
points to either:

1. the current policy is improving but the average strategy network or strategy
   memory is failing to represent it, or
2. the advantage networks are fitting targets that do not translate into better
   game play under the current traversal distribution.

The current metrics cannot distinguish those two cases. The next diagnostic
should evaluate the same checkpoint under both policy views:

- current advantage networks + regret matching,
- average strategy network.

There is also a target-consistency issue worth auditing before a new long run.
The recursive Cython traversal honors `outcome_unsampled_regret=zero` by setting
unsampled legal targets to zero at
`src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:930`. The
interleaved traversal currently sets every legal target to `-node_value` before
overwriting the sampled action at
`src/coolrl_lost_cities/games/classic/deep_cfr/interleaved_traversal.py:386`.
That makes the interleaved path behave like the `negative_node_value` variant
even when the resolved config says `zero`.

Theoretically, this is a state-wise constant shift and regret matching is
invariant to action-independent constants. With function approximation and MSE,
however, it changes training dynamics and should be treated as an experimental
variable unless explicitly made consistent.

## Deep CFR Improvement Candidates

### 1. Evaluate current policy vs average policy first

This is the highest-value diagnostic. If current regret-matching policy is
materially stronger than the strategy network, focus on strategy memory,
strategy weighting, and average-policy fitting. If both are weak, focus on
advantage targets, exploration, and representation.

### 2. Make interleaved outcome targets match the config

Either implement `outcome_unsampled_regret` in the interleaved scheduler or
change the default/config documentation to state that interleaved outcome
sampling uses the baseline-subtracted target. The former is cleaner because it
lets `zero` vs `negative_node_value` become a controlled ablation.

### 3. Run a lower-variance traversal ablation

Outcome sampling is correct but high variance for Lost Cities entry decisions.
Before changing model size, test whether increasing `traversals_per_player`
improves strict-heuristic score and expedition quality. A useful paired
ablation would keep model/config fixed and change only traversal count, for
example 280 vs 560 per player.

### 4. Add first-open regret diagnostics

The current bad-open metric is outcome-facing and heuristic-visible. It tells us
that opened expeditions are bad by recoverable score, but not whether the
counterfactual regret target said the open action was bad at the decision point.
Add diagnostics over advantage-memory targets by action class:

- first-open action targets,
- non-open alternatives in the same information state,
- color/action class buckets,
- target quantiles, not only means.

If bad first-open actions have negative advantage targets but the policy still
opens, suspect representation or strategy fitting. If their targets are not
negative, suspect exploration distribution or the metric/objective mismatch.

### 5. Treat model-size scaling as a follow-up, not the first fix

The earlier 1024x6 run showed a better 200-iteration signal than 512x3, but this
2000-iteration baseline shows a more basic policy-quality issue. Scaling capacity
may help representation, but it will not tell us whether the average-policy
training path is currently suppressing or distorting a better current policy.

## Recommended Next Plan

1. Implement or script current-vs-average evaluation for a saved checkpoint.
2. Run it on `iteration_00070.pt`, `iteration_00200.pt`, `iteration_01000.pt`,
   and `iteration_02000.pt` from the source run.
3. Audit and fix interleaved `outcome_unsampled_regret` handling.
4. Establish a new deterministic 512x3 baseline only after the target behavior
   is intentional and documented.
5. Then compare 512x3 vs 1024x6 under matched deterministic settings and W&B
   grouping.

## Practical Baseline Rule

Do not promote `baseline-512x3-2000-dense-eval` as the canonical baseline. It is
better classified as a diagnostic pre-determinism long run. The next canonical
baseline should be rerun with:

- deterministic traversal enabled,
- `evaluation.eval_every=5`,
- `checkpoint.save_every=50`,
- explicit W&B group/job type,
- and target semantics that match the resolved config.
