coorl-lost-cities/docs/plans/lost-cities-league-selfplay-spec.md

Lost Cities JAX PPO Snapshot-Pool League Spec

Status: ready for implementation after ladder v2. Last updated: 2026-07-05. Warm start: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest.

Preconditions

The static-opponent ladder v2 has passed:

• discard_only: win rate 1.0000, mean score diff +182.0670.
• heuristic_balanced: win rate 0.9409, mean score diff +119.9250.
• heuristic_expert: win rate 0.8382, mean score diff +43.3099, score-diff CI95 lower bound +42.7065.

Canary warnings remain:

• Gate-3 opened colors are 4.5509/game, not the desired 2-3 range.
• The v2 gate-3 checkpoint is exploitable: a 250-update PPO exploiter reached win rate 0.83135 and mean score diff +54.7721 against it.

Anchor Semantics

Anchors are not certificates of strength. They serve two narrower purposes:

• Fix the Elo scale so curves remain comparable over time.
• Provide style diversity so the league does not train only against recent policy snapshots.

Strength certification is tracked on separate axes:

• Duplicate mean score difference against heuristic_expert.
• Exploiter win-rate trend under the fixed exploiter protocol.
• Later human play, once the automated diagnostics are stable.

Initial Pool

Permanent anchors are never removed:

• Static anchors: discard_only, heuristic_balanced, heuristic_cautious, heuristic_expert.
• Learned ladder anchors:
  • Gate 1: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest
  • Gate 2: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest
  • Gate 3: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest

The learning policy starts from the v2 gate-3 checkpoint. Shaping coefficient is fixed at 0 by default. If warm-start collapse is observed, the implementation may expose a config switch for random initialization plus shaping anneal, but that is not the default path.

Opponent Sampling

For each episode:

• Randomize the learner seat.
• With probability 50%, play mirror self-play against the current policy copy.
• With probability 50%, sample from the pool.

Pool sampling is PFSP-like:

• Track recent win rate versus each pool member.
• Weight non-anchor snapshots by (1 - win_rate)^2.
• Mix in a small uniform component to avoid starvation.
• Give every permanent anchor only a small floor probability, such as 2-3% per anchor, so stalling or weak anchors do not waste most throughput.
• Allocate the remaining probability mass to snapshots according to the PFSP weights.

Snapshot Lifecycle

• Add the current policy to the pool at a configurable interval.
• If the pool size cap is reached, remove the oldest non-anchor snapshot.
• Never remove permanent anchors.
• Store large checkpoints and raw evaluation JSON under /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/.
• Store one-line tracked JSON summaries under docs/reports/ so results do not disappear with artifact cleanup.

Evaluation

At each snapshot interval, evaluate the current policy with fixed shuffle-bank duplicate play against:

• The six ladder anchors.
• heuristic_expert.
• A configurable set of recent snapshots.

Update logistic Elo from the pairwise results, using the heuristic anchors as fixed reference points. Elo is a scale diagnostic, not a strength certificate.

Run the exploiter protocol periodically against the current best checkpoint:

• Randomly initialized PPO.
• Same 250-update budget used in ladder v2 unless explicitly overridden.
• Duplicate evaluation versus the frozen target checkpoint.
• Track exploiter win rate and mean score difference over time.

Canary Metrics

Log these metrics at every evaluation:

• Opened colors per game.
• Play action rate.
• Mean and quantile game length.
• Max-steps rate.
• Positive expeditions per game.
• Duplicate score-diff distribution against heuristic_expert.

Canaries are diagnostics only. Do not put them directly into the reward.

Regression Guards

• heuristic_expert replaces heuristic_cautious as the main regression opponent.
• If win rate versus heuristic_expert falls below 90% of the v2 gate-3 value, flag the snapshot.
• If max-steps rate rises above the gate-3 baseline of 0.0 by a material amount, flag the snapshot.
• If exploiter win rate rises above the v2 baseline of 0.83135, flag the snapshot.

Stop Conditions

Stop the league run when either:

• Elo is statistically flat over a configured recent-snapshot window and exploiter win rate is at or below the configured threshold, for example 0.55.
• The wall-clock budget is exhausted.

Decisions

• The earlier heuristic_cautious remains a permanent anchor for scale and style diversity only. It is a stalling anchor and no longer certifies strength.
• discard_only is also kept only as a low-probability anchor and regression sanity check.
• The league spec file was not present in the repository when this update was requested, so this tracked plan file is the authoritative spec location.