140 lines
5.1 KiB
Markdown
140 lines
5.1 KiB
Markdown
# Lost Cities JAX PPO Snapshot-Pool League Spec
|
|
|
|
**Status:** ready for implementation after ladder v2.
|
|
**Last updated:** 2026-07-05.
|
|
**Warm start:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`.
|
|
|
|
## Preconditions
|
|
|
|
The static-opponent ladder v2 has passed:
|
|
|
|
- `discard_only`: win rate 1.0000, mean score diff +182.0670.
|
|
- `heuristic_balanced`: win rate 0.9409, mean score diff +119.9250.
|
|
- `heuristic_expert`: win rate 0.8382, mean score diff +43.3099, score-diff
|
|
CI95 lower bound +42.7065.
|
|
|
|
Canary warnings remain:
|
|
|
|
- Gate-3 opened colors are 4.5509/game, not the desired 2-3 range.
|
|
- The v2 gate-3 checkpoint is exploitable: a 250-update PPO exploiter reached
|
|
win rate 0.83135 and mean score diff +54.7721 against it.
|
|
|
|
## Anchor Semantics
|
|
|
|
Anchors are not certificates of strength. They serve two narrower purposes:
|
|
|
|
- Fix the Elo scale so curves remain comparable over time.
|
|
- Provide style diversity so the league does not train only against recent
|
|
policy snapshots.
|
|
|
|
Strength certification is tracked on separate axes:
|
|
|
|
- Duplicate mean score difference against `heuristic_expert`.
|
|
- Exploiter win-rate trend under the fixed exploiter protocol.
|
|
- Later human play, once the automated diagnostics are stable.
|
|
|
|
## Initial Pool
|
|
|
|
Permanent anchors are never removed:
|
|
|
|
- Static anchors: `discard_only`, `heuristic_balanced`, `heuristic_cautious`,
|
|
`heuristic_expert`.
|
|
- Learned ladder anchors:
|
|
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest`
|
|
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest`
|
|
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`
|
|
|
|
The learning policy starts from the v2 gate-3 checkpoint. Shaping coefficient is
|
|
fixed at 0 by default. If warm-start collapse is observed, the implementation
|
|
may expose a config switch for random initialization plus shaping anneal, but
|
|
that is not the default path.
|
|
|
|
## Opponent Sampling
|
|
|
|
For each episode:
|
|
|
|
- Randomize the learner seat.
|
|
- With probability 50%, play mirror self-play against the current policy copy.
|
|
- With probability 50%, sample from the pool.
|
|
|
|
Pool sampling is PFSP-like:
|
|
|
|
- Track recent win rate versus each pool member.
|
|
- Weight non-anchor snapshots by `(1 - win_rate)^2`.
|
|
- Mix in a small uniform component to avoid starvation.
|
|
- Give every permanent anchor only a small floor probability, such as 2-3% per
|
|
anchor, so stalling or weak anchors do not waste most throughput.
|
|
- Allocate the remaining probability mass to snapshots according to the PFSP
|
|
weights.
|
|
|
|
## Snapshot Lifecycle
|
|
|
|
- Add the current policy to the pool at a configurable interval.
|
|
- If the pool size cap is reached, remove the oldest non-anchor snapshot.
|
|
- Never remove permanent anchors.
|
|
- Store large checkpoints and raw evaluation JSON under
|
|
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/`.
|
|
- Store one-line tracked JSON summaries under `docs/reports/` so results do
|
|
not disappear with artifact cleanup.
|
|
|
|
## Evaluation
|
|
|
|
At each snapshot interval, evaluate the current policy with fixed shuffle-bank
|
|
duplicate play against:
|
|
|
|
- The six ladder anchors.
|
|
- `heuristic_expert`.
|
|
- A configurable set of recent snapshots.
|
|
|
|
Update logistic Elo from the pairwise results, using the heuristic anchors as
|
|
fixed reference points. Elo is a scale diagnostic, not a strength certificate.
|
|
|
|
Run the exploiter protocol periodically against the current best checkpoint:
|
|
|
|
- Randomly initialized PPO.
|
|
- Same 250-update budget used in ladder v2 unless explicitly overridden.
|
|
- Duplicate evaluation versus the frozen target checkpoint.
|
|
- Track exploiter win rate and mean score difference over time.
|
|
|
|
## Canary Metrics
|
|
|
|
Log these metrics at every evaluation:
|
|
|
|
- Opened colors per game.
|
|
- Play action rate.
|
|
- Mean and quantile game length.
|
|
- Max-steps rate.
|
|
- Positive expeditions per game.
|
|
- Duplicate score-diff distribution against `heuristic_expert`.
|
|
|
|
Canaries are diagnostics only. Do not put them directly into the reward.
|
|
|
|
## Regression Guards
|
|
|
|
- `heuristic_expert` replaces `heuristic_cautious` as the main regression
|
|
opponent.
|
|
- If win rate versus `heuristic_expert` falls below 90% of the v2 gate-3 value,
|
|
flag the snapshot.
|
|
- If max-steps rate rises above the gate-3 baseline of 0.0 by a material
|
|
amount, flag the snapshot.
|
|
- If exploiter win rate rises above the v2 baseline of 0.83135, flag the
|
|
snapshot.
|
|
|
|
## Stop Conditions
|
|
|
|
Stop the league run when either:
|
|
|
|
- Elo is statistically flat over a configured recent-snapshot window and
|
|
exploiter win rate is at or below the configured threshold, for example 0.55.
|
|
- The wall-clock budget is exhausted.
|
|
|
|
## Decisions
|
|
|
|
- The earlier `heuristic_cautious` remains a permanent anchor for scale and
|
|
style diversity only. It is a stalling anchor and no longer certifies
|
|
strength.
|
|
- `discard_only` is also kept only as a low-probability anchor and regression
|
|
sanity check.
|
|
- The league spec file was not present in the repository when this update was
|
|
requested, so this tracked plan file is the authoritative spec location.
|