Add JAX PPO ladder v2 expert pass
This commit is contained in:
@@ -0,0 +1,139 @@
|
||||
# Lost Cities JAX PPO Snapshot-Pool League Spec
|
||||
|
||||
**Status:** ready for implementation after ladder v2.
|
||||
**Last updated:** 2026-07-05.
|
||||
**Warm start:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`.
|
||||
|
||||
## Preconditions
|
||||
|
||||
The static-opponent ladder v2 has passed:
|
||||
|
||||
- `discard_only`: win rate 1.0000, mean score diff +182.0670.
|
||||
- `heuristic_balanced`: win rate 0.9409, mean score diff +119.9250.
|
||||
- `heuristic_expert`: win rate 0.8382, mean score diff +43.3099, score-diff
|
||||
CI95 lower bound +42.7065.
|
||||
|
||||
Canary warnings remain:
|
||||
|
||||
- Gate-3 opened colors are 4.5509/game, not the desired 2-3 range.
|
||||
- The v2 gate-3 checkpoint is exploitable: a 250-update PPO exploiter reached
|
||||
win rate 0.83135 and mean score diff +54.7721 against it.
|
||||
|
||||
## Anchor Semantics
|
||||
|
||||
Anchors are not certificates of strength. They serve two narrower purposes:
|
||||
|
||||
- Fix the Elo scale so curves remain comparable over time.
|
||||
- Provide style diversity so the league does not train only against recent
|
||||
policy snapshots.
|
||||
|
||||
Strength certification is tracked on separate axes:
|
||||
|
||||
- Duplicate mean score difference against `heuristic_expert`.
|
||||
- Exploiter win-rate trend under the fixed exploiter protocol.
|
||||
- Later human play, once the automated diagnostics are stable.
|
||||
|
||||
## Initial Pool
|
||||
|
||||
Permanent anchors are never removed:
|
||||
|
||||
- Static anchors: `discard_only`, `heuristic_balanced`, `heuristic_cautious`,
|
||||
`heuristic_expert`.
|
||||
- Learned ladder anchors:
|
||||
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest`
|
||||
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest`
|
||||
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`
|
||||
|
||||
The learning policy starts from the v2 gate-3 checkpoint. Shaping coefficient is
|
||||
fixed at 0 by default. If warm-start collapse is observed, the implementation
|
||||
may expose a config switch for random initialization plus shaping anneal, but
|
||||
that is not the default path.
|
||||
|
||||
## Opponent Sampling
|
||||
|
||||
For each episode:
|
||||
|
||||
- Randomize the learner seat.
|
||||
- With probability 50%, play mirror self-play against the current policy copy.
|
||||
- With probability 50%, sample from the pool.
|
||||
|
||||
Pool sampling is PFSP-like:
|
||||
|
||||
- Track recent win rate versus each pool member.
|
||||
- Weight non-anchor snapshots by `(1 - win_rate)^2`.
|
||||
- Mix in a small uniform component to avoid starvation.
|
||||
- Give every permanent anchor only a small floor probability, such as 2-3% per
|
||||
anchor, so stalling or weak anchors do not waste most throughput.
|
||||
- Allocate the remaining probability mass to snapshots according to the PFSP
|
||||
weights.
|
||||
|
||||
## Snapshot Lifecycle
|
||||
|
||||
- Add the current policy to the pool at a configurable interval.
|
||||
- If the pool size cap is reached, remove the oldest non-anchor snapshot.
|
||||
- Never remove permanent anchors.
|
||||
- Store large checkpoints and raw evaluation JSON under
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/`.
|
||||
- Store one-line tracked JSON summaries under `docs/reports/` so results do
|
||||
not disappear with artifact cleanup.
|
||||
|
||||
## Evaluation
|
||||
|
||||
At each snapshot interval, evaluate the current policy with fixed shuffle-bank
|
||||
duplicate play against:
|
||||
|
||||
- The six ladder anchors.
|
||||
- `heuristic_expert`.
|
||||
- A configurable set of recent snapshots.
|
||||
|
||||
Update logistic Elo from the pairwise results, using the heuristic anchors as
|
||||
fixed reference points. Elo is a scale diagnostic, not a strength certificate.
|
||||
|
||||
Run the exploiter protocol periodically against the current best checkpoint:
|
||||
|
||||
- Randomly initialized PPO.
|
||||
- Same 250-update budget used in ladder v2 unless explicitly overridden.
|
||||
- Duplicate evaluation versus the frozen target checkpoint.
|
||||
- Track exploiter win rate and mean score difference over time.
|
||||
|
||||
## Canary Metrics
|
||||
|
||||
Log these metrics at every evaluation:
|
||||
|
||||
- Opened colors per game.
|
||||
- Play action rate.
|
||||
- Mean and quantile game length.
|
||||
- Max-steps rate.
|
||||
- Positive expeditions per game.
|
||||
- Duplicate score-diff distribution against `heuristic_expert`.
|
||||
|
||||
Canaries are diagnostics only. Do not put them directly into the reward.
|
||||
|
||||
## Regression Guards
|
||||
|
||||
- `heuristic_expert` replaces `heuristic_cautious` as the main regression
|
||||
opponent.
|
||||
- If win rate versus `heuristic_expert` falls below 90% of the v2 gate-3 value,
|
||||
flag the snapshot.
|
||||
- If max-steps rate rises above the gate-3 baseline of 0.0 by a material
|
||||
amount, flag the snapshot.
|
||||
- If exploiter win rate rises above the v2 baseline of 0.83135, flag the
|
||||
snapshot.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
Stop the league run when either:
|
||||
|
||||
- Elo is statistically flat over a configured recent-snapshot window and
|
||||
exploiter win rate is at or below the configured threshold, for example 0.55.
|
||||
- The wall-clock budget is exhausted.
|
||||
|
||||
## Decisions
|
||||
|
||||
- The earlier `heuristic_cautious` remains a permanent anchor for scale and
|
||||
style diversity only. It is a stalling anchor and no longer certifies
|
||||
strength.
|
||||
- `discard_only` is also kept only as a low-probability anchor and regression
|
||||
sanity check.
|
||||
- The league spec file was not present in the repository when this update was
|
||||
requested, so this tracked plan file is the authoritative spec location.
|
||||
@@ -0,0 +1 @@
|
||||
{"date":"2026-07-05","status":"pass_with_canary_warnings","expert_bot":{"mirror_max_steps_rate":0.0,"mirror_opened_colors_per_game":2.256,"mirror_play_action_rate":0.3117858091947914,"vs_discard_mean_score_diff":8.9716,"vs_balanced_mean_score_diff":54.36725,"vs_cautious_mean_score_diff":51.9397},"gates":{"discard_only":{"win_rate":1.0,"wilson_low":0.9998079639438954,"mean_score_diff":182.067,"opened_colors_per_game":4.2163,"max_steps_rate":0.0},"heuristic_balanced":{"win_rate":0.9409,"wilson_low":0.9375464241082592,"mean_score_diff":119.925,"opened_colors_per_game":4.9862,"max_steps_rate":0.1859},"heuristic_expert":{"win_rate":0.83815,"wilson_low":0.8329806740267004,"mean_score_diff":43.3099,"score_diff_ci95_low":42.706527573078894,"opened_colors_per_game":4.5509,"max_steps_rate":0.0}},"exploiter":{"target":"ladder_v2_gate3","win_rate":0.83135,"wilson_low":0.8260970642435626,"mean_score_diff":54.7721,"score_diff_ci95_low":53.98710920050297,"opened_colors_per_game":4.99135,"max_steps_rate":0.0001},"artifact_root":"/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/"}
|
||||
@@ -0,0 +1,114 @@
|
||||
# JAX PPO Ladder v2 Report - 2026-07-05
|
||||
|
||||
**Status:** PASS for the ordered ladder gates; canary warnings remain.
|
||||
**Hardware:** RTX 3090 via `uv run --with 'jax[cuda12]'`.
|
||||
**Artifact roots:**
|
||||
|
||||
- Expert bot: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/`
|
||||
- Ladder v2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/`
|
||||
|
||||
## Protocol
|
||||
|
||||
Part A added a pure-JAX `heuristic_expert` opponent and evaluated it with a
|
||||
fixed shuffle bank plus duplicate play. Part B trained PPO from random
|
||||
initialization for each gate with the same 250-update configuration used by the
|
||||
original ladder, changing only the opponent and artifact root:
|
||||
|
||||
- Gate 1: `discard_only`
|
||||
- Gate 2: `heuristic_balanced`
|
||||
- Gate 3: `heuristic_expert`
|
||||
|
||||
All evaluations below used 10,000 deck orders with duplicate seat-swapped play,
|
||||
for 20,000 evaluated games.
|
||||
|
||||
## Part A - Expert Bot Gate
|
||||
|
||||
| Check | Result | Win rate | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Expert mirror, 1,000 duplicate pairs | PASS | 0.4815 | 0.0000 | [-0.6956, +0.6956] | 45.1560 | 0.0000 | 2.2560 | 0.3118 |
|
||||
| Expert vs `discard_only` | PASS | 0.6643 | +8.9716 | [+8.7886, +9.1546] | 45.2125 | 0.0000 | 1.8971 | 0.2828 |
|
||||
| Expert vs `heuristic_balanced` | PASS | 0.9961 | +54.3673 | [+54.0791, +54.6554] | 64.9181 | 0.0000 | 2.8003 | 0.2728 |
|
||||
| Expert vs `heuristic_cautious` | PASS | 0.9972 | +51.9397 | [+51.6960, +52.1834] | 75.2329 | 0.0000 | 2.6090 | 0.2240 |
|
||||
|
||||
The new script bot is non-stalling in mirror play, symmetric under duplicate
|
||||
seat swap, and beats all three previous static bots by a clear positive score
|
||||
margin. Its own opened-color profile is in the intended 2-3 color band.
|
||||
|
||||
## Part B - Ladder v2
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.0000 [0.9998, 1.0000] | +182.0670 | [+181.6390, +182.4950] | 82.1558 | 0.0000 | 4.2163 | 0.6398 | 3.1845 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.9409 [0.9375, 0.9441] | +119.9250 | [+118.9950, +120.8550] | 151.2468 | 0.1859 | 4.9862 | 0.2937 | 3.5316 |
|
||||
| 3 | `heuristic_expert` | PASS | 0.8382 [0.8330, 0.8432] | +43.3099 | [+42.7065, +43.9133] | 61.4025 | 0.0000 | 4.5509 | 0.6661 | 2.6121 |
|
||||
|
||||
Gate 1 passed the original win-rate and positive-expedition conditions. Gate 2
|
||||
passed the mean-score-difference condition. Gate 3 passed the v2 condition:
|
||||
duplicate mean score difference is positive and its 95% confidence lower bound
|
||||
is above zero.
|
||||
|
||||
## Canary Findings
|
||||
|
||||
The ladder did not produce the desired 2-3 color selectivity in the learned PPO
|
||||
policies. The gate-3 policy still opens 4.5509 colors/game against the expert.
|
||||
This is lower than the original gate-3 checkpoint but still far outside the
|
||||
target band. Treat the selectivity hypothesis as not confirmed.
|
||||
|
||||
Gate 2 also has a high forced-end canary: `max_steps_rate=0.1859` in duplicate
|
||||
evaluation versus `heuristic_balanced`. It passed its score gate, but the
|
||||
balanced opponent still permits long games and 5-color farming.
|
||||
|
||||
## Exploiter Baseline
|
||||
|
||||
After gate 3 passed, a new PPO exploiter was trained from random initialization
|
||||
against the frozen v2 gate-3 checkpoint with the same 250-update budget and
|
||||
duplicate-evaluated against that checkpoint.
|
||||
|
||||
| Target checkpoint | Exploiter win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Old gate-3 checkpoint | 0.6712 [0.6646, 0.6776] | +24.1371 | n/a | 48.0017 | 0.0000 | 4.8971 | 0.7461 | 2.1287 |
|
||||
| New v2 gate-3 checkpoint | 0.8314 [0.8261, 0.8365] | +54.7721 | [+53.9871, +55.5571] | 60.8744 | 0.0001 | 4.9914 | 0.7028 | 2.8921 |
|
||||
|
||||
The new gate-3 checkpoint beats `heuristic_expert`, but is more exploitable
|
||||
under this protocol than the old checkpoint. That does not invalidate the gate,
|
||||
but it makes exploiter reduction a primary objective for the league phase.
|
||||
|
||||
## Old vs New Gate-3 Canary Comparison
|
||||
|
||||
| Metric | Old gate-3 vs cautious | New gate-3 vs expert | Interpretation |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Mean score diff | +142.8993 | +43.3099 | New gate is harder and less suspiciously clean. |
|
||||
| Win rate | 0.9596 | 0.8382 | New result is less inflated. |
|
||||
| Opened colors | 4.9896 | 4.5509 | Improved, but still not selective enough. |
|
||||
| Max-step rate | not recorded in old eval | 0.0000 | New expert evaluation does not stall. |
|
||||
| Play action rate | 0.2763 | 0.6661 | New policy plays much more actively. |
|
||||
| Exploiter win rate | 0.6712 | 0.8314 | New checkpoint is currently more exploitable. |
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Item | Path |
|
||||
| --- | --- |
|
||||
| Expert mirror JSON | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/expert-mirror-1000-duplicate.json` |
|
||||
| Expert vs old bots JSONs | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/` |
|
||||
| Gate 1 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/` |
|
||||
| Gate 2 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/` |
|
||||
| Gate 3 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/` |
|
||||
| Exploiter run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_014713_jax-ppo-ladder-v2-exploiter/` |
|
||||
|
||||
## Decisions
|
||||
|
||||
- The requested `lost-cities-league-selfplay-spec.md` file was not present in
|
||||
the repository. I created `docs/plans/lost-cities-league-selfplay-spec.md` as
|
||||
the tracked league spec location, following the repository docs routing for
|
||||
active plans.
|
||||
- Gate 2's high max-step rate is treated as a canary warning rather than a
|
||||
gate failure because the v2 instruction keeps gate 2's original score-based
|
||||
pass condition.
|
||||
|
||||
## Next
|
||||
|
||||
Proceed to league implementation only with the updated interpretation: anchors
|
||||
fix the Elo scale and provide style diversity, but they do not certify strength.
|
||||
The league's main success criterion should be lowering exploiter win rate from
|
||||
the v2 baseline of `0.83135` while preserving positive duplicate performance
|
||||
against `heuristic_expert`.
|
||||
Reference in New Issue
Block a user