Add JAX PPO ladder v2 expert pass
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# JAX PPO Ladder v2 Report - 2026-07-05
|
||||
|
||||
**Status:** PASS for the ordered ladder gates; canary warnings remain.
|
||||
**Hardware:** RTX 3090 via `uv run --with 'jax[cuda12]'`.
|
||||
**Artifact roots:**
|
||||
|
||||
- Expert bot: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/`
|
||||
- Ladder v2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/`
|
||||
|
||||
## Protocol
|
||||
|
||||
Part A added a pure-JAX `heuristic_expert` opponent and evaluated it with a
|
||||
fixed shuffle bank plus duplicate play. Part B trained PPO from random
|
||||
initialization for each gate with the same 250-update configuration used by the
|
||||
original ladder, changing only the opponent and artifact root:
|
||||
|
||||
- Gate 1: `discard_only`
|
||||
- Gate 2: `heuristic_balanced`
|
||||
- Gate 3: `heuristic_expert`
|
||||
|
||||
All evaluations below used 10,000 deck orders with duplicate seat-swapped play,
|
||||
for 20,000 evaluated games.
|
||||
|
||||
## Part A - Expert Bot Gate
|
||||
|
||||
| Check | Result | Win rate | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Expert mirror, 1,000 duplicate pairs | PASS | 0.4815 | 0.0000 | [-0.6956, +0.6956] | 45.1560 | 0.0000 | 2.2560 | 0.3118 |
|
||||
| Expert vs `discard_only` | PASS | 0.6643 | +8.9716 | [+8.7886, +9.1546] | 45.2125 | 0.0000 | 1.8971 | 0.2828 |
|
||||
| Expert vs `heuristic_balanced` | PASS | 0.9961 | +54.3673 | [+54.0791, +54.6554] | 64.9181 | 0.0000 | 2.8003 | 0.2728 |
|
||||
| Expert vs `heuristic_cautious` | PASS | 0.9972 | +51.9397 | [+51.6960, +52.1834] | 75.2329 | 0.0000 | 2.6090 | 0.2240 |
|
||||
|
||||
The new script bot is non-stalling in mirror play, symmetric under duplicate
|
||||
seat swap, and beats all three previous static bots by a clear positive score
|
||||
margin. Its own opened-color profile is in the intended 2-3 color band.
|
||||
|
||||
## Part B - Ladder v2
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.0000 [0.9998, 1.0000] | +182.0670 | [+181.6390, +182.4950] | 82.1558 | 0.0000 | 4.2163 | 0.6398 | 3.1845 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.9409 [0.9375, 0.9441] | +119.9250 | [+118.9950, +120.8550] | 151.2468 | 0.1859 | 4.9862 | 0.2937 | 3.5316 |
|
||||
| 3 | `heuristic_expert` | PASS | 0.8382 [0.8330, 0.8432] | +43.3099 | [+42.7065, +43.9133] | 61.4025 | 0.0000 | 4.5509 | 0.6661 | 2.6121 |
|
||||
|
||||
Gate 1 passed the original win-rate and positive-expedition conditions. Gate 2
|
||||
passed the mean-score-difference condition. Gate 3 passed the v2 condition:
|
||||
duplicate mean score difference is positive and its 95% confidence lower bound
|
||||
is above zero.
|
||||
|
||||
## Canary Findings
|
||||
|
||||
The ladder did not produce the desired 2-3 color selectivity in the learned PPO
|
||||
policies. The gate-3 policy still opens 4.5509 colors/game against the expert.
|
||||
This is lower than the original gate-3 checkpoint but still far outside the
|
||||
target band. Treat the selectivity hypothesis as not confirmed.
|
||||
|
||||
Gate 2 also has a high forced-end canary: `max_steps_rate=0.1859` in duplicate
|
||||
evaluation versus `heuristic_balanced`. It passed its score gate, but the
|
||||
balanced opponent still permits long games and 5-color farming.
|
||||
|
||||
## Exploiter Baseline
|
||||
|
||||
After gate 3 passed, a new PPO exploiter was trained from random initialization
|
||||
against the frozen v2 gate-3 checkpoint with the same 250-update budget and
|
||||
duplicate-evaluated against that checkpoint.
|
||||
|
||||
| Target checkpoint | Exploiter win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Old gate-3 checkpoint | 0.6712 [0.6646, 0.6776] | +24.1371 | n/a | 48.0017 | 0.0000 | 4.8971 | 0.7461 | 2.1287 |
|
||||
| New v2 gate-3 checkpoint | 0.8314 [0.8261, 0.8365] | +54.7721 | [+53.9871, +55.5571] | 60.8744 | 0.0001 | 4.9914 | 0.7028 | 2.8921 |
|
||||
|
||||
The new gate-3 checkpoint beats `heuristic_expert`, but is more exploitable
|
||||
under this protocol than the old checkpoint. That does not invalidate the gate,
|
||||
but it makes exploiter reduction a primary objective for the league phase.
|
||||
|
||||
## Old vs New Gate-3 Canary Comparison
|
||||
|
||||
| Metric | Old gate-3 vs cautious | New gate-3 vs expert | Interpretation |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Mean score diff | +142.8993 | +43.3099 | New gate is harder and less suspiciously clean. |
|
||||
| Win rate | 0.9596 | 0.8382 | New result is less inflated. |
|
||||
| Opened colors | 4.9896 | 4.5509 | Improved, but still not selective enough. |
|
||||
| Max-step rate | not recorded in old eval | 0.0000 | New expert evaluation does not stall. |
|
||||
| Play action rate | 0.2763 | 0.6661 | New policy plays much more actively. |
|
||||
| Exploiter win rate | 0.6712 | 0.8314 | New checkpoint is currently more exploitable. |
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Item | Path |
|
||||
| --- | --- |
|
||||
| Expert mirror JSON | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/expert-mirror-1000-duplicate.json` |
|
||||
| Expert vs old bots JSONs | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/` |
|
||||
| Gate 1 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/` |
|
||||
| Gate 2 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/` |
|
||||
| Gate 3 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/` |
|
||||
| Exploiter run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_014713_jax-ppo-ladder-v2-exploiter/` |
|
||||
|
||||
## Decisions
|
||||
|
||||
- The requested `lost-cities-league-selfplay-spec.md` file was not present in
|
||||
the repository. I created `docs/plans/lost-cities-league-selfplay-spec.md` as
|
||||
the tracked league spec location, following the repository docs routing for
|
||||
active plans.
|
||||
- Gate 2's high max-step rate is treated as a canary warning rather than a
|
||||
gate failure because the v2 instruction keeps gate 2's original score-based
|
||||
pass condition.
|
||||
|
||||
## Next
|
||||
|
||||
Proceed to league implementation only with the updated interpretation: anchors
|
||||
fix the Elo scale and provide style diversity, but they do not certify strength.
|
||||
The league's main success criterion should be lowering exploiter win rate from
|
||||
the v2 baseline of `0.83135` while preserving positive duplicate performance
|
||||
against `heuristic_expert`.
|
||||
Reference in New Issue
Block a user