Document JAX PPO ladder results
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Plan: JAX PPO Static-Opponent Ladder
|
||||
|
||||
**Status:** Ready to implement.
|
||||
**Status:** Static-opponent ladder passed on 2026-07-04.
|
||||
**Owner:** Codex implements; operator reviews training gates.
|
||||
**Scope:** A compact PPO training stack on top of `lost_cities_jax`, using GPU
|
||||
via optional CUDA JAX execution.
|
||||
@@ -68,7 +68,6 @@ Prefer a small number of files while keeping testable boundaries:
|
||||
src/lost_cities_jax/
|
||||
ppo.py # config, network, rollout, PPO update, train loop
|
||||
opponents.py # pure-JAX static opponent policy functions
|
||||
eval.py # duplicate evaluation and Wilson intervals
|
||||
ppo_cli.py # argparse CLI
|
||||
configs/jax_ppo/
|
||||
discard-only.yaml
|
||||
@@ -168,7 +167,7 @@ ppo:
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 8
|
||||
minibatches: 128
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
@@ -223,6 +222,45 @@ Gates:
|
||||
|
||||
Only after all three gates pass should league self-play begin.
|
||||
|
||||
## 2026-07-04 Gate Results
|
||||
|
||||
All three static-opponent gates passed on an RTX 3090 using optional CUDA JAX:
|
||||
|
||||
```bash
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo train --config <config>
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval --config <config> \
|
||||
--checkpoint <run>/latest --games 10000 --duplicate --output <eval.json>
|
||||
```
|
||||
|
||||
Training used `batch_games=8192`, `rollout_steps=400`, `total_updates=250`,
|
||||
`seed=20260704`, and artifact root
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
|
||||
The implementation commit used for the runs was `4c0c2e9`.
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
|
||||
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
|
||||
|
||||
Artifact directories:
|
||||
|
||||
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/`
|
||||
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/`
|
||||
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/`
|
||||
|
||||
Final training rollout canaries:
|
||||
|
||||
| Opponent | Final train return_mean | Final play_action_rate | Final max_steps_rate |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `discard_only` | 0.99281 | 0.59007 | 0.00000 |
|
||||
| `heuristic_balanced` | 0.93545 | 0.37737 | 0.00415 |
|
||||
| `heuristic_cautious` | 0.97101 | 0.37473 | 0.00391 |
|
||||
|
||||
The balanced and cautious policies win decisively but produce longer games than
|
||||
the discard-only diagnostic. That is not a gate failure, but the next training
|
||||
phase should keep game length and forced-end rate as canaries.
|
||||
|
||||
## Artifact Policy
|
||||
|
||||
Generated training output stays out of git. Use:
|
||||
@@ -313,10 +351,10 @@ tmux new-session -s coolrl-jax-ppo-discard \
|
||||
Duplicate evaluation:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
|
||||
--config configs/jax_ppo/discard-only.yaml \
|
||||
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/latest \
|
||||
--opponent discard_only \
|
||||
--shuffle-bank-seed 20260704 \
|
||||
--games 10000 \
|
||||
--duplicate \
|
||||
--output /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/eval_duplicate.json
|
||||
@@ -336,5 +374,5 @@ flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
|
||||
|
||||
## Current Next Action
|
||||
|
||||
Implement Phase 1: static opponents plus random-policy batched rollout with the
|
||||
full dashboard, then record the random baseline before adding PPO updates.
|
||||
The static-opponent ladder is complete. The next eligible phase is snapshot-pool
|
||||
league self-play design, using the three passed checkpoints as initial anchors.
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
# JAX PPO Static-Opponent Ladder Report - 2026-07-04
|
||||
|
||||
**Status:** PASS.
|
||||
**Implementation commit:** `4c0c2e9`.
|
||||
**Hardware:** RTX 3090 with optional CUDA JAX via `uv run --with 'jax[cuda12]'`.
|
||||
**Artifact root:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
|
||||
|
||||
## Protocol
|
||||
|
||||
Each opponent was trained from scratch with the same PPO configuration:
|
||||
|
||||
- `batch_games=8192`
|
||||
- `rollout_steps=400`
|
||||
- `total_updates=250`
|
||||
- `gamma=1.0`
|
||||
- `gae_lambda=0.95`
|
||||
- `clip_epsilon=0.2`
|
||||
- `entropy_coef=0.01`
|
||||
- `seed=20260704`
|
||||
|
||||
Evaluation used a fixed 10,000-deck shuffle bank and duplicate play: each deck
|
||||
was played once with the learner in seat 0 and once with the learner in seat 1,
|
||||
for 20,000 evaluated games per gate. Win rates include Wilson 95% intervals.
|
||||
|
||||
## Results
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean game length | Positive expeditions/game |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
|
||||
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
|
||||
|
||||
Gate 1 passed both required conditions: win rate >= 90% and at least two
|
||||
positive expeditions per game. Gates 2 and 3 passed their mean-score-difference
|
||||
condition.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Opponent | Run directory | Evaluation JSON |
|
||||
| --- | --- | --- |
|
||||
| `discard_only` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` | `eval_discard_only_duplicate.json` |
|
||||
| `heuristic_balanced` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` | `eval_balanced_duplicate.json` |
|
||||
| `heuristic_cautious` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` | `eval_cautious_duplicate.json` |
|
||||
|
||||
The random rollout baseline against `discard_only` is stored at:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/random_baseline_discard_only.json
|
||||
```
|
||||
|
||||
## Canary Notes
|
||||
|
||||
The random-policy baseline against `discard_only` had `play_action_rate=0.28855`
|
||||
and only `0.41797` positive expeditions per game. The trained discard-only
|
||||
policy ended duplicate evaluation at `play_action_rate=0.64598` and `3.2783`
|
||||
positive expeditions per game, so the intended anti-collapse signal is present.
|
||||
|
||||
Balanced and cautious checkpoints pass their score gates decisively, but their
|
||||
duplicate evaluations are longer: mean game lengths are `167.81450` and
|
||||
`185.38690`. The final training rollout forced-end rates were low but nonzero
|
||||
for those opponents (`0.00415` and `0.00391`). Treat game length and forced-end
|
||||
rate as canaries in the next self-play phase.
|
||||
|
||||
## Next Step
|
||||
|
||||
The static ladder has cleared. The next phase can start snapshot-pool league
|
||||
self-play, using these three passed checkpoints as initial anchors and keeping
|
||||
duplicate evaluation as the regression gate.
|
||||
@@ -14,7 +14,7 @@ The core algorithmic components have been ported to Cython to ensure C-level per
|
||||
- `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:406`: `def encode_info_state` generates the information-state feature vectors required for network inference.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives.
|
||||
|
||||
## Analysis
|
||||
@@ -39,4 +39,4 @@ Until this transition is made, optimization efforts should focus on reducing the
|
||||
|
||||
- `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`: Original status and gap analysis.
|
||||
- `docs/research/deep-cfr-v0-feature-parity.md`: Detailed subsystem coverage report.
|
||||
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
|
||||
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
|
||||
|
||||
Reference in New Issue
Block a user