coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Document JAX PPO ladder results

Browse Source
This commit is contained in:
coolguy
2026-07-04 23:19:41 +09:00
parent 4c0c2e9add
commit bb52ef9ec1
4 changed files with 127 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+12 -1
View File
@@ -233,9 +233,20 @@ opened_colors_mean=4.942626953125
positive_expeditions_mean=0.41796875
```
Static-opponent gate results, measured on 2026-07-04 with 10,000 fixed shuffles
and duplicate seat-swapped evaluation:
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Large PPO artifacts are written under
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
Generated checkpoints and evaluation JSON are not committed to git.
Generated checkpoints and evaluation JSON are not committed to git. The full
run summary is in
[docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md](docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md).
## Basic Usage
docs/plans/jax-ppo-static-opponents.md
+45 -7
View File
@@ -1,6 +1,6 @@
# Plan: JAX PPO Static-Opponent Ladder
**Status:** Ready to implement.
**Status:** Static-opponent ladder passed on 2026-07-04.
**Owner:** Codex implements; operator reviews training gates.
**Scope:** A compact PPO training stack on top of `lost_cities_jax`, using GPU
via optional CUDA JAX execution.
@@ -68,7 +68,6 @@ Prefer a small number of files while keeping testable boundaries:
src/lost_cities_jax/
ppo.py # config, network, rollout, PPO update, train loop
opponents.py # pure-JAX static opponent policy functions
eval.py # duplicate evaluation and Wilson intervals
ppo_cli.py # argparse CLI
configs/jax_ppo/
discard-only.yaml
@@ -168,7 +167,7 @@ ppo:
max_grad_norm: 0.5
learning_rate: 0.0003
epochs: 4
minibatches: 8
minibatches: 128
reward:
terminal_scale: 50.0
potential_shaping_initial: 1.0
@@ -223,6 +222,45 @@ Gates:
Only after all three gates pass should league self-play begin.
## 2026-07-04 Gate Results
All three static-opponent gates passed on an RTX 3090 using optional CUDA JAX:
```bash
uv run --with 'jax[cuda12]' lost-cities-jax-ppo train --config <config>
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval --config <config> \
--checkpoint <run>/latest --games 10000 --duplicate --output <eval.json>
```
Training used `batch_games=8192`, `rollout_steps=400`, `total_updates=250`,
`seed=20260704`, and artifact root
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
The implementation commit used for the runs was `4c0c2e9`.
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Artifact directories:
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/`
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/`
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/`
Final training rollout canaries:
| Opponent | Final train return_mean | Final play_action_rate | Final max_steps_rate |
| --- | ---: | ---: | ---: |
| `discard_only` | 0.99281 | 0.59007 | 0.00000 |
| `heuristic_balanced` | 0.93545 | 0.37737 | 0.00415 |
| `heuristic_cautious` | 0.97101 | 0.37473 | 0.00391 |
The balanced and cautious policies win decisively but produce longer games than
the discard-only diagnostic. That is not a gate failure, but the next training
phase should keep game length and forced-end rate as canaries.
## Artifact Policy
Generated training output stays out of git. Use:
@@ -313,10 +351,10 @@ tmux new-session -s coolrl-jax-ppo-discard \
Duplicate evaluation:
```bash
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
--config configs/jax_ppo/discard-only.yaml \
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/latest \
--opponent discard_only \
--shuffle-bank-seed 20260704 \
--games 10000 \
--duplicate \
--output /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/eval_duplicate.json
@@ -336,5 +374,5 @@ flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
## Current Next Action
Implement Phase 1: static opponents plus random-policy batched rollout with the
full dashboard, then record the random baseline before adding PPO updates.
The static-opponent ladder is complete. The next eligible phase is snapshot-pool
league self-play design, using the three passed checkpoints as initial anchors.
docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md
+68
View File
@@ -0,0 +1,68 @@
# JAX PPO Static-Opponent Ladder Report - 2026-07-04
**Status:** PASS.
**Implementation commit:** `4c0c2e9`.
**Hardware:** RTX 3090 with optional CUDA JAX via `uv run --with 'jax[cuda12]'`.
**Artifact root:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
## Protocol
Each opponent was trained from scratch with the same PPO configuration:
- `batch_games=8192`
- `rollout_steps=400`
- `total_updates=250`
- `gamma=1.0`
- `gae_lambda=0.95`
- `clip_epsilon=0.2`
- `entropy_coef=0.01`
- `seed=20260704`
Evaluation used a fixed 10,000-deck shuffle bank and duplicate play: each deck
was played once with the learner in seat 0 and once with the learner in seat 1,
for 20,000 evaluated games per gate. Win rates include Wilson 95% intervals.
## Results
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean game length | Positive expeditions/game |
| --- | --- | --- | --- | ---: | ---: | ---: |
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
Gate 1 passed both required conditions: win rate >= 90% and at least two
positive expeditions per game. Gates 2 and 3 passed their mean-score-difference
condition.
## Artifacts
| Opponent | Run directory | Evaluation JSON |
| --- | --- | --- |
| `discard_only` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` | `eval_discard_only_duplicate.json` |
| `heuristic_balanced` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` | `eval_balanced_duplicate.json` |
| `heuristic_cautious` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` | `eval_cautious_duplicate.json` |
The random rollout baseline against `discard_only` is stored at:
```text
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/random_baseline_discard_only.json
```
## Canary Notes
The random-policy baseline against `discard_only` had `play_action_rate=0.28855`
and only `0.41797` positive expeditions per game. The trained discard-only
policy ended duplicate evaluation at `play_action_rate=0.64598` and `3.2783`
positive expeditions per game, so the intended anti-collapse signal is present.
Balanced and cautious checkpoints pass their score gates decisively, but their
duplicate evaluations are longer: mean game lengths are `167.81450` and
`185.38690`. The final training rollout forced-end rates were low but nonzero
for those opponents (`0.00415` and `0.00391`). Treat game length and forced-end
rate as canaries in the next self-play phase.
## Next Step
The static ladder has cleared. The next phase can start snapshot-pool league
self-play, using these three passed checkpoints as initial anchors and keeping
duplicate evaluation as the regression gate.
docs/research/deep-cfr-v0-gap-vs-coolrl.md
+1 -1
View File
@@ -14,7 +14,7 @@ The core algorithmic components have been ported to Cython to ensure C-level per
- `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring.
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine.
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling.
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference.
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:406`: `def encode_info_state` generates the information-state feature vectors required for network inference.
- `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives.
## Analysis