From bb52ef9ec1fb8fb678e5c2f76cf3a0af1524e82d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=A0=95=EC=8B=9C=EC=9B=90?= Date: Sat, 4 Jul 2026 23:19:41 +0900 Subject: [PATCH] Document JAX PPO ladder results --- README.md | 13 +++- docs/plans/jax-ppo-static-opponents.md | 52 ++++++++++++-- ...x-ppo-static-opponent-ladder-2026-07-04.md | 68 +++++++++++++++++++ docs/research/deep-cfr-v0-gap-vs-coolrl.md | 4 +- 4 files changed, 127 insertions(+), 10 deletions(-) create mode 100644 docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md diff --git a/README.md b/README.md index 3c6f0e9..af0dbf3 100644 --- a/README.md +++ b/README.md @@ -233,9 +233,20 @@ opened_colors_mean=4.942626953125 positive_expeditions_mean=0.41796875 ``` +Static-opponent gate results, measured on 2026-07-04 with 10,000 fixed shuffles +and duplicate seat-swapped evaluation: + +| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game | +| --- | --- | --- | --- | ---: | ---: | ---: | +| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 | +| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 | +| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 | + Large PPO artifacts are written under `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`. -Generated checkpoints and evaluation JSON are not committed to git. +Generated checkpoints and evaluation JSON are not committed to git. The full +run summary is in +[docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md](docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md). ## Basic Usage diff --git a/docs/plans/jax-ppo-static-opponents.md b/docs/plans/jax-ppo-static-opponents.md index b23f856..a8b79f7 100644 --- a/docs/plans/jax-ppo-static-opponents.md +++ b/docs/plans/jax-ppo-static-opponents.md @@ -1,6 +1,6 @@ # Plan: JAX PPO Static-Opponent Ladder -**Status:** Ready to implement. +**Status:** Static-opponent ladder passed on 2026-07-04. **Owner:** Codex implements; operator reviews training gates. **Scope:** A compact PPO training stack on top of `lost_cities_jax`, using GPU via optional CUDA JAX execution. @@ -68,7 +68,6 @@ Prefer a small number of files while keeping testable boundaries: src/lost_cities_jax/ ppo.py # config, network, rollout, PPO update, train loop opponents.py # pure-JAX static opponent policy functions - eval.py # duplicate evaluation and Wilson intervals ppo_cli.py # argparse CLI configs/jax_ppo/ discard-only.yaml @@ -168,7 +167,7 @@ ppo: max_grad_norm: 0.5 learning_rate: 0.0003 epochs: 4 - minibatches: 8 + minibatches: 128 reward: terminal_scale: 50.0 potential_shaping_initial: 1.0 @@ -223,6 +222,45 @@ Gates: Only after all three gates pass should league self-play begin. +## 2026-07-04 Gate Results + +All three static-opponent gates passed on an RTX 3090 using optional CUDA JAX: + +```bash +uv run --with 'jax[cuda12]' lost-cities-jax-ppo train --config +uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval --config \ + --checkpoint /latest --games 10000 --duplicate --output +``` + +Training used `batch_games=8192`, `rollout_steps=400`, `total_updates=250`, +`seed=20260704`, and artifact root +`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`. +The implementation commit used for the runs was `4c0c2e9`. + +| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game | +| --- | --- | --- | --- | ---: | ---: | ---: | +| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 | +| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 | +| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 | + +Artifact directories: + +- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` +- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` +- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` + +Final training rollout canaries: + +| Opponent | Final train return_mean | Final play_action_rate | Final max_steps_rate | +| --- | ---: | ---: | ---: | +| `discard_only` | 0.99281 | 0.59007 | 0.00000 | +| `heuristic_balanced` | 0.93545 | 0.37737 | 0.00415 | +| `heuristic_cautious` | 0.97101 | 0.37473 | 0.00391 | + +The balanced and cautious policies win decisively but produce longer games than +the discard-only diagnostic. That is not a gate failure, but the next training +phase should keep game length and forced-end rate as canaries. + ## Artifact Policy Generated training output stays out of git. Use: @@ -313,10 +351,10 @@ tmux new-session -s coolrl-jax-ppo-discard \ Duplicate evaluation: ```bash -flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \ +uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \ + --config configs/jax_ppo/discard-only.yaml \ --checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents//latest \ --opponent discard_only \ - --shuffle-bank-seed 20260704 \ --games 10000 \ --duplicate \ --output /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents//eval_duplicate.json @@ -336,5 +374,5 @@ flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \ ## Current Next Action -Implement Phase 1: static opponents plus random-policy batched rollout with the -full dashboard, then record the random baseline before adding PPO updates. +The static-opponent ladder is complete. The next eligible phase is snapshot-pool +league self-play design, using the three passed checkpoints as initial anchors. diff --git a/docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md b/docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md new file mode 100644 index 0000000..d0611a5 --- /dev/null +++ b/docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md @@ -0,0 +1,68 @@ +# JAX PPO Static-Opponent Ladder Report - 2026-07-04 + +**Status:** PASS. +**Implementation commit:** `4c0c2e9`. +**Hardware:** RTX 3090 with optional CUDA JAX via `uv run --with 'jax[cuda12]'`. +**Artifact root:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`. + +## Protocol + +Each opponent was trained from scratch with the same PPO configuration: + +- `batch_games=8192` +- `rollout_steps=400` +- `total_updates=250` +- `gamma=1.0` +- `gae_lambda=0.95` +- `clip_epsilon=0.2` +- `entropy_coef=0.01` +- `seed=20260704` + +Evaluation used a fixed 10,000-deck shuffle bank and duplicate play: each deck +was played once with the learner in seat 0 and once with the learner in seat 1, +for 20,000 evaluated games per gate. Win rates include Wilson 95% intervals. + +## Results + +| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean game length | Positive expeditions/game | +| --- | --- | --- | --- | ---: | ---: | ---: | +| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 | +| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 | +| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 | + +Gate 1 passed both required conditions: win rate >= 90% and at least two +positive expeditions per game. Gates 2 and 3 passed their mean-score-difference +condition. + +## Artifacts + +| Opponent | Run directory | Evaluation JSON | +| --- | --- | --- | +| `discard_only` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` | `eval_discard_only_duplicate.json` | +| `heuristic_balanced` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` | `eval_balanced_duplicate.json` | +| `heuristic_cautious` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` | `eval_cautious_duplicate.json` | + +The random rollout baseline against `discard_only` is stored at: + +```text +/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/random_baseline_discard_only.json +``` + +## Canary Notes + +The random-policy baseline against `discard_only` had `play_action_rate=0.28855` +and only `0.41797` positive expeditions per game. The trained discard-only +policy ended duplicate evaluation at `play_action_rate=0.64598` and `3.2783` +positive expeditions per game, so the intended anti-collapse signal is present. + +Balanced and cautious checkpoints pass their score gates decisively, but their +duplicate evaluations are longer: mean game lengths are `167.81450` and +`185.38690`. The final training rollout forced-end rates were low but nonzero +for those opponents (`0.00415` and `0.00391`). Treat game length and forced-end +rate as canaries in the next self-play phase. + +## Next Step + +The static ladder has cleared. The next phase can start snapshot-pool league +self-play, using these three passed checkpoints as initial anchors and keeping +duplicate evaluation as the regression gate. diff --git a/docs/research/deep-cfr-v0-gap-vs-coolrl.md b/docs/research/deep-cfr-v0-gap-vs-coolrl.md index dcb7396..5800232 100644 --- a/docs/research/deep-cfr-v0-gap-vs-coolrl.md +++ b/docs/research/deep-cfr-v0-gap-vs-coolrl.md @@ -14,7 +14,7 @@ The core algorithmic components have been ported to Cython to ensure C-level per - `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring. - `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine. - `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling. -- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference. +- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:406`: `def encode_info_state` generates the information-state feature vectors required for network inference. - `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives. ## Analysis @@ -39,4 +39,4 @@ Until this transition is made, optimization efforts should focus on reducing the - `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`: Original status and gap analysis. - `docs/research/deep-cfr-v0-feature-parity.md`: Detailed subsystem coverage report. -- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server. \ No newline at end of file +- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.