Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 coolguy
|
dafb7a6a1d | ||
|
coolguy
|
1ee329250e |
@@ -1,46 +0,0 @@
|
||||
name: Deploy web client
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "web/**"
|
||||
- ".github/workflows/deploy-web.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
concurrency:
|
||||
group: github-pages
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 22
|
||||
cache: npm
|
||||
cache-dependency-path: web/package-lock.json
|
||||
- run: npm ci
|
||||
working-directory: web
|
||||
- run: npm run build -- --base=/${{ github.event.repository.name }}/
|
||||
working-directory: web
|
||||
- uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: web/dist
|
||||
|
||||
deploy:
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
-15
@@ -15,9 +15,6 @@ src/**/*.c
|
||||
# Local session / lock files
|
||||
.compute.lock
|
||||
.claude/
|
||||
# ...except checked-in project skills, which are documentation for the next session.
|
||||
!web/.claude/
|
||||
!web/.claude/**
|
||||
|
||||
# Rust build output
|
||||
target/
|
||||
@@ -30,15 +27,3 @@ tools/julia/
|
||||
|
||||
.pytest_cache
|
||||
.ruff_cache
|
||||
|
||||
# Web dependencies and locally exported models. The verified public browser
|
||||
# policies below are the exception: they are deliberately served as static assets,
|
||||
# and the deploy builds straight from the repo, so an ignored model ships as a 404.
|
||||
web/node_modules/
|
||||
web/*.tsbuildinfo
|
||||
web/public/models/*.onnx
|
||||
web/public/models/*.json
|
||||
!web/public/models/jax-ppo.onnx
|
||||
!web/public/models/jax-ppo.json
|
||||
!web/public/models/borealis.onnx
|
||||
!web/public/models/borealis.json
|
||||
|
||||
@@ -1,18 +0,0 @@
|
||||
stages:
|
||||
- deploy
|
||||
|
||||
deploy-pages:
|
||||
stage: deploy
|
||||
image: node:22
|
||||
rules:
|
||||
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
|
||||
changes:
|
||||
- web/**/*
|
||||
- .gitlab-ci.yml
|
||||
script:
|
||||
- cd web
|
||||
- npm ci
|
||||
- npm run build -- --base="${CI_PAGES_URL}/"
|
||||
- mv dist ../public
|
||||
pages:
|
||||
publish: public
|
||||
@@ -1,98 +1,442 @@
|
||||
# AGENTS.md
|
||||
|
||||
This repository is managed with `uv`. Use `uv run ...` so the project
|
||||
environment and Cython extensions are built and loaded consistently.
|
||||
This repository is managed with `uv`. Use `uv run ...` for commands so the
|
||||
project environment and Cython extensions are built/loaded consistently.
|
||||
|
||||
## Active project path
|
||||
## Project Layout
|
||||
|
||||
The active product and research path is **JAX + PPO**:
|
||||
- `src/coolrl_lost_cities/games/classic/game.pyx`: Cython Lost Cities engine.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/`: Deep CFR training,
|
||||
traversal, evaluation, analysis, and CLI code.
|
||||
- `configs/deep_cfr/`: active Deep CFR YAML configs (kebab-case filenames).
|
||||
Currently holds two:
|
||||
- `default.yaml`: the canonical "best-known" baseline. Start here, then
|
||||
override fields via `--set` for experiments/ablations.
|
||||
- `smoke.yaml`: 1-iter sanity check for the training loop.
|
||||
- `configs/archive/`: retired/historical configs. Don't modify; reference
|
||||
if you need to reproduce an old run.
|
||||
- `runs/`: generated training runs. Gitignored, may be a symlink to larger
|
||||
storage. Layout:
|
||||
- `runs/archive/`: past runs. **Do not modify or delete.**
|
||||
- `runs/tmp/`: smoke, tests, throwaway. Free to `rm -rf` anytime.
|
||||
- `runs/<YYYY-MM-DD_HHMMSS>_<kebab-name>/`: real experiments (flat).
|
||||
Promote a `runs/<...>` directory to `runs/archive/` with a manual `mv`
|
||||
once analysis is complete.
|
||||
- `docs/`: profiling notes, migration notes, and experiment documentation.
|
||||
|
||||
- `src/lost_cities_jax/`: pure JAX rules, observations, PPO, evaluation,
|
||||
league, and human-play tools.
|
||||
- `configs/jax_ppo/`: active YAML configurations.
|
||||
- `web/`: static TypeScript client with the final JAX PPO ONNX policy.
|
||||
- `web/public/models/jax-ppo.onnx`: verified browser policy. Keep its manifest
|
||||
in sync and do not replace it without running the export validation.
|
||||
## Core Commands
|
||||
|
||||
Historical Deep CFR and ISMCTS implementations live under
|
||||
`src/coolrl_lost_cities/games/classic/`. Their configs are in `legacy/`; they
|
||||
are reproduction-only and must not be used as the default for new work. See
|
||||
`docs/legacy.md`.
|
||||
|
||||
## Core commands
|
||||
Run lint:
|
||||
|
||||
```bash
|
||||
uv run ruff check .
|
||||
uv run pytest -q tests/lost_cities_jax
|
||||
```
|
||||
|
||||
Run all tests:
|
||||
|
||||
```bash
|
||||
uv run pytest -q
|
||||
uv run lost-cities-jax-ppo --help
|
||||
```
|
||||
|
||||
Web checks:
|
||||
Run focused Deep CFR tests:
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npm test
|
||||
npm run build
|
||||
uv run pytest -q tests/games/classic/test_deep_cfr_trainer.py
|
||||
```
|
||||
|
||||
## JAX PPO workflow
|
||||
|
||||
CPU smoke runs write disposable artifacts under `runs/tmp/`:
|
||||
Run the CLI through the console script:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
|
||||
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
|
||||
uv run lost-cities-deep-cfr --help
|
||||
```
|
||||
|
||||
GPU training must hold the shared compute lock. Use a real `tmux` session for
|
||||
long user-observable jobs:
|
||||
Equivalent module form:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
|
||||
--config configs/jax_ppo/balanced.yaml \
|
||||
--set run.artifact_root=runs/jax-ppo
|
||||
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli --help
|
||||
```
|
||||
|
||||
Evaluation is deterministic and does not need the lock:
|
||||
## Deep CFR Training
|
||||
|
||||
The CLI auto-derives the run directory from `run.experiment_name` plus a
|
||||
timestamp. By default runs land under `runs/tmp/`; pass `--keep` for a real
|
||||
experiment that should live under `runs/`.
|
||||
|
||||
Smoke / throwaway run (lands in `runs/tmp/`):
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo eval \
|
||||
--config configs/jax_ppo/balanced.yaml \
|
||||
--checkpoint runs/jax-ppo/<run>/latest \
|
||||
--opponent heuristic_balanced --games 10000 --duplicate
|
||||
uv run lost-cities-deep-cfr train --config configs/deep_cfr/smoke.yaml
|
||||
# → runs/tmp/<YYYY-MM-DD_HHMMSS>_smoke/
|
||||
```
|
||||
|
||||
Generated artifacts under `runs/` are gitignored. Do not modify or delete
|
||||
`runs/archive/`.
|
||||
|
||||
## Static browser policy
|
||||
|
||||
The checked-in ONNX policy is small enough for static distribution. To replace
|
||||
it, export a validated checkpoint and rebuild the web app:
|
||||
Real experiment (lands in `runs/`):
|
||||
|
||||
```bash
|
||||
uv run --with onnx scripts/export_jax_ppo_onnx.py \
|
||||
--checkpoint /path/to/checkpoint \
|
||||
--output web/public/models/jax-ppo.onnx
|
||||
cd web && npm test && npm run build
|
||||
uv run lost-cities-deep-cfr train \
|
||||
--config configs/deep_cfr/default.yaml \
|
||||
--keep
|
||||
# → runs/<YYYY-MM-DD_HHMMSS>_deep-cfr-default/
|
||||
```
|
||||
|
||||
Commit the ONNX file and its JSON manifest together. The app must continue to
|
||||
use a base-relative model URL so static subpath hosts work.
|
||||
Variant / ablation (override one field; keep slug informative):
|
||||
|
||||
## Documentation
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr train \
|
||||
--config configs/deep_cfr/default.yaml \
|
||||
--keep \
|
||||
--set training_weighting.mode=none \
|
||||
--set run.experiment_name=ablation-no-lcfr
|
||||
# → runs/<YYYY-MM-DD_HHMMSS>_ablation-no-lcfr/
|
||||
```
|
||||
|
||||
- Active plans: `docs/plans/`.
|
||||
- Retired plans and historical evidence: `docs/plans/archive/`,
|
||||
`docs/archive/`, and `docs/research/`.
|
||||
- Do not edit archival documents; add a new dated note or active plan instead.
|
||||
Short fixed-iteration run:
|
||||
|
||||
When changing docs, run `scripts/librarian.sh` to validate Markdown links and
|
||||
`file:line` citations.
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr train \
|
||||
--config configs/deep_cfr/default.yaml \
|
||||
--set run.max_iterations=100 \
|
||||
--set checkpoint.save_every=0
|
||||
```
|
||||
|
||||
## Git policy
|
||||
Resume (path required, no shortcut):
|
||||
|
||||
Do not create git branches unless the user explicitly asks in the current
|
||||
task. Work on the checked-out branch. Before committing, run `uv run ruff
|
||||
check .` and the relevant tests. Never commit generated run artifacts.
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr train \
|
||||
--config configs/deep_cfr/default.yaml \
|
||||
--resume runs/<YYYY-MM-DD_HHMMSS>_<slug>/latest.pt
|
||||
```
|
||||
|
||||
When `--resume` is given, the trainer reuses the resumed checkpoint's parent
|
||||
directory; no new timestamped folder is created.
|
||||
|
||||
Useful train controls:
|
||||
|
||||
- `--keep`: real experiment, write under `runs/` (default is `runs/tmp/`).
|
||||
- `--resume PATH`: resume from a specific checkpoint. `PATH` is required.
|
||||
- `--set PATH=VALUE`: override config fields. Repeatable, parses values as
|
||||
YAML (e.g. `--set traversal.num_workers=4`, `--set run.max_minutes=null`).
|
||||
|
||||
Common `--set` overrides:
|
||||
|
||||
- `--set run.device=cuda`: set the trainer device.
|
||||
- `--set run.experiment_name=foo-v2`: change the slug used in the run dir
|
||||
name (kebab-case).
|
||||
- `--set checkpoint.exact_resume=true`: require checkpoint config compatibility.
|
||||
- `--set checkpoint.save_latest=false --set checkpoint.save_every=0`:
|
||||
disable checkpoint writes.
|
||||
- `--set checkpoint.save_every=0`: keep only `latest.pt` (no archives).
|
||||
- `--set checkpoint.save_every=N`: archive every N iterations.
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Directory names, run dirs, config filenames, `experiment_name` values**:
|
||||
kebab-case (`deep-cfr-color-shared-512x3.yaml`,
|
||||
`runs/2026-05-08_103045_color-attn-v2/`).
|
||||
- **YAML keys, Python identifiers, config field names**: snake_case
|
||||
(unchanged: `hidden_size`, `traversals_per_player`, `experiment_name`).
|
||||
- The CLI converts `run.experiment_name` to a kebab slug when building the
|
||||
run directory, so values may contain spaces or mixed case.
|
||||
|
||||
## Long Runs
|
||||
|
||||
Run long jobs in a real `tmux` session so the user can attach and stop them.
|
||||
Do not rely on Codex command sessions for long user-observable training runs.
|
||||
|
||||
Start a long unbounded run:
|
||||
|
||||
```bash
|
||||
tmux new-session -s coolrl-deepcfr-unbounded \
|
||||
-c /home/coolguy/dev/coolrl-lost-cities \
|
||||
'uv run lost-cities-deep-cfr train \
|
||||
--config configs/deep_cfr/default.yaml \
|
||||
--set run.max_iterations=null \
|
||||
--set run.max_minutes=null \
|
||||
--keep'
|
||||
```
|
||||
|
||||
Attach later:
|
||||
|
||||
```bash
|
||||
tmux attach -t coolrl-deepcfr-unbounded
|
||||
```
|
||||
|
||||
Detach without stopping:
|
||||
|
||||
```text
|
||||
Ctrl+B, D
|
||||
```
|
||||
|
||||
Stop training:
|
||||
|
||||
```text
|
||||
Ctrl+C
|
||||
```
|
||||
|
||||
Follow logs from another terminal:
|
||||
|
||||
```bash
|
||||
tail -f runs/<YYYY-MM-DD_HHMMSS>_<slug>/train.log
|
||||
```
|
||||
|
||||
The unbounded config intentionally has:
|
||||
|
||||
```yaml
|
||||
run:
|
||||
max_iterations: null
|
||||
max_minutes: null
|
||||
checkpoint:
|
||||
save_every: 100
|
||||
```
|
||||
|
||||
`latest.pt` is updated continuously; archive checkpoints are written every 100
|
||||
iterations. If disk is tight, set `--set checkpoint.save_every=0` (keep only
|
||||
`latest.pt`) or increase `save_every`.
|
||||
|
||||
## Compute Lock
|
||||
|
||||
여러 에이전트가 한 머신을 공유하므로, train과 속도 벤치마크는
|
||||
repo 루트 `.compute.lock`을 잡고 실행한다:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run lost-cities-deep-cfr train ...
|
||||
```
|
||||
|
||||
승률/점수만 뽑는 eval과 `analyze`는 결정적이라 락 불필요.
|
||||
|
||||
## Evaluation And Analysis
|
||||
|
||||
Evaluate a checkpoint:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr eval \
|
||||
--checkpoint runs/<run-dir>/latest.pt \
|
||||
--opponent random \
|
||||
--games 100 \
|
||||
--device cpu
|
||||
```
|
||||
|
||||
Save evaluation game records:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr eval \
|
||||
--checkpoint runs/<run-dir>/latest.pt \
|
||||
--opponent random \
|
||||
--games 100 \
|
||||
--device cpu \
|
||||
--save-games runs/<run-dir>/eval_random_games.json
|
||||
```
|
||||
|
||||
Generate analysis plots from `metrics.jsonl`:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr analyze \
|
||||
--run runs/<run-dir>
|
||||
```
|
||||
|
||||
Write plots to a separate directory:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-deep-cfr analyze \
|
||||
--run runs/<run-dir> \
|
||||
--output-dir runs/<run-dir>/analysis
|
||||
```
|
||||
|
||||
The analyzer reads `metrics.jsonl` and writes PNG files grouped by diagnostic
|
||||
section. Opponents are compared within each plot using fixed colors. The
|
||||
`lost-cities-deep-cfr analyze` subcommand uses the analyzer default smoothing
|
||||
window, currently 1 iteration (no smoothing), and supports `--max-iteration`.
|
||||
|
||||
For smoothing controls, run the analyzer module directly:
|
||||
|
||||
```bash
|
||||
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.analyze \
|
||||
--run runs/<run-dir> \
|
||||
--smoothing-window 5
|
||||
```
|
||||
|
||||
Use `--no-smoothing` to force no moving average.
|
||||
|
||||
Current output files:
|
||||
|
||||
- `analysis_01_loss.png`
|
||||
- `analysis_02_match.png`
|
||||
- `analysis_03_action.png`
|
||||
- `analysis_04_gameflow.png`
|
||||
- `analysis_05_open_quality.png`
|
||||
- `analysis_06_expedition_outcomes.png`
|
||||
- `analysis_07_calibration.png`
|
||||
- `analysis_08_traversal.png`
|
||||
- `analysis_09_selectivity.png`
|
||||
- `analysis_final_eval_summary.png`
|
||||
|
||||
## Runtime Artifacts
|
||||
|
||||
Each training run writes:
|
||||
|
||||
- `metrics.jsonl`: structured metrics, one completed iteration per line.
|
||||
- `train.log`: human-readable timestamped logs.
|
||||
- `runtime_progress.json`: latest progress snapshot.
|
||||
- `latest.pt`: latest checkpoint.
|
||||
- `iteration_*.pt`: archive checkpoints when enabled.
|
||||
- `config.json`: resolved config for the run.
|
||||
|
||||
If a run is stopped mid-iteration, the in-progress iteration may not appear in
|
||||
`metrics.jsonl`. Analyze the latest completed metric row.
|
||||
|
||||
## Weights & Biases (Optional)
|
||||
|
||||
Metrics can be mirrored to W&B. `wandb` is an optional extra; default
|
||||
installs and runs do not require it.
|
||||
|
||||
Install:
|
||||
|
||||
```bash
|
||||
uv sync --extra wandb
|
||||
```
|
||||
|
||||
Run with W&B:
|
||||
|
||||
```bash
|
||||
# Offline: no login, writes to <run_dir>/wandb/offline-run-*/
|
||||
uv run lost-cities-deep-cfr train --config <...> --wandb --wandb-mode offline
|
||||
|
||||
# Online: requires `uv run wandb login` once, then real-time upload
|
||||
uv run lost-cities-deep-cfr train --config <...> --wandb
|
||||
```
|
||||
|
||||
W&B data is stored **per run** at `<run_dir>/wandb/`, not at a global
|
||||
`runs/wandb/`. Each training run gets its own subfolder, so moving or
|
||||
deleting a run directory carries its W&B data along with it.
|
||||
|
||||
Sync offline runs to wandb.ai later:
|
||||
|
||||
```bash
|
||||
wandb sync runs/<run-dir>/wandb/offline-run-*
|
||||
```
|
||||
|
||||
Flags:
|
||||
|
||||
- `--wandb`: enable W&B mirroring.
|
||||
- `--wandb-project <name>`: defaults to `coolrl-lost-cities`.
|
||||
- `--wandb-name <name>`: W&B run name; defaults to `run.experiment_name`.
|
||||
- `--wandb-mode {online,offline,disabled}`: default `online`.
|
||||
- `--wandb-group <name>`: group related runs from one experiment/hypothesis.
|
||||
- `--wandb-job-type <type>`: role of this run, e.g. `train`, `eval`, `sweep`, or `smoke`.
|
||||
- `--wandb-tag <tag>`: tag the run; repeatable.
|
||||
|
||||
W&B is purely additive — `metrics.jsonl` remains the source of truth, and
|
||||
`analyze` reads `metrics.jsonl`, not W&B. Disabling W&B never breaks
|
||||
training, resume, or analysis.
|
||||
|
||||
### Groups, notes, and tags
|
||||
|
||||
Use `--wandb-group` for the experiment or hypothesis family, e.g.
|
||||
`model-size-grid-2026-05-08` or `strict-curriculum-v1`. Use `--wandb-name`
|
||||
for the individual run, e.g. `512x3-seed79`, and `--wandb-job-type` for the
|
||||
run role (`train`, `eval`, `sweep`, `smoke`).
|
||||
|
||||
Use `--wandb-notes` for the run's *purpose* (free-form prose) and `--wandb-tag`
|
||||
for *categories you might filter on later* (short kebab-case keywords,
|
||||
repeatable). Otherwise use them however you like. Just avoid:
|
||||
|
||||
- Tags that duplicate `config` (`lr-1e-4`, `traversal-280`) — W&B already
|
||||
indexes config fields.
|
||||
- Tags that are unique per run (`test-1`, `2026-05-07`) — that's the run
|
||||
name and timestamp's job.
|
||||
- Tag-as-sentence (`tested-bigger-traversal-with-lcfr`) — that belongs in
|
||||
`--wandb-notes`.
|
||||
|
||||
**Notes length**: 3–5 lines, commit-message-body length. Should answer
|
||||
*why* (hypothesis), *what* (key config delta), and *baseline* (run/iter
|
||||
to compare against). Long analyses go in `docs/` and are linked from
|
||||
notes; don't paste them in.
|
||||
|
||||
### Comparing two runs
|
||||
|
||||
Default is **sequential, single seed**. Run baseline first, then the
|
||||
treatment with exactly one config change, both with the same `run.seed`.
|
||||
Put both in the same W&B group (e.g. `--wandb-group lr-bump-v1`) and optionally
|
||||
tag both with a shared hypothesis tag (e.g. `--wandb-tag lr-bump`) so they show
|
||||
up together in W&B's Compare Runs view.
|
||||
|
||||
Do **not** run multiple seeds per condition unless explicitly asked —
|
||||
that doubles or quintuples wall-clock and isn't the default protocol.
|
||||
Single-seed comparison is enough to surface a signal; multi-seed is a
|
||||
follow-up to confirm it.
|
||||
|
||||
Do **not** run two trainings in parallel on the same GPU — VRAM/SM
|
||||
contention slows both unevenly and breaks the comparison.
|
||||
|
||||
## Docs & Experiment Workflow
|
||||
|
||||
### Where to write what
|
||||
|
||||
| 기록 내용 | 쓸 곳 |
|
||||
| --- | --- |
|
||||
| 진행 중인 계획/가설 | `docs/plans/<topic>.md` (1 주제 1 파일) |
|
||||
| 끝난 계획 | `docs/plans/archive/` (수동 `mv`) |
|
||||
| 날짜 박힌 실험 기록 | `docs/archive/<name>-YYYY-MM-DD.md` (immutable) |
|
||||
| 항구적 알고리즘 노트 | `docs/research/<name>.md` (`Last verified:` 헤더) |
|
||||
| 비용/프로파일 | `docs/reports/<name>-YYYY-MM-DD.md` (dated) |
|
||||
| 스크래치 / 연구 스레드 인덱스 | `ideas.md` |
|
||||
|
||||
### Rules
|
||||
|
||||
- `docs/archive/`, `runs/archive/`는 read-only.
|
||||
- archive 본문 복붙 금지 — 대신 `Source:` 링크 + distill.
|
||||
- `file.py:NN` 인용은 작성 시점에 `rg`로 검증.
|
||||
- 한 주제 한 파일 — `foo-v2.md` 만들지 말 것.
|
||||
- 파일당 ~500줄 soft cap — 강제 분할이 아니라 "다른 위치(archive/research)로
|
||||
가야 할 내용이 누적됐는지" routing 점검 트리거. 초과 시 dated 실험은
|
||||
`docs/archive/`로, 항구적 분석은 `docs/research/`로 보내고 본 파일은
|
||||
현재 상태 reference만 남긴다.
|
||||
|
||||
## Notes For Future Agents
|
||||
|
||||
- Prefer `rg`/`rg --files` for search.
|
||||
- Use `apply_patch` for manual edits.
|
||||
- Do not commit generated run artifacts from `runs/`.
|
||||
- Cython-generated `.c` files are gitignored; edit `.pyx`/`.pxd` sources.
|
||||
- Before committing, run `uv run ruff check .` and at least the relevant pytest
|
||||
subset. For Deep CFR changes, run
|
||||
`uv run pytest -q tests/games/classic/test_deep_cfr_trainer.py`.
|
||||
- When changing docs, run `scripts/librarian.sh` to lint markdown link
|
||||
integrity and `file:line` code citations across `docs/**`. See
|
||||
`docs/plans/librarian.md` for the full design.
|
||||
|
||||
## Git Branching Policy (READ THIS)
|
||||
|
||||
**DO NOT create new git branches unless the user explicitly asks for one in
|
||||
the current task. Work on whatever branch is currently checked out (default
|
||||
`main`). This is a hard rule — no exceptions for "safety", "isolation",
|
||||
"experiments", "work-in-progress", or any other self-justified reason.**
|
||||
|
||||
Why this rule exists:
|
||||
- This project intentionally develops on `main` with frequent small commits.
|
||||
- Auto-created branches like `experiments/foo`, `feature/bar`, `wip/baz`
|
||||
fragment review, hide work from the user, and require manual cleanup.
|
||||
- The user has not authorized branch creation as a default behavior. If
|
||||
they want a branch, they will say so explicitly ("make a branch", "PR
|
||||
this", "isolate this in a branch", etc.).
|
||||
|
||||
What you MUST do instead:
|
||||
- Make commits directly on the currently checked-out branch.
|
||||
- If you think a branch is justified, **stop and ask the user first** —
|
||||
do not preemptively create one.
|
||||
- If a tool or subagent invocation auto-suggests creating a branch
|
||||
(e.g. PR-style workflows), refuse the branch creation step and commit
|
||||
to the current branch.
|
||||
- If you find yourself already on a non-default branch you didn't expect,
|
||||
stop and ask the user — do not switch, do not create a new one, do not
|
||||
reset.
|
||||
|
||||
Forbidden without explicit user instruction:
|
||||
- `git checkout -b <name>`
|
||||
- `git switch -c <name>`
|
||||
- `git branch <name>`
|
||||
- `gh pr create` from an auto-created branch
|
||||
- Any worktree creation that implicitly creates a new branch
|
||||
|
||||
This rule applies to the main agent and to every subagent or tool the main
|
||||
agent invokes. Pass it through in subagent prompts when delegating
|
||||
git-touching work.
|
||||
|
||||
@@ -1,282 +1,38 @@
|
||||
# coolrl-lost-cities
|
||||
|
||||
JAX PPO training and on-device browser play for the two-player card game Lost
|
||||
Cities. The project provides a pure JAX rules engine, PPO training and
|
||||
evaluation tools, and a static web client that runs the shipped final policy
|
||||
with WebGPU when available and WebAssembly otherwise.
|
||||
Focused Lost Cities extraction from the legacy `coolrl` repository.
|
||||
|
||||
The active path is **JAX + PPO**. Deep CFR and ISMCTS are retained only as
|
||||
historical research implementations; see [legacy notes](docs/legacy.md).
|
||||
The current implementation starts with the classic two-player card game:
|
||||
|
||||
## Quick start
|
||||
- classic 5-expedition rules by default
|
||||
- Python/Cython game engine
|
||||
- env wrapper
|
||||
- random, discard-only, and safe-heuristic bots
|
||||
- core rule, scoring, mask, env, canonical-state, bot, and GUI smoke tests
|
||||
|
||||
Install the project, then run a complete CPU-sized sanity check:
|
||||
Training code, Deep CFR, learned-policy evaluation, GUI, and web client are
|
||||
intentionally outside the first port.
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
uv sync
|
||||
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
|
||||
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
|
||||
uv run pytest tests/games/classic
|
||||
uv run lost-cities-classic
|
||||
```
|
||||
|
||||
The final command prints a run directory containing `latest`, `config.json`,
|
||||
and `metrics.jsonl` under `runs/tmp/jax-ppo-artifacts/`.
|
||||
|
||||
## Train and evaluate a PPO policy
|
||||
|
||||
The committed configurations describe the opponent and training budget. For a
|
||||
GPU run, use CUDA JAX and keep generated artifacts outside git:
|
||||
For future GUI work, install the optional GUI dependencies:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
|
||||
--config configs/jax_ppo/balanced.yaml \
|
||||
--set run.artifact_root=runs/jax-ppo
|
||||
uv sync --extra gui
|
||||
```
|
||||
|
||||
Evaluate a saved checkpoint against a fixed opponent. Duplicate evaluation
|
||||
swaps seats over the same shuffled games:
|
||||
Run the classic pygame GUI:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo eval \
|
||||
--config configs/jax_ppo/balanced.yaml \
|
||||
--checkpoint runs/jax-ppo/<run>/latest \
|
||||
--opponent heuristic_balanced \
|
||||
--games 10000 \
|
||||
--duplicate
|
||||
uv run lost-cities-classic-gui --mode pvc --bot safe-heuristic
|
||||
```
|
||||
|
||||
Run `uv run lost-cities-jax-ppo --help` for training against saved opponents,
|
||||
league runs, gates, human-play logs, and evaluation variants.
|
||||
|
||||
## Browser client
|
||||
|
||||
The final verified JAX PPO policy is shipped as a 3.1 MB static ONNX asset.
|
||||
No server or local checkpoint is required to play it:
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npm ci
|
||||
npm run dev
|
||||
```
|
||||
|
||||
`npm run build` produces a fully static site. The model path is deployment-base
|
||||
aware, so the build can be served from GitHub Pages, GitLab Pages, or a normal
|
||||
web root. Pushes to `main` deploy that build to both configured Pages hosts.
|
||||
See [web/README.md](web/README.md) for model replacement, tests, and the
|
||||
cross-runtime parity fixture.
|
||||
|
||||
## JAX Rules Engine
|
||||
|
||||
`lost_cities_jax` is a standalone pure rules simulator for one two-player
|
||||
Lost Cities round. It does not contain neural networks, PPO, CFR, MCTS, match
|
||||
wrappers, bots, or rule variants.
|
||||
|
||||
Public API:
|
||||
|
||||
```python
|
||||
from lost_cities_jax import (
|
||||
OBS_DIM,
|
||||
N_ACTIONS,
|
||||
State,
|
||||
batched_legal_mask,
|
||||
batched_obs,
|
||||
batched_reset,
|
||||
batched_step,
|
||||
board_score,
|
||||
legal_action_mask,
|
||||
observation,
|
||||
reset,
|
||||
reset_from_order,
|
||||
score,
|
||||
step,
|
||||
)
|
||||
```
|
||||
|
||||
Core functions are pure JAX functions:
|
||||
|
||||
- `reset(rng) -> State`
|
||||
- `reset_from_order(deck_order) -> State`
|
||||
- `legal_action_mask(state) -> bool[96]`
|
||||
- `step(state, action) -> (State, float32[2], bool)`
|
||||
- `score(state) -> float32[2]`
|
||||
- `board_score(state) -> float32[2]`
|
||||
- `observation(state, player) -> float32[454]`
|
||||
|
||||
The batched exports are `jax.jit(jax.vmap(...))` wrappers. Illegal actions and
|
||||
done-state actions are defined as no-op transitions with zero reward; training
|
||||
code should still sample only from `legal_action_mask`.
|
||||
|
||||
### Rule Summary
|
||||
|
||||
One round uses 60 cards: five colors, each with three handshakes and ranks
|
||||
2 through 10. Each player starts with eight cards, then each ply must place one
|
||||
hand card to the matching expedition or discard pile and draw one card from the
|
||||
deck or a discard pile. A player may not draw the card they just discarded.
|
||||
|
||||
Expedition numbers must be strictly increasing. Handshakes may be played only
|
||||
before any number in that color. The round ends immediately when the final deck
|
||||
card is drawn, or at `MAX_STEPS == 400`; forced termination is scored exactly
|
||||
like natural termination.
|
||||
|
||||
Scoring per player/color:
|
||||
|
||||
```text
|
||||
empty column: 0
|
||||
non-empty: (sum(number ranks) - 20) * (1 + handshake_count)
|
||||
bonus: +20 if total column length >= 8, not multiplied
|
||||
```
|
||||
|
||||
### Encodings
|
||||
|
||||
Cards:
|
||||
|
||||
| Field | Encoding |
|
||||
| --- | --- |
|
||||
| `card_id` | `color * 12 + slot` |
|
||||
| `color` | `0..4` |
|
||||
| `slot 0..2` | handshake |
|
||||
| `slot 3..11` | ranks `2..10`, with `rank = slot - 1` |
|
||||
|
||||
Actions (`N_ACTIONS == 96`):
|
||||
|
||||
```text
|
||||
action_id = hand_slot * 12 + place_type * 6 + draw_source
|
||||
hand_slot = 0..7, current player's hand sorted by card_id
|
||||
place_type = 0 play, 1 discard
|
||||
draw_source = 0 deck, 1..5 discard pile color 0..4
|
||||
```
|
||||
|
||||
Observation (`OBS_DIM == 454`):
|
||||
|
||||
- 60 cards x 7 one-hot channels:
|
||||
my hand, my board, opponent board, discard top, discard non-top,
|
||||
opponent public hand, unknown.
|
||||
- 34 scalar features:
|
||||
remaining deck `/44`, opponent unknown hand count `/8`, step count `/400`,
|
||||
current-player then opponent `col_top /10`, `col_hs /3`, `col_len /12`,
|
||||
and current board score difference `(player - opponent) /780`.
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
uv run pytest -q tests/lost_cities_jax
|
||||
uv run pytest -q
|
||||
uv run ruff check .
|
||||
```
|
||||
|
||||
Large differential profiles:
|
||||
|
||||
```bash
|
||||
# CI profile: 100,000 random legal-policy games
|
||||
CI=1 uv run pytest -q tests/lost_cities_jax/test_differential.py
|
||||
|
||||
# Full profile: 1,000,000 random legal-policy games
|
||||
uv run pytest -q tests/lost_cities_jax/test_differential.py --full
|
||||
```
|
||||
|
||||
Observed differential results on 2026-07-04 with CPU JAX backend:
|
||||
|
||||
```text
|
||||
CI=1 ... test_differential.py
|
||||
1 passed in 247.61s (0:04:07)
|
||||
|
||||
... test_differential.py --full
|
||||
1 passed in 2514.14s (0:41:54)
|
||||
elapsed=41:54.48
|
||||
```
|
||||
|
||||
Throughput benchmark:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run python benchmarks/throughput.py
|
||||
|
||||
# Optional CUDA check without making CUDA a project dependency:
|
||||
flock -n .compute.lock uv run --with 'jax[cuda12]' python benchmarks/throughput.py
|
||||
```
|
||||
|
||||
Measured on 2026-07-04 with CPU JAX backend:
|
||||
|
||||
```text
|
||||
backend=cpu
|
||||
batch_size=8192
|
||||
steps=256
|
||||
elapsed_sec=4.655526
|
||||
steps_per_sec=450465.14
|
||||
```
|
||||
|
||||
Measured on 2026-07-04 with CUDA JAX backend on RTX 3090, using the optional
|
||||
`uv run --with 'jax[cuda12]' ...` command:
|
||||
|
||||
```text
|
||||
backend=gpu
|
||||
batch_size=8192
|
||||
steps=256
|
||||
elapsed_sec=0.530039
|
||||
steps_per_sec=3956598.38
|
||||
```
|
||||
|
||||
### DECISIONS.md
|
||||
|
||||
- Explicit `deck_order` dealing uses the first eight cards for player 0 and
|
||||
the next eight for player 1. The remaining cards are drawn from index 16.
|
||||
This is equivalent under a uniform shuffle and is fixed by tests.
|
||||
- After a legal terminal transition, `to_move` is advanced to the next player,
|
||||
but `done=True` makes all later steps complete no-ops.
|
||||
- Terminal reward is emitted only on the transition that reaches `done=True`.
|
||||
Done-state no-op steps return zero reward.
|
||||
- Observation scalar normalization is implementation-defined as documented
|
||||
above and locked by the exported `OBS_DIM`.
|
||||
|
||||
## JAX PPO Static-Opponent Ladder
|
||||
|
||||
The first training stack above `lost_cities_jax` is exposed as:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo --help
|
||||
```
|
||||
|
||||
CPU smoke:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/smoke.yaml
|
||||
uv run lost-cities-jax-ppo train --config configs/jax_ppo/smoke.yaml
|
||||
```
|
||||
|
||||
GPU training uses optional CUDA JAX, keeping CUDA wheels out of the default
|
||||
project dependency set:
|
||||
|
||||
```bash
|
||||
tmux new-session -s coolrl-jax-ppo-discard \
|
||||
-c /home/coolguy/dev/coolrl-lost-cities \
|
||||
"flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
|
||||
--config configs/jax_ppo/discard-only.yaml"
|
||||
```
|
||||
|
||||
Random-policy baseline vs `discard_only`, measured on 2026-07-04 with
|
||||
8192 games x 400 plies on GPU:
|
||||
|
||||
```text
|
||||
return_mean=-55.582763671875
|
||||
game_length_mean=69.6085205078125
|
||||
max_steps_rate=0.0
|
||||
play_action_rate=0.28854578733444214
|
||||
opened_colors_mean=4.942626953125
|
||||
positive_expeditions_mean=0.41796875
|
||||
```
|
||||
|
||||
Static-opponent gate results, measured on 2026-07-04 with 10,000 fixed shuffles
|
||||
and duplicate seat-swapped evaluation:
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
|
||||
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
|
||||
|
||||
Large PPO artifacts are written under
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
|
||||
Generated checkpoints and evaluation JSON are not committed to git. The full
|
||||
run summary is in
|
||||
[docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md](docs/reports/jax-ppo-static-opponent-ladder-2026-07-04.md).
|
||||
The GUI uses the in-process Cython game engine.
|
||||
|
||||
## Basic Usage
|
||||
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import time
|
||||
from functools import partial
|
||||
|
||||
import jax
|
||||
import jax.numpy as jnp
|
||||
|
||||
from lost_cities_jax.engine import legal_action_mask, reset, step
|
||||
|
||||
|
||||
@partial(jax.jit, static_argnames=("steps",))
|
||||
def rollout(states, rng, *, steps: int):
|
||||
def body(carry, _):
|
||||
state, key = carry
|
||||
key, action_key = jax.random.split(key)
|
||||
mask = jax.vmap(legal_action_mask)(state)
|
||||
logits = jnp.where(mask, 0.0, -1.0e9)
|
||||
actions = jax.random.categorical(action_key, logits, axis=1).astype(jnp.int32)
|
||||
state, _, _ = jax.vmap(step, in_axes=(0, 0))(state, actions)
|
||||
return (state, key), None
|
||||
|
||||
(states, rng), _ = jax.lax.scan(body, (states, rng), xs=None, length=steps)
|
||||
return states, rng
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Lost Cities JAX random-policy throughput")
|
||||
parser.add_argument("--batch-size", type=int, default=8192)
|
||||
parser.add_argument("--steps", type=int, default=256)
|
||||
parser.add_argument("--warmup-steps", type=int, default=32)
|
||||
args = parser.parse_args()
|
||||
|
||||
key = jax.random.PRNGKey(0)
|
||||
reset_keys = jax.random.split(key, args.batch_size)
|
||||
states = jax.jit(jax.vmap(reset))(reset_keys)
|
||||
|
||||
warm_states, key = rollout(states, jax.random.PRNGKey(1), steps=args.warmup_steps)
|
||||
jax.tree_util.tree_leaves(warm_states)[0].block_until_ready()
|
||||
|
||||
start = time.perf_counter()
|
||||
states, _ = rollout(states, key, steps=args.steps)
|
||||
jax.tree_util.tree_leaves(states)[0].block_until_ready()
|
||||
elapsed = time.perf_counter() - start
|
||||
|
||||
transitions = args.batch_size * args.steps
|
||||
print(f"backend={jax.default_backend()}")
|
||||
print(f"batch_size={args.batch_size}")
|
||||
print(f"steps={args.steps}")
|
||||
print(f"elapsed_sec={elapsed:.6f}")
|
||||
print(f"steps_per_sec={transitions / elapsed:.2f}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,8 +1,8 @@
|
||||
run:
|
||||
experiment_name: ismcts-default
|
||||
max_iterations: 500
|
||||
max_iterations: 100
|
||||
seed: 1
|
||||
device: cuda
|
||||
device: auto
|
||||
rules:
|
||||
n_colors: 5
|
||||
n_ranks: 9
|
||||
@@ -17,20 +17,15 @@ encoding:
|
||||
slot_aware_playability: true
|
||||
network:
|
||||
kind: mlp
|
||||
hidden_size: 768
|
||||
num_layers: 4
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
activation: relu
|
||||
mcts:
|
||||
n_simulations: 50
|
||||
c_puct: 5.0
|
||||
c_puct: 1.5
|
||||
max_depth: 200
|
||||
parallel_simulations: 64
|
||||
virtual_loss_value: 5.0
|
||||
eval_n_simulations: 16
|
||||
rollout_policy: heuristic_balanced
|
||||
use_rollout_value: false
|
||||
root_dirichlet_alpha: 0.3
|
||||
root_dirichlet_epsilon: 0.4
|
||||
parallel_simulations: 8
|
||||
virtual_loss_value: 1.0
|
||||
temperature:
|
||||
training: 1.0
|
||||
eval: 0.0
|
||||
@@ -41,17 +36,19 @@ training:
|
||||
replay_capacity: 100000
|
||||
interleave_games: 8
|
||||
interleave_max_batch: 64
|
||||
num_workers: 8
|
||||
worker_device: cuda
|
||||
use_central_scheduler: false
|
||||
use_inference_server: false
|
||||
inference_server_max_batch: 128
|
||||
inference_server_batch_timeout_ms: 10.0
|
||||
optimization:
|
||||
learning_rate: 0.0003
|
||||
grad_clip: 5.0
|
||||
checkpoint:
|
||||
save_every: 20
|
||||
save_every: 10
|
||||
save_latest: true
|
||||
evaluation:
|
||||
eval_every: 5
|
||||
games: 5
|
||||
eval_every: 10
|
||||
games: 20
|
||||
opponents: [random, discard-only, heuristic-cautious]
|
||||
max_steps: 500
|
||||
num_workers: 8
|
||||
max_steps: 10000
|
||||
num_workers: 1
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-balanced
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
|
||||
|
||||
opponent:
|
||||
name: heuristic_balanced
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-cautious
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
|
||||
|
||||
opponent:
|
||||
name: heuristic_cautious
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-discard-only
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents
|
||||
|
||||
opponent:
|
||||
name: discard_only
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-gate3-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/verification
|
||||
|
||||
opponent:
|
||||
name: gate3_checkpoint_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: gates-1-2-long-random-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 1200
|
||||
log_every: 10
|
||||
checkpoint_every: 100
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
|
||||
|
||||
opponent:
|
||||
name: league_v1_update_500_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 2000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: gates-1-2-replay-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 900
|
||||
log_every: 10
|
||||
checkpoint_every: 100
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
|
||||
|
||||
opponent:
|
||||
name: league_v1_update_500_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 0.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 0
|
||||
|
||||
evaluation:
|
||||
games: 2000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: gates-1-2-smoke-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 1
|
||||
log_every: 1
|
||||
checkpoint_every: 1
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/exploiters
|
||||
|
||||
opponent:
|
||||
name: league_v1_update_500_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8
|
||||
rollout_steps: 16
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 1
|
||||
minibatches: 2
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 2
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 2
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: gates-1-2-warmstart-gate3-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 900
|
||||
log_every: 10
|
||||
checkpoint_every: 100
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/exploiters
|
||||
|
||||
opponent:
|
||||
name: league_v1_update_500_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 0.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 0
|
||||
|
||||
evaluation:
|
||||
games: 2000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,35 +0,0 @@
|
||||
run:
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke
|
||||
report_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/report.md
|
||||
summary_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2-smoke/summary.jsonl
|
||||
shuffle_bank_seed: 20260704
|
||||
gpu_budget_hours: 1.0
|
||||
|
||||
target:
|
||||
config: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500
|
||||
|
||||
gate1:
|
||||
games: 2
|
||||
batch_games: 2
|
||||
delta_games: 2
|
||||
delta_target_events: 2
|
||||
delta_pairs: 2
|
||||
delta_batch_events: 1
|
||||
|
||||
gate2:
|
||||
eval_games: 2
|
||||
pass_threshold: 0.55
|
||||
exploiters:
|
||||
- name: long_random_smoke
|
||||
config: configs/jax_ppo/gates-1-2-exploiter-smoke.yaml
|
||||
|
||||
repair:
|
||||
max_cycles: 1
|
||||
league_template: configs/jax_ppo/league-smoke.yaml
|
||||
experiment_name: jax-ppo-gates-1-2-repair-smoke
|
||||
artifact_subdir: gate2c
|
||||
evaluation_games: 2
|
||||
evaluation_batch_games: 2
|
||||
guard_expert_ci_low: -1000.0
|
||||
guard_max_steps_rate: 1.0
|
||||
@@ -1,44 +0,0 @@
|
||||
run:
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2
|
||||
report_path: docs/reports/gates-1-2-2026-07-05.md
|
||||
summary_path: docs/reports/gates-1-2-2026-07-05-summary.jsonl
|
||||
shuffle_bank_seed: 20260704
|
||||
gpu_budget_hours: 8.0
|
||||
|
||||
target:
|
||||
config: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500
|
||||
|
||||
gate1:
|
||||
games: 2000
|
||||
batch_games: 8192
|
||||
delta_games: 200
|
||||
delta_target_events: 500
|
||||
delta_pairs: 64
|
||||
delta_batch_events: 16
|
||||
|
||||
gate2:
|
||||
eval_games: 2000
|
||||
pass_threshold: 0.55
|
||||
exploiters:
|
||||
- name: long_random
|
||||
config: configs/jax_ppo/gates-1-2-exploiter-long-random.yaml
|
||||
notes: random init + shaping anneal, extended budget
|
||||
- name: warmstart_gate3
|
||||
config: configs/jax_ppo/gates-1-2-exploiter-warmstart.yaml
|
||||
resume: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
|
||||
notes: ladder v2 gate-3 warm start, shaping disabled
|
||||
- name: replay_exploiter
|
||||
config: configs/jax_ppo/gates-1-2-exploiter-replay.yaml
|
||||
resume: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest
|
||||
notes: league v1 exploiter warm start, shaping disabled
|
||||
|
||||
repair:
|
||||
max_cycles: 3
|
||||
league_template: configs/jax_ppo/league-v1.yaml
|
||||
experiment_name: jax-ppo-gates-1-2-repair
|
||||
artifact_subdir: gate2c
|
||||
evaluation_games: 2000
|
||||
evaluation_batch_games: 8192
|
||||
guard_expert_ci_low: 0.0
|
||||
guard_max_steps_rate: 0.02
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-ladder-v2-balanced
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
|
||||
|
||||
opponent:
|
||||
name: heuristic_balanced
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-ladder-v2-discard-only
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
|
||||
|
||||
opponent:
|
||||
name: discard_only
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-ladder-v2-expert
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
|
||||
|
||||
opponent:
|
||||
name: heuristic_expert
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-ladder-v2-exploiter
|
||||
seed: 20260705
|
||||
learner_seat: 0
|
||||
total_updates: 250
|
||||
log_every: 1
|
||||
checkpoint_every: 10
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2
|
||||
|
||||
opponent:
|
||||
name: ladder_v2_gate3_checkpoint_frozen
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5000000
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
@@ -1,77 +0,0 @@
|
||||
base_config: configs/jax_ppo/ladder-v2-expert.yaml
|
||||
warm_start_checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
|
||||
|
||||
run:
|
||||
experiment_name: jax-ppo-league-smoke
|
||||
seed: 20260705
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke
|
||||
|
||||
league:
|
||||
cycles: 1
|
||||
league_updates_per_cycle: 1
|
||||
snapshot_interval_updates: 1
|
||||
mirror_probability: 0.5
|
||||
uniform_mix: 0.1
|
||||
stalling_anchor_floor: 0.02
|
||||
stalling_anchor_cap: 0.05
|
||||
pool_max_size: 8
|
||||
max_active_pool_members: 7
|
||||
exploiter_member_fraction_cap: 0.3333333333333333
|
||||
wall_clock_hours: 1.0
|
||||
success_exploiter_win_rate: 0.60
|
||||
stagnation_window: 5
|
||||
stagnation_min_delta: 0.02
|
||||
|
||||
evaluation:
|
||||
games: 2
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 2
|
||||
recent_snapshot_evals: 1
|
||||
|
||||
exploiter:
|
||||
config: configs/jax_ppo/ladder-v2-exploiter.yaml
|
||||
updates: 1
|
||||
|
||||
guards:
|
||||
expert_ci_low: -1000.0
|
||||
max_steps_rate: 1.0
|
||||
|
||||
tracking:
|
||||
tracked_summary_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke/summary.jsonl
|
||||
report_path: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league-smoke/report.md
|
||||
|
||||
anchors:
|
||||
- name: discard_only
|
||||
kind: static
|
||||
policy_name: discard_only
|
||||
anchor: true
|
||||
stalling: true
|
||||
- name: heuristic_balanced
|
||||
kind: static
|
||||
policy_name: heuristic_balanced
|
||||
anchor: true
|
||||
- name: heuristic_cautious
|
||||
kind: static
|
||||
policy_name: heuristic_cautious
|
||||
anchor: true
|
||||
stalling: true
|
||||
- name: heuristic_expert
|
||||
kind: static
|
||||
policy_name: heuristic_expert
|
||||
anchor: true
|
||||
- name: ladder_v2_gate1_discard
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-discard-only.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest
|
||||
anchor: true
|
||||
- name: ladder_v2_gate2_balanced
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-balanced.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest
|
||||
anchor: true
|
||||
- name: ladder_v2_gate3_expert
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-expert.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
|
||||
anchor: true
|
||||
@@ -1,77 +0,0 @@
|
||||
base_config: configs/jax_ppo/ladder-v2-expert.yaml
|
||||
warm_start_checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
|
||||
|
||||
run:
|
||||
experiment_name: jax-ppo-league-v1
|
||||
seed: 20260705
|
||||
artifact_root: /mnt/2tbhdd/coolrl-lost-cities-artifacts/league
|
||||
|
||||
league:
|
||||
cycles: 5
|
||||
league_updates_per_cycle: 500
|
||||
snapshot_interval_updates: 250
|
||||
mirror_probability: 0.5
|
||||
uniform_mix: 0.1
|
||||
stalling_anchor_floor: 0.02
|
||||
stalling_anchor_cap: 0.05
|
||||
pool_max_size: 24
|
||||
max_active_pool_members: 12
|
||||
exploiter_member_fraction_cap: 0.3333333333333333
|
||||
wall_clock_hours: 6.0
|
||||
success_exploiter_win_rate: 0.60
|
||||
stagnation_window: 5
|
||||
stagnation_min_delta: 0.02
|
||||
|
||||
evaluation:
|
||||
games: 10000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 8192
|
||||
recent_snapshot_evals: 3
|
||||
|
||||
exploiter:
|
||||
config: configs/jax_ppo/ladder-v2-exploiter.yaml
|
||||
updates: 250
|
||||
|
||||
guards:
|
||||
expert_ci_low: 0.0
|
||||
max_steps_rate: 0.02
|
||||
|
||||
tracking:
|
||||
tracked_summary_path: docs/reports/league-v1-2026-07-05-summary.jsonl
|
||||
report_path: docs/reports/league-v1-2026-07-05.md
|
||||
|
||||
anchors:
|
||||
- name: discard_only
|
||||
kind: static
|
||||
policy_name: discard_only
|
||||
anchor: true
|
||||
stalling: true
|
||||
- name: heuristic_balanced
|
||||
kind: static
|
||||
policy_name: heuristic_balanced
|
||||
anchor: true
|
||||
- name: heuristic_cautious
|
||||
kind: static
|
||||
policy_name: heuristic_cautious
|
||||
anchor: true
|
||||
stalling: true
|
||||
- name: heuristic_expert
|
||||
kind: static
|
||||
policy_name: heuristic_expert
|
||||
anchor: true
|
||||
- name: ladder_v2_gate1_discard
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-discard-only.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest
|
||||
anchor: true
|
||||
- name: ladder_v2_gate2_balanced
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-balanced.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest
|
||||
anchor: true
|
||||
- name: ladder_v2_gate3_expert
|
||||
kind: checkpoint
|
||||
config: configs/jax_ppo/ladder-v2-expert.yaml
|
||||
checkpoint: /mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest
|
||||
anchor: true
|
||||
@@ -1,50 +0,0 @@
|
||||
# Classic Lost Cities: three rounds, self-play.
|
||||
#
|
||||
# rollout_steps is deliberately shorter than a match. Early self-play stalls
|
||||
# rounds badly (untrained matches run ~900 plies), and the env carries across
|
||||
# updates, so a long match simply spans several rollouts -- the GAE truncation
|
||||
# bootstrap keeps that unbiased. Sizing the scan to the worst-case match instead
|
||||
# would blow up the rollout tensors, which are already doubled by training both
|
||||
# seats and widened by the critic's 681-dim privileged view.
|
||||
run:
|
||||
experiment_name: match-selfplay
|
||||
seed: 20260715
|
||||
learner_seat: 0
|
||||
total_updates: 300
|
||||
log_every: 10
|
||||
checkpoint_every: 50
|
||||
artifact_root: runs/jax-ppo-match
|
||||
|
||||
opponent:
|
||||
name: discard_only # unused: self-play
|
||||
|
||||
network:
|
||||
hidden_size: 512
|
||||
num_layers: 3
|
||||
|
||||
ppo:
|
||||
batch_games: 512
|
||||
rollout_steps: 256
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.97 # a match is ~3x a round; 0.95 reaches too little of it
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 32
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
# Counted in learner actions now, not padded scan steps. Held slightly above
|
||||
# zero: the match-terminal signal alone is one bounded number per ~160 plies.
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.05
|
||||
potential_shaping_anneal_steps: 20000000
|
||||
|
||||
evaluation:
|
||||
games: 2000
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260715
|
||||
batch_games: 512
|
||||
@@ -1,40 +0,0 @@
|
||||
run:
|
||||
experiment_name: jax-ppo-smoke
|
||||
seed: 20260704
|
||||
learner_seat: 0
|
||||
total_updates: 1
|
||||
log_every: 1
|
||||
checkpoint_every: 1
|
||||
artifact_root: runs/tmp/jax-ppo-artifacts
|
||||
|
||||
opponent:
|
||||
name: discard_only
|
||||
|
||||
network:
|
||||
hidden_size: 64
|
||||
num_layers: 2
|
||||
|
||||
ppo:
|
||||
batch_games: 64
|
||||
rollout_steps: 64
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 1
|
||||
minibatches: 4
|
||||
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 100000
|
||||
|
||||
evaluation:
|
||||
games: 128
|
||||
duplicate: true
|
||||
shuffle_bank_seed: 20260704
|
||||
batch_games: 64
|
||||
File diff suppressed because one or more lines are too long
@@ -1,43 +0,0 @@
|
||||
{
|
||||
"_scheme": {
|
||||
"naming": "Astronomical names, alphabetically ordered. The first letter is the generation; a new letter means the observation space broke, not that the model got better.",
|
||||
"rule": "A codename never encodes quality. The record this replaces stored 'FINAL PPO', which stops meaning anything the moment there is a second final model.",
|
||||
"identity": "The hash is the truth -- it is what actually played. The codename is for humans, and it is assigned here, not derived. Records should carry the hash; look the codename up.",
|
||||
"next": "cygnus, deneb, ..."
|
||||
},
|
||||
|
||||
"models": {
|
||||
"altair": {
|
||||
"hash": "e8241e305c01",
|
||||
"hash_kind": "sha256 of web/public/models/jax-ppo.onnx",
|
||||
"generation": "a",
|
||||
"game": "single round",
|
||||
"observation_size": 454,
|
||||
"hidden_size": 512,
|
||||
"num_layers": 3,
|
||||
"trained": "self-play league, 122.6M learner actions",
|
||||
"source": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/latest",
|
||||
"deployed": "web/public/models/jax-ppo.onnx",
|
||||
"displayed_as": "WASM · FINAL PPO",
|
||||
"note": "Every game in data/human-play/game-records.jsonl (format v1, 111 games, 2026-07-14) was played against this model. The v1 schema has no model field -- it stores the on-screen label -- so this line is the record of what they played."
|
||||
},
|
||||
|
||||
"borealis": {
|
||||
"hash": "4ae613b010ca",
|
||||
"hash_kind": "sha256 over the orbax checkpoint files (no ONNX export yet)",
|
||||
"generation": "b",
|
||||
"game": "three-round match (classic rules)",
|
||||
"observation_size": 501,
|
||||
"critic_observation_size": 681,
|
||||
"hidden_size": 512,
|
||||
"num_layers": 3,
|
||||
"trained": "self-play, 131.1M learner actions, linear total-score reward, both seats, privileged critic",
|
||||
"source": "runs/jax-ppo-match/2026-07-15_031529_match-scaled/latest",
|
||||
"deployed": null,
|
||||
"results": {
|
||||
"vs_altair_3round": "0.6094 win rate [0.599, 0.620], +22.0 points, 8192 duplicate matches",
|
||||
"exploitability": "a from-scratch exploiter funded to 131M reaches 0.4657 against it, and 0.6295 against altair -- lower bound, neither exploiter had plateaued"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,25 +0,0 @@
|
||||
# Legacy research stacks
|
||||
|
||||
The supported project path is the JAX rules engine and JAX PPO tooling exposed
|
||||
by `lost-cities-jax-ppo`. The historical Deep CFR and ISMCTS implementations
|
||||
are preserved for research reproduction, but they are not current training
|
||||
recipes and no longer have public console-script entry points.
|
||||
|
||||
## What is preserved
|
||||
|
||||
- Deep CFR sources: `src/coolrl_lost_cities/games/classic/deep_cfr/`
|
||||
- ISMCTS sources: `src/coolrl_lost_cities/games/classic/ismcts/`
|
||||
- Their retired configurations: `legacy/deep-cfr/configs/` and
|
||||
`legacy/ismcts/configs/`
|
||||
- Earlier plans and dated findings: `docs/plans/archive/`, `docs/archive/`,
|
||||
and `docs/research/`
|
||||
|
||||
These files remain in the repository so a historical experiment can be read or
|
||||
reproduced against its original commit. They should not be chosen for new
|
||||
models, benchmarks, or product work.
|
||||
|
||||
## Current workflow
|
||||
|
||||
Start from `configs/jax_ppo/`, run `lost-cities-jax-ppo`, and use the browser
|
||||
client in `web/`. The root [README](../README.md) contains the short training,
|
||||
evaluation, and static-web recipes.
|
||||
+5
-8
@@ -1,10 +1,7 @@
|
||||
# Legacy Deep CFR Performance Notes
|
||||
# Deep CFR Performance Notes
|
||||
|
||||
> Historical record only. The supported training stack is JAX PPO; see the
|
||||
> root [README](../README.md) and [legacy notes](legacy.md).
|
||||
|
||||
This document records the Deep CFR runtime bottlenecks observed before the JAX
|
||||
PPO transition. The numbers below are historical, not a benchmark contract.
|
||||
This document tracks current runtime bottlenecks for the active Deep CFR
|
||||
training path. The numbers below are observational, not a benchmark contract.
|
||||
|
||||
## Current Default Runtime
|
||||
|
||||
@@ -14,7 +11,7 @@ Source run:
|
||||
runs/tmp/2026-05-07_171535_deep-cfr-default/metrics.jsonl
|
||||
```
|
||||
|
||||
The run used `legacy/deep-cfr/configs/default.yaml` with CUDA enabled. At the time of
|
||||
The run used `configs/deep_cfr/default.yaml` with CUDA enabled. At the time of
|
||||
inspection, completed metrics covered iterations 70 through 95. The training
|
||||
process was still running, so later rows may differ.
|
||||
|
||||
@@ -91,7 +88,7 @@ This gives 560 traversals per iteration, split into 70 worker batches.
|
||||
## Device Use
|
||||
|
||||
The trainer constructs the advantage and strategy networks on `run.device`.
|
||||
`legacy/deep-cfr/configs/default.yaml` sets:
|
||||
`configs/deep_cfr/default.yaml` sets:
|
||||
|
||||
```yaml
|
||||
run:
|
||||
|
||||
@@ -1,100 +0,0 @@
|
||||
# Plan: JAX PPO Model-Size A/B
|
||||
|
||||
**Status:** Ready to execute
|
||||
**Priority:** Low — 진단상 용량은 병목이 아닐 가능성이 높다. 그러나 PPO 스택에서
|
||||
한 번도 측정된 적이 없어 정황증거뿐이다. 이 A/B의 목적은 **스케일링 라인을 열지,
|
||||
닫을지 확정**하는 것이다.
|
||||
**Background:** [docs/reports/jax-ppo-model-capacity-2026-07-12.md](../reports/jax-ppo-model-capacity-2026-07-12.md)
|
||||
|
||||
> 이 계획은 은퇴한 Deep CFR/PyTorch 스택용
|
||||
> [model_size_experiment.md](archive/model_size_experiment.md)를 **대체하지 않는다** —
|
||||
> 그쪽은 다른 스택(`input_dim=365`, `legacy/deep-cfr/configs/`) 이야기다. 서로 무관하다.
|
||||
|
||||
## 가설
|
||||
|
||||
`hidden_size=512, num_layers=3` (808K 파라미터)은 구 스택에서 복사돼 온 상속값이며
|
||||
PPO에서 검증된 적이 없다. 셀프플레이 샘플:파라미터 비가 약 2,028:1이므로
|
||||
용량을 키우면 **응수 품질(expert 앵커)은 오르지만, exploitability 바닥(~0.54)은
|
||||
움직이지 않을 것**이다.
|
||||
|
||||
이 예측이 맞으면 → 병목은 용량이 아니라 셀프플레이 스킴이며, 스케일링 라인을
|
||||
닫고 리그/착취 구조 쪽에 투자한다.
|
||||
틀리면 (exploitability가 유의하게 내려가면) → 스케일링이 열린다.
|
||||
|
||||
## 실험 설계
|
||||
|
||||
**2점 A/B. 그리드 아님.** 그리드는 이 우선순위에 비해 과하다.
|
||||
|
||||
| 이름 | hidden_size | num_layers | 파라미터 |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `size-512x3` (baseline) | 512 | 3 | 808,033 |
|
||||
| `size-1024x4` (treatment) | 1024 | 4 | 3,714,145 |
|
||||
|
||||
- 두 config는 `network` 블록과 `run.experiment_name`만 다르고 **나머지는
|
||||
final candidate와 byte-identical**이어야 한다. 기준 config:
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/main_ppo_config.json`
|
||||
- **같은 `run.seed`**, 순차 실행 (AGENTS.md "Comparing two runs" 규약).
|
||||
같은 GPU에서 두 학습을 병렬로 돌리지 말 것.
|
||||
- 리그 **1 사이클**, 500 updates.
|
||||
|
||||
## 판정 기준 (중요)
|
||||
|
||||
**H2H로 판정하지 말 것.** 더 큰 모델이 더 작은 모델을 이기는 건 거의 자명하고,
|
||||
그건 우리가 알고 싶은 게 아니다. 두 축으로만 본다:
|
||||
|
||||
1. **Expert 앵커** — `heuristic_expert` 상대 duplicate 평가
|
||||
(shuffle bank seed 20260704, 2,000쌍). *응수 품질*을 본다.
|
||||
2. **Exploitability** — 각 체크포인트에 대해 exploiter를 새로 학습시켜 승률 측정.
|
||||
`long_random_shaping` 프로토콜(1200 updates, shaping anneal, warm start
|
||||
`random`) 하나로 충분하다.
|
||||
[diminishing-returns-2026-07-05.md](../reports/diminishing-returns-2026-07-05.md)의
|
||||
기존 측정과 **같은 프로토콜 안에서만** 비교할 것 — 프로토콜이 다르면 섞지 않는다.
|
||||
|
||||
### 결정 트리
|
||||
|
||||
| Expert 앵커 | Exploiter 승률 | 결론 | 조치 |
|
||||
| --- | --- | --- | --- |
|
||||
| 유의하게 상승 | **변화 없음** (~0.54 유지) | **예측대로.** 용량은 병목이 아니다 | 스케일링 라인 종료. 리그/착취 구조로 이동. 1024×4를 default로 올릴지는 비용 대비 선택 사항 |
|
||||
| 유의하게 상승 | **유의하게 하락** | 가설 기각. 용량이 실제로 제약이었다 | `network` default를 1024×4로 올리고, 2048×4까지 한 점 더 확장 |
|
||||
| 변화 없음 | 변화 없음 | 용량 무관 확정 | 스케일링 라인 종료 |
|
||||
| 하락 | — | 최적화 문제 (LR/KL 등이 큰 모델에 안 맞음) | default 유지. 스케일 재시도 전에 하이퍼파라미터 점검 |
|
||||
|
||||
"유의하게"는 Wilson CI(승률) / score CI(점수차) 비중첩 기준. 기존 리포트들이
|
||||
쓰는 것과 동일하다.
|
||||
|
||||
## 비용
|
||||
|
||||
리포트 기준 리그 1사이클 ≈ 1.5시간 GPU (512×3). 1024×4는 파라미터 4.6배이나
|
||||
병목이 롤아웃(엔진 시뮬)이라 학습 시간이 4.6배가 되지는 않는다.
|
||||
exploiter 학습이 별도로 붙는다.
|
||||
|
||||
`.compute.lock`을 잡고 돌릴 것 (AGENTS.md "Compute Lock"):
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock <train command>
|
||||
```
|
||||
|
||||
## Non-goals
|
||||
|
||||
- 그리드 스윕(768×4, 1536×8 등) — 2점으로 충분하다. 결과가 애매할 때만 확장.
|
||||
- 아키텍처 변경 (residual, transformer, per-color 공유 인코더 등) — 별도 주제.
|
||||
- 인코딩 변경 (`OBS_DIM=454` 고정).
|
||||
- 하이퍼파라미터 동시 변경 — 한 번에 하나만 바꾼다.
|
||||
|
||||
## Definition of done
|
||||
|
||||
산출될 파일 (아직 존재하지 않음):
|
||||
|
||||
```text
|
||||
configs/jax_ppo/size-512x3.yaml
|
||||
configs/jax_ppo/size-1024x4.yaml
|
||||
docs/reports/jax-ppo-model-size-ab-<YYYY-MM-DD>.md
|
||||
```
|
||||
|
||||
- [ ] 위 두 config 추가 (final candidate config 기준, `network` /
|
||||
`run.experiment_name`만 차이)
|
||||
- [ ] 두 학습 순차 실행, 같은 seed, `.compute.lock` 사용
|
||||
- [ ] 두 체크포인트 expert 앵커 duplicate 평가
|
||||
- [ ] 두 체크포인트 exploiter 학습 + 승률 측정 (`long_random_shaping`)
|
||||
- [ ] 결과표 + 결정 트리 적용 결과를 위 dated 리포트에 기록
|
||||
- [ ] 결정 트리에 따라: `network` default 갱신 **또는** 스케일링 라인 종료를 문서화
|
||||
@@ -1,378 +0,0 @@
|
||||
# Plan: JAX PPO Static-Opponent Ladder
|
||||
|
||||
**Status:** Static-opponent ladder passed on 2026-07-04.
|
||||
**Owner:** Codex implements; operator reviews training gates.
|
||||
**Scope:** A compact PPO training stack on top of `lost_cities_jax`, using GPU
|
||||
via optional CUDA JAX execution.
|
||||
|
||||
## Goal
|
||||
|
||||
Build the first learning layer above the JAX Lost Cities rules engine:
|
||||
|
||||
1. A batched rollout driver using `batched_reset`, `batched_step`,
|
||||
`batched_legal_mask`, and `batched_obs`.
|
||||
2. Three static pure-JAX opponent policies:
|
||||
`discard_only`, `heuristic_balanced`, and `heuristic_cautious`.
|
||||
3. A 3 x 512 MLP actor-critic with 96-action policy logits and scalar value.
|
||||
4. Standard PPO training against one static opponent at a time.
|
||||
5. Duplicate evaluation on a fixed 10,000-deck shuffle bank.
|
||||
6. Orbax checkpoints, metrics logging, and a CLI:
|
||||
`lost-cities-jax-ppo train --config ...`.
|
||||
|
||||
The near-term objective is not league self-play. It is to pass the
|
||||
static-opponent diagnostic ladder and preserve enough artifacts that a later
|
||||
self-play failure can be localized cleanly.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No league self-play, snapshot pool, Elo, or opponent matchmaking in this
|
||||
phase.
|
||||
- No MCTS, CFR, search, or neural opponent ensemble.
|
||||
- No multi-round match wrapper.
|
||||
- No rule variants, expanded colors, or extra players.
|
||||
- No large binary checkpoints committed to git. Generated artifacts are stored
|
||||
outside the code repo, with small manifests and summaries committed.
|
||||
|
||||
## GPU Execution
|
||||
|
||||
Keep the project dependency portable. Do not make CUDA wheels mandatory in
|
||||
`pyproject.toml`.
|
||||
|
||||
Use optional CUDA JAX for training and GPU benchmarks:
|
||||
|
||||
```bash
|
||||
flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
|
||||
--config configs/jax_ppo/discard-only.yaml
|
||||
```
|
||||
|
||||
The CPU path must still work for tests:
|
||||
|
||||
```bash
|
||||
uv run pytest -q tests/lost_cities_jax
|
||||
```
|
||||
|
||||
Current benchmark evidence on RTX 3090:
|
||||
|
||||
```text
|
||||
backend=gpu
|
||||
batch_size=8192
|
||||
steps=256
|
||||
steps_per_sec=3956598.38
|
||||
```
|
||||
|
||||
## Implementation Shape
|
||||
|
||||
Prefer a small number of files while keeping testable boundaries:
|
||||
|
||||
```text
|
||||
src/lost_cities_jax/
|
||||
ppo.py # config, network, rollout, PPO update, train loop
|
||||
opponents.py # pure-JAX static opponent policy functions
|
||||
ppo_cli.py # argparse CLI
|
||||
configs/jax_ppo/
|
||||
discard-only.yaml
|
||||
balanced.yaml
|
||||
cautious.yaml
|
||||
```
|
||||
|
||||
If the PPO implementation remains readable in one main file, keep it there.
|
||||
Split only when a file becomes hard to test or review.
|
||||
|
||||
Dependencies likely needed:
|
||||
|
||||
- `flax` for model modules and train state.
|
||||
- `optax` for Adam and PPO losses.
|
||||
- `orbax-checkpoint` for checkpointing.
|
||||
|
||||
Add them through `uv add`, not pip/conda/poetry.
|
||||
|
||||
## Static Opponents
|
||||
|
||||
All opponent policies are pure JAX functions:
|
||||
|
||||
```python
|
||||
policy_fn(state: State, player: jax.Array) -> jax.Array # int32 action
|
||||
```
|
||||
|
||||
They must sample no Python-side randomness. If tie-breaking needs randomness,
|
||||
pass a JAX key explicitly:
|
||||
|
||||
```python
|
||||
policy_fn(state, player, rng) -> action
|
||||
```
|
||||
|
||||
Policies:
|
||||
|
||||
- `discard_only`: always discard a legal hand slot and draw from deck when
|
||||
legal. This opponent should make free expedition building easy.
|
||||
- `heuristic_balanced`: prefer legal plays that improve expedition prospects,
|
||||
avoid obviously toxic openings, draw useful discard tops when available.
|
||||
- `heuristic_cautious`: stricter opening threshold, fewer negative expedition
|
||||
commitments, more conservative discard/draw behavior.
|
||||
|
||||
Opponent policies are not learning targets. They are diagnostic fixtures.
|
||||
|
||||
## Rollout Driver
|
||||
|
||||
Run 8192 games in parallel by default.
|
||||
|
||||
The learner controls one fixed seat per rollout batch. The opponent occupies
|
||||
the other seat. Because games alternate turns, each environment step chooses:
|
||||
|
||||
- learner action from the actor policy when `state.to_move == learner_seat`;
|
||||
- static opponent action otherwise.
|
||||
|
||||
Done states remain no-op through the engine, so the rollout can keep a fixed
|
||||
time axis. Use `MAX_STEPS == 400` as the scan length unless a shorter config
|
||||
value is explicitly introduced.
|
||||
|
||||
The first milestone is random-policy rollout with the full dashboard. This is
|
||||
not throwaway; it defines the baseline canaries.
|
||||
|
||||
Log canaries:
|
||||
|
||||
- episode return;
|
||||
- score difference;
|
||||
- game length distribution;
|
||||
- `max_steps` termination rate;
|
||||
- `play_action_rate`;
|
||||
- opened color count;
|
||||
- positive expedition count per game;
|
||||
- average entropy under legal-action masking.
|
||||
|
||||
Record the random-policy baseline in `README.md` before PPO training starts.
|
||||
|
||||
## PPO Details
|
||||
|
||||
Network:
|
||||
|
||||
- Input: `OBS_DIM` observation vector.
|
||||
- Body: MLP 512 -> 512 -> 512, ReLU.
|
||||
- Policy head: 96 logits.
|
||||
- Value head: scalar.
|
||||
- Illegal actions are masked to a large negative value before sampling and
|
||||
before log-prob/loss computation.
|
||||
|
||||
Training defaults:
|
||||
|
||||
```yaml
|
||||
ppo:
|
||||
batch_games: 8192
|
||||
rollout_steps: 400
|
||||
gamma: 1.0
|
||||
gae_lambda: 0.95
|
||||
clip_epsilon: 0.2
|
||||
entropy_coef: 0.01
|
||||
value_coef: 0.5
|
||||
max_grad_norm: 0.5
|
||||
learning_rate: 0.0003
|
||||
epochs: 4
|
||||
minibatches: 128
|
||||
reward:
|
||||
terminal_scale: 50.0
|
||||
potential_shaping_initial: 1.0
|
||||
potential_shaping_final: 0.0
|
||||
potential_shaping_anneal_steps: 5_000_000
|
||||
```
|
||||
|
||||
Terminal reward:
|
||||
|
||||
```text
|
||||
tanh((learner_score - opponent_score) / terminal_scale)
|
||||
```
|
||||
|
||||
Potential shaping:
|
||||
|
||||
```text
|
||||
coef(t) * (board_score_diff_after - board_score_diff_before)
|
||||
```
|
||||
|
||||
The shaping coefficient anneals linearly from `initial` to `final`. Expose all
|
||||
four shaping fields in config. This shaping exists to prevent the previous
|
||||
play-action-rate collapse by giving immediate credit for board progress; its
|
||||
schedule is a controlled experiment variable, not a hidden constant.
|
||||
|
||||
## Evaluation Gates
|
||||
|
||||
Evaluation uses a fixed shuffle bank:
|
||||
|
||||
- Generate 10,000 explicit `deck_order` permutations from a fixed seed.
|
||||
- For each deck, play twice:
|
||||
- learner as player 0, opponent as player 1;
|
||||
- opponent as player 0, learner as player 1.
|
||||
- Aggregate duplicate-pair results.
|
||||
|
||||
Report:
|
||||
|
||||
- win rate;
|
||||
- Wilson confidence interval;
|
||||
- mean score difference;
|
||||
- mean game length;
|
||||
- positive expedition count per game;
|
||||
- opened colors;
|
||||
- `play_action_rate`.
|
||||
|
||||
Gates:
|
||||
|
||||
| Gate | Opponent | Pass condition | Diagnosis if failed |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | `discard_only` | win rate >= 90% and >= 2 positive expeditions/game | Audit reward pipeline, observations, and masks. Self-play is irrelevant. |
|
||||
| 2 | `heuristic_balanced` | mean score difference > 0 | Learner works; tune shaping schedule and batch/variance. |
|
||||
| 3 | `heuristic_cautious` | mean score difference > 0 | Same as gate 2; static ladder still not cleared. |
|
||||
|
||||
Only after all three gates pass should league self-play begin.
|
||||
|
||||
## 2026-07-04 Gate Results
|
||||
|
||||
All three static-opponent gates passed on an RTX 3090 using optional CUDA JAX:
|
||||
|
||||
```bash
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo train --config <config>
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval --config <config> \
|
||||
--checkpoint <run>/latest --games 10000 --duplicate --output <eval.json>
|
||||
```
|
||||
|
||||
Training used `batch_games=8192`, `rollout_steps=400`, `total_updates=250`,
|
||||
`seed=20260704`, and artifact root
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
|
||||
The implementation commit used for the runs was `4c0c2e9`.
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean length | Positive expeditions/game |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
|
||||
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
|
||||
|
||||
Artifact directories:
|
||||
|
||||
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/`
|
||||
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/`
|
||||
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/`
|
||||
|
||||
Final training rollout canaries:
|
||||
|
||||
| Opponent | Final train return_mean | Final play_action_rate | Final max_steps_rate |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| `discard_only` | 0.99281 | 0.59007 | 0.00000 |
|
||||
| `heuristic_balanced` | 0.93545 | 0.37737 | 0.00415 |
|
||||
| `heuristic_cautious` | 0.97101 | 0.37473 | 0.00391 |
|
||||
|
||||
The balanced and cautious policies win decisively but produce longer games than
|
||||
the discard-only diagnostic. That is not a gate failure, but the next training
|
||||
phase should keep game length and forced-end rate as canaries.
|
||||
|
||||
## Artifact Policy
|
||||
|
||||
Generated training output stays out of git. Use:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/
|
||||
```
|
||||
|
||||
Per gate, preserve:
|
||||
|
||||
- checkpoint directory;
|
||||
- resolved config;
|
||||
- duplicate evaluation JSON;
|
||||
- metrics JSONL;
|
||||
- short summary markdown with command, git commit, seed, and pass/fail result.
|
||||
|
||||
Commit only small, durable metadata to the code repo:
|
||||
|
||||
- config files;
|
||||
- evaluator/training code;
|
||||
- README baseline and gate summaries;
|
||||
- optional dated summary under docs/reports.
|
||||
|
||||
If binary artifact tracking is later required, add DVC/Git LFS explicitly
|
||||
instead of committing large checkpoint files directly.
|
||||
|
||||
## Work Order
|
||||
|
||||
### Phase 1: Rollout Dashboard
|
||||
|
||||
1. Add static opponent policies.
|
||||
2. Add random-policy batched rollout.
|
||||
3. Log canary metrics:
|
||||
returns, game length, max-step rate, `play_action_rate`, opened colors,
|
||||
positive expeditions.
|
||||
4. Run CPU tests and one GPU random rollout.
|
||||
5. Record the random baseline in README.
|
||||
|
||||
Exit criteria:
|
||||
|
||||
- deterministic smoke rollout passes;
|
||||
- canary metrics look finite and stable;
|
||||
- `play_action_rate` and game length are logged before any PPO code is judged.
|
||||
|
||||
### Phase 2: PPO Against `discard_only`
|
||||
|
||||
1. Add MLP actor-critic and masked action sampling.
|
||||
2. Add GAE and PPO update.
|
||||
3. Add Orbax checkpoints and resume.
|
||||
4. Add CLI and the discard-only config under configs/jax_ppo.
|
||||
5. Train on GPU.
|
||||
|
||||
First-run checks:
|
||||
|
||||
- with shaping enabled, `play_action_rate` rises above random baseline within
|
||||
the first few hundred thousand environment steps;
|
||||
- average game length converges near natural 44-ply games;
|
||||
- `max_steps` termination rate goes to zero.
|
||||
|
||||
Then anneal shaping and run duplicate evaluation for gate 1.
|
||||
|
||||
### Phase 3: Static Ladder
|
||||
|
||||
1. Reuse the same config and checkpoint flow.
|
||||
2. Change only the opponent config for `heuristic_balanced`.
|
||||
3. Train/evaluate until gate 2 passes or diagnostics point to variance.
|
||||
4. Repeat for `heuristic_cautious`.
|
||||
|
||||
No self-play work starts before gate 3 passes.
|
||||
|
||||
## Commands
|
||||
|
||||
Smoke rollout:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo rollout-smoke --config configs/jax_ppo/discard-only.yaml
|
||||
```
|
||||
|
||||
GPU training in tmux:
|
||||
|
||||
```bash
|
||||
tmux new-session -s coolrl-jax-ppo-discard \
|
||||
-c /home/coolguy/dev/coolrl-lost-cities \
|
||||
"flock -n .compute.lock uv run --with 'jax[cuda12]' lost-cities-jax-ppo train \
|
||||
--config configs/jax_ppo/discard-only.yaml"
|
||||
```
|
||||
|
||||
Duplicate evaluation:
|
||||
|
||||
```bash
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo eval \
|
||||
--config configs/jax_ppo/discard-only.yaml \
|
||||
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/latest \
|
||||
--opponent discard_only \
|
||||
--games 10000 \
|
||||
--duplicate \
|
||||
--output /mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/<run>/eval_duplicate.json
|
||||
```
|
||||
|
||||
## Risks
|
||||
|
||||
- The engine is fast on GPU, but Python logging/evaluation can dominate if
|
||||
metrics are copied every step. Aggregate in JAX and transfer per rollout.
|
||||
- Static heuristics can accidentally become too weak or too strong. Keep them
|
||||
deterministic and versioned by config.
|
||||
- Potential shaping can teach score-chasing artifacts if it never anneals.
|
||||
Treat the coefficient schedule as part of the experiment identity.
|
||||
- Checkpoint volume can grow quickly. Store large artifacts under `/mnt/2tbhdd`
|
||||
and keep only `latest` plus gate checkpoints unless a run is explicitly
|
||||
archived.
|
||||
|
||||
## Current Next Action
|
||||
|
||||
The static-opponent ladder is complete. The next eligible phase is snapshot-pool
|
||||
league self-play design, using the three passed checkpoints as initial anchors.
|
||||
@@ -117,7 +117,7 @@ operator applies the patch.
|
||||
|
||||
- ✅ AGENTS.md "Docs & Experiment Workflow" section landed
|
||||
(commit `09d5815`, 2026-05-07).
|
||||
- ✅ Plan drafted at `docs/plans/archive/librarian.md` (this file).
|
||||
- ✅ Plan drafted at `docs/plans/librarian.md` (this file).
|
||||
- ✅ Prompt moved: `.claude/agents/librarian.md` →
|
||||
`scripts/librarian-prompt.md`. Claude-specific subagent registration
|
||||
removed.
|
||||
@@ -1,617 +0,0 @@
|
||||
# Plan: 클래식 3라운드 로스트시티 에이전트
|
||||
|
||||
**Status:** Ready to execute (Fable 2차 검토 반영 완료)
|
||||
**Priority:** High — 현재 스택은 **단판(1라운드)** 게임을 학습하는데, 원작 로스트시티는
|
||||
**3라운드 합산**이 승패를 가른다. 즉 지금 에이전트는 애초에 다른 게임을 배우고 있다.
|
||||
**Background:** Fable 설계 검토 2회 (2026-07-15), 인간 대 AI 기보
|
||||
(`data/human-play/game-records.jsonl`, 계속 증가 중)
|
||||
|
||||
## 목표
|
||||
|
||||
**원작 규칙 그대로의 로스트시티에서 가장 강한 에이전트.** 레거시 코드 호환은 고려하지 않는다.
|
||||
env/obs/보상/학습 루프 재작성 모두 허용.
|
||||
|
||||
## 확정된 규칙 (룰북 원문 검증 완료)
|
||||
|
||||
코스모스 공식 영문 룰북(691820-02) 및 Board Game Arena 구현체로 교차 확인:
|
||||
|
||||
- 3라운드를 두고 **누적 총점**이 높은 쪽이 매치 승리.
|
||||
- 1라운드 선공: "가장 나이 많은 사람" → 게임 내적으로는 **임의**.
|
||||
- **2·3라운드 선공: 누적 점수가 더 많은 쪽** ("The player who has more points begins.")
|
||||
— 번갈아 가는 것이 아니다.
|
||||
- **정확한 동점일 때의 선공은 룰북에 없다.** 공식 룰북·BGA 모두 침묵.
|
||||
→ **무작위(동전 던지기)로 정한다.** 결정론적 규칙은 대칭 제로섬 셀프플레이에
|
||||
자리(seat) 비대칭을 주입해 착취 가능한 구멍이 된다.
|
||||
|
||||
점수 계산(`(랭크합 − 20) × (1 + 악수) + (8장 이상 +20)`)과 "방금 버린 카드는 즉시
|
||||
회수 불가"(`just_discarded`)는 **현재 엔진이 이미 정확하다**
|
||||
(`engine.py:244`, `engine.py:124`).
|
||||
|
||||
### 선공이 중요한 이유: 덱 시계
|
||||
|
||||
라운드는 **덱의 마지막 카드를 뽑는 순간** 끝난다. 카드는 놓은 뒤에 뽑으므로 마지막
|
||||
카드를 뽑은 쪽은 그 카드를 쓰지 못한다. 총 턴 수가 홀수면 **선공이 한 장 더 놓는다**.
|
||||
총 턴 수 = 44 + (버림패 드로우 횟수)이므로 **플레이어가 홀짝을 조작할 수 있다**
|
||||
(실측 게임당 버림패 드로우 ≈6.6회). 선공권이 누적 점수에 달려 있으므로, 2라운드
|
||||
마진에는 **점수를 넘어 3라운드 선공권이라는 추가 가치**가 붙는다.
|
||||
|
||||
### 측정된 사실: stalling은 정적 상대 착취이지 균형이 아니다 (Phase 1 사이징의 근거)
|
||||
|
||||
버림패 드로우는 덱을 줄이지 않으므로, **양쪽이 계속 버림패만 뒤지면 라운드가 끝나지
|
||||
않는다.** 기존 학습 로그를 보면 이 착취가 실제로 학습된다:
|
||||
|
||||
| 학습 체제 | 라운드 길이 추이 | 400수 상한 도달 |
|
||||
|---|---|---|
|
||||
| 정적 휴리스틱 상대 (`balanced.yaml`) | 82 → **116수** (eval greedy **168수**) | 0.4~0.7% |
|
||||
| **셀프플레이 (league)** | 71 → **53.6수로 수렴** | **0%** |
|
||||
|
||||
정적 약체 상대에게는 턴을 늘릴수록 원정 깊이·8장 보너스에서 강자가 이득이라 질질 끄는
|
||||
것이 합리적이다. 그러나 **셀프플레이에서는 상대도 똑같이 끌 수 있어 상쇄되고, 라운드가
|
||||
자연 길이(≈54수 = 44 덱 드로우 + ~10 버림패 드로우)로 수렴한다.**
|
||||
|
||||
**결론:** 우리의 목표 체제는 셀프플레이/league이므로 **3라운드 매치 ≈ 160수**로 잡으면
|
||||
되고, 기존 `rollout_steps=400` 안에 여유롭게 들어간다. 단 두 가지를 유의한다:
|
||||
|
||||
1. **정적 상대로 3라운드를 학습시키면 매치가 350~500수까지 늘어날 수 있다** — 정적 상대
|
||||
워밍업 단계를 쓴다면 스캔 길이를 따로 잡아야 한다.
|
||||
2. **라운드 상한(~120수)은 그 자체가 착취 가능한 구멍이다.** 앞선 쪽이 상한까지 끌면
|
||||
라운드가 유리한 상태로 얼어붙는다. `max_steps_rate`를 게이트 지표로 계속 감시할 것
|
||||
(셀프플레이에서는 0이어야 정상).
|
||||
|
||||
## 현재 코드의 결함 (Fable 2회 교차 검증 — 8건 전부 사실 확인)
|
||||
|
||||
1. **env가 1라운드짜리** — `State`에 라운드/누적 점수 없음 (`types.py:46`),
|
||||
`to_move=0` 하드코딩 (`engine.py:63`).
|
||||
2. **롤아웃의 87%가 죽은 스텝** — 한 라운드는 실측 평균 50.6수(최대 56)인데
|
||||
`rollout_steps = MAX_STEPS = 400` (`ppo.py:66`)이고, 스캔 **안에는 리셋이 없다**
|
||||
(`reset_done_envs`는 업데이트 사이에서만, `ppo.py:275`).
|
||||
3. **GAE 부트스트랩 초기값이 0** (`ppo.py:691`). 지금은 모든 에피소드가 스캔 안에서 끝나
|
||||
무해하나, **in-scan 리셋을 켜는 순간 편향**이 된다.
|
||||
4. **shaping 어닐링 회계 버그** — `update * batch_games * rollout_steps` (`ppo.py:222`)로
|
||||
**죽은 스텝까지 세어서**, 5M 어닐링이 250 업데이트 중 2번째에 끝난다. league는 shaping을
|
||||
아예 끈다 (`league.py:592`).
|
||||
→ **"shaping은 불필요하다"는 과거 결론이 있다면 무효다. 켜진 적이 없다.**
|
||||
5. **obs에 `to_move`가 없다.** 그런데 value loss는 `active` 마스크로 학습되어 (`ppo.py:624`)
|
||||
**상대 차례 상태에서도** 가치를 맞추라고 요구한다. critic이 차례를 구분할 단서가
|
||||
`step_count / 400`뿐 — MLP에게 부동소수점에서 홀짝을 뽑으라는 요구다.
|
||||
6. **점수차 정규화가 ÷780** (`MAX_ABS_SCORE`, `obs.py:68`). ±50점 차가 ±0.06 → 사실상
|
||||
안 보인다.
|
||||
7. **셀프플레이에서 상대 자리 결정을 전부 버린다** (`stop_gradient`, `ppo.py:425`) —
|
||||
의사결정의 절반을 낭비.
|
||||
8. **평가가 단판 승률** (`gates.py`) — 3라운드 개편의 성패를 측정할 자가 없다.
|
||||
|
||||
## 핵심 설계 결정
|
||||
|
||||
### PRNG: 모든 무작위성을 `reset()`에서 미리 뽑는다 (필수)
|
||||
|
||||
라운드 전환 리셔플과 동점 코인플립을 `step()` 안에서 샘플링하면 `step`이 확률적이 되어
|
||||
서명이 바뀌고, 모든 rollout/eval/gates 바디에 키를 꿰어야 하며 duplicate 미러링이 꼬인다.
|
||||
|
||||
**`reset()`에서 3라운드 덱 순서 전부(3×60), 동점 코인플립 비트 2개, 1라운드 선공 비트
|
||||
1개를 미리 샘플링해 `State`에 저장한다.** `step()`은 결정론을 유지하고 라운드 전환은
|
||||
다음 덱 슬라이스로 스위치만 한다. 덤: **미러 매치(같은 3딜 + 자리 교대 + 같은 코인플립
|
||||
비트)가 공짜로 따라온다** — Phase 5의 antithetic 페어링이 그대로 성립.
|
||||
|
||||
### 라운드 전환 vs 매치 리셋의 분리
|
||||
|
||||
- **라운드 전환은 엔진 `step()` 내부**에서 (`done=False` 유지, carry 갱신, 덱 슬라이스 전환)
|
||||
- **매치 done 리셋은 rollout 바디**에서 (in-scan auto-reset)
|
||||
|
||||
이러면 두 메커니즘이 깨끗이 분리된다. 단 위의 pre-sampled 덱 설계가 전제다.
|
||||
|
||||
- `State`에 **라운드 스텝 카운터와 매치 스텝 카운터를 둘 다** 둔다 (forced-done은 라운드
|
||||
단위, obs 정규화는 라운드 상대 진행도 + round 원-핫).
|
||||
- 중간 라운드 forced-done(라운드 상한 초과) 시 **자연 종료와 동일하게 라운드 종료 → 전환**.
|
||||
|
||||
### 보상 스케일에 대한 해소 (혼동 방지)
|
||||
|
||||
`sign()`에 가까운 작은 scale은 **틀린 선택**이다 — 150수에 걸쳐 모든 수가 동일한 ±1을
|
||||
받아 크레딧 할당이 전부 critic에 떠넘겨지고, 5점 차 패배와 80점 차 패배의 그래디언트가
|
||||
같아진다. **scale 30~50의 tanh가 절충점**이다: |마진| ≳ 60에서는 `sign`을 근사하면서
|
||||
크레딧 그래디언트를 보존한다. 대신 5점 승과 80점 승을 여전히 구분하므로 순수 승률
|
||||
최적화 대비 **마진 쪽으로 약간 왜곡**된다 — 이 잔여 왜곡은 Phase 5의 후기 fine-tune으로
|
||||
scale을 낮춰서(예: 50 → 25) 제거한다.
|
||||
|
||||
명시적 risk 항은 **넣지 않는다.** carry + round를 obs에 넣은 종료 보상이 올바른 리스크
|
||||
태도(뒤지면 도박, 앞서면 잠금)를 자동으로 유도한다.
|
||||
|
||||
## 체크리스트
|
||||
|
||||
### Phase 0a — 학습 루프 수정 (최우선, 단판 체제에서 검증 가능)
|
||||
|
||||
- [x] **GAE 부트스트랩 수정**: 잘린 에피소드를 `V(s_final)`로 부트스트랩 (`ppo.py:691`).
|
||||
- [x] **in-scan auto-reset**: 롤아웃 스캔 안에서 done env 리셋 → 죽은 스텝 제거.
|
||||
**실측(rollout_steps=400, batch_games=64, 3 업데이트):** main은 활성 17.3%
|
||||
(learner 액션 2,221/업데이트), 수정 후 **활성 100% (12,842/업데이트) — 동일 연산량에
|
||||
샘플 5.8배.** 활성 비율은 게임 길이에 비례하므로, 학습 초기 정책(~69수)에서 5.8배,
|
||||
학습된 정책(~50수)이면 8배에 가까워진다. 3라운드 매치(~150수)가 되면 약 2.7배로
|
||||
줄지만 그때는 스캔 전체가 실제 매치로 채워진다.
|
||||
- [x] **메트릭 파이프라인 재작업 (필수, 놓치면 조용히 오염됨)**: `rollout_metrics`는
|
||||
`final_env`에서 통계를 읽고 `episode_return = jnp.sum(transitions.reward, axis=0)`은
|
||||
**env당 에피소드 1개를 가정**한다 (`ppo.py:1069`). in-scan 리셋 후에는 final_env가
|
||||
에피소드 중간이고 여러 에피소드의 보상이 합산된다 → **metrics.jsonl 전체가 쓰레기가
|
||||
된다.** done 경계에서 누적하는 에피소드 단위 집계로 바꿀 것.
|
||||
- [x] **league assignments를 scan carry로**: `learner_seat`/`opponent_index`/`use_mirror`가
|
||||
업데이트 사이에서만 재샘플링된다 (`ppo.py:568`). in-scan 리셋을 켜면 리셋 시점에
|
||||
스캔 내부에서 재샘플링해야 한다 — 안 하면 자리/상대가 에피소드 간 고정되어 편향.
|
||||
- [x] **shaping 어닐링 회계 수정**: 패딩된 스캔 스텝이 아니라 **learner 액션 수** 기준
|
||||
(`ppo.py:222`). `shaping_coef`는 호스트에서 계산해 jit 함수에 넘기므로
|
||||
(`ppo.py:221`), 직전 업데이트의 액션 수를 호스트에서 누적하는 **1-업데이트 지연**
|
||||
구조가 된다 (무해).
|
||||
- [x] **회귀 게이트 — 통과 (단, 게이트 설계를 고쳐야 했다).**
|
||||
|
||||
**(1) "동일 learner-액션 예산"은 잘못된 게이트였다.** PPO 진척도는 샘플 수와
|
||||
**옵티마이저 스텝 수** 둘 다에 달려 있다. 샘플을 맞추면(64 vs 250 업데이트) 새 코드는
|
||||
그래디언트 스텝이 1/4이 되어 덜 수렴한다. 업데이트당 연산량은 동일하므로
|
||||
**올바른 게이트는 250 대 250**(같은 컴퓨트·같은 그래디언트 스텝, 샘플만 5.8배)이다.
|
||||
|
||||
**(2) 동일 컴퓨트 게이트 결과 (2만판 duplicate eval):**
|
||||
|
||||
| 앵커 | 지표 | 베이스라인 | Phase 0a |
|
||||
|---|---|---|---|
|
||||
| **expert** (강함) | 승률 | 0.4525 | **0.5071** |
|
||||
| **expert** | 평균 점수차 | −4.28 | **−0.56** |
|
||||
| balanced (약함) | 승률 | 0.9866 | 0.9292 |
|
||||
| balanced | 평균 점수차 | 116.8 | **128.1** |
|
||||
|
||||
학습 곡선은 **모든 업데이트 지점에서** 새 코드가 위(최종 return 0.963 vs 0.935).
|
||||
|
||||
**(3) balanced 승률 하락은 전부 `MAX_STEPS=400` 아티팩트다.** 새 정책 2,048판 분해:
|
||||
|
||||
| | 판수 | 패배율 |
|
||||
|---|---|---|
|
||||
| 자연 종료 | 1,214 | **0.08%** |
|
||||
| 400수 상한 도달 | 834 (40.7%) | **20.62%** |
|
||||
|
||||
**전체 패배의 99.4%가 상한에 부딪힌 판에서 발생.** 새 정책은 자연 종료 게임에서
|
||||
**99.92% 승률**이다. 샘플이 5.8배라 stalling 착취를 더 깊이 배웠고(greedy 225수),
|
||||
그 결과 40%가 인위적 벽에 박아 원정 미완성인 채 얼어붙는다.
|
||||
|
||||
**결론:** Phase 0a는 성공. expert(=stalling을 허용하지 않는 앵커)에서 45%→51%로
|
||||
실제 실력이 올랐다.
|
||||
|
||||
### Phase 1 — 3라운드 env
|
||||
|
||||
**구현: `src/lost_cities_jax/match.py` (커밋 `7cd299d`).** 단판 엔진을 고치지 않고 **감쌌다**
|
||||
— `engine.py`는 TS 클라이언트와의 차분 테스트가 지키는 규칙 오라클이고, 라운드 자체는
|
||||
매치가 생겨도 변하지 않는다. `MatchState`가 내부에 단판 `State`를 든다.
|
||||
|
||||
- [x] `MatchState`: `round`(단판 State), `deck_orders`(3×60), `coin_flips`(3,),
|
||||
`round_idx`, `carry`(2, 플레이어별 누적 점수), `done`.
|
||||
별도 매치 스텝 카운터는 두지 않았다 — 매치는 3라운드가 끝나면 종료하므로 종료 판정에
|
||||
불필요하고, 라운드 진행도는 내부 `State.step_count`(라운드마다 리셋)가 이미 준다.
|
||||
- [x] `match_step()`: 덱 소진 시 라운드 < 3이면 carry 갱신 → 다음 덱 슬라이스 → `round_idx += 1`,
|
||||
`done` 유지. 라운드 == 3이면 `done = True`. **보상은 3라운드 끝에서만.**
|
||||
- [x] **선공 규칙**: `carry[0] > carry[1]` → p0, `<` → p1, `==` → pre-sampled 코인플립.
|
||||
1라운드는 특수 케이스가 **필요 없다** — carry가 (0,0)이라 동점 분기가 자동으로 코인을
|
||||
뽑고, 그게 룰북의 "가장 나이 많은 사람"(= 임의)과 정확히 같다.
|
||||
- [x] **PRNG**: 딜 3벌 + 코인 3개를 `match_reset`에서 전부 미리 뽑아 상태에 저장.
|
||||
`match_step`은 결정론적이라 키를 rollout/eval/gates에 꿸 필요가 없고, **미러 매치가
|
||||
자리 라벨 교체만으로 성립**한다 (Phase 5 antithetic 페어링이 여기 의존).
|
||||
- [x] ~~라운드당 스텝 상한을 400 → ~120으로 분리~~ — **철회한다. 라운드 상한은 400을 유지.**
|
||||
|
||||
**이 항목의 원래 명분이 사라졌다.** 근거는 "3라운드(~360수)가 400스텝 스캔에 들어가게"
|
||||
였는데, **Phase 0a의 GAE 절단 부트스트랩이 스캔 길이 제약을 없앴다.** 잘린 매치는
|
||||
올바르게 부트스트랩되므로 스캔 길이는 이제 자유 파라미터다.
|
||||
|
||||
**그리고 상한을 낮추면 아티팩트가 악화된다.** Phase 0a 실측: 400수 벽에 부딪힌 판의
|
||||
20.6%가 패배로 뒤집혔고 **전체 패배의 99.4%가 거기서 발생**했다. 상한을 120으로 내리면
|
||||
**더 많은 판이 벽에 박고**, "앞선 쪽이 상한까지 끌어 라운드를 얼린다"는 착취가 훨씬
|
||||
쉬워진다. 셀프플레이에서는 라운드가 54수로 수렴해 120이든 400이든 안 걸리므로,
|
||||
**덜 걸리는 쪽을 두고 `max_steps_rate`를 감시**하는 것이 맞다.
|
||||
|
||||
- [ ] **학습 파이프라인 배선** (원래 계획서에서 누락된 항목): `ppo.py`의 rollout/eval,
|
||||
`gates.py`, `league.py`가 전부 단판 `State`를 받는다. `MatchState`를 받도록 배선한다.
|
||||
- 매치용 obs 래퍼: 우선 `observation(state.round, player)`로 carry-blind하게 연결
|
||||
(carry/round 피처는 Phase 3에서 추가).
|
||||
- `match_legal_action_mask`, `match_step`, `match_score`로 교체.
|
||||
- 스캔 길이는 자유 파라미터가 되었으므로 셀프플레이 매치 길이(~160수) 기준으로 잡되,
|
||||
잘려도 부트스트랩이 처리한다.
|
||||
- [ ] **league 풀 재구축**: 기존 스냅샷 멤버는 obs 변경으로 전부 무효화된다 — Phase 3 이후로
|
||||
미룬다 (obs가 확정되기 전에 풀을 다시 채우면 두 번 일한다).
|
||||
- [x] **테스트 12개** (`tests/lost_cities_jax/test_match.py`): 라운드당 덱 드로우 = 44 +
|
||||
버림패 드로우만큼 연장, `carry`가 각 라운드를 정확히 한 번씩 적립, 선공 규칙 3분기,
|
||||
1라운드 코인 공정성(≈50/50), 라운드 경계에서 누적 점수 불연속 없음(PBRS 전제),
|
||||
미러 매치 대칭성, 보상은 3라운드 끝에서만 지급.
|
||||
|
||||
### Phase 0b — 측정 자 (첫 장기 3라운드 학습 **전에** 착지)
|
||||
|
||||
- [ ] **매치 단위 평가**: 3딜 전부 미러링 + 자리 교대 + **동일 코인플립 비트**,
|
||||
매치 승률 + Wilson CI, 평균 총 마진. `gates.py`/`league.py`의 단판 기준을 대체.
|
||||
**`gates.py:884`, `gates.py:961`의 `MAX_STEPS` 길이 단판 스캔 2곳 포팅 포함** —
|
||||
exploiter 학습·평가 경로 전체가 3라운드로 가야 성공 기준 3을 잴 수 있다.
|
||||
- [ ] **carry 조건부 프로브**: carry ∈ {−60, −25, −1, +1, +25, +60}을 주입한 3라운드 시작
|
||||
위치에서 승률·행동 변화(원정 개수, 악수 비율, 덱 레이스 비율) 측정.
|
||||
- [ ] **shuffle bank 포맷 확장**: 덱과 함께 코인플립·선공 비트도 뽑도록. 안 그러면
|
||||
`jax.random.PRNGKey(0)` 고정 eval(`ppo.py:1025`)의 재현성이 매치 정의와 얽힌다.
|
||||
- [ ] **legacy obs 버전 보존**: 구 정책을 3라운드 매치에 투입해 베이스라인을 재려면 구
|
||||
obs(454차원)로 추론해야 하는데, `checkpoint_policy_from_params`는 전역 `observation`을
|
||||
호출한다 (`ppo.py:802`) → obs 개편 후 구 체크포인트는 **로드 자체가 실패**한다.
|
||||
policy 로더가 체크포인트별 obs 버전을 받도록 할 것. **성공 기준 2의 분모가 여기 달렸다.**
|
||||
|
||||
### Phase 2 — 보상
|
||||
|
||||
- [ ] 종료 보상: 3라운드 끝에서만 `tanh(총_점수차 / scale)`, **scale = 30~50**
|
||||
(근거는 위 "보상 스케일에 대한 해소" 절).
|
||||
- [ ] 마진 shaping 부활: `Φ(s) = carry + 현재 보드 점수차`, **작은 계수**(종료 보상 스케일의
|
||||
0.05~0.2). 현재의 계수 1.0 원점수 shaping은 ±1 종료 보상보다 10~30배 크다.
|
||||
learner 스텝 기준으로 어닐링하되 **후반까지 정확히 0으로 내리지 않는다.**
|
||||
- [ ] γ=1.0에서 `Φ(s') − Φ(s)`는 **올바른 PBRS 형태다** (γ 누락 아님 — 검토에서 확인).
|
||||
|
||||
### Phase 3 — 관측
|
||||
|
||||
- [ ] `carry`: **÷75 스칼라 + 구간 원-핫**(약 9구간). 3라운드 정책은 "1점만 더" 임계값
|
||||
근처에서 급격히 꺾여야 한다. 기존 `score_diff`의 ÷780 정규화도 같이 고친다.
|
||||
- [ ] `round_idx` 원-핫 + 남은 라운드 수.
|
||||
- [ ] **`to_move` 비트** + "내가 이번 라운드 선공인가" + "현재 홀짝에서 마지막 덱 카드를
|
||||
누가 뽑는가"(덱 시계).
|
||||
- [ ] 색깔별 **살아있는 점수 3분할**: 내 `col_top` 위로 아직 나올 수 있는 점수를
|
||||
(a) **내 손패**, (b) **버림패 더미**(공개돼 있고 회수 가능 — 빠뜨리기 쉬움),
|
||||
(c) **미공개**(덱 ∪ 상대 은닉 손패)로 나눠 넣는다. 상대에 대해서도 동일
|
||||
(상대 `col_top`은 공개). MLP가 7×60 채널에서 뽑아내기 어려운 비선형 집계이고,
|
||||
모든 개시/연장/차단 판단을 좌우한다.
|
||||
|
||||
### Phase 4 — 학습 효율
|
||||
|
||||
- [ ] **전지적 critic (CTDE)**: critic에만 상대 손패 + 덱 구성을 준다. 행동과 무관한
|
||||
정보이므로 정책 그래디언트를 편향시키지 않는다. 딜 운 분산을 정면으로 깎는다.
|
||||
**가치 경로를 정책 트렁크에서 분리해야 한다** (현재 공유 트렁크, `ppo.py:104`) —
|
||||
특권 정보가 정책 로짓으로 새면 안 된다.
|
||||
→ 이 critic은 나중에 **PIMC 탐색의 리프 평가기로 그대로 재활용**된다.
|
||||
- [ ] **양쪽 자리 학습**: `stop_gradient`된 상대 자리 전이(`ppo.py:425`)도 학습에 쓴다
|
||||
(샘플 효율 2배). 같은 게임의 두 자리는 반상관이므로 같은 배치에 두고 advantage
|
||||
정규화에 맡긴다.
|
||||
- [ ] `gae_lambda` 재검토: 0.95는 150수 지평에서 너무 짧다(중반 수 직접 가중치 0.02).
|
||||
전지적 critic이 있으면 유지, 없으면 0.97~0.99. **λ=1은 금지**(딜 분산).
|
||||
- [ ] `entropy_coef` **스윕으로 재결정**. (주의: "0.01이 원점수 shaping 기준으로 잡혔다"는
|
||||
추론은 **틀렸다** — league는 shaping을 끄고 학습했으므로 0.01은 이미 ±1 tanh 체제에서
|
||||
동작해온 값이다. 다만 3라운드에서 보상 빈도가 1/150로 희석되고 작은 shaping이
|
||||
추가되므로 재튜닝 자체는 타당하다. **근거 없이 10배 낮추면 과소탐색으로 직행한다.**)
|
||||
|
||||
### Phase 5 (나중) — 최종 강함
|
||||
|
||||
- [ ] **후기 fine-tune**: 학습 말미에 종료 보상 scale을 낮춰(50 → 25) 마진 왜곡을 제거하고
|
||||
순수 승률 쪽으로 당긴다.
|
||||
- [ ] 페어드 antithetic 딜(같은 3딜 + 자리 교대 + 같은 코인플립 비트)을 **학습에** 도입.
|
||||
단순 포함이 아니라 **쌍으로 묶어** control variate로 써야 효과가 있다.
|
||||
(Phase 1의 pre-sampled PRNG 설계 덕에 사실상 공짜.)
|
||||
- [ ] MMD식 정규화 셀프플레이 (loss에 ~20줄) — 2인 제로섬 근사 내시 보험.
|
||||
- [ ] **추론 시점 탐색 (PIMC / ISMCTS)** — 최종 강함의 가장 큰 이득. 로스트시티는 블러핑
|
||||
경제가 없는 저기만성 불완전정보 게임이라 결정화 탐색이 잘 맞는다. raw net 대
|
||||
net+search 맞대결로 측정.
|
||||
- [ ] 네트워크 용량 A/B (512×3 → 1024×3 또는 residual) — **파이프라인 변경이 끝난 뒤에.**
|
||||
|
||||
## 범위 밖 (명시적 동결)
|
||||
|
||||
- **웹 클라이언트/ONNX는 별도 계획 전까지 레거시 단판 모델로 동결한다.** obs 개편 즉시
|
||||
export 파이프라인(`scripts/export_jax_ppo_onnx.py:70`, manifest `observation_size: 454`),
|
||||
TS obs 빌더, TS 단판 엔진이 전부 비호환이 된다. 이 선언이 없으면 실행 중 스코프가
|
||||
웹 재작성으로 샌다.
|
||||
- **Deep CFR 복귀** — 이미 BC 천장을 쳤고, 3라운드는 트리만 키운다. PPO+league를 학습
|
||||
백본으로 유지한다.
|
||||
- 레거시 호환을 위한 타협.
|
||||
|
||||
## 성공 기준
|
||||
|
||||
0. **(Phase 0a 게이트)** 루프 수정 후 단판 체제에서 동일 learner-액션 예산으로 기존 anchor
|
||||
성적 재현 ≥ 동등. **없으면 auto-reset 버그가 3라운드 결과에 섞여 원인 분리가 불가능해진다.**
|
||||
1. **carry 프로브에서 행동이 단조롭게 변한다** — carry 6개 수준에 걸쳐 원정 개수/악수 비율의
|
||||
**단조 추세**(CI 포함). ("행동이 변한다"는 노이즈로도 통과 가능하므로 단조성으로 정의.)
|
||||
2. **매치 승률이 구 정책(3라운드에 그대로 투입)을 이긴다** — 페어드 매치 **≥ 1만 쌍**,
|
||||
매치 승률 **Wilson 하한 > 0.5** *및* 평균 총 마진 **CI 하한 > 0**.
|
||||
3. 착취자(exploiter) 승률이 악화되지 않는다.
|
||||
4. **(anchor 비회귀)** 휴리스틱 anchor 상대 매치 승률·라운드당 마진이 구 정책 대비 악화되지
|
||||
않는다. **기준 2의 구멍을 막는 항목**: carry-blind인 구 정책만 상대로 이기는 것은
|
||||
**카드 플레이가 퇴보해도 carry 착취만으로 달성 가능**하다.
|
||||
|
||||
**단, anchor 선택에 주의:** Phase 0a에서 실증됐듯 **약한 anchor(`heuristic_balanced`)는
|
||||
무한 stalling을 허용해서 신호를 오염시킨다** — 승률이 실력이 아니라 상한 도달률을 잰다.
|
||||
**`heuristic_expert`처럼 stalling을 허용하지 않는 anchor**(게임이 자연 길이로 끝남)만
|
||||
실력 지표로 쓰고, 약한 anchor는 **자연 종료 게임만 필터링해서** 보거나 `max_steps_rate`를
|
||||
함께 보고할 것.
|
||||
|
||||
---
|
||||
|
||||
## 실행 결과 (2026-07-15, 커밋 `9ba5a07`)
|
||||
|
||||
### 동작하는 것
|
||||
|
||||
- **셀프플레이가 stalling을 스스로 제거한다.** 매치가 146.8수(라운드 ≈49수)로 수렴하고
|
||||
덱 레이스 비율 91%. Phase 0a에서 정적 상대가 225수까지 끌던 것과 대조된다.
|
||||
- **미러 매치 평가가 정확하다.** 셀프플레이 duplicate 승률 0.4968, **평균 lead 정확히 0.0** —
|
||||
같은 3딜 + 같은 코인 + 자리 교대가 딜 운을 완전히 상쇄한다. 미완료 0%.
|
||||
- **전지적 critic이 격리돼 있다.** 특권 입력을 흔들어도 정책 로짓은 비트 단위로 동일하고,
|
||||
가치만 움직인다 (테스트로 고정).
|
||||
|
||||
### 성공 기준 1 — **약하게만 충족. 사실상 미달.**
|
||||
|
||||
carry 프로브(3라운드 시작 시 carry 주입)에서:
|
||||
|
||||
| | scale 50 | scale 25 | scale 12 |
|
||||
|---|---|---|---|
|
||||
| 악수 스프레드 (carry −60 → +60) | 0.19 | 0.08 | **0.31** |
|
||||
| 원정 개수 | 5.00 고정 | 4.99 고정 | **5.00 고정** |
|
||||
|
||||
- **악수 사용은 6개 carry 수준 전체에서 단조**로 움직인다 (뒤지면 배수 베팅 ↑). 방향은 맞다.
|
||||
- 그러나 **원정 개수는 carry와 무관하게 5.00에 붙어 있다.** 굳은 습관이지 조건부 플레이가 아니다.
|
||||
- 효과 크기가 작다.
|
||||
|
||||
### 확정된 진단
|
||||
|
||||
1. **shaping 가설은 틀렸다.** `potential_shaping_final=0.0`으로 완전히 꺼도 프로브는 평평했다.
|
||||
2. **원인은 `terminal_scale`이다.** carry = −60에서 `tanh((margin − 60)/50)`의 인자는 현실적
|
||||
마진 범위에서 **거의 선형**이고, 선형 구간에서 `E[tanh]` 최대화는 `E[margin]` 최대화와 같다
|
||||
→ 도박할 이유가 없다. 위험 추구는 tanh가 **강하게 볼록한** 구간에서만 나오며 scale을 낮춰야
|
||||
그 구간에 들어간다. scale 50 → 12에서 스프레드가 0.19 → 0.31로 커진 것이 이를 확인한다.
|
||||
3. **프로브 설계 결함:** scale 12에서 `tanh(60/12) ≈ 1.0`이라 **±60은 포화 = 그래디언트 0**.
|
||||
거기서 정책은 학습된 바가 없다. **의미 있는 측정 구간은 `|carry| ≲ 2 × terminal_scale`.**
|
||||
프로브 수준을 scale에 맞춰 재설계해야 한다.
|
||||
|
||||
### 다음에 할 일
|
||||
|
||||
- [ ] **프로브 재설계**: carry 수준을 `terminal_scale`에 맞춰 잡는다 (포화 구간 측정 금지).
|
||||
- [ ] **학습 중 carry 분포 확인**: 셀프플레이에서 3라운드 진입 시 carry가 실제로 얼마나
|
||||
퍼지는가. 좁으면 정책이 큰 deficit을 본 적이 없다는 뜻이고, 그게 진짜 원인일 수 있다.
|
||||
- [ ] **"항상 5색 개시"가 정상인지 검증**: 인간 기보에서 AI는 4.81, 인간은 4.19를 열었다.
|
||||
5.00 고정은 의심스럽다. 셀프플레이 균형인지, 탐색 붕괴인지 (엔트로피 3.5 → 1.03).
|
||||
- [ ] scale 스케줄(50 → 12 후기 fine-tune)이 처음부터 12로 학습하는 것보다 나은지 A/B.
|
||||
|
||||
---
|
||||
|
||||
## 판정: 보상은 **선형 총점**이다 (2026-07-15, Fable 3차 검토 + 실측)
|
||||
|
||||
사용자 제안("그냥 3판 총점이 크기만 하면 되는 것 아니냐")이 **맞았다. tanh 종료 보상은
|
||||
오버엔지니어링이었다.**
|
||||
|
||||
### 근거 1 — 분해 논증 (실측)
|
||||
|
||||
보상이 총점에 선형이면 라운드가 독립이므로 3라운드 게임이 3개의 독립 단판으로 분해된다.
|
||||
셀프플레이 2,048매치 실측: `corr(m1, m2) = 0.004`, `corr(m1+m2, m3) = 0.05`.
|
||||
라운드를 잇는 유일한 고리인 **선공 어드밴티지는 +0.73 ± 0.84점** — 0과 구분 불가.
|
||||
|
||||
### 근거 2 — 리스크 태도는 값어치가 없다 (실측)
|
||||
|
||||
Fable이 리스크 태도를 직접 구현해 greedy 클론과 duplicate 6,144판 맞대결:
|
||||
|
||||
| 도박 정책 | 매치 승률 |
|
||||
|---|---|
|
||||
| 3R에서 20점 이상 뒤지면 온도 샘플링 | **0.482** (진다) |
|
||||
| 40점 이상 뒤질 때만 | 0.498 (본전) |
|
||||
|
||||
**일부러 도박을 시켜도 지거나 본전.** 로스트시티의 분산 레버(한계 악수 ≈ Δσ +1.7에 마진
|
||||
−2~3점)가 근본적으로 약해서, 분산을 사는 비용이 볼록성 이득을 먹는다.
|
||||
carry 조건부 플레이의 가치 상한: **1승점 미만.**
|
||||
|
||||
### 근거 3 — 맞대결에서 단순한 쪽이 **이겼다**
|
||||
|
||||
duplicate 10,000판 (같은 3딜 + 같은 코인 + 자리 교대), 동일 컴퓨트 300 업데이트:
|
||||
|
||||
| A | B | A 승률 | A 평균 총점차 |
|
||||
|---|---|---|---|
|
||||
| **선형 총점** | tanh(총점/12) | **0.5859** (CI 0.576–0.596) | **+20.3점** |
|
||||
|
||||
버려도 손해가 없는 게 아니라 **버리니 더 강해졌다.** 이유는 리스크가 아니라 **신호 밀도**다:
|
||||
tanh는 ~150수 매치에 포화된 ±1 하나를 주고, 선형은 매 수마다 그 수가 총점차를 움직인 만큼을
|
||||
준다. γ=1에서 후자의 합이 정확히 최종 총점차로 telescoping되므로 **목적함수는 동일한데 크레딧
|
||||
할당만 150배 조밀**하다.
|
||||
|
||||
### 근거 4 — 탐색 붕괴가 아니었다
|
||||
|
||||
프로브가 평평했던 이유는 탐색 붕괴가 아니다. 샘플링 프로브에서도 원정 5.00 ± 0.05,
|
||||
엔트로피 1.36나트(유효 행동 3.9개)로 정상 수렴. 그리고 **critic은 carry를 완벽히 읽고 있었다**
|
||||
(3R 시작 가치 −0.87 → +0.86 단조). 신호는 있었고, **정책이 그걸로 살 수 있는 물건이
|
||||
없었을 뿐**이다.
|
||||
|
||||
### 최종 설계
|
||||
|
||||
```python
|
||||
# match_ppo.py 롤아웃 바디
|
||||
reward0 = jnp.where(active, (after - before) / cfg.reward.terminal_scale, 0.0)
|
||||
```
|
||||
|
||||
- **성공 기준 1(carry 프로브 단조성)은 게이트에서 제거한다.** 최적 반응 자체가 이 게임에서
|
||||
거의 존재하지 않는다는 것이 측정 결과다. 기준 2·4(맞대결 + anchor 비회귀)가 옳은 자다.
|
||||
- `carry`는 obs에 **남긴다** (비용 0, 선공 규칙이 키로 쓰는 상태, critic이 잘 읽음).
|
||||
|
||||
### 목적함수와 무관하게 살아남은 것 (전부 순이득)
|
||||
|
||||
in-scan auto-reset + GAE 절단 부트스트랩(샘플 5.8배), 메트릭 재작업, **비대칭 CTDE critic**,
|
||||
**양쪽 좌석 학습**, **duplicate 매치 평가**, pre-sampled PRNG/미러 설계, 그리고 match_obs의
|
||||
단판 결함 수정분(`to_move`, 덱 시계, ÷780 → ÷75, 살아있는 점수 3분할).
|
||||
|
||||
**죽은 것은 tanh 종료 보상과 carry 구간 원-핫뿐이다.**
|
||||
|
||||
---
|
||||
|
||||
## 신구 대결: 진짜 3라운드 클래식에서 (2026-07-15)
|
||||
|
||||
**질문: "기존 학습 방법 대비 실제로 뭐가 나아졌나?"**
|
||||
|
||||
duplicate 매치 8,192판(같은 3딜 + 같은 코인 + 자리 교대), 진짜 3라운드 게임:
|
||||
|
||||
| 신(매치 스택) | 상대 | 신 승률 | 신 평균 총점차 | 상대 learner 액션 |
|
||||
|---|---|---|---|---|
|
||||
| 39.3M | 기존 baseline (정적 상대 학습) | 0.6077 | +19.5 | 104M |
|
||||
| 39.3M | Phase 0a 게이트 (루프 수정만) | **0.5842** | +16.0 | **411M (10.5배)** |
|
||||
| 39.3M | league (셀프플레이, 웹 배포판) | 0.3142 | −37.0 | 122.6M (3.1배) |
|
||||
| **131M** | **league (셀프플레이, 웹 배포판)** | **0.6094** (CI 0.599–0.620) | **+22.0** | 122.6M |
|
||||
|
||||
**결론:**
|
||||
|
||||
1. **샘플 효율이 크게 올랐다.** 39.3M짜리가 **411M(10.5배)짜리를 이긴다.**
|
||||
2. **동일 예산에서 기존 최강(league)을 이긴다** — 131M vs 122.6M에서 0.6094.
|
||||
3. 39.3M에서 league에 졌던 것(0.3142)은 **약해서가 아니라 예산이 1/3이어서**였다.
|
||||
|
||||
**남은 단서:** league는 **착취자(exploiter) 구조**를 포함한 런이다. 평균 강함은 우리가 이기지만
|
||||
**exploitability는 아직 재지 않았다.** "덜 착취당한다"는 증명되지 않았다.
|
||||
|
||||
---
|
||||
|
||||
## 기여도 A/B (2026-07-15)
|
||||
|
||||
각 조각을 하나씩 끄고 동일 컴퓨트(300 업데이트, batch 512)로 학습시킨 뒤,
|
||||
full 스택과 duplicate 매치 8,192판 맞대결. **0.5 미만 = 그 조각이 기여하고 있었다.**
|
||||
|
||||
| 제거한 것 | full 대비 승률 | 95% CI | 평균 총점차 | 판정 |
|
||||
|---|---|---|---|---|
|
||||
| **양쪽 좌석 학습** | **0.3317** | [0.322, 0.342] | −28.6 | **압도적 기여** |
|
||||
| 매치 관측 (carry/to_move/덱시계/살아있는점수) | 0.4751 | [0.464, 0.486] | −4.0 | 유의하게 기여 |
|
||||
| **전지적 critic (CTDE)** | **0.5160** | [0.505, 0.527] | +2.6 | **오히려 해가 된다** |
|
||||
|
||||
### 1. 양쪽 좌석 학습 — 가장 큰 기여
|
||||
|
||||
끄면 승률이 0.33으로 무너진다. 같은 업데이트 수에서 learner 액션이 **정확히 절반**이 되므로
|
||||
상당 부분은 샘플 수 효과다 — 그러나 **그게 요점이다.** 셀프플레이에서 상대 좌석의 수는 같은
|
||||
네트워크가 둔 것인데 기존 트레이너는 `stop_gradient`로 버렸다. **공짜로 데이터가 2배**가 된다.
|
||||
|
||||
### 2. 매치 관측 — 작지만 실재
|
||||
|
||||
0.4751 (CI 상한 0.486 < 0.5). `to_move`, 덱 시계, ÷780 → ÷75 스케일 수정, 살아있는 점수
|
||||
3분할이 합쳐서 약 2.5%p 값어치. carry 자체는 (분해 논증대로) 거의 기여하지 않으므로,
|
||||
이 이득의 대부분은 **단판 obs의 결함 수정분**으로 보인다.
|
||||
|
||||
### 3. 전지적 critic — **Fable의 최우선 권고가 틀렸다**
|
||||
|
||||
Fable은 이것을 "가장 큰 누락 아이디어"로 꼽았다. 실측은 **반대**다: 끄면 오히려
|
||||
**0.5160 (CI 하한 0.505 > 0.5)** 으로 유의하게 **더 강해진다.**
|
||||
|
||||
그럴듯한 이유: 특권 critic은 덱을 알기 때문에 가치를 아주 잘 맞추지만, 그 결과
|
||||
`V(전체상태) ≠ E[리턴 | 마스킹된 관측]`이 되어, **어드밴티지에 정책이 통제할 수 없는 성분**이
|
||||
섞인다. 액터 입장에서 그건 분산 감소가 아니라 노이즈다. 부분관측 환경에서 비대칭 critic이
|
||||
해가 되는 것은 알려진 현상이다.
|
||||
|
||||
**조치: 전지적 critic을 기본에서 끈다.** (PIMC 탐색의 리프 평가기로 재활용한다는 계획은
|
||||
별개로 유효할 수 있으나, 정책 학습용으로는 손해다.)
|
||||
|
||||
### 종합: 실제로 값어치 있었던 것
|
||||
|
||||
1. **in-scan auto-reset + GAE 절단 부트스트랩** — 샘플 5.8배, expert 0.4525 → 0.5071
|
||||
2. **양쪽 좌석 학습** — 데이터 2배, 단독으로 승률 0.33 → 0.5
|
||||
3. **선형 총점 보상** — tanh 대비 0.5859 (사용자 제안)
|
||||
4. **매치 관측 결함 수정** — 약 2.5%p
|
||||
5. ~~전지적 critic~~ — **해가 된다. 끈다.**
|
||||
|
||||
### 정정: 전지적 critic의 가치는 **스케일에 따라 뒤집힌다**
|
||||
|
||||
위 A/B는 39.3M learner 액션 규모에서 돌렸다. 실전 규모(131M)에서 동일 설정으로 둘을 직접
|
||||
맞붙이면 **결론이 반대로 나온다.**
|
||||
|
||||
| 규모 | critic OFF의 승률 (critic ON 상대) | 판정 |
|
||||
|---|---|---|
|
||||
| 39.3M (A/B 규모) | **0.5160** [0.505, 0.527] | OFF가 낫다 |
|
||||
| **131M (실전 규모)** | **0.4633** [0.453, 0.474] | **ON이 낫다** |
|
||||
|
||||
**전지적 critic은 그것을 학습시킬 데이터가 충분해질 때 값어치를 한다.** 39.3M에서는 특권
|
||||
정보를 쓰는 큰 가치 트렁크를 제대로 못 맞춰서 어드밴티지에 노이즈만 얹었고, 131M에서는
|
||||
제대로 맞춰서 분산 감소가 실현된다.
|
||||
|
||||
**교훈: 배포할 예산이 아닌 규모에서 ablation을 돌리면 결론이 뒤집힐 수 있다.**
|
||||
`privileged_critic` 기본값은 **ON으로 되돌린다.**
|
||||
|
||||
### 최종 순위 (진짜 3라운드 클래식, duplicate 8,192판)
|
||||
|
||||
| 모델 | learner 액션 | vs league |
|
||||
|---|---|---|
|
||||
| **매치 스택 (critic ON)** | 131M | **0.6094** (+22.0점) |
|
||||
| 매치 스택 (critic OFF) | 131M | 0.5526 (+10.9점) |
|
||||
| league (기존 최강, 웹 배포판) | 122.6M | — |
|
||||
|
||||
### 기여도 재측정: 동일 규모(131M)에서
|
||||
|
||||
39.3M A/B는 critic에게 가장 불리한 지점이었다. 두 조각을 **같은 131M 예산**에서 다시 쟀다:
|
||||
|
||||
| 조각 | full의 승률 (그 조각 뺀 버전 상대) | 순이득 |
|
||||
|---|---|---|
|
||||
| **양쪽 좌석 학습** | 0.6707 | **+17.1 %p** |
|
||||
| **전지적 critic** | 0.5367 | **+3.7 %p** |
|
||||
|
||||
**양쪽 좌석이 약 4.6배 크다.** 그리고 성격이 다르다:
|
||||
|
||||
| 규모 | 양쪽 좌석 | 전지적 critic |
|
||||
|---|---|---|
|
||||
| 39.3M | +16.8 %p | **−1.6 %p** |
|
||||
| 131M | **+17.1 %p** | +3.7 %p |
|
||||
|
||||
**양쪽 좌석은 규모와 무관하게 안정적**(+17%p). **critic은 규모를 탄다.**
|
||||
|
||||
이유: 양쪽 좌석은 **데이터를 2배로** 만든다(65.5M → 131M, 같은 컴퓨트에서). critic은 데이터를
|
||||
안 늘리고 **채점 정확도만** 올리는데, 코치가 배울 게 많아져서 데이터가 부족하면 오히려
|
||||
엉터리 채점을 한다.
|
||||
|
||||
**Fable은 critic을 "가장 큰 누락 아이디어" 1순위로, 양쪽 좌석을 4순위로 꼽았다. 정확히 거꾸로였다.**
|
||||
|
||||
---
|
||||
|
||||
## Exploitability: 우리가 league보다 **덜 착취당한다** (2026-07-15)
|
||||
|
||||
**남겨뒀던 단서를 해소했다.** 평균 실력에서 이기는 것과 "파먹을 약점이 없는 것"은 다른
|
||||
속성이다. 가위바위보에서 바위를 60% 내는 놈은 아무한테나 이길 수도 있지만, 그 습관을
|
||||
알아챈 놈에게는 매번 진다.
|
||||
|
||||
**측정 방법:** 정책을 얼려놓고, **오직 그놈만 이기도록 특화된 새 정책을 처음부터 학습**시킨다
|
||||
(`src/lost_cities_jax/exploit.py`). 착취자가 도달한 승률이 곧 그 정책이 못 막아낸 습관의 크기다.
|
||||
|
||||
**동일 착취자 예산 (250 업데이트 × batch 1024 = 32.5M learner 액션):**
|
||||
|
||||
| 얼려놓은 정책 | 착취자 승률 | 95% CI |
|
||||
|---|---|---|
|
||||
| **우리 매치 스택 (131M)** | **0.2278** | [0.219, 0.237] |
|
||||
| league (웹 배포판, 122.6M) | **0.3213** | [0.311, 0.331] |
|
||||
|
||||
**league가 9.4%p 더 털린다.** 그리고 둘 다 0.5에서 한참 멀다 — 어느 쪽도 만만한 상대는 아니다.
|
||||
|
||||
### 최종 정리: 두 축 모두에서 이긴다
|
||||
|
||||
| 축 | 결과 |
|
||||
|---|---|
|
||||
| **평균 실력** (정면 대결) | 0.6094, +22.0점 ✅ |
|
||||
| **약점 적음** (exploitability) | 0.2278 vs 0.3213 ✅ |
|
||||
|
||||
### 단서
|
||||
|
||||
1. **하한선이다.** 더 세거나 더 오래 학습한 착취자는 더 찾아낼 수 있다. **절대값이 아니라
|
||||
동일 예산에서의 비교로만 의미가 있다.**
|
||||
2. **착취자 예산이 목표의 약 1/4에 불과하다 (32.5M vs 131M).** 약한 공격자다.
|
||||
따라서 **"우리는 22.8%밖에 안 털린다"고 말할 수 없다** — 그건 이 약한 공격자 기준일 뿐이다.
|
||||
살아남는 주장은 **"동일 예산에서 league가 더 털린다"** 하나뿐이다.
|
||||
착취자를 목표와 같은 예산(131M)으로 키워서 **순서가 유지되는지** 확인해야 한다.
|
||||
3. league는 단판 정책이라 3라운드 게임에선 다소 제 물이 아니다 — 다만 carry가 무의미하다는
|
||||
것이 이미 증명됐으므로 큰 불리함은 아니다.
|
||||
|
||||
**흥미로운 점:** 우리는 **순수 셀프플레이**이고 league는 **착취자 구조를 학습에 넣은** 런인데,
|
||||
그런데도 우리가 덜 착취당한다. 착취자 구조가 exploitability를 낮춰줄 것이라는 기대가
|
||||
이 게임에서는 확인되지 않았다.
|
||||
|
||||
### 착취자에 제대로 자금을 대고 재측정 (131M, 목표와 동일 예산)
|
||||
|
||||
앞선 32.5M 측정은 **공격자가 목표보다 4배 부족**했다. 예산을 목표와 맞춰 다시 쟀다.
|
||||
실측 확인: 착취자 learner 액션 **130.4M** (우리 상대) / **131.1M** (league 상대).
|
||||
|
||||
| 얼려놓은 정책 | 착취자 승률 | 95% CI | 착취자 평균 마진 | 이전(32.5M) |
|
||||
|---|---|---|---|---|
|
||||
| **우리 매치 스택** (131.1M 학습) | **0.4657** | [0.455, 0.477] | **−5.5점** | 0.2278 |
|
||||
| league (122.6M 학습) | **0.6295** | [0.619, 0.640] | **+28.8점** | 0.3213 |
|
||||
|
||||
**CI가 전혀 겹치지 않고, 격차가 오히려 벌어졌다 (0.094 → 0.164).**
|
||||
|
||||
**질적으로 선을 넘었다:** 제대로 자금 댄 전담 공격자는 **league를 아예 이긴다**(0.63, +28.8점).
|
||||
반면 **우리는 여전히 못 뚫는다**(0.47, −5.5점).
|
||||
|
||||
#### 우리에게 불리한 단서 (반드시 함께 읽을 것)
|
||||
|
||||
- **둘 다 1000 업데이트 끝까지 정체 없이 오르고 있었다. 이 수치도 여전히 하한선이다.**
|
||||
- **우리 쪽 착취자의 말단 기울기가 더 가파르다** (+0.031 vs +0.016 / 100 업데이트).
|
||||
league 착취자 곡선은 꺾이기 시작했는데 우리 쪽은 아직 갈 길이 남았다.
|
||||
**공격자 예산을 크게 더 키우면 격차가 좁혀질 수 있고, 뒤집힐 가능성도 배제 못 한다.**
|
||||
|
||||
#### 구조적 교란 (순수한 "학습법 A vs B"가 아니다)
|
||||
|
||||
1. **league는 단판 정책이다.** carry도 매치 점수도 모른다. 더 털리는 것의 일부는
|
||||
**학습 방법이 아니라 구조적 맹점** 때문일 수 있다.
|
||||
2. **둘 다 greedy(argmax)로 뒀다.** 결정론적 정책은 **정의상 최대로 착취당한다.**
|
||||
양쪽을 똑같이 대우했으니 비교는 공정하나, **절대값은 부풀려져 있다.**
|
||||
|
||||
#### 살아남는 주장 / 못 하는 주장
|
||||
|
||||
- ✅ **"동일 예산의 전담 공격자 앞에서 league는 뚫리고 우리는 안 뚫린다."**
|
||||
- ❌ "우리 정책은 착취 불가능하다." — 하한선일 뿐이고, 우리 쪽 곡선은 아직 오르는 중이다.
|
||||
@@ -1,139 +0,0 @@
|
||||
# Lost Cities JAX PPO Snapshot-Pool League Spec
|
||||
|
||||
**Status:** ready for implementation after ladder v2.
|
||||
**Last updated:** 2026-07-05.
|
||||
**Warm start:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`.
|
||||
|
||||
## Preconditions
|
||||
|
||||
The static-opponent ladder v2 has passed:
|
||||
|
||||
- `discard_only`: win rate 1.0000, mean score diff +182.0670.
|
||||
- `heuristic_balanced`: win rate 0.9409, mean score diff +119.9250.
|
||||
- `heuristic_expert`: win rate 0.8382, mean score diff +43.3099, score-diff
|
||||
CI95 lower bound +42.7065.
|
||||
|
||||
Canary warnings remain:
|
||||
|
||||
- Gate-3 opened colors are 4.5509/game, not the desired 2-3 range.
|
||||
- The v2 gate-3 checkpoint is exploitable: a 250-update PPO exploiter reached
|
||||
win rate 0.83135 and mean score diff +54.7721 against it.
|
||||
|
||||
## Anchor Semantics
|
||||
|
||||
Anchors are not certificates of strength. They serve two narrower purposes:
|
||||
|
||||
- Fix the Elo scale so curves remain comparable over time.
|
||||
- Provide style diversity so the league does not train only against recent
|
||||
policy snapshots.
|
||||
|
||||
Strength certification is tracked on separate axes:
|
||||
|
||||
- Duplicate mean score difference against `heuristic_expert`.
|
||||
- Exploiter win-rate trend under the fixed exploiter protocol.
|
||||
- Later human play, once the automated diagnostics are stable.
|
||||
|
||||
## Initial Pool
|
||||
|
||||
Permanent anchors are never removed:
|
||||
|
||||
- Static anchors: `discard_only`, `heuristic_balanced`, `heuristic_cautious`,
|
||||
`heuristic_expert`.
|
||||
- Learned ladder anchors:
|
||||
- Gate 1: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/latest`
|
||||
- Gate 2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/latest`
|
||||
- Gate 3: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest`
|
||||
|
||||
The learning policy starts from the v2 gate-3 checkpoint. Shaping coefficient is
|
||||
fixed at 0 by default. If warm-start collapse is observed, the implementation
|
||||
may expose a config switch for random initialization plus shaping anneal, but
|
||||
that is not the default path.
|
||||
|
||||
## Opponent Sampling
|
||||
|
||||
For each episode:
|
||||
|
||||
- Randomize the learner seat.
|
||||
- With probability 50%, play mirror self-play against the current policy copy.
|
||||
- With probability 50%, sample from the pool.
|
||||
|
||||
Pool sampling is PFSP-like:
|
||||
|
||||
- Track recent win rate versus each pool member.
|
||||
- Weight non-anchor snapshots by `(1 - win_rate)^2`.
|
||||
- Mix in a small uniform component to avoid starvation.
|
||||
- Give every permanent anchor only a small floor probability, such as 2-3% per
|
||||
anchor, so stalling or weak anchors do not waste most throughput.
|
||||
- Allocate the remaining probability mass to snapshots according to the PFSP
|
||||
weights.
|
||||
|
||||
## Snapshot Lifecycle
|
||||
|
||||
- Add the current policy to the pool at a configurable interval.
|
||||
- If the pool size cap is reached, remove the oldest non-anchor snapshot.
|
||||
- Never remove permanent anchors.
|
||||
- Store large checkpoints and raw evaluation JSON under
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/`.
|
||||
- Store one-line tracked JSON summaries under `docs/reports/` so results do
|
||||
not disappear with artifact cleanup.
|
||||
|
||||
## Evaluation
|
||||
|
||||
At each snapshot interval, evaluate the current policy with fixed shuffle-bank
|
||||
duplicate play against:
|
||||
|
||||
- The six ladder anchors.
|
||||
- `heuristic_expert`.
|
||||
- A configurable set of recent snapshots.
|
||||
|
||||
Update logistic Elo from the pairwise results, using the heuristic anchors as
|
||||
fixed reference points. Elo is a scale diagnostic, not a strength certificate.
|
||||
|
||||
Run the exploiter protocol periodically against the current best checkpoint:
|
||||
|
||||
- Randomly initialized PPO.
|
||||
- Same 250-update budget used in ladder v2 unless explicitly overridden.
|
||||
- Duplicate evaluation versus the frozen target checkpoint.
|
||||
- Track exploiter win rate and mean score difference over time.
|
||||
|
||||
## Canary Metrics
|
||||
|
||||
Log these metrics at every evaluation:
|
||||
|
||||
- Opened colors per game.
|
||||
- Play action rate.
|
||||
- Mean and quantile game length.
|
||||
- Max-steps rate.
|
||||
- Positive expeditions per game.
|
||||
- Duplicate score-diff distribution against `heuristic_expert`.
|
||||
|
||||
Canaries are diagnostics only. Do not put them directly into the reward.
|
||||
|
||||
## Regression Guards
|
||||
|
||||
- `heuristic_expert` replaces `heuristic_cautious` as the main regression
|
||||
opponent.
|
||||
- If win rate versus `heuristic_expert` falls below 90% of the v2 gate-3 value,
|
||||
flag the snapshot.
|
||||
- If max-steps rate rises above the gate-3 baseline of 0.0 by a material
|
||||
amount, flag the snapshot.
|
||||
- If exploiter win rate rises above the v2 baseline of 0.83135, flag the
|
||||
snapshot.
|
||||
|
||||
## Stop Conditions
|
||||
|
||||
Stop the league run when either:
|
||||
|
||||
- Elo is statistically flat over a configured recent-snapshot window and
|
||||
exploiter win rate is at or below the configured threshold, for example 0.55.
|
||||
- The wall-clock budget is exhausted.
|
||||
|
||||
## Decisions
|
||||
|
||||
- The earlier `heuristic_cautious` remains a permanent anchor for scale and
|
||||
style diversity only. It is a stalling anchor and no longer certifies
|
||||
strength.
|
||||
- `discard_only` is also kept only as a low-probability anchor and regression
|
||||
sanity check.
|
||||
- The league spec file was not present in the repository when this update was
|
||||
requested, so this tracked plan file is the authoritative spec location.
|
||||
@@ -1,12 +1,5 @@
|
||||
# Plan: Model-Size Experiment (Keystone for Model-Scale Optimizations)
|
||||
|
||||
> **⚠️ 스택 주의 — 이 문서는 Deep CFR / PyTorch 스택 전용이다.**
|
||||
> `input_dim=365`, `legacy/deep-cfr/configs/`, `DeepCFRMLP` 기준으로 쓰였다.
|
||||
> 현행 JAX PPO 스택(`OBS_DIM=454`, `configs/jax_ppo/`, `ActorCritic`)에는
|
||||
> **적용되지 않는다.** PPO 쪽 모델 크기 문제는
|
||||
> [jax-ppo-model-size-ab.md](../jax-ppo-model-size-ab.md)와
|
||||
> [../../reports/jax-ppo-model-capacity-2026-07-12.md](../../reports/jax-ppo-model-capacity-2026-07-12.md)를 볼 것.
|
||||
|
||||
**Status:** Ready to execute
|
||||
**Owner:** Operator (runs grid on `home`); Codex (adds configs and runner script)
|
||||
**Background:** See `docs/performance.md` → "Post-A Optimization Calculus",
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 66 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 89 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 57 KiB |
@@ -1,17 +0,0 @@
|
||||
{"current": "league_v1_update_250", "current_label": "league v1 update 250", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_league_v1_update_250_vs_ladder_v2_gate3.json", "max_steps_rate": 0.0, "mean_game_length": 57.4935, "mean_score_diff": 42.87575, "opened_colors_per_game": 4.911, "play_action_rate": 0.6949448449578737, "positive_expeditions_per_game": 2.56825, "previous": "ladder_v2_gate3", "previous_label": "ladder v2 gate3", "score_diff_ci95_high": 44.43765342421665, "score_diff_ci95_low": 41.313846575783344, "score_diff_std": 50.40064352917072, "wilson_high": 0.8135690832007875, "wilson_low": 0.7888523724955226, "win_rate": 0.8015}
|
||||
{"current": "league_v1_update_500", "current_label": "league v1 update 500", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_league_v1_update_500_vs_league_v1_update_250.json", "max_steps_rate": 0.0, "mean_game_length": 50.7755, "mean_score_diff": 5.20975, "opened_colors_per_game": 4.9415, "play_action_rate": 0.7485131356099098, "positive_expeditions_per_game": 2.351, "previous": "league_v1_update_250", "previous_label": "league v1 update 250", "score_diff_ci95_high": 6.664905283640598, "score_diff_ci95_low": 3.7545947163594007, "score_diff_std": 46.95601635366284, "wilson_high": 0.5558977420904201, "wilson_low": 0.525024543003114, "win_rate": 0.5405}
|
||||
{"current": "repair_c01_update_500", "current_label": "repair c01 update 500", "event": "h2h_adjacent", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/h2h_repair_c01_update_500_vs_league_v1_update_500.json", "max_steps_rate": 0.0, "mean_game_length": 49.2915, "mean_score_diff": 3.64125, "opened_colors_per_game": 4.9395, "play_action_rate": 0.7722021038110018, "positive_expeditions_per_game": 2.43225, "previous": "league_v1_update_500", "previous_label": "league v1 update 500", "score_diff_ci95_high": 5.0464015232057395, "score_diff_ci95_low": 2.23609847679426, "score_diff_std": 45.34245839244679, "wilson_high": 0.5476742803350019, "wilson_low": 0.5167638355728124, "win_rate": 0.53225}
|
||||
{"checkpoint": "ladder_v2_gate3", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_ladder_v2_gate3.json", "label": "ladder v2 gate3", "max_steps_rate": 0.0, "mean_game_length": 61.398, "mean_score_diff": 43.456, "opened_colors_per_game": 4.55925, "play_action_rate": 0.6680377805945128, "positive_expeditions_per_game": 2.61475, "score_diff_ci95_high": 44.81720694691559, "score_diff_ci95_low": 42.094793053084416, "score_diff_std": 43.92442262256479, "stage_index": 0, "wilson_high": 0.8490930348441887, "wilson_low": 0.8262583814919731, "win_rate": 0.838}
|
||||
{"checkpoint": "league_v1_update_250", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_league_v1_update_250.json", "label": "league v1 update 250", "max_steps_rate": 0.0, "mean_game_length": 56.52425, "mean_score_diff": 26.42325, "opened_colors_per_game": 4.54775, "play_action_rate": 0.6422064732439839, "positive_expeditions_per_game": 2.19725, "score_diff_ci95_high": 27.654455989944754, "score_diff_ci95_low": 25.192044010055245, "score_diff_std": 39.729456538778834, "stage_index": 1, "wilson_high": 0.7570405983993863, "wilson_low": 0.7299916729968844, "win_rate": 0.74375}
|
||||
{"checkpoint": "league_v1_update_500", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_league_v1_update_500.json", "label": "league v1 update 500", "max_steps_rate": 0.0, "mean_game_length": 53.29175, "mean_score_diff": 13.967, "opened_colors_per_game": 4.84725, "play_action_rate": 0.6934619506966774, "positive_expeditions_per_game": 2.14075, "score_diff_ci95_high": 15.054322005906956, "score_diff_ci95_low": 12.879677994093045, "score_diff_std": 35.086502770569304, "stage_index": 2, "wilson_high": 0.6609204517986065, "wilson_low": 0.6312989110391558, "win_rate": 0.64625}
|
||||
{"checkpoint": "repair_c01_update_500", "event": "expert_anchor", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05/expert_anchor_repair_c01_update_500.json", "label": "repair c01 update 500", "max_steps_rate": 0.0, "mean_game_length": 54.61975, "mean_score_diff": 19.654, "opened_colors_per_game": 4.7015, "play_action_rate": 0.6708126417051431, "positive_expeditions_per_game": 2.1505, "score_diff_ci95_high": 20.81558482366198, "score_diff_ci95_low": 18.49241517633802, "score_diff_std": 37.482869759149246, "stage_index": 3, "wilson_high": 0.7068566468456556, "wilson_low": 0.6782734877661645, "win_rate": 0.69275}
|
||||
{"budget": "same-budget phase-1 exploiter", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 54.7721, "phase": "ladder_v2", "protocol": "short_random_shaping", "source": "docs/reports/ladder-v2-2026-07-05-summary.jsonl", "target": "ladder_v2_gate3", "warmstart": "random", "win_rate": 0.83135}
|
||||
{"budget": "league v1 cycle exploiter", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 0.6818, "phase": "league_v1", "protocol": "league_cycle_exploiter", "source": "docs/reports/league-v1-2026-07-05-summary.jsonl", "target": "league_v1_update_250", "warmstart": "random", "win_rate": 0.50225}
|
||||
{"budget": "same exploiter vs final", "delta_from_previous_same_protocol": -0.031299999999999994, "event": "exploiter_trajectory", "mean_score_diff": -2.63775, "phase": "league_v1", "protocol": "league_cycle_exploiter", "source": "docs/reports/league-v1-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "random", "win_rate": 0.47095}
|
||||
{"budget": "1200 updates, shaping anneal", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 10.95125, "phase": "gates_1_2_original", "protocol": "long_random_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "random", "win_rate": 0.587}
|
||||
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 11.38075, "phase": "gates_1_2_original", "protocol": "warmstart_gate3_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "ladder_v2_gate3", "win_rate": 0.59}
|
||||
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": null, "event": "exploiter_trajectory", "mean_score_diff": 10.42525, "phase": "gates_1_2_original", "protocol": "replay_exploiter_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "league_v1_update_500", "warmstart": "league_v1_cycle_1_exploiter", "win_rate": 0.582}
|
||||
{"budget": "1200 updates, shaping anneal", "delta_from_previous_same_protocol": -0.04799999999999993, "event": "exploiter_trajectory", "mean_score_diff": 6.27925, "phase": "repair_c01", "protocol": "long_random_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "random", "win_rate": 0.539}
|
||||
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": -0.040999999999999925, "event": "exploiter_trajectory", "mean_score_diff": 7.0285, "phase": "repair_c01", "protocol": "warmstart_gate3_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "ladder_v2_gate3", "win_rate": 0.549}
|
||||
{"budget": "900 updates, shaping 0", "delta_from_previous_same_protocol": -0.042749999999999955, "event": "exploiter_trajectory", "mean_score_diff": 6.10225, "phase": "repair_c01", "protocol": "replay_exploiter_no_shaping", "source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl", "target": "repair_c01_update_500", "warmstart": "league_v1_cycle_1_exploiter", "win_rate": 0.53925}
|
||||
{"artifact_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05", "elapsed_seconds": 15.667906627990305, "event": "diminishing_returns_complete", "report": "docs/reports/diminishing-returns-2026-07-05.md"}
|
||||
@@ -1,65 +0,0 @@
|
||||
# Diminishing Returns Diagnostic - 2026-07-05
|
||||
|
||||
신규 학습 없이 기존 체크포인트와 평가 롤아웃만 사용했다. 모든 duplicate 평가는 셔플 뱅크 seed 20260704, 2,000쌍(4000 games) 기준이다.
|
||||
|
||||
Raw artifacts: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05`
|
||||
|
||||
## Checkpoints
|
||||
|
||||
| Order | Name | Config | Checkpoint |
|
||||
| ---: | --- | --- | --- |
|
||||
| 0 | `ladder_v2_gate3` | `configs/jax_ppo/ladder-v2-expert.yaml` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest` |
|
||||
| 1 | `league_v1_update_250` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250` |
|
||||
| 2 | `league_v1_update_500` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500` |
|
||||
| 3 | `repair_c01_update_500` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500` |
|
||||
|
||||
## 1. Adjacent Head-to-Head
|
||||
|
||||
|
||||
|
||||
| Later | Earlier | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| `league_v1_update_250` | `ladder_v2_gate3` | 0.8015 | [0.7889, 0.8136] | +42.8757 | [+41.3138, +44.4377] | 4.9110 | 0.0000 |
|
||||
| `league_v1_update_500` | `league_v1_update_250` | 0.5405 | [0.5250, 0.5559] | +5.2097 | [+3.7546, +6.6649] | 4.9415 | 0.0000 |
|
||||
| `repair_c01_update_500` | `league_v1_update_500` | 0.5323 | [0.5168, 0.5477] | +3.6412 | [+2.2361, +5.0464] | 4.9395 | 0.0000 |
|
||||
|
||||
## 2. External Anchor Trajectory
|
||||
|
||||
|
||||
|
||||
| Checkpoint | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| `ladder_v2_gate3` | 0.8380 | [0.8263, 0.8491] | +43.4560 | [+42.0948, +44.8172] | 4.5592 | 0.0000 |
|
||||
| `league_v1_update_250` | 0.7438 | [0.7300, 0.7570] | +26.4232 | [+25.1920, +27.6545] | 4.5477 | 0.0000 |
|
||||
| `league_v1_update_500` | 0.6462 | [0.6313, 0.6609] | +13.9670 | [+12.8797, +15.0543] | 4.8472 | 0.0000 |
|
||||
| `repair_c01_update_500` | 0.6927 | [0.6783, 0.7069] | +19.6540 | [+18.4924, +20.8156] | 4.7015 | 0.0000 |
|
||||
|
||||
Recent expert-anchor slope: `+5.6870` points (league_v1_update_500 -> repair_c01_update_500).
|
||||
|
||||
## 3. Exploitability Trajectory
|
||||
|
||||
|
||||
|
||||
프로토콜이 다른 피탈률은 한 곡선에 섞지 않았다. `Delta`는 같은 프로토콜 안에서만 계산했다.
|
||||
|
||||
| Phase | Target | Protocol | Budget | Warm start | Win rate | Delta same protocol | Mean diff | Source |
|
||||
| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
|
||||
| `ladder_v2` | `ladder_v2_gate3` | `short_random_shaping` | same-budget phase-1 exploiter | `random` | 0.83135 | n/a | +54.7721 | [ladder-v2-2026-07-05-summary.jsonl](ladder-v2-2026-07-05-summary.jsonl) |
|
||||
| `league_v1` | `league_v1_update_250` | `league_cycle_exploiter` | league v1 cycle exploiter | `random` | 0.50225 | n/a | +0.6818 | [league-v1-2026-07-05-summary.jsonl](league-v1-2026-07-05-summary.jsonl) |
|
||||
| `league_v1` | `league_v1_update_500` | `league_cycle_exploiter` | same exploiter vs final | `random` | 0.47095 | -0.03130 | -2.6378 | [league-v1-2026-07-05-summary.jsonl](league-v1-2026-07-05-summary.jsonl) |
|
||||
| `gates_1_2_original` | `league_v1_update_500` | `long_random_shaping` | 1200 updates, shaping anneal | `random` | 0.58700 | n/a | +10.9512 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
| `gates_1_2_original` | `league_v1_update_500` | `warmstart_gate3_no_shaping` | 900 updates, shaping 0 | `ladder_v2_gate3` | 0.59000 | n/a | +11.3808 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
| `gates_1_2_original` | `league_v1_update_500` | `replay_exploiter_no_shaping` | 900 updates, shaping 0 | `league_v1_cycle_1_exploiter` | 0.58200 | n/a | +10.4253 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
| `repair_c01` | `repair_c01_update_500` | `long_random_shaping` | 1200 updates, shaping anneal | `random` | 0.53900 | -0.04800 | +6.2793 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
| `repair_c01` | `repair_c01_update_500` | `warmstart_gate3_no_shaping` | 900 updates, shaping 0 | `ladder_v2_gate3` | 0.54900 | -0.04100 | +7.0285 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
| `repair_c01` | `repair_c01_update_500` | `replay_exploiter_no_shaping` | 900 updates, shaping 0 | `league_v1_cycle_1_exploiter` | 0.53925 | -0.04275 | +6.1022 | [gates-1-2-2026-07-05-summary.jsonl](gates-1-2-2026-07-05-summary.jsonl) |
|
||||
|
||||
## Judgment
|
||||
|
||||
추가 학습 1사이클(~1.5시간 GPU)의 기대 개선은 인접 H2H 최근 이득 +3.64점, expert 앵커 최근 변화 +5.69점, 동일 프로토콜 exploiter 평균 피탈률 감소 0.044에 따라 대략 +3.6~+5.7점 또는 피탈률 -0.044 내외 수준으로 추정되며, 수확체감 구간에 진입했다. 근거는 최근 H2H 이득이 첫 리그 전이보다 작고, expert 앵커 성능이 v1 update 250 이후 하락/회복을 반복하며, 보수 후 exploiter 최악값이 0.549로 게이트 바로 아래에 머문다는 세 측정의 일치다. 단, repair가 expert 앵커를 유의하게 회복시킨 점은 별도 긍정 신호다.
|
||||
|
||||
## Notes
|
||||
|
||||
- H2H와 expert 앵커 평가는 이번 작업에서 새로 실행했다.
|
||||
- Exploiter 표는 지금까지 생성된 리포트/JSONL의 기존 측정값만 재정리했다.
|
||||
- 이 작업에서는 신규 학습, 봇 수정, 체크포인트 수정이 없었다.
|
||||
@@ -1,8 +0,0 @@
|
||||
{"cycle": 1, "event": "final_cycle_league", "league_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/league_config.yaml", "league_run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "worst_protocol_used": "warmstart_gate3"}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 919.0, "max_steps_rate": 0.0005, "mean_game_length": 65.69525, "mean_score_diff": 55.799, "notes": "random init + shaping anneal, 1200 updates", "opened_colors_per_game": 4.92425, "play_action_rate": 0.5457588946038896, "positive_expeditions_per_game": 2.7735, "resume": null, "score_diff_ci95_high": 57.963152016622004, "score_diff_ci95_low": 53.634847983377995, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 22.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_205759_final-c01-long-random-exploiter", "wilson_high": 0.7776365926294233, "wilson_low": 0.7513553821486798, "win_rate": 0.76475, "wins": 3059.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 961.0, "max_steps_rate": 0.00075, "mean_game_length": 65.13875, "mean_score_diff": 53.30925, "notes": "ladder v2 gate-3 warm start, shaping disabled, 900 updates", "opened_colors_per_game": 4.90075, "play_action_rate": 0.5480372138722999, "positive_expeditions_per_game": 2.71225, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 55.52225515395847, "score_diff_ci95_low": 51.09624484604153, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 18.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_221402_final-c01-warmstart-gate3-exploiter", "wilson_high": 0.7683246594694194, "wilson_low": 0.741685544731188, "win_rate": 0.75525, "wins": 3021.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter/latest", "cycle": 1, "event": "final_cycle_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter/eval_vs_final_cycle_01_duplicate.json", "losses": 866.0, "max_steps_rate": 0.00025, "mean_game_length": 65.467, "mean_score_diff": 59.351, "notes": "league v1 cycle-1 exploiter warm start, shaping disabled, 900 updates", "opened_colors_per_game": 4.989, "play_action_rate": 0.5882132820136853, "positive_expeditions_per_game": 2.7905, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 61.580508518304526, "score_diff_ci95_low": 57.12149148169547, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 17.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/battery/2026-07-05_231108_final-c01-replay-exploiter-exploiter", "wilson_high": 0.7918317711217939, "wilson_low": 0.7661323798009079, "win_rate": 0.77925, "wins": 3117.0}
|
||||
{"cycle": 1, "event": "final_cycle_battery_judgment", "passed": false, "threshold": 0.52, "worst_exploiter": "replay_exploiter", "worst_win_rate": 0.77925}
|
||||
{"cycle": 1, "event": "final_cycle_guard_expert", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/guard_vs_heuristic_expert.json", "losses": 852.0, "max_steps_rate": 0.0, "mean_game_length": 60.38125, "mean_score_diff": 36.08475, "opened_colors_per_game": 4.15225, "passed": true, "play_action_rate": 0.5845773004045489, "positive_expeditions_per_game": 2.05975, "score_diff_ci95_high": 37.5625235448534, "score_diff_ci95_low": 34.6069764551466, "target": "final_cycle_01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "ties": 33.0, "wilson_high": 0.7913426526326875, "wilson_low": 0.7656224577333041, "win_rate": 0.77875, "wins": 3115.0}
|
||||
{"current": "final_cycle_01", "current_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "cycle": 1, "event": "final_cycle_h2h", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/cycle_01/h2h_final_cycle_01_vs_repair_c01_update_500.json", "losses": 1987.0, "max_steps_rate": 0.0, "mean_game_length": 49.54275, "mean_score_diff": 0.35425, "opened_colors_per_game": 4.91275, "play_action_rate": 0.7646239835347768, "positive_expeditions_per_game": 2.422, "previous": "repair_c01_update_500", "previous_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "score_diff_ci95_high": 1.7891388254913125, "score_diff_ci95_low": -1.0806388254913126, "ties": 51.0, "wilson_high": 0.505993762454951, "wilson_low": 0.47502446696755335, "win_rate": 0.4905, "wins": 1962.0}
|
||||
{"elapsed_seconds": 17320.886204754002, "event": "final_cycles_complete", "final_candidate": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate", "final_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json", "source_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/snapshots/cycle_01_update_000500", "stop_reason": "h2h_stagnation"}
|
||||
@@ -1,83 +0,0 @@
|
||||
# Final Cycles and Human Play - 2026-07-05
|
||||
|
||||
## Part A - Closing Reinforcement Cycles
|
||||
|
||||
Raw artifacts: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05`
|
||||
Stop reason: `h2h_stagnation`
|
||||
Final candidate: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate`
|
||||
Final candidate config: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/main_ppo_config.json`
|
||||
|
||||
The initial 4 hour GPU budget was treated as an estimate after user confirmation,
|
||||
not a hard cutoff. The run completed cycle 1 fully, then stopped by the written
|
||||
H2H stagnation rule. Cycle 2 was not started.
|
||||
|
||||
Bottom line: the closing reinforcement cycle did not produce a materially better
|
||||
policy. The cycle-1 snapshot passed the expert guard, but the strengthened
|
||||
exploiter battery got substantially worse than the incoming baseline target:
|
||||
worst exploiter win rate was 0.7792, far above the 0.5200 success threshold. The
|
||||
adjacent H2H against `repair_c01_update_500` was statistically indistinguishable
|
||||
from zero, so this recipe is stopped here. Per instruction, this report does not
|
||||
recommend more training with the same observation/network/PPO recipe.
|
||||
|
||||
### Expert Guard
|
||||
|
||||
| Cycle | Passed | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | True | 0.7788 | +36.0847 | +34.6070 | 4.1523 | 0.0000 |
|
||||
|
||||
### Adjacent H2H
|
||||
|
||||
| Cycle | Current | Previous | Win rate | Mean diff | Score CI |
|
||||
| ---: | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `final_cycle_01` | `repair_c01_update_500` | 0.4905 | +0.3543 | [-1.0806, +1.7891] |
|
||||
|
||||
### Strengthened Exploiter Battery
|
||||
|
||||
| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| ---: | --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | `long_random` | 0.7648 | +55.7990 | +53.6348 | 4.9242 | 0.0005 |
|
||||
| 1 | `warmstart_gate3` | 0.7552 | +53.3092 | +51.0962 | 4.9008 | 0.0008 |
|
||||
| 1 | `replay_exploiter` | 0.7792 | +59.3510 | +57.1215 | 4.9890 | 0.0003 |
|
||||
|
||||
### Battery Judgment
|
||||
|
||||
| Cycle | Worst exploiter | Worst win rate | Threshold | Passed |
|
||||
| ---: | --- | ---: | ---: | ---: |
|
||||
| 1 | `replay_exploiter` | 0.7792 | 0.5200 | False |
|
||||
|
||||
## Part B - Human Play Interface
|
||||
|
||||
Start a single game:
|
||||
|
||||
```bash
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \
|
||||
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \
|
||||
--seat 0
|
||||
```
|
||||
|
||||
Start a duplicate set with one shared shuffle and swapped seats:
|
||||
|
||||
```bash
|
||||
uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \
|
||||
--checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \
|
||||
--seat 0 --duplicate
|
||||
```
|
||||
|
||||
Summarize logged human games:
|
||||
|
||||
```bash
|
||||
uv run lost-cities-jax-ppo human-play summarize \
|
||||
--log-dir /mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/
|
||||
```
|
||||
|
||||
Move syntax: `play R7 draw deck`, `discard G3 draw Y`, or `play RHS draw deck`.
|
||||
The renderer shows only the human hand, both boards, all public discard piles, deck count, and current board score differential. Opponent hand and deck order are not rendered.
|
||||
Every game is appended to `/mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/games.jsonl` with deck seed/index, full action list, AI top-3 policy actions/probabilities, value outputs, scoring breakdown, and optional human comment.
|
||||
|
||||
Validation performed before final report:
|
||||
|
||||
- `uv run ruff check .` passed.
|
||||
- `uv run pytest -q` passed: 257 passed, 1 skipped.
|
||||
- `uv run lost-cities-jax-ppo play --help` passed.
|
||||
- `uv run lost-cities-jax-ppo human-play summarize --log-dir /tmp/nonexistent-human-play-log` passed.
|
||||
- `scripts/librarian.sh` found no link or code-citation errors; it still exits non-zero on the known pre-existing `docs/plans/archive/deep-cfr-selectivity.md` 500-line soft cap.
|
||||
@@ -1,19 +0,0 @@
|
||||
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_expert_cap3.json", "learner": "expert_cap2", "losses": 2102.0, "max_steps_rate": 0.0, "mean_game_length": 46.549, "mean_score_diff": -1.89325, "opened_colors_per_game": 1.77925, "opponent": "expert_cap3", "play_action_rate": 0.2414775169804832, "positive_expeditions_per_game": 0.94175, "score_diff_ci95_high": -1.4203123965597364, "score_diff_ci95_low": -2.366187603440264, "score_diff_std": 15.261096936571331, "ties": 132.0, "wilson_high": 0.45693730157215046, "wilson_low": 0.42617495329274474, "win_rate": 0.4415, "wins": 1766.0}
|
||||
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_expert_capN.json", "learner": "expert_cap2", "losses": 2092.0, "max_steps_rate": 0.0, "mean_game_length": 46.354, "mean_score_diff": -1.88075, "opened_colors_per_game": 1.7815, "opponent": "expert_capN", "play_action_rate": 0.24223387949830538, "positive_expeditions_per_game": 0.93725, "score_diff_ci95_high": -1.4070318876291175, "score_diff_ci95_low": -2.3544681123708826, "score_diff_std": 15.286282970338549, "ties": 130.0, "wilson_high": 0.45994507591745637, "wilson_low": 0.4291614222877006, "win_rate": 0.4445, "wins": 1778.0}
|
||||
{"event": "gate1_tournament", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap3_vs_expert_capN.json", "learner": "expert_cap3", "losses": 1927.0, "max_steps_rate": 0.0, "mean_game_length": 45.35975, "mean_score_diff": 0.0285, "opened_colors_per_game": 2.21525, "opponent": "expert_capN", "play_action_rate": 0.3101634934459303, "positive_expeditions_per_game": 1.20625, "score_diff_ci95_high": 0.5258735279417411, "score_diff_ci95_low": -0.46887352794174114, "score_diff_std": 16.04961324366783, "ties": 137.0, "wilson_high": 0.49949486790031544, "wilson_low": 0.4685358342849551, "win_rate": 0.484, "wins": 1936.0}
|
||||
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap2_vs_league_v1_update_500.json", "learner": "expert_cap2", "losses": 2871.0, "max_steps_rate": 0.00025, "mean_game_length": 55.372, "mean_score_diff": -20.88625, "opened_colors_per_game": 1.9735, "opponent": "league_v1_update_500", "play_action_rate": 0.22526015834941476, "positive_expeditions_per_game": 1.09725, "score_diff_ci95_high": -19.787867065349186, "score_diff_ci95_low": -21.984632934650815, "score_diff_std": 35.44342491958145, "ties": 39.0, "wilson_high": 0.2865114339331819, "wilson_low": 0.2589251127636324, "win_rate": 0.2725, "wins": 1090.0}
|
||||
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_cap3_vs_league_v1_update_500.json", "learner": "expert_cap3", "losses": 2590.0, "max_steps_rate": 0.0, "mean_game_length": 53.52525, "mean_score_diff": -14.35125, "opened_colors_per_game": 2.7385, "opponent": "league_v1_update_500", "play_action_rate": 0.320897958422324, "positive_expeditions_per_game": 1.52675, "score_diff_ci95_high": -13.256641796659636, "score_diff_ci95_low": -15.445858203340364, "score_diff_std": 35.32161912528796, "ties": 50.0, "wilson_high": 0.3548273870268761, "wilson_low": 0.32547963482582853, "win_rate": 0.34, "wins": 1360.0}
|
||||
{"event": "gate1_vs_league", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/expert_capN_vs_league_v1_update_500.json", "learner": "expert_capN", "losses": 2585.0, "max_steps_rate": 0.0, "mean_game_length": 53.29175, "mean_score_diff": -13.967, "opened_colors_per_game": 3.004, "opponent": "league_v1_update_500", "play_action_rate": 0.3410701746172932, "positive_expeditions_per_game": 1.56475, "score_diff_ci95_high": -12.879677994093045, "score_diff_ci95_low": -15.054322005906956, "score_diff_std": 35.086502770569304, "ties": 49.0, "wilson_high": 0.3563415432992672, "wilson_low": 0.3269626002235683, "win_rate": 0.3415, "wins": 1366.0}
|
||||
{"ci95_high": 0.3559042062015777, "ci95_low": -0.32834170620157765, "deltas_npy": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate1/delta_open_deltas.npy", "event": "gate1_delta_open", "histogram": {"counts": [107, 1643, 5209, 7192, 10627, 5447, 1662, 113], "edges": [-200.0, -100.0, -50.0, -20.0, 0.0, 20.0, 50.0, 100.0, 200.0]}, "histogram_png": "docs/reports/gates-1-2-delta-open-hist.png", "judgment": "near_zero", "mean_delta": 0.01378125, "pairs": 64, "quantiles": {"max": 149.0, "min": -167.0, "p05": -52.0, "p25": -17.0, "p50": 0.0, "p75": 17.0, "p95": 51.0}, "samples": 32000, "states": 500, "std_delta": 31.225479356323433}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter/latest", "event": "gate2_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1610.0, "max_steps_rate": 0.0, "mean_game_length": 50.74375, "mean_score_diff": 10.95125, "notes": "random init + shaping anneal, extended budget", "opened_colors_per_game": 4.99575, "play_action_rate": 0.8010700772505124, "positive_expeditions_per_game": 2.53625, "resume": null, "score_diff_ci95_high": 12.423541316332845, "score_diff_ci95_low": 9.478958683667155, "score_diff_std": 47.50897440589277, "ties": 42.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter", "wilson_high": 0.6021679468487495, "wilson_low": 0.5716651100188423, "win_rate": 0.587, "wins": 2348.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter/latest", "event": "gate2_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1605.0, "max_steps_rate": 0.0, "mean_game_length": 50.8715, "mean_score_diff": 11.38075, "notes": "ladder v2 gate-3 warm start, shaping disabled", "opened_colors_per_game": 4.9945, "play_action_rate": 0.798003736847281, "positive_expeditions_per_game": 2.5195, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 12.872234578510747, "score_diff_ci95_low": 9.889265421489254, "score_diff_std": 48.128316645747006, "ties": 35.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter", "wilson_high": 0.6051483732088389, "wilson_low": 0.5746789269990148, "win_rate": 0.59, "wins": 2360.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter/latest", "event": "gate2_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter/eval_vs_league_v1_update_500_duplicate.json", "losses": 1638.0, "max_steps_rate": 0.0, "mean_game_length": 51.4305, "mean_score_diff": 10.42525, "notes": "league v1 exploiter warm start, shaping disabled", "opened_colors_per_game": 4.99775, "play_action_rate": 0.7904596181514173, "positive_expeditions_per_game": 2.54225, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 11.913719629678091, "score_diff_ci95_low": 8.936780370321909, "score_diff_std": 48.0310280688623, "ties": 34.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter", "wilson_high": 0.5971992743148633, "wilson_low": 0.5666433769856256, "win_rate": 0.582, "wins": 2328.0}
|
||||
{"event": "gate2_judgment", "passed": false, "threshold": 0.55, "worst_exploiter": "warmstart_gate3", "worst_win_rate": 0.59}
|
||||
{"elapsed_seconds": 11425.08623591601, "event": "gates_complete", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2"}
|
||||
{"cycle": 1, "event": "gate2c_league_cycle", "league_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/league_config.yaml", "league_run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "worst_protocol": "warmstart_gate3"}
|
||||
{"cycle": 1, "event": "gate2c_guard_expert", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/guard_vs_heuristic_expert.json", "losses": 1188.0, "max_steps_rate": 0.0, "mean_game_length": 54.61975, "mean_score_diff": 19.654, "opened_colors_per_game": 4.7015, "passed": true, "play_action_rate": 0.6708126417051431, "positive_expeditions_per_game": 2.1505, "score_diff_ci95_high": 20.81558482366198, "score_diff_ci95_low": 18.49241517633802, "score_diff_std": 37.482869759149246, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "ties": 41.0, "wilson_high": 0.7068566468456556, "wilson_low": 0.6782734877661645, "win_rate": 0.69275, "wins": 2771.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "long_random", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1810.0, "max_steps_rate": 0.0, "mean_game_length": 51.4185, "mean_score_diff": 6.27925, "notes": "random init + shaping anneal, extended budget", "opened_colors_per_game": 4.99425, "play_action_rate": 0.7816023796562719, "positive_expeditions_per_game": 2.52, "resume": null, "score_diff_ci95_high": 7.76660187726116, "score_diff_ci95_low": 4.79189812273884, "score_diff_std": 47.99495961530356, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 34.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_143807_gates-1-2-long-random-exploiter", "wilson_high": 0.5544028831760525, "wilson_low": 0.5235222802473508, "win_rate": 0.539, "wins": 2156.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "warmstart_gate3", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1763.0, "max_steps_rate": 0.0, "mean_game_length": 50.434, "mean_score_diff": 7.0285, "notes": "ladder v2 gate-3 warm start, shaping disabled", "opened_colors_per_game": 4.9935, "play_action_rate": 0.8006960073764884, "positive_expeditions_per_game": 2.4845, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest", "score_diff_ci95_high": 8.490821414519859, "score_diff_ci95_low": 5.5661785854801415, "score_diff_std": 47.18725831288327, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 41.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_155430_gates-1-2-warmstart-gate3-exploiter", "wilson_high": 0.5643659496573598, "wilson_low": 0.5335400249002497, "win_rate": 0.549, "wins": 2196.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter/latest", "cycle": 1, "event": "gate2c_exploiter", "exploiter": "replay_exploiter", "games": 4000, "json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter/eval_vs_repair_cycle_01_duplicate.json", "losses": 1798.0, "max_steps_rate": 0.0, "mean_game_length": 51.116, "mean_score_diff": 6.10225, "notes": "league v1 exploiter warm start, shaping disabled", "opened_colors_per_game": 4.99425, "play_action_rate": 0.7845902569268536, "positive_expeditions_per_game": 2.50025, "resume": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "score_diff_ci95_high": 7.583110013023179, "score_diff_ci95_low": 4.62138998697682, "score_diff_std": 47.785475385853, "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500", "target_config": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json", "target_name": "repair_cycle_01", "ties": 45.0, "train_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/cycle_01/battery/2026-07-05_165151_gates-1-2-replay-exploiter", "wilson_high": 0.5546520360903574, "wilson_low": 0.523772647611401, "win_rate": 0.53925, "wins": 2157.0}
|
||||
{"cycle": 1, "event": "gate2c_judgment", "passed": true, "threshold": 0.55, "worst_exploiter": "warmstart_gate3", "worst_win_rate": 0.549}
|
||||
{"elapsed_seconds_total": 28945.184503759025, "event": "gates_repair_complete", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2"}
|
||||
@@ -1,67 +0,0 @@
|
||||
# Gates 1-2 Report - 2026-07-05
|
||||
|
||||
**Run dir:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2`.
|
||||
**Target:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500`.
|
||||
|
||||
## Gate 1A - Policy Class Tournament
|
||||
|
||||
| Learner | Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| `expert_cap2` | `expert_cap3` | 0.4415 | -1.8933 | -2.3662 | 1.7792 | 0.0000 |
|
||||
| `expert_cap2` | `expert_capN` | 0.4445 | -1.8807 | -2.3545 | 1.7815 | 0.0000 |
|
||||
| `expert_cap3` | `expert_capN` | 0.4840 | +0.0285 | -0.4689 | 2.2153 | 0.0000 |
|
||||
|
||||
## Gate 1A - Variants vs League v1
|
||||
|
||||
| Learner | Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| `expert_cap2` | `league_v1_update_500` | 0.2725 | -20.8863 | -21.9846 | 1.9735 | 0.0003 |
|
||||
| `expert_cap3` | `league_v1_update_500` | 0.3400 | -14.3513 | -15.4459 | 2.7385 | 0.0000 |
|
||||
| `expert_capN` | `league_v1_update_500` | 0.3415 | -13.9670 | -15.0543 | 3.0040 | 0.0000 |
|
||||
|
||||
## Gate 1B - Delta Open Audit
|
||||
|
||||
- States: 500
|
||||
- Paired samples: 32000
|
||||
- Mean delta: +0.0138
|
||||
- CI95: [-0.3283, +0.3559]
|
||||
- Judgment: `near_zero`
|
||||
- Histogram: `docs/reports/gates-1-2-delta-open-hist.png`
|
||||
|
||||
## Gate 1C - Selectivity Judgment
|
||||
|
||||
통념 기각/미결: capN은 집중 변형보다 유의하게 나쁘지 않고, 리그 정책의 4번째+ 오픈 delta는 0 근처다. selectivity는 현재 주요 성능 병목으로 보이지 않는다.
|
||||
|
||||
## Gate 2A - Strengthened Exploiter Battery
|
||||
|
||||
| Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step | Run |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| `long_random` | 0.5870 | +10.9512 | +9.4790 | 4.9958 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_094311_gates-1-2-long-random-exploiter` |
|
||||
| `warmstart_gate3` | 0.5900 | +11.3808 | +9.8893 | 4.9945 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_105902_gates-1-2-warmstart-gate3-exploiter` |
|
||||
| `replay_exploiter` | 0.5820 | +10.4253 | +8.9368 | 4.9977 | 0.0000 | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/exploiters/2026-07-05_115602_gates-1-2-replay-exploiter` |
|
||||
|
||||
## Gate 2B - Robustness Judgment
|
||||
|
||||
보수 필요: worst exploiter `warmstart_gate3` win rate 0.5900 vs threshold 0.5500.
|
||||
|
||||
## Gate 2C - Conditional Repair League
|
||||
|
||||
| Cycle | Worst protocol used | Guard pass | Battery worst | Passed | Checkpoint |
|
||||
| ---: | --- | ---: | ---: | ---: | --- |
|
||||
| 1 | `warmstart_gate3` | True | 0.5490 | True | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500` |
|
||||
|
||||
| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| ---: | --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | `long_random` | 0.5390 | +6.2793 | +4.7919 | 4.9943 | 0.0000 |
|
||||
| 1 | `warmstart_gate3` | 0.5490 | +7.0285 | +5.5662 | 4.9935 | 0.0000 |
|
||||
| 1 | `replay_exploiter` | 0.5393 | +6.1022 | +4.6214 | 4.9943 | 0.0000 |
|
||||
|
||||
## Human Play Recommendation
|
||||
|
||||
조건부 예: 강화 exploiter 관문은 통과했다. 다만 selectivity 관문이 미결/부분 지지이면 인간 대전은 실력 인증이 아니라 행동 양식 진단으로 시작해야 한다.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Existing `heuristic_expert` remains unchanged. The cap variants add only a hard `max_open_colors` gate around new-color openings.
|
||||
- `expert_capN` means no hard cap; EV thresholds and the existing soft concentration penalties are retained.
|
||||
- Warm-started exploiters use shaping coefficient 0 to measure target-specific exploitation without reintroducing early shaping rewards.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 19 KiB |
@@ -1,119 +0,0 @@
|
||||
# JAX PPO Model-Capacity Diagnostic - 2026-07-12
|
||||
|
||||
"MLP를 더 키우면 아직 얻을 게 남아 있나?"에 대한 진단이다.
|
||||
|
||||
신규 학습 없이 기존 체크포인트, `league_metrics.jsonl`, 소스만 읽어 작성했다.
|
||||
새로 실행한 학습·평가·벤치마크는 없다.
|
||||
|
||||
Subject: final candidate
|
||||
`/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate`
|
||||
Metrics: `.../final-cycles/2026-07-05/league/2026-07-05_191933_jax-ppo-final-cycle-c01/league_metrics.jsonl` (500 updates)
|
||||
|
||||
## 요약
|
||||
|
||||
용량이 **현재의 병목일 가능성은 낮다.** 다만 PPO 스택에서 모델 크기는
|
||||
**한 번도 검증된 적이 없으므로**, 이 판단은 아직 측정으로 뒷받침되지 않았다.
|
||||
단발 A/B로 결론을 확정할 것을 권한다 —
|
||||
[docs/plans/jax-ppo-model-size-ab.md](../plans/jax-ppo-model-size-ab.md).
|
||||
|
||||
## 1. 현재 크기는 근거 없는 상속값
|
||||
|
||||
`configs/jax_ppo/` 의 실 학습 config는 **전부** `hidden_size: 512, num_layers: 3`
|
||||
이다 (`smoke.yaml`만 64×2). final candidate도 동일하다
|
||||
(`main_ppo_config.json` → `network: {hidden_size: 512, num_layers: 3}`).
|
||||
|
||||
`hidden_size`를 다룬 기존 문서는 전부 은퇴한 Deep CFR/PyTorch 스택 것이다
|
||||
(`input_dim=365`, `legacy/deep-cfr/configs/` 기준). 특히
|
||||
[docs/plans/model_size_experiment.md](../plans/archive/model_size_experiment.md)는
|
||||
**현 JAX PPO 스택과 무관하다.**
|
||||
|
||||
→ 512×3은 실험으로 고른 값이 아니라 구 스택에서 복사돼 온 값이다.
|
||||
|
||||
## 2. 데이터 대비 모델이 매우 작다 (스케일링에 유리한 신호)
|
||||
|
||||
| 항목 | 값 |
|
||||
| --- | ---: |
|
||||
| `OBS_DIM` (`types.py:27`) | 454 |
|
||||
| `N_ACTIONS` | 96 |
|
||||
| 512×3 파라미터 수 | **808,033** |
|
||||
| 1024×4 파라미터 수 | 3,714,145 |
|
||||
| 2048×4 파라미터 수 | 13,719,649 |
|
||||
| 최종 사이클 소비 transition | **1,638,400,000** |
|
||||
| 샘플 : 파라미터 | **약 2,028 : 1** |
|
||||
|
||||
transition 수는 `rollout_steps=400 × batch_games=8192 × updates=500`.
|
||||
|
||||
셀프플레이라 데이터는 사실상 무한하다. 샘플:파라미터 2,000:1은 **데이터 제약이
|
||||
아니라 용량/최적화 제약** 구간의 전형이며, 보통 이런 영역에서 스케일링이 먹힌다.
|
||||
0.8M 파라미터는 절대적으로도 작다.
|
||||
|
||||
여기까지만 보면 "키우면 이득"이다. 그러나 3절이 반대 방향을 가리킨다.
|
||||
|
||||
## 3. 학습 로그에 용량 부족의 지문이 없다
|
||||
|
||||
최종 사이클 500 업데이트 구간 평균:
|
||||
|
||||
| update | value_loss | entropy_mean | approx_kl | return_mean | return_std |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| 0–50 | 0.0468 | 1.1159 | 0.0433 | 0.2696 | 0.6299 |
|
||||
| 50–100 | 0.0462 | 1.1167 | 0.0438 | 0.2799 | 0.6275 |
|
||||
| 100–200 | 0.0461 | 1.1241 | 0.0441 | 0.2822 | 0.6316 |
|
||||
| 200–300 | 0.0459 | 1.1143 | 0.0437 | 0.2857 | 0.6306 |
|
||||
| 300–400 | 0.0460 | 1.1239 | 0.0424 | 0.2900 | 0.6323 |
|
||||
| 400–500 | 0.0463 | 1.1163 | 0.0423 | 0.2916 | 0.6340 |
|
||||
|
||||
세 가지를 읽을 수 있다.
|
||||
|
||||
**(a) 크리틱은 이미 잘 맞춘다.** `value_loss`는 순수 MSE다
|
||||
(`ppo.py:624`: `masked_mean((mb_returns - value) ** 2, active_weight)`).
|
||||
`return_std ≈ 0.634` → 에피소드 리턴 분산 ≈ 0.402. MSE 0.0463과 비교하면
|
||||
**설명분산 ≈ 0.88**.
|
||||
|
||||
> 주의 — 이 0.88은 **상한 추정치**다. `return_std`는 에피소드 리턴의 표준편차이고
|
||||
> (`ppo.py:1072`: `episode_return = jnp.sum(transitions.reward, axis=0)`),
|
||||
> `value_loss`의 타깃은 GAE 리턴(`mb_returns`)이다. final candidate는 shaping이
|
||||
> 0이고 `gamma=1.0`이라 두 분포가 가깝지만, `gae_lambda=0.95`의 부트스트랩이
|
||||
> 타깃 분산을 축소시키므로 실제 설명분산은 이보다 낮을 수 있다.
|
||||
|
||||
용량이 모자란 네트워크는 loss가 **높은 지점에서** 정체한다. 여기는 **낮은
|
||||
지점에서** 정체한다. 잔여 오차 상당 부분은 불완전정보 + 덱 셔플에서 오는
|
||||
**환원 불가능한 분산**으로 보인다.
|
||||
|
||||
**(b) 엔트로피는 계수가 잡아둔 평형이다.** `entropy_mean`이 1.114–1.124에서
|
||||
500 업데이트 내내 미동도 없다. `entropy_coef=0.01`과 정책 그래디언트가 이룬
|
||||
평형이지 용량 한계가 아니다. **파라미터를 늘려도 이 값은 안 변한다.**
|
||||
(불완전정보에서 혼합 전략은 정상이므로 이 자체가 결함은 아니다.)
|
||||
|
||||
**(c) 죽은 게 아니라 느린 것이다.** `return_mean`은 0.2696 → 0.2916으로
|
||||
여전히 오르고 있다. `approx_kl`도 0.042–0.044로 유지되어 정책이 계속 움직인다.
|
||||
수렴해서 멈춘 상태가 아니다.
|
||||
|
||||
## 4. 실제 천장은 exploitability이고, 이건 용량 문제가 아니다
|
||||
|
||||
[diminishing-returns-2026-07-05.md](diminishing-returns-2026-07-05.md) 기준,
|
||||
final 계열을 상대로 새로 학습시킨 exploiter가 **여전히 승률 0.539–0.549로 이긴다**
|
||||
(`repair_c01_update_500`, 3개 프로토콜).
|
||||
|
||||
불완전정보 게임에서 이 잔여 착취가능성은 **게임이론적 문제**다. PPO 셀프플레이는
|
||||
내쉬로 수렴하지 않고 전략공간을 순환한다. MLP를 키우면 *같은 순환 역학 안에서
|
||||
더 나은 응수*를 찾을 뿐, 이 0.54 바닥 자체를 무너뜨리지는 못한다.
|
||||
|
||||
## 판정
|
||||
|
||||
- 스케일링이 **소폭 이득**을 줄 가능성은 있다 (2절: 샘플:파라미터 2,000:1).
|
||||
- 그러나 **용량 부족의 직접 증거는 없다** (3절: 낮은 지점에서 평평한 value_loss,
|
||||
계수가 고정한 엔트로피).
|
||||
- 그리고 **모델을 실제로 가두고 있는 것은 exploitability**이며, 이건 스케일로
|
||||
풀리지 않는다 (4절).
|
||||
|
||||
따라서 우선순위는 낮다. 다만 **한 번도 측정한 적이 없다**는 사실 때문에 위
|
||||
판단은 전부 정황증거다. 결론을 확정하려면 단발 A/B가 필요하다 →
|
||||
[docs/plans/jax-ppo-model-size-ab.md](../plans/jax-ppo-model-size-ab.md).
|
||||
|
||||
## Notes
|
||||
|
||||
- 이 작업에서 신규 학습, config 변경, 체크포인트 수정은 없었다.
|
||||
- 파라미터 수는 `ActorCritic(hidden, layers).init()` 후 리프 크기 합으로 계산했다
|
||||
(`ppo.py:104`).
|
||||
- GUI 영향 참고: 현재 CPU 추론 약 50ms/수. 1024×4로 키워도 150–200ms 수준이라
|
||||
`lost-cities-play` 대국에는 지장이 없다.
|
||||
@@ -1,68 +0,0 @@
|
||||
# JAX PPO Static-Opponent Ladder Report - 2026-07-04
|
||||
|
||||
**Status:** PASS.
|
||||
**Implementation commit:** `4c0c2e9`.
|
||||
**Hardware:** RTX 3090 with optional CUDA JAX via `uv run --with 'jax[cuda12]'`.
|
||||
**Artifact root:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/`.
|
||||
|
||||
## Protocol
|
||||
|
||||
Each opponent was trained from scratch with the same PPO configuration:
|
||||
|
||||
- `batch_games=8192`
|
||||
- `rollout_steps=400`
|
||||
- `total_updates=250`
|
||||
- `gamma=1.0`
|
||||
- `gae_lambda=0.95`
|
||||
- `clip_epsilon=0.2`
|
||||
- `entropy_coef=0.01`
|
||||
- `seed=20260704`
|
||||
|
||||
Evaluation used a fixed 10,000-deck shuffle bank and duplicate play: each deck
|
||||
was played once with the learner in seat 0 and once with the learner in seat 1,
|
||||
for 20,000 evaluated games per gate. Win rates include Wilson 95% intervals.
|
||||
|
||||
## Results
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean score diff | Mean game length | Positive expeditions/game |
|
||||
| --- | --- | --- | --- | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.00000 [0.99981, 1.00000] | 204.56335 | 82.45495 | 3.2783 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.98655 [0.98486, 0.98806] | 116.83400 | 167.81450 | 3.9573 |
|
||||
| 3 | `heuristic_cautious` | PASS | 0.95955 [0.95673, 0.96219] | 142.89930 | 185.38690 | 4.0946 |
|
||||
|
||||
Gate 1 passed both required conditions: win rate >= 90% and at least two
|
||||
positive expeditions per game. Gates 2 and 3 passed their mean-score-difference
|
||||
condition.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Opponent | Run directory | Evaluation JSON |
|
||||
| --- | --- | --- |
|
||||
| `discard_only` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_223401_jax-ppo-discard-only/` | `eval_discard_only_duplicate.json` |
|
||||
| `heuristic_balanced` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_224749_jax-ppo-balanced/` | `eval_balanced_duplicate.json` |
|
||||
| `heuristic_cautious` | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/` | `eval_cautious_duplicate.json` |
|
||||
|
||||
The random rollout baseline against `discard_only` is stored at:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/random_baseline_discard_only.json
|
||||
```
|
||||
|
||||
## Canary Notes
|
||||
|
||||
The random-policy baseline against `discard_only` had `play_action_rate=0.28855`
|
||||
and only `0.41797` positive expeditions per game. The trained discard-only
|
||||
policy ended duplicate evaluation at `play_action_rate=0.64598` and `3.2783`
|
||||
positive expeditions per game, so the intended anti-collapse signal is present.
|
||||
|
||||
Balanced and cautious checkpoints pass their score gates decisively, but their
|
||||
duplicate evaluations are longer: mean game lengths are `167.81450` and
|
||||
`185.38690`. The final training rollout forced-end rates were low but nonzero
|
||||
for those opponents (`0.00415` and `0.00391`). Treat game length and forced-end
|
||||
rate as canaries in the next self-play phase.
|
||||
|
||||
## Next Step
|
||||
|
||||
The static ladder has cleared. The next phase can start snapshot-pool league
|
||||
self-play, using these three passed checkpoints as initial anchors and keeping
|
||||
duplicate evaluation as the regression gate.
|
||||
@@ -1 +0,0 @@
|
||||
{"date":"2026-07-05","status":"pass_with_canary_warnings","expert_bot":{"mirror_max_steps_rate":0.0,"mirror_opened_colors_per_game":2.256,"mirror_play_action_rate":0.3117858091947914,"vs_discard_mean_score_diff":8.9716,"vs_balanced_mean_score_diff":54.36725,"vs_cautious_mean_score_diff":51.9397},"gates":{"discard_only":{"win_rate":1.0,"wilson_low":0.9998079639438954,"mean_score_diff":182.067,"opened_colors_per_game":4.2163,"max_steps_rate":0.0},"heuristic_balanced":{"win_rate":0.9409,"wilson_low":0.9375464241082592,"mean_score_diff":119.925,"opened_colors_per_game":4.9862,"max_steps_rate":0.1859},"heuristic_expert":{"win_rate":0.83815,"wilson_low":0.8329806740267004,"mean_score_diff":43.3099,"score_diff_ci95_low":42.706527573078894,"opened_colors_per_game":4.5509,"max_steps_rate":0.0}},"exploiter":{"target":"ladder_v2_gate3","win_rate":0.83135,"wilson_low":0.8260970642435626,"mean_score_diff":54.7721,"score_diff_ci95_low":53.98710920050297,"opened_colors_per_game":4.99135,"max_steps_rate":0.0001},"artifact_root":"/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/"}
|
||||
@@ -1,114 +0,0 @@
|
||||
# JAX PPO Ladder v2 Report - 2026-07-05
|
||||
|
||||
**Status:** PASS for the ordered ladder gates; canary warnings remain.
|
||||
**Hardware:** RTX 3090 via `uv run --with 'jax[cuda12]'`.
|
||||
**Artifact roots:**
|
||||
|
||||
- Expert bot: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/`
|
||||
- Ladder v2: `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/`
|
||||
|
||||
## Protocol
|
||||
|
||||
Part A added a pure-JAX `heuristic_expert` opponent and evaluated it with a
|
||||
fixed shuffle bank plus duplicate play. Part B trained PPO from random
|
||||
initialization for each gate with the same 250-update configuration used by the
|
||||
original ladder, changing only the opponent and artifact root:
|
||||
|
||||
- Gate 1: `discard_only`
|
||||
- Gate 2: `heuristic_balanced`
|
||||
- Gate 3: `heuristic_expert`
|
||||
|
||||
All evaluations below used 10,000 deck orders with duplicate seat-swapped play,
|
||||
for 20,000 evaluated games.
|
||||
|
||||
## Part A - Expert Bot Gate
|
||||
|
||||
| Check | Result | Win rate | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Expert mirror, 1,000 duplicate pairs | PASS | 0.4815 | 0.0000 | [-0.6956, +0.6956] | 45.1560 | 0.0000 | 2.2560 | 0.3118 |
|
||||
| Expert vs `discard_only` | PASS | 0.6643 | +8.9716 | [+8.7886, +9.1546] | 45.2125 | 0.0000 | 1.8971 | 0.2828 |
|
||||
| Expert vs `heuristic_balanced` | PASS | 0.9961 | +54.3673 | [+54.0791, +54.6554] | 64.9181 | 0.0000 | 2.8003 | 0.2728 |
|
||||
| Expert vs `heuristic_cautious` | PASS | 0.9972 | +51.9397 | [+51.6960, +52.1834] | 75.2329 | 0.0000 | 2.6090 | 0.2240 |
|
||||
|
||||
The new script bot is non-stalling in mirror play, symmetric under duplicate
|
||||
seat swap, and beats all three previous static bots by a clear positive score
|
||||
margin. Its own opened-color profile is in the intended 2-3 color band.
|
||||
|
||||
## Part B - Ladder v2
|
||||
|
||||
| Gate | Opponent | Result | Win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | `discard_only` | PASS | 1.0000 [0.9998, 1.0000] | +182.0670 | [+181.6390, +182.4950] | 82.1558 | 0.0000 | 4.2163 | 0.6398 | 3.1845 |
|
||||
| 2 | `heuristic_balanced` | PASS | 0.9409 [0.9375, 0.9441] | +119.9250 | [+118.9950, +120.8550] | 151.2468 | 0.1859 | 4.9862 | 0.2937 | 3.5316 |
|
||||
| 3 | `heuristic_expert` | PASS | 0.8382 [0.8330, 0.8432] | +43.3099 | [+42.7065, +43.9133] | 61.4025 | 0.0000 | 4.5509 | 0.6661 | 2.6121 |
|
||||
|
||||
Gate 1 passed the original win-rate and positive-expedition conditions. Gate 2
|
||||
passed the mean-score-difference condition. Gate 3 passed the v2 condition:
|
||||
duplicate mean score difference is positive and its 95% confidence lower bound
|
||||
is above zero.
|
||||
|
||||
## Canary Findings
|
||||
|
||||
The ladder did not produce the desired 2-3 color selectivity in the learned PPO
|
||||
policies. The gate-3 policy still opens 4.5509 colors/game against the expert.
|
||||
This is lower than the original gate-3 checkpoint but still far outside the
|
||||
target band. Treat the selectivity hypothesis as not confirmed.
|
||||
|
||||
Gate 2 also has a high forced-end canary: `max_steps_rate=0.1859` in duplicate
|
||||
evaluation versus `heuristic_balanced`. It passed its score gate, but the
|
||||
balanced opponent still permits long games and 5-color farming.
|
||||
|
||||
## Exploiter Baseline
|
||||
|
||||
After gate 3 passed, a new PPO exploiter was trained from random initialization
|
||||
against the frozen v2 gate-3 checkpoint with the same 250-update budget and
|
||||
duplicate-evaluated against that checkpoint.
|
||||
|
||||
| Target checkpoint | Exploiter win rate (Wilson 95%) | Mean diff | CI95 diff | Length | Max-step | Opened colors | Play rate | Positive exp. |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Old gate-3 checkpoint | 0.6712 [0.6646, 0.6776] | +24.1371 | n/a | 48.0017 | 0.0000 | 4.8971 | 0.7461 | 2.1287 |
|
||||
| New v2 gate-3 checkpoint | 0.8314 [0.8261, 0.8365] | +54.7721 | [+53.9871, +55.5571] | 60.8744 | 0.0001 | 4.9914 | 0.7028 | 2.8921 |
|
||||
|
||||
The new gate-3 checkpoint beats `heuristic_expert`, but is more exploitable
|
||||
under this protocol than the old checkpoint. That does not invalidate the gate,
|
||||
but it makes exploiter reduction a primary objective for the league phase.
|
||||
|
||||
## Old vs New Gate-3 Canary Comparison
|
||||
|
||||
| Metric | Old gate-3 vs cautious | New gate-3 vs expert | Interpretation |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Mean score diff | +142.8993 | +43.3099 | New gate is harder and less suspiciously clean. |
|
||||
| Win rate | 0.9596 | 0.8382 | New result is less inflated. |
|
||||
| Opened colors | 4.9896 | 4.5509 | Improved, but still not selective enough. |
|
||||
| Max-step rate | not recorded in old eval | 0.0000 | New expert evaluation does not stall. |
|
||||
| Play action rate | 0.2763 | 0.6661 | New policy plays much more actively. |
|
||||
| Exploiter win rate | 0.6712 | 0.8314 | New checkpoint is currently more exploitable. |
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Item | Path |
|
||||
| --- | --- |
|
||||
| Expert mirror JSON | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/expert-mirror-1000-duplicate.json` |
|
||||
| Expert vs old bots JSONs | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/expert-bot/2026-07-05/` |
|
||||
| Gate 1 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_010429_jax-ppo-ladder-v2-discard-only/` |
|
||||
| Gate 2 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_011827_jax-ppo-ladder-v2-balanced/` |
|
||||
| Gate 3 run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/` |
|
||||
| Exploiter run | `/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_014713_jax-ppo-ladder-v2-exploiter/` |
|
||||
|
||||
## Decisions
|
||||
|
||||
- The requested `lost-cities-league-selfplay-spec.md` file was not present in
|
||||
the repository. I created `docs/plans/lost-cities-league-selfplay-spec.md` as
|
||||
the tracked league spec location, following the repository docs routing for
|
||||
active plans.
|
||||
- Gate 2's high max-step rate is treated as a canary warning rather than a
|
||||
gate failure because the v2 instruction keeps gate 2's original score-based
|
||||
pass condition.
|
||||
|
||||
## Next
|
||||
|
||||
Proceed to league implementation only with the updated interpretation: anchors
|
||||
fix the Elo scale and provide style diversity, but they do not certify strength.
|
||||
The league's main success criterion should be lowering exploiter win rate from
|
||||
the v2 baseline of `0.83135` while preserving positive duplicate performance
|
||||
against `heuristic_expert`.
|
||||
@@ -1,18 +0,0 @@
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 914.3801500956258, "event": "snapshot_eval", "games": 20000, "losses": 96.0, "max_steps_rate": 0.0, "mean_game_length": 71.11445, "mean_score_diff": 125.14285, "opened_colors_per_game": 4.4374, "opponent": "discard_only", "opponent_kind": "static", "play_action_rate": 0.6509964599542043, "positive_expeditions_per_game": 2.9703, "score_diff_ci95_high": 125.74118659681464, "score_diff_ci95_low": 124.54451340318535, "snapshot": "league_c01_u000250", "ties": 7.0, "update": 250, "wilson_high": 0.9957514261405391, "wilson_low": 0.9937585157747342, "win_rate": 0.99485, "wins": 19897.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 637.8568047136434, "event": "snapshot_eval", "games": 20000, "losses": 479.0, "max_steps_rate": 0.0231, "mean_game_length": 84.2736, "mean_score_diff": 100.8997, "opened_colors_per_game": 4.8582, "opponent": "heuristic_balanced", "opponent_kind": "static", "play_action_rate": 0.49003797546566286, "positive_expeditions_per_game": 2.6421, "score_diff_ci95_high": 101.6030587557229, "score_diff_ci95_low": 100.1963412442771, "snapshot": "league_c01_u000250", "ties": 17.0, "update": 250, "wilson_high": 0.9772657590483588, "wilson_low": 0.9729517298839193, "win_rate": 0.9752, "wins": 19504.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 522.2171364943674, "event": "snapshot_eval", "games": 20000, "losses": 931.0, "max_steps_rate": 0.1264, "mean_game_length": 139.4069, "mean_score_diff": 116.66435, "opened_colors_per_game": 4.9623, "opponent": "heuristic_cautious", "opponent_kind": "static", "play_action_rate": 0.3169406240961486, "positive_expeditions_per_game": 3.0465, "score_diff_ci95_high": 117.46585342388337, "score_diff_ci95_low": 115.86284657611662, "snapshot": "league_c01_u000250", "ties": 12.0, "update": 250, "wilson_high": 0.9557015978586921, "wilson_low": 0.9498244750852939, "win_rate": 0.95285, "wins": 19057.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 186.84131209092277, "event": "snapshot_eval", "games": 20000, "losses": 4910.0, "max_steps_rate": 0.0, "mean_game_length": 56.74285, "mean_score_diff": 26.7149, "opened_colors_per_game": 4.5478, "opponent": "heuristic_expert", "opponent_kind": "static", "play_action_rate": 0.6407205946284102, "positive_expeditions_per_game": 2.20505, "score_diff_ci95_high": 27.262035512864674, "score_diff_ci95_low": 26.167764487135326, "snapshot": "league_c01_u000250", "ties": 177.0, "update": 250, "wilson_high": 0.7516379759261593, "wilson_low": 0.7395676767594765, "win_rate": 0.74565, "wins": 14913.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 816.73245801838, "event": "snapshot_eval", "games": 20000, "losses": 178.0, "max_steps_rate": 0.03575, "mean_game_length": 124.3454, "mean_score_diff": 163.52095, "opened_colors_per_game": 4.99725, "opponent": "ladder_v2_gate1_discard", "opponent_kind": "checkpoint", "play_action_rate": 0.36905599758881236, "positive_expeditions_per_game": 3.6202, "score_diff_ci95_high": 164.3139031956684, "score_diff_ci95_low": 162.7279968043316, "snapshot": "league_c01_u000250", "ties": 2.0, "update": 250, "wilson_high": 0.9922178307682814, "wilson_low": 0.9895935898246238, "win_rate": 0.991, "wins": 19820.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 90.70646999257946, "event": "snapshot_eval", "games": 20000, "losses": 7275.0, "max_steps_rate": 0.0, "mean_game_length": 49.7744, "mean_score_diff": 16.1786, "opened_colors_per_game": 4.7712, "opponent": "ladder_v2_gate2_balanced", "opponent_kind": "checkpoint", "play_action_rate": 0.7223761262392399, "positive_expeditions_per_game": 2.14855, "score_diff_ci95_high": 16.80698121804704, "score_diff_ci95_low": 15.550218781952958, "snapshot": "league_c01_u000250", "ties": 172.0, "update": 250, "wilson_high": 0.6343247760445404, "wilson_low": 0.6209261971503363, "win_rate": 0.62765, "wins": 12553.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "cycle": 1, "elo_estimate": 246.30450760214026, "event": "snapshot_eval", "games": 20000, "losses": 3778.0, "max_steps_rate": 5e-05, "mean_game_length": 57.6601, "mean_score_diff": 43.7116, "opened_colors_per_game": 4.9119, "opponent": "ladder_v2_gate3_expert", "opponent_kind": "checkpoint", "play_action_rate": 0.6936249212033888, "positive_expeditions_per_game": 2.57955, "score_diff_ci95_high": 44.41070179719779, "score_diff_ci95_low": 43.012498202802206, "snapshot": "league_c01_u000250", "ties": 122.0, "update": 250, "wilson_high": 0.8104321748965465, "wilson_low": 0.7994506831092297, "win_rate": 0.805, "wins": 16100.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 630.8113677090081, "event": "snapshot_eval", "games": 20000, "losses": 494.0, "max_steps_rate": 0.0, "mean_game_length": 69.40915, "mean_score_diff": 107.6151, "opened_colors_per_game": 4.88455, "opponent": "discard_only", "opponent_kind": "static", "play_action_rate": 0.6697188054808169, "positive_expeditions_per_game": 2.9962, "score_diff_ci95_high": 108.30476814184044, "score_diff_ci95_low": 106.92543185815956, "snapshot": "league_c01_u000500", "ties": 22.0, "update": 500, "wilson_high": 0.9763077989189339, "wilson_low": 0.9719100740854563, "win_rate": 0.9742, "wins": 19484.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 560.1723021956866, "event": "snapshot_eval", "games": 20000, "losses": 740.0, "max_steps_rate": 0.0271, "mean_game_length": 82.88495, "mean_score_diff": 88.5699, "opened_colors_per_game": 4.93005, "opponent": "heuristic_balanced", "opponent_kind": "static", "play_action_rate": 0.5006923105759661, "positive_expeditions_per_game": 2.53425, "score_diff_ci95_high": 89.24524287663027, "score_diff_ci95_low": 87.89455712336974, "snapshot": "league_c01_u000500", "ties": 25.0, "update": 500, "wilson_high": 0.9643207034494167, "wilson_low": 0.9590019512527707, "win_rate": 0.96175, "wins": 19235.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 455.44923699476067, "event": "snapshot_eval", "games": 20000, "losses": 1336.0, "max_steps_rate": 0.16445, "mean_game_length": 148.75985, "mean_score_diff": 94.1284, "opened_colors_per_game": 4.9832, "opponent": "heuristic_cautious", "opponent_kind": "static", "play_action_rate": 0.28929498432939504, "positive_expeditions_per_game": 2.8439, "score_diff_ci95_high": 94.9311528056073, "score_diff_ci95_low": 93.3256471943927, "snapshot": "league_c01_u000500", "ties": 19.0, "update": 500, "wilson_high": 0.9356506507735038, "wilson_low": 0.9286833340559937, "win_rate": 0.93225, "wins": 18645.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 111.06146709621818, "event": "snapshot_eval", "games": 20000, "losses": 6672.0, "max_steps_rate": 0.0, "mean_game_length": 53.33205, "mean_score_diff": 14.2458, "opened_colors_per_game": 4.8409, "opponent": "heuristic_expert", "opponent_kind": "static", "play_action_rate": 0.6926969797194702, "positive_expeditions_per_game": 2.16835, "score_diff_ci95_high": 14.735340147409243, "score_diff_ci95_low": 13.756259852590755, "snapshot": "league_c01_u000500", "ties": 236.0, "update": 500, "wilson_high": 0.6611596974601753, "wilson_low": 0.647980924991277, "win_rate": 0.6546, "wins": 13092.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 683.7735272455216, "event": "snapshot_eval", "games": 20000, "losses": 373.0, "max_steps_rate": 0.0419, "mean_game_length": 123.1991, "mean_score_diff": 141.3215, "opened_colors_per_game": 4.99905, "opponent": "ladder_v2_gate1_discard", "opponent_kind": "checkpoint", "play_action_rate": 0.3599053985607506, "positive_expeditions_per_game": 3.56165, "score_diff_ci95_high": 142.1693389362133, "score_diff_ci95_low": 140.47366106378666, "snapshot": "league_c01_u000500", "ties": 10.0, "update": 500, "wilson_high": 0.9826591308155324, "wilson_low": 0.9788561881093119, "win_rate": 0.98085, "wins": 19617.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 118.72098430949596, "event": "snapshot_eval", "games": 20000, "losses": 6562.0, "max_steps_rate": 5e-05, "mean_game_length": 49.765, "mean_score_diff": 20.6296, "opened_colors_per_game": 4.9362, "opponent": "ladder_v2_gate2_balanced", "opponent_kind": "checkpoint", "play_action_rate": 0.771107881039547, "positive_expeditions_per_game": 2.2327, "score_diff_ci95_high": 21.259362634675494, "score_diff_ci95_low": 19.999837365324506, "snapshot": "league_c01_u000500", "ties": 148.0, "update": 500, "wilson_high": 0.6710116102233363, "wilson_low": 0.6579252099142052, "win_rate": 0.6645, "wins": 13290.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 229.5355020840115, "event": "snapshot_eval", "games": 20000, "losses": 4093.0, "max_steps_rate": 5e-05, "mean_game_length": 55.7303, "mean_score_diff": 38.96295, "opened_colors_per_game": 4.96905, "opponent": "ladder_v2_gate3_expert", "opponent_kind": "checkpoint", "play_action_rate": 0.7273133036132009, "positive_expeditions_per_game": 2.5661, "score_diff_ci95_high": 39.63449150676593, "score_diff_ci95_low": 38.291408493234066, "snapshot": "league_c01_u000500", "ties": 119.0, "update": 500, "wilson_high": 0.7949949671932197, "wilson_low": 0.7836938823375068, "win_rate": 0.7894, "wins": 15788.0}
|
||||
{"checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "cycle": 1, "elo_estimate": 30.163477370688103, "event": "snapshot_eval", "games": 20000, "losses": 8961.0, "max_steps_rate": 0.0001, "mean_game_length": 50.86105, "mean_score_diff": 5.5848, "opened_colors_per_game": 4.94555, "opponent": "league_c01_u000250", "opponent_kind": "checkpoint", "play_action_rate": 0.7468741704926414, "positive_expeditions_per_game": 2.3624, "score_diff_ci95_high": 6.236320508695035, "score_diff_ci95_low": 4.933279491304966, "snapshot": "league_c01_u000500", "ties": 173.0, "update": 500, "wilson_high": 0.5501945130045903, "wilson_low": 0.536388856672951, "win_rate": 0.5433, "wins": 10866.0}
|
||||
{"cycle": 1, "eval_json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/eval_vs_league_c01_u000250_duplicate.json", "event": "exploiter_eval", "exploiter": "exploiter_c1", "exploiter_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "games": 20000, "losses": 9759.0, "max_steps_rate": 0.0, "mean_game_length": 48.76485, "mean_score_diff": 0.6818, "opened_colors_per_game": 4.99735, "play_action_rate": 0.7938232266689632, "positive_expeditions_per_game": 2.5895, "score_diff_ci95_high": 1.283273913015642, "score_diff_ci95_low": 0.08032608698435773, "target": "league_c01_u000250", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250", "ties": 196.0, "wilson_high": 0.5091783515083876, "wilson_low": 0.4953207843293599, "win_rate": 0.50225, "wins": 10045.0}
|
||||
{"cycle": 1, "eval_json": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/eval_vs_league_c01_u000500_duplicate.json", "event": "exploiter_final_eval", "exploiter": "exploiter_c1", "exploiter_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/exploiters/2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest", "games": 20000, "losses": 10376.0, "max_steps_rate": 0.0, "mean_game_length": 48.23185, "mean_score_diff": -2.63775, "opened_colors_per_game": 4.9971, "play_action_rate": 0.797814966505273, "positive_expeditions_per_game": 2.569, "score_diff_ci95_high": -2.039857230939478, "score_diff_ci95_low": -3.2356427690605223, "target": "league_c01_u000500", "target_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "ties": 205.0, "wilson_high": 0.4778727301883191, "wilson_low": 0.46403842710654053, "win_rate": 0.47095, "wins": 9419.0}
|
||||
{"cycles_completed": 1, "event": "league_complete", "final_checkpoint": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500", "run_dir": "/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1", "stop_reason": "success", "updates": 500}
|
||||
@@ -1,53 +0,0 @@
|
||||
# League v1 Report - 2026-07-05
|
||||
|
||||
**Status:** `success`.
|
||||
**Run dir:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1`.
|
||||
**Final checkpoint:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500`.
|
||||
|
||||
## Summary
|
||||
|
||||
- Updates: 500
|
||||
- Cycles completed: 1
|
||||
- Success threshold: exploiter win rate <= 0.60 and expert CI low > 0.00
|
||||
|
||||
## Exploiter Series
|
||||
|
||||
| Type | Cycle | Win rate | Mean diff | Opened colors | Max-step | Target |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| cycle-target | 1 | 0.5022 | +0.6818 | 4.9973 | 0.0000 | `league_c01_u000250` |
|
||||
| final-checkpoint | 1 | 0.4709 | -2.6378 | 4.9971 | 0.0000 | `league_c01_u000500` |
|
||||
|
||||
## Expert Evaluation Series
|
||||
|
||||
| Cycle | Update | Win rate | Mean diff | CI low | Opened colors | Max-step |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 250 | 0.7457 | +26.7149 | +26.1678 | 4.5478 | 0.0000 |
|
||||
| 1 | 500 | 0.6546 | +14.2458 | +13.7563 | 4.8409 | 0.0000 |
|
||||
|
||||
## Final Anchor Table
|
||||
|
||||
| Opponent | Win rate | Mean diff | CI low | Opened colors | Max-step | Elo est. |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| `discard_only` | 0.9742 | +107.6151 | +106.9254 | 4.8845 | 0.0000 | +630.8 |
|
||||
| `heuristic_balanced` | 0.9617 | +88.5699 | +87.8946 | 4.9300 | 0.0271 | +560.2 |
|
||||
| `heuristic_cautious` | 0.9323 | +94.1284 | +93.3256 | 4.9832 | 0.1645 | +455.4 |
|
||||
| `heuristic_expert` | 0.6546 | +14.2458 | +13.7563 | 4.8409 | 0.0000 | +111.1 |
|
||||
| `ladder_v2_gate1_discard` | 0.9808 | +141.3215 | +140.4737 | 4.9991 | 0.0419 | +683.8 |
|
||||
| `ladder_v2_gate2_balanced` | 0.6645 | +20.6296 | +19.9998 | 4.9362 | 0.0001 | +118.7 |
|
||||
| `ladder_v2_gate3_expert` | 0.7894 | +38.9629 | +38.2914 | 4.9691 | 0.0001 | +229.5 |
|
||||
| `league_c01_u000250` | 0.5433 | +5.5848 | +4.9333 | 4.9455 | 0.0001 | +30.2 |
|
||||
|
||||
## Plots
|
||||
|
||||
- `docs/reports/league-v1-exploiter-win-rate.png`
|
||||
- `docs/reports/league-v1-opened-colors.png`
|
||||
- `docs/reports/league-v1-elo-estimate.png`
|
||||
|
||||
## Hypothesis Read
|
||||
|
||||
부분 지지: 피탈률 목표는 한 사이클 만에 달성했다. 다만 expert 상대 오픈 색은 4.55에서 4.84로 증가해, 적대적 압력이 selectivity를 유도한다는 하위 가설은 아직 지지되지 않는다.
|
||||
|
||||
## Decisions
|
||||
|
||||
- Training uses a PFSP active subset when the pool exceeds `max_active_pool_members=12`. The full pool is retained for lifecycle and evaluation; the subset keeps single-GPU cycle time within budget.
|
||||
- Stalling anchors are capped between the configured floor and cap during PFSP sampling.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 24 KiB |
File diff suppressed because it is too large
Load Diff
@@ -1,191 +0,0 @@
|
||||
# Verification Pass - 2026-07-04
|
||||
|
||||
**Status:** FAIL - stop before league self-play.
|
||||
**Implementation start commit:** `bb52ef9`.
|
||||
**Primary checkpoint under audit:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/jax-ppo-static-opponents/2026-07-04_230150_jax-ppo-cautious/latest`.
|
||||
**Verification artifacts:** `/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/`.
|
||||
|
||||
## Decision
|
||||
|
||||
Do not start snapshot-pool league self-play from the current ladder artifacts.
|
||||
The duplicate/Wilson protocol is reproducible, and the obvious seat-perspective
|
||||
bug was not found, but the cautious anchor is not a trustworthy permanent Elo
|
||||
anchor: it stalls heavily in mirror play and gives the trained policy a very
|
||||
clean win while the trained policy opens almost all colors.
|
||||
|
||||
## Transcript Audit
|
||||
|
||||
Transcript file:
|
||||
|
||||
```text
|
||||
docs/reports/verification-pass-2026-07-04-transcripts.txt
|
||||
```
|
||||
|
||||
Rendered protocol: 10 fixed shuffle-bank deck orders, duplicate seat-swapped,
|
||||
for 20 total games of gate-3 checkpoint vs `heuristic_cautious`.
|
||||
|
||||
Summary from the transcript dump:
|
||||
|
||||
| Metric | Value |
|
||||
| --- | ---: |
|
||||
| Games | 20 |
|
||||
| Agent mean score diff | +144.500 |
|
||||
| Agent score diff min/max | -93 / +230 |
|
||||
| Agent opened colors mean | 5.000 |
|
||||
| Mean game length | 159.200 |
|
||||
| Length min / p50 / p95 / max | 63 / 109 / 400 / 400 |
|
||||
| Max-steps rate | 0.150 |
|
||||
| Handshake play events | 86 |
|
||||
| Cautious openings | 78 |
|
||||
| Cautious low openings below rank 7 | 3 |
|
||||
| Cautious discards immediately playable by opponent | 162 |
|
||||
|
||||
Interpretation: the sample confirms the concern behind this pass. The agent is
|
||||
not winning by a restrained 2-3 color expert pattern; it opens all 5 colors on
|
||||
average and still wins by a wide margin. The cautious policy is also leaking
|
||||
many immediately useful discard tops.
|
||||
|
||||
## Protocol Recheck
|
||||
|
||||
The original ladder evaluation was audited in code. The `eval` path uses:
|
||||
|
||||
- explicit shuffle banks generated from `evaluation.shuffle_bank_seed`;
|
||||
- duplicate seat-swapped loops over learner seat 0 and learner seat 1;
|
||||
- Wilson confidence intervals from aggregate wins.
|
||||
|
||||
The protocol itself is not the main failure. The failure is that the original
|
||||
gate summary omitted `max_steps_rate` and score-diff distribution canaries.
|
||||
|
||||
Recomputed gate-3 duplicate evaluation with distribution canaries:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/gate3_reverification_distribution.json
|
||||
```
|
||||
|
||||
| Metric | Value |
|
||||
| --- | ---: |
|
||||
| Games | 20,000 |
|
||||
| Win rate | 0.95960 |
|
||||
| Wilson 95% | [0.95678, 0.96224] |
|
||||
| Mean score diff | +142.9178 |
|
||||
| Score diff p05 / p50 / p95 | +21 / +152 / +227 |
|
||||
| Opened colors/game | 4.98965 |
|
||||
| Positive expeditions/game | 4.09470 |
|
||||
| Play action rate | 0.27639 |
|
||||
| Mean game length | 185.3179 |
|
||||
| Game length p50 / p95 / max | 115 / 400 / 400 |
|
||||
| Max-steps rate | 0.24070 |
|
||||
|
||||
The 24.07% forced-end rate is a high-severity canary failure for using this
|
||||
gate as a clean league baseline.
|
||||
|
||||
## Perspective And Mirror Tests
|
||||
|
||||
New tests added:
|
||||
|
||||
- hand-authored heuristic behavior checks for weak unopened hands, strong
|
||||
balanced opens, and P1 own-board perspective;
|
||||
- static-policy duplicate mirror checks for all three heuristic policies;
|
||||
- gate-3 checkpoint duplicate self-mirror check, skipped on CPU-only runs and
|
||||
executed under CUDA JAX locally.
|
||||
|
||||
Commands run:
|
||||
|
||||
```bash
|
||||
uv run pytest -q tests/lost_cities_jax/test_ppo_stack.py
|
||||
uv run --with 'jax[cuda12]' pytest -q \
|
||||
tests/lost_cities_jax/test_ppo_stack.py::test_gate3_checkpoint_duplicate_self_mirror_score_diff_is_zero
|
||||
```
|
||||
|
||||
Results:
|
||||
|
||||
- CPU focused tests: 11 passed, 1 skipped.
|
||||
- CUDA checkpoint self-mirror test: 1 passed.
|
||||
|
||||
Mirror evaluation artifacts:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_discard_only.json
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_heuristic_balanced.json
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_heuristic_cautious.json
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/mirror_gate3_checkpoint.json
|
||||
```
|
||||
|
||||
| Mirror policy | Mean diff | Win/loss symmetry | Max-steps rate | Play action rate | Mean length |
|
||||
| --- | ---: | --- | ---: | ---: | ---: |
|
||||
| `discard_only` | 0.0 | exact | 0.0000 | 0.0000 | 44.0000 |
|
||||
| `heuristic_balanced` | 0.0 | exact | 0.4683 | 0.0422 | 231.1210 |
|
||||
| `heuristic_cautious` | 0.0 | exact | 0.8956 | 0.0053 | 368.8435 |
|
||||
| gate-3 checkpoint | 0.0 | exact | 0.0000 | 0.7839 | 45.4292 |
|
||||
|
||||
Interpretation: the broad P0/P1 perspective bug is unlikely. Duplicate mirror
|
||||
score differences cancel exactly. The serious issue is heuristic quality:
|
||||
`heuristic_cautious` mirror play almost never opens expeditions and reaches
|
||||
forced termination in 89.56% of games.
|
||||
|
||||
## Exploiter Baseline
|
||||
|
||||
Frozen opponent: gate-3 checkpoint. New exploiter: random init PPO, same 8192 x
|
||||
400 rollout shape, 250 updates, shaping annealed by the standard schedule.
|
||||
|
||||
Training run:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_235948_jax-ppo-gate3-exploiter/
|
||||
```
|
||||
|
||||
Final train-row canaries:
|
||||
|
||||
| Metric | Value |
|
||||
| --- | ---: |
|
||||
| return_mean | +0.24118 |
|
||||
| play_action_rate | 0.71362 |
|
||||
| opened_colors_mean | 4.91016 |
|
||||
| positive_expeditions_mean | 2.07922 |
|
||||
| game_length_mean | 49.12524 |
|
||||
| max_steps_rate | 0.00000 |
|
||||
|
||||
Duplicate exploiter evaluation:
|
||||
|
||||
```text
|
||||
/mnt/2tbhdd/coolrl-lost-cities-artifacts/verification/2026-07-04_phase1/exploiter_vs_gate3_duplicate.json
|
||||
```
|
||||
|
||||
| Metric | Value |
|
||||
| --- | ---: |
|
||||
| Games | 20,000 |
|
||||
| Exploiter win rate | 0.67115 |
|
||||
| Wilson 95% | [0.66461, 0.67763] |
|
||||
| Mean score diff | +24.1371 |
|
||||
| Score diff p05 / p50 / p95 | -50 / +21 / +110 |
|
||||
| Opened colors/game | 4.89710 |
|
||||
| Positive expeditions/game | 2.12865 |
|
||||
| Play action rate | 0.74612 |
|
||||
| Mean game length | 48.0017 |
|
||||
| Max-steps rate | 0.00000 |
|
||||
|
||||
This is now the measured exploitability baseline. It is not by itself a stop
|
||||
condition, but it reinforces that the current gate-3 checkpoint is not a
|
||||
finished robust policy.
|
||||
|
||||
## Findings
|
||||
|
||||
| Severity | Finding | Evidence | Consequence |
|
||||
| --- | --- | --- | --- |
|
||||
| High | `heuristic_cautious` is a stalling anchor, not a strong cautious baseline. | Mirror max-steps rate 89.56%, play action rate 0.0053. | Do not use it as a permanent Elo anchor without fixing/replacing it. |
|
||||
| High | Gate-3 result is contaminated by forced-end games and over-opening. | Gate-3 max-steps rate 24.07%, opened colors/game 4.98965. | The ladder PASS remains reproducible, but it should not become a league baseline. |
|
||||
| Medium | Cautious discards many immediately playable cards to the opponent. | Transcript sample: 162 such discards in 20 games. | The agent may be exploiting discard leakage rather than learning robust play. |
|
||||
| Medium | A same-budget PPO exploiter beats the gate-3 checkpoint. | Exploiter duplicate win rate 67.115%, mean diff +24.1371. | League success should reduce this number, but Phase 2 should wait for anchor repair. |
|
||||
| Low | No broad seat perspective bug found. | Static and checkpoint duplicate mirrors have exact zero mean diff. | Seat symmetry is not the likely explanation for the ladder result. |
|
||||
|
||||
## Stop Condition
|
||||
|
||||
Phase 1 did not pass cleanly. Per the work order, Phase 2 is not started.
|
||||
|
||||
Recommended next work:
|
||||
|
||||
1. Replace or repair `heuristic_balanced` and `heuristic_cautious` so mirror
|
||||
play has near-zero forced-end rate and realistic play/open rates.
|
||||
2. Add `max_steps_rate` and game-length quantiles to every gate report.
|
||||
3. Rerun the static ladder gates with the repaired anchors before initializing
|
||||
a snapshot-pool league.
|
||||
@@ -1,47 +0,0 @@
|
||||
# Research Notes Catalog
|
||||
|
||||
This file catalogs permanent research notes in `docs/research/`. `AGENTS.md` is
|
||||
the workflow document for deciding where new documentation belongs.
|
||||
|
||||
## Deep CFR
|
||||
|
||||
- [Lost Cities Deep CFR selectivity ideas](lost_cities_selectivity.md) — Consolidates selectivity failure hypotheses, diagnostics, and interventions before expensive architecture changes. (2026-05-10)
|
||||
- [Deep CFR 512x3 2000-iteration baseline analysis](deep-cfr-baseline-2000-analysis.md) — Finds advantage loss improving while evaluated average-policy quality degrades, requiring current-vs-average diagnostics. (2026-05-08)
|
||||
- [Batched and parallel evaluation in Deep CFR](deep-cfr-batched-evaluation.md) — Batching games and parallelizing opponents cuts CUDA evaluation overhead after entropy stays on device. (2026-05-08)
|
||||
- [Deep CFR evaluation profiling](deep-cfr-evaluation-profile-plan.md) — Defines runtime counters that separate policy inference, encoding, opponent logic, and engine costs. (2026-05-08)
|
||||
- [Deep CFR evaluation performance](deep-cfr-evaluation-profile.md) — Shows serial batch-size-one CUDA evaluation is slower than CPU until evaluation is batched. (2026-05-08)
|
||||
- [Deep CFR legacy parity and hyperparameter mapping](deep-cfr-legacy-experiment-reproduction.md) — Maps legacy features, architecture, and traversal knobs needed for meaningful reproduction comparisons. (2026-05-08)
|
||||
- [Deep CFR runtime: legacy vs. current implementation](deep-cfr-legacy-runtime-comparison.md) — Attributes current speedups to Cython traversal, batched evaluation, and opponent-parallel execution. (2026-05-08)
|
||||
- [Deep CFR performance optimization and scaling](deep-cfr-performance-experiments.md) — Concludes small models regress under AMP or compile; interleaved traversal is the real win. (2026-05-08)
|
||||
- [Advantage memory split performance optimization](deep-cfr-profile-advantage-memory-split.md) — Splitting advantage reservoirs by player removes linear sample filtering and stabilizes iteration time. (2026-05-08)
|
||||
- [Deep CFR performance profile: advantage training bottlenecks](deep-cfr-profile.md) — Identifies shared-memory player filtering as the original advantage-training scaling bottleneck. (2026-05-08)
|
||||
- [Deep CFR regret matching fallback and early over-opening](deep-cfr-regret-fallback-audit.md) — Shows uniform all-negative fallback frequently fires early and over-samples expedition-opening actions. (2026-05-08)
|
||||
- [Deep CFR reproducibility policy](deep-cfr-reproducibility-policy.md) — Sets debug versus research reproducibility expectations and required matched-seed reporting practice. (2026-05-08)
|
||||
- [Deep CFR reproducibility](deep-cfr-reproducibility.md) — Traces same-seed multi-worker divergence to completion-order-sensitive reservoir insertion and sampling. (2026-05-08)
|
||||
- [Deep CFR v0: architectural parity and performance gaps](deep-cfr-v0-gap-vs-coolrl.md) — Confirms functional parity while naming synchronous recursive policy inference as the scaling bottleneck. (2026-05-08)
|
||||
- [Deep CFR Cython traversal architecture](deep-cfr-v0-plan.md) — Explains mutation-based Cython traversal, chance sampling, and non-leaking information-state encoding. (2026-05-08)
|
||||
- [Option A benchmark and structural ceiling](option-a-bench-result.md) — Shows centralized inference regressed because sync-blocking recursion capped realized GPU batch size. (2026-05-08)
|
||||
- [Post-A optimization calculus](post-a-optimization-calculus.md) — Defers compile and TensorRT until model scale or evaluation density makes inference compute-bound. (2026-05-08)
|
||||
- [Deep CFR package architecture and design rationale](deep-cfr-architecture.md) — Documents the Cython/Python module split that follows traversal hot paths versus orchestration. (2026-05-07)
|
||||
- [Batched traversal inference decision](batched-traversal-inference-decision.md) — Records the initial Option A rationale and why later benchmarks redirected work toward interleaving. (2026-05-07)
|
||||
- [Deep CFR v0 subsystem coverage vs. legacy reference](deep-cfr-v0-feature-parity.md) — Establishes that remaining legacy gaps are tooling and performance, not core algorithm correctness. (2026-05-07)
|
||||
- [Opponent policy: network vs. self-play league](opponent-policy-network-divergence.md) — Explains why a live network opponent collapses while snapshot leagues preserve stationarity. (2026-05-07)
|
||||
- [Outcome-sampling MCCFR advantage target](outcome-sampling-target.md) — Defends zero unsampled-action targets as the textbook importance-weighted outcome-sampling estimator. (2026-05-07)
|
||||
- [Regret-matching all-negative fallback](regret-matching-fallback.md) — Keeps uniform as the safe default while documenting argmax-tiebreak's unsettled early-training effects. (2026-05-07)
|
||||
- [Strategy memory recording location](strategy-memory-location.md) — Concludes traverser-node strategy samples are fine for outcome sampling, but external sampling needs OpenSpiel flags. (2026-05-07)
|
||||
|
||||
## SO-ISMCTS
|
||||
|
||||
- [SO-ISMCTS BC ceiling](ismcts-bc-ceiling-2026-05-11.md) — Finds behavior cloning remains the ceiling under current search and compute budgets. (2026-05-11)
|
||||
|
||||
## Engine / Performance
|
||||
|
||||
- [Classic game port architecture](classic-port-notes.md) — Describes the standalone Cython classic engine as the stable rules layer for all consumers. (2026-05-08)
|
||||
- [Lost Cities match record v1](lost-cities-match-record-v1.md) — Defines the versioned JSONL format for complete hidden-state replay and post-game analysis. (2026-07-12)
|
||||
- [Fast engine optimization architecture](fast-engine-next-optimizations.md) — Prioritizes C-level APIs, contiguous allocation, and zero-copy extraction for high-throughput RL. (2026-05-08)
|
||||
- [Optimization sequencing](optimization_sequencing.md) — Orders runtime, traversal, model-scale, and inference optimizations to avoid invalidating experiments. (2026-05-07)
|
||||
|
||||
## Other
|
||||
|
||||
- [Julia port evaluation](julia_port_evaluation.md) — Rejects a full Julia port for now because GPU MLP inference misses the threshold. (2026-05-10)
|
||||
- [Test coverage strategy for Python and Cython modules](test-coverage-notes.md) — Recommends Python-first coverage and isolated Cython tracing to protect performance artifacts. (2026-05-08)
|
||||
@@ -54,5 +54,5 @@ The 8-worker interleaved path is more effective than a single-process CUDA path
|
||||
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py`: Implementation of AMP and training loops.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/networks.py`: `DeepCFRMLP` architecture.
|
||||
- `legacy/deep-cfr/configs/default.yaml`: Configuration for interleaved scheduler.
|
||||
- `scripts/profile_gpu_forward.py`: GPU forward pass micro-benchmarks.
|
||||
- `configs/deep_cfr/default.yaml`: Configuration for interleaved scheduler.
|
||||
- `scripts/profile_gpu_forward.py`: GPU forward pass micro-benchmarks.
|
||||
@@ -14,7 +14,7 @@ The core algorithmic components have been ported to Cython to ensure C-level per
|
||||
- `src/coolrl_lost_cities/games/classic/game.pyx:217`: `cdef class GameState` provides high-speed state mutation, legal action generation, and scoring.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:228`: `cpdef traverse` serves as the entry point for the recursive Deep CFR traversal engine.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx:253`: `cdef _traverse` implements the core recursive tree-walking logic, including traverser/opponent node handling and outcome sampling.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:406`: `def encode_info_state` generates the information-state feature vectors required for network inference.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/encoding.pyx:425`: `def encode_info_state` generates the information-state feature vectors required for network inference.
|
||||
- `src/coolrl_lost_cities/games/classic/deep_cfr/cfr_math.pyx`: Contains optimized regret-matching and advantage calculation primitives.
|
||||
|
||||
## Analysis
|
||||
@@ -39,4 +39,4 @@ Until this transition is made, optimization efforts should focus on reducing the
|
||||
|
||||
- `docs/archive/deep-cfr-v0-gap-vs-coolrl.md`: Original status and gap analysis.
|
||||
- `docs/research/deep-cfr-v0-feature-parity.md`: Detailed subsystem coverage report.
|
||||
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
|
||||
- `docs/research/batched-traversal-inference-decision.md`: Architectural decision record for the next-generation inference server.
|
||||
@@ -1,169 +0,0 @@
|
||||
# SO-ISMCTS BC Ceiling — 2026-05-11 Autonomous Session
|
||||
|
||||
**Last verified:** 2026-05-11, commit `cba6cae` (branch `autonomous/trap-exploration`)
|
||||
|
||||
## Short answer
|
||||
|
||||
Under our current compute budget (1 GPU, 12 CPU cores, 50 MCTS sims/move,
|
||||
768x4 MLP), **behavior-cloning the heuristic-balanced bot is the ceiling**.
|
||||
Across 13 self-play training variants, no run cleared the BC baseline of
|
||||
21/100 wins vs `heuristic-cautious` in 100-game evaluation. Every variant
|
||||
either preserved BC (KL anchor, mirror-descent target) or regressed toward
|
||||
catastrophic forgetting (naive finetune, high-fraction mixed opponent).
|
||||
|
||||
The single largest improvement of the session came from PUCT Q-value
|
||||
normalization at the *search* level, not from any learning change.
|
||||
|
||||
## Headline numbers (vs heuristic-cautious, 100 games, n_sims = 16)
|
||||
|
||||
| Run | Setup | W/100 | Notes |
|
||||
|-------------------|------------------------------------------------|------:|-------|
|
||||
| BC pretrain | 5k heuristic-vs-heuristic games, 20 epochs CE+MSE | 21 | baseline |
|
||||
| C9 naive finetune | BC + plain self-play (no regularizer) | 0 | catastrophic forgetting |
|
||||
| C10 KL β=1.0 | BC + self-play + KL(current ‖ BC) | ~21 | preserved BC, no improvement |
|
||||
| C11 KL β=0.3 | weaker anchor | ~21 | preserved BC, no improvement |
|
||||
| C12 mirror desc. | target = softmax(α log π_mcts + (1−α) log π_BC) | 19 | preserved BC, no improvement |
|
||||
| C13 mixed=0.5 | 50 % games vs heuristic-balanced, opponent-aware MCTS, no KL | 0 | forgetting (worse than naive) |
|
||||
| C14 mixed=0.2 | mixed-opponent + opponent-aware + KL β=1.0 | 17 | preserved BC, no improvement |
|
||||
|
||||
CIs (Wilson 95 %) overlap across all "preserved BC" rows; the 17–22 band
|
||||
is statistically indistinguishable from the BC baseline.
|
||||
|
||||
## What actually moved the needle: PUCT Q normalization
|
||||
|
||||
`mcts.pyx _select_action` previously used the raw score-unit Q:
|
||||
|
||||
```
|
||||
score = q_eff + c_puct * prior * sqrt(N) / (1 + n)
|
||||
```
|
||||
|
||||
With `value_scale = 100` (Lost Cities score units), a single backup could
|
||||
swing `q_eff` by ±100, while the exploration bonus is ~1–10. A noisy value
|
||||
at the root permanently buried low-prior actions before they could be
|
||||
explored.
|
||||
|
||||
Fix (`b9fc569`): divide Q by `q_scale` (defaults to 100) before scoring:
|
||||
|
||||
```
|
||||
score = q_eff / q_scale + c_puct * prior * sqrt(N) / (1 + n)
|
||||
```
|
||||
|
||||
Replaying the exact same BC checkpoint with this fix took win rate vs
|
||||
heuristic-cautious from **5/100 → 21/100** — a 4× improvement from a
|
||||
~10-line search change, with no retraining. Worth holding onto as the
|
||||
load-bearing finding of the session.
|
||||
|
||||
## Hypotheses we negated
|
||||
|
||||
1. **Symmetric self-play eventually escapes the weak fixed point.**
|
||||
Random-init + KL-free self-play ran for 100s of iterations across
|
||||
C1–C8 without exceeding the noise floor (0–25 wins, all CIs overlap
|
||||
each other and zero).
|
||||
|
||||
2. **Mixed-opponent self-play (Codex top pick) breaks the weak
|
||||
equilibrium.** With opponent-aware MCTS so the search distribution
|
||||
reflects the real opponent (per Codex's "pitfall" warning), C13
|
||||
regressed to 0/100. The training signal from vs-heuristic games is
|
||||
structurally negative — BC cannot beat the heuristic, so every mixed
|
||||
sample is a loss, and the gradient labels every BC action as bad.
|
||||
C14 cut the fraction to 0.2 and added a strong KL anchor (β = 1.0),
|
||||
which preserved BC but did not lift it.
|
||||
|
||||
3. **Deeper search compensates for weak learning.** Increasing
|
||||
`n_simulations` from 50 → 200 on the BC checkpoint *reduced* wins
|
||||
vs `heuristic-balanced` from 28/64 → 14/64 in earlier probing.
|
||||
Deeper search amplifies the network's preferences, including its
|
||||
weaker ones, without supplying new information.
|
||||
|
||||
4. **A different regularizer would let self-play improve on BC.**
|
||||
KL anchor (β ∈ {0.3, 1.0}) and mirror-descent target mixing (α
|
||||
annealed 0.3 → 0.8) both kept the network glued to BC. Neither
|
||||
supplied a positive gradient to walk away from it.
|
||||
|
||||
## Why BC is the ceiling — the mechanism
|
||||
|
||||
Self-play seeded from a strong heuristic faces a structural trap:
|
||||
|
||||
- BC has internalized the heuristic. Two BC copies playing each other
|
||||
produce a near-symmetric outcome distribution; the visit counts at
|
||||
most nodes give little policy-improvement signal beyond what BC
|
||||
already encodes.
|
||||
- Against the real heuristic, BC loses systematically (the heuristic
|
||||
beats its own clone in approx. 79 % of games at our scale). The
|
||||
resulting training signal is uniformly negative; learning that
|
||||
signal pushes the policy *away* from BC without pointing anywhere
|
||||
productive.
|
||||
- With 50 MCTS sims/move on a 768x4 network, the search cannot
|
||||
reliably *find* moves that beat the heuristic. So the only way out
|
||||
of the trap — discovering a positive improvement direction —
|
||||
is closed by the search-depth budget.
|
||||
|
||||
The result is consistent with the standard SO-ISMCTS picture: π_weak
|
||||
(the symmetric weak fixed point) sits at roughly BC strength, π_Nash
|
||||
is unreachable at this compute, and every variant we tried collapses
|
||||
onto π_weak.
|
||||
|
||||
## Things left as configurable dials (no behavior change at defaults)
|
||||
|
||||
The `autonomous/trap-exploration` branch leaves the following in place
|
||||
for future runs with more compute:
|
||||
|
||||
- `MctsConfig.q_scale` — PUCT Q normalization (defaults to 100, keep).
|
||||
- `MctsConfig.root_dirichlet_alpha / epsilon` — AlphaZero exploration noise.
|
||||
- `MctsConfig.opponent_aware_search` — when true, MCTS treats the
|
||||
opponent seat as an external bot (skips tree expansion on opponent
|
||||
turns, traverser-centered values).
|
||||
- `TrainingConfig.mixed_opponent_fraction` — 0 disables (pure self-play).
|
||||
- `TrainingConfig.mixed_opponent_bot` — bot name from
|
||||
`coolrl_lost_cities.games.classic.bots.registry`.
|
||||
- `TrainingConfig.kl_anchor_ckpt` / `kl_anchor_beta` — frozen reference
|
||||
network for `KL(current ‖ ref)` regularization.
|
||||
- `TrainingConfig.md_target_ref_ckpt` / `md_target_alpha_*` — mirror-
|
||||
descent policy target with annealed mixing.
|
||||
- `lost-cities-ismcts pretrain` — heuristic behavior cloning subcommand.
|
||||
- `lost-cities-ismcts eval --ckpt … --n-sims N --games N --device cpu`
|
||||
— standalone evaluator with Wilson CIs (`eval_checkpoint.py`).
|
||||
|
||||
## What would be worth trying with more compute
|
||||
|
||||
Not implemented here. These are the directions that the mechanism above
|
||||
*does not rule out*:
|
||||
|
||||
- **Deeper search at training time** (n_sims ≫ 200, e.g. 800–1600).
|
||||
Enough simulations should eventually surface a heuristic-beating
|
||||
action somewhere in the search tree; that's a positive gradient.
|
||||
- **Population training with frozen snapshots.** Periodically snapshot
|
||||
the trainer and route 10–20 % of self-play games against the snapshot
|
||||
pool. Combined with opponent-aware search, this gives a stationary
|
||||
diverse-opponent gradient without the all-negative-signal problem of
|
||||
pure heuristic mixing.
|
||||
- **Value-weighted replay.** Prioritize high-error samples in the
|
||||
buffer so the value head sees the cases where it disagrees with the
|
||||
search rollout.
|
||||
- **Larger / better-shaped networks.** 768x4 MLP may simply lack the
|
||||
capacity to represent the conjunctions Lost Cities needs (color ×
|
||||
expedition × hand composition). Attention or factored heads could be
|
||||
worth probing.
|
||||
|
||||
## Code references
|
||||
|
||||
- Search-side: `src/coolrl_lost_cities/games/classic/ismcts/mcts.pyx`
|
||||
(`_select_action`, `prepare_simulation`, `_expand_with_prior`).
|
||||
- Python parity: `src/coolrl_lost_cities/games/classic/ismcts/mcts.py`.
|
||||
- Mixed-opponent / opponent-aware wiring:
|
||||
`src/coolrl_lost_cities/games/classic/ismcts/interleaved_self_play.py`.
|
||||
- Regularization (KL anchor, mirror-descent) and metrics:
|
||||
`src/coolrl_lost_cities/games/classic/ismcts/trainer.py`.
|
||||
- BC pretrain: `src/coolrl_lost_cities/games/classic/ismcts/pretrain.py`.
|
||||
- Eval CLI with Wilson CIs:
|
||||
`src/coolrl_lost_cities/games/classic/ismcts/eval_checkpoint.py`.
|
||||
- BC checkpoint (load with `--resume-from`):
|
||||
`runs/pretrain/heuristic_balanced_5kg_20ep.pt` (5 k games, 20 epochs).
|
||||
|
||||
## Related memory
|
||||
|
||||
- `opponent-policy-network-divergence.md` — the Deep CFR analogue:
|
||||
using the live network as its own opponent breaks stationarity and
|
||||
diverges. The SO-ISMCTS picture here is the same family of failure:
|
||||
bootstrapping from oneself does not provide a positive learning
|
||||
signal.
|
||||
@@ -1,42 +0,0 @@
|
||||
# Lost Cities Match Record v1
|
||||
|
||||
Last verified: 2026-07-12
|
||||
|
||||
`coolrl.lost-cities.match.v1` is the canonical portable record for one complete
|
||||
or in-progress classic Lost Cities match. Files use UTF-8 JSON Lines (`.jsonl`)
|
||||
so metadata can be inspected without loading the full replay and steps can be
|
||||
streamed in order.
|
||||
|
||||
## Row 1: metadata
|
||||
|
||||
The first row has `type: "metadata"` and `format:
|
||||
"coolrl.lost-cities.match.v1"`. It records creation time, classic rules,
|
||||
initial seed, human seat, opponent identity, and completion status. Producers
|
||||
may add fields; v1 readers must ignore unknown metadata fields.
|
||||
|
||||
## Rows 2+: steps
|
||||
|
||||
Every remaining row has `type: "step"` and a contiguous zero-based `index`.
|
||||
Step 0 is the initial position and has null `actor`, `phase_before`, and
|
||||
`action_id`. Later steps contain:
|
||||
|
||||
- `actor`: player that took the action (`0` or `1`).
|
||||
- `phase_before`: `card` or `draw`.
|
||||
- `action_id`: the classic engine unified action ID.
|
||||
- `state`: complete `GameState.to_snapshot()` output after the action.
|
||||
- `public_hands`: cards known to be in each player's hand because they were
|
||||
drawn from a discard pile. Entries use `{color, value, count}`; wagers have
|
||||
value 0 and numbered cards use their printed value.
|
||||
|
||||
Full state snapshots intentionally include hidden hands and deck order. A match
|
||||
record is therefore suitable for post-game analysis and deterministic replay,
|
||||
but must not be exposed to a player during a live match.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Readers must reject an unknown `format` value rather than guessing. New
|
||||
optional fields may be added within v1. Any incompatible change to action IDs,
|
||||
state encoding, or required row semantics requires `v2`.
|
||||
|
||||
The reference reader, writer, and validator are in
|
||||
`src/coolrl_lost_cities/games/classic/match_record.py`.
|
||||
@@ -6,10 +6,10 @@
|
||||
|
||||
| Lever | 어디서 nail되는지 | 현재 상태 |
|
||||
| --- | --- | --- |
|
||||
| **Model size growth** (hidden ≥ 1024 / layers ≥ 6) | `docs/plans/archive/model_size_experiment.md` | 인프라 미설치 |
|
||||
| **Model size growth** (hidden ≥ 1024 / layers ≥ 6) | `docs/plans/model_size_experiment.md` | 인프라 미설치 |
|
||||
| **Option B** (per-worker interleaved traversal) | plan 미작성 | 미시작 |
|
||||
| **AMP** trainer | `docs/plans/archive/amp_trainer.md` (구현 됨, default off) | 모델 키운 후 재측정 |
|
||||
| **torch.compile** trainer | `docs/plans/archive/torch_compile.md` | 모델 키운 후 재측정 |
|
||||
| **torch.compile** trainer | `docs/plans/torch_compile.md` | 모델 키운 후 재측정 |
|
||||
| **TensorRT** inference | plan 미작성 | 모델 키운 + eval dense 시점 |
|
||||
| **Option A re-enable** | 코드 있음 (default off) | 모델 키운 후 또는 Option B 후 |
|
||||
| **Julia port** | `docs/research/julia_port_evaluation.md` | Torch.jl 결과 대기 중. Flux FAIL. |
|
||||
|
||||
@@ -19,7 +19,7 @@ with torch.inference_mode():
|
||||
advantages = networks[player](x).squeeze(0).detach().cpu().numpy().astype(np.float32)
|
||||
```
|
||||
|
||||
When `traversal.inference_backend` is set to `server` in `legacy/deep-cfr/configs/default.yaml`, the `networks[player]` call is intercepted by a `NetworkProxy` (instantiated in `workers.py`, around line 91). This proxy posts a request to the `InferenceServer` and blocks until a response is received via a per-slot event.
|
||||
When `traversal.inference_backend` is set to `server` in `configs/deep_cfr/default.yaml`, the `networks[player]` call is intercepted by a `NetworkProxy` (instantiated in `workers.py`, around line 91). This proxy posts a request to the `InferenceServer` and blocks until a response is received via a per-slot event.
|
||||
|
||||
The server's batching logic in `src/coolrl_lost_cities/games/classic/deep_cfr/inference_server.py` (around line 221) reports the realized batch size:
|
||||
|
||||
@@ -46,7 +46,7 @@ The "structural ceiling" is that `batch_window_us` and `max_batch` tuning cannot
|
||||
|
||||
## Practical implication
|
||||
|
||||
Option A is deferred for the current small MLP models (512x3). The `local` backend remains the default in `legacy/deep-cfr/configs/default.yaml`.
|
||||
Option A is deferred for the current small MLP models (512x3). The `local` backend remains the default in `configs/deep_cfr/default.yaml`.
|
||||
|
||||
To unlock the projected GPU gains, the traversal must be restructured to drive batch sizes up. This leads to two primary paths:
|
||||
1. **Option B (Interleaved Traversal):** Refactor the Cython traversal into a state machine that can advance multiple traversals concurrently per worker. Each worker would suspend at a policy call, batch its own requests, and resume continuations once the results return.
|
||||
|
||||
@@ -9,7 +9,7 @@ Why do architectural optimizations like `torch.compile` and TensorRT integration
|
||||
|
||||
## Code reference
|
||||
|
||||
The historical baseline configuration is defined in `legacy/deep-cfr/configs/default.yaml`:
|
||||
The current baseline configuration is defined in `configs/deep_cfr/default.yaml`:
|
||||
|
||||
```yaml
|
||||
network:
|
||||
@@ -51,4 +51,4 @@ Avoid premature optimization with `torch.compile` or TensorRT on the current sma
|
||||
|
||||
- `docs/archive/deep-cfr-performance-experiments-2026-05-07.md` (Small-model regression data)
|
||||
- `docs/archive/option-a-bench-result-2026-05-07.md` (Batched traversal benchmarks)
|
||||
- `docs/performance.md` (Current runtime bottleneck profile)
|
||||
- `docs/performance.md` (Current runtime bottleneck profile)
|
||||
@@ -46,7 +46,7 @@ sampling-mode branch. The `info_state` is computed by `_policy(state,
|
||||
player, ...)` for the *current acting player*, which is the right thing in
|
||||
both conventions.
|
||||
|
||||
`legacy/deep-cfr/configs/default.yaml` sets:
|
||||
`configs/deep_cfr/default.yaml` sets:
|
||||
|
||||
```yaml
|
||||
store_strategy_on_traverser_nodes: true
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Option B Interleaved Traversal Prototype
|
||||
|
||||
Experiment-only prototype for `docs/plans/archive/option_b_interleaved_traversal.md`.
|
||||
Experiment-only prototype for `docs/plans/option_b_interleaved_traversal.md`.
|
||||
It does not wire into the trainer and does not replace the production Cython
|
||||
recursive traversal path.
|
||||
|
||||
@@ -37,7 +37,7 @@ uv run python experiments/option_b_interleaved_traversal/prototype_interleaved.p
|
||||
--output experiments/option_b_interleaved_traversal/results_cuda.json
|
||||
```
|
||||
|
||||
2026-05-07 results, `legacy/deep-cfr/configs/default.yaml`, RTX 3090 host:
|
||||
2026-05-07 results, `configs/deep_cfr/default.yaml`, RTX 3090 host:
|
||||
|
||||
| Device | Mode | total s | forward s | scheduler s | batch mean | batch max | speedup |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
|
||||
@@ -650,7 +650,7 @@ def _build_proto_config(cfg: Any, max_depth: int | None, max_nodes: int | None)
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", default="legacy/deep-cfr/configs/default.yaml")
|
||||
parser.add_argument("--config", default="configs/deep_cfr/default.yaml")
|
||||
parser.add_argument("--device", default="cpu")
|
||||
parser.add_argument("--traversals", type=int, default=64)
|
||||
parser.add_argument("--interleave-width", type=int, default=32)
|
||||
|
||||
@@ -30,7 +30,7 @@ uv run python experiments/traversal_policy_boundary/bench_policy_boundary.py \
|
||||
--output experiments/traversal_policy_boundary/results_cuda.json
|
||||
```
|
||||
|
||||
2026-05-07 results, `legacy/deep-cfr/configs/default.yaml`, RTX 3090 host:
|
||||
2026-05-07 results, `configs/deep_cfr/default.yaml`, RTX 3090 host:
|
||||
|
||||
| Device | Component | Median us/call | p95 us/call |
|
||||
| --- | --- | ---: | ---: |
|
||||
|
||||
@@ -321,7 +321,7 @@ def _print_table(result: dict[str, Any]) -> None:
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", default="legacy/deep-cfr/configs/default.yaml")
|
||||
parser.add_argument("--config", default="configs/deep_cfr/default.yaml")
|
||||
parser.add_argument("--device", default="cpu")
|
||||
parser.add_argument("--traversals", type=int, default=32)
|
||||
parser.add_argument("--runs", type=int, default=5)
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
# Deep CFR archive
|
||||
|
||||
These YAML files are retained only to reproduce historical Deep CFR work. The
|
||||
supported training stack is JAX PPO; start with `configs/jax_ppo/` instead.
|
||||
@@ -1,4 +0,0 @@
|
||||
# ISMCTS archive
|
||||
|
||||
These YAML files are retained only to reproduce historical ISMCTS work. The
|
||||
supported training stack is JAX PPO; start with `configs/jax_ppo/` instead.
|
||||
@@ -1,31 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Autonomous cycle eval helper.
|
||||
# Usage: ./autonomous_cycle_eval.sh <run-prefix> [extra eval args...]
|
||||
# Finds latest run matching prefix, runs eval --ckpt latest.pt with 30 games,
|
||||
# and reports: timeouts, natural-end wins per opponent.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PREFIX="${1:-}"
|
||||
shift || true
|
||||
|
||||
if [ -z "$PREFIX" ]; then
|
||||
echo "usage: $0 <run-prefix> [extra eval args...]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RUN=$(ls -td runs/*${PREFIX}* 2>/dev/null | head -1)
|
||||
if [ -z "$RUN" ]; then
|
||||
echo "no run matching ${PREFIX}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
CKPT="$RUN/latest.pt"
|
||||
if [ ! -f "$CKPT" ]; then
|
||||
echo "no checkpoint at $CKPT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "=== eval $CKPT ==="
|
||||
uv run python -m coolrl_lost_cities.games.classic.ismcts.cli \
|
||||
eval --ckpt "$CKPT" --games 30 --verbose "$@" 2>&1
|
||||
+5
-10
@@ -1,7 +1,7 @@
|
||||
[project]
|
||||
name = "coolrl-lost-cities"
|
||||
version = "0.1.0"
|
||||
description = "JAX PPO training and browser play for Lost Cities"
|
||||
description = "Focused Lost Cities classic game extraction"
|
||||
readme = "README.md"
|
||||
authors = [
|
||||
{ name = "정시원", email = "sebastianrcnt@gmail.com" }
|
||||
@@ -13,16 +13,11 @@ dependencies = [
|
||||
"pydantic>=2.7",
|
||||
"PyYAML>=6.0",
|
||||
"torch>=2.3.0",
|
||||
"jax>=0.10.2",
|
||||
"chex>=0.1.92",
|
||||
"flax>=0.12.7",
|
||||
"optax>=0.2.8",
|
||||
"orbax-checkpoint>=0.12.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
gui = [
|
||||
"pygame-ce>=2.5.3",
|
||||
"pygame>=2.6.1",
|
||||
"pygame-gui>=0.6.14",
|
||||
]
|
||||
wandb = [
|
||||
@@ -33,8 +28,8 @@ wandb = [
|
||||
lost-cities-classic = "coolrl_lost_cities.games.classic:main"
|
||||
lost-cities-eval = "coolrl_lost_cities.games.classic.evaluation:main"
|
||||
lost-cities-classic-gui = "coolrl_lost_cities.games.classic.pygame_pvp:main"
|
||||
lost-cities-play = "coolrl_lost_cities.games.classic.pygame_table:main"
|
||||
lost-cities-jax-ppo = "lost_cities_jax.ppo_cli:main"
|
||||
lost-cities-deep-cfr = "coolrl_lost_cities.games.classic.deep_cfr.cli:main"
|
||||
lost-cities-ismcts = "coolrl_lost_cities.games.classic.ismcts.cli:main"
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
@@ -50,7 +45,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
where = ["src"]
|
||||
include = ["coolrl_lost_cities*", "lost_cities_jax*"]
|
||||
include = ["coolrl_lost_cities*"]
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
"coolrl_lost_cities.games.classic" = [
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""Reference implementations used by tests."""
|
||||
@@ -1,289 +0,0 @@
|
||||
"""Independent pure-Python Lost Cities rules reference.
|
||||
|
||||
This module intentionally uses ordinary Python containers rather than the JAX
|
||||
state representation. It is deterministic under an explicit ``deck_order`` and
|
||||
the shared flat action encoding.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from bisect import insort
|
||||
from dataclasses import dataclass
|
||||
from random import Random
|
||||
|
||||
N_PLAYERS = 2
|
||||
N_COLORS = 5
|
||||
CARDS_PER_COLOR = 12
|
||||
N_CARDS = N_COLORS * CARDS_PER_COLOR
|
||||
HAND_SIZE = 8
|
||||
INITIAL_DEAL = N_PLAYERS * HAND_SIZE
|
||||
MAX_STEPS = 400
|
||||
N_ACTIONS = HAND_SIZE * 2 * 6
|
||||
|
||||
LOC_DECK = 0
|
||||
LOC_P0_HAND = 1
|
||||
LOC_P1_HAND = 2
|
||||
LOC_P0_BOARD = 3
|
||||
LOC_P1_BOARD = 4
|
||||
LOC_DISCARD = 5
|
||||
|
||||
NO_CARD = -1
|
||||
PLAY = 0
|
||||
DISCARD = 1
|
||||
DRAW_DECK = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class RefState:
|
||||
deck_order: list[int]
|
||||
draw_ptr: int
|
||||
card_loc: list[int]
|
||||
hand_public: list[bool]
|
||||
hands: list[list[int]]
|
||||
board: list[list[list[int]]]
|
||||
col_top: list[list[int]]
|
||||
piles: list[list[int]]
|
||||
to_move: int
|
||||
just_discarded: int
|
||||
step_count: int
|
||||
done: bool
|
||||
|
||||
|
||||
def reset(seed: int | None = None) -> RefState:
|
||||
rng = Random(seed)
|
||||
order = list(range(N_CARDS))
|
||||
rng.shuffle(order)
|
||||
return reset_from_order(order)
|
||||
|
||||
|
||||
def reset_from_order(deck_order: list[int] | tuple[int, ...]) -> RefState:
|
||||
order = [int(card) for card in deck_order]
|
||||
card_loc = [LOC_DECK] * N_CARDS
|
||||
p0_hand = sorted(order[:HAND_SIZE])
|
||||
p1_hand = sorted(order[HAND_SIZE:INITIAL_DEAL])
|
||||
for card in p0_hand:
|
||||
card_loc[card] = LOC_P0_HAND
|
||||
for card in p1_hand:
|
||||
card_loc[card] = LOC_P1_HAND
|
||||
return RefState(
|
||||
deck_order=order,
|
||||
draw_ptr=INITIAL_DEAL,
|
||||
card_loc=card_loc,
|
||||
hand_public=[False] * N_CARDS,
|
||||
hands=[p0_hand, p1_hand],
|
||||
board=[[[] for _ in range(N_COLORS)] for _ in range(N_PLAYERS)],
|
||||
col_top=[[0 for _ in range(N_COLORS)] for _ in range(N_PLAYERS)],
|
||||
piles=[[] for _ in range(N_COLORS)],
|
||||
to_move=0,
|
||||
just_discarded=NO_CARD,
|
||||
step_count=0,
|
||||
done=False,
|
||||
)
|
||||
|
||||
|
||||
def clone_state(state: RefState) -> RefState:
|
||||
return RefState(
|
||||
deck_order=list(state.deck_order),
|
||||
draw_ptr=state.draw_ptr,
|
||||
card_loc=list(state.card_loc),
|
||||
hand_public=list(state.hand_public),
|
||||
hands=[list(hand) for hand in state.hands],
|
||||
board=[[list(col) for col in player] for player in state.board],
|
||||
col_top=[list(player) for player in state.col_top],
|
||||
piles=[list(pile) for pile in state.piles],
|
||||
to_move=state.to_move,
|
||||
just_discarded=state.just_discarded,
|
||||
step_count=state.step_count,
|
||||
done=state.done,
|
||||
)
|
||||
|
||||
|
||||
def decode_action(action: int) -> tuple[int, int, int]:
|
||||
action = int(action)
|
||||
hand_slot = action // 12
|
||||
rem = action % 12
|
||||
place_type = rem // 6
|
||||
draw_source = rem % 6
|
||||
return hand_slot, place_type, draw_source
|
||||
|
||||
|
||||
def hand_cards(state: RefState, player: int | None = None) -> list[int]:
|
||||
if player is None:
|
||||
player = state.to_move
|
||||
return list(state.hands[player])
|
||||
|
||||
|
||||
def legal_action_mask(state: RefState) -> list[bool]:
|
||||
bits = legal_action_bits(state)
|
||||
return [bool(bits & (1 << action)) for action in range(N_ACTIONS)]
|
||||
|
||||
|
||||
def legal_action_bits(state: RefState) -> int:
|
||||
bits = 0
|
||||
if state.done:
|
||||
return bits
|
||||
|
||||
player = state.to_move
|
||||
hand = state.hands[player]
|
||||
draw_bits = 1
|
||||
for color, pile in enumerate(state.piles):
|
||||
if pile and pile[-1] != state.just_discarded:
|
||||
draw_bits |= 1 << (color + 1)
|
||||
|
||||
for hand_slot, card in enumerate(hand[:HAND_SIZE]):
|
||||
base = hand_slot * 12
|
||||
if can_play_card(state, player, card):
|
||||
bits |= draw_bits << base
|
||||
|
||||
color = card_color(card)
|
||||
discard_bits = draw_bits & ~(1 << (color + 1))
|
||||
bits |= discard_bits << (base + 6)
|
||||
return bits
|
||||
|
||||
|
||||
def nth_legal_action(bits: int, index: int) -> int:
|
||||
"""Return the ``index``-th set action bit from low to high."""
|
||||
|
||||
remaining = int(index)
|
||||
action = 0
|
||||
while bits:
|
||||
if bits & 1:
|
||||
if remaining == 0:
|
||||
return action
|
||||
remaining -= 1
|
||||
action += 1
|
||||
bits >>= 1
|
||||
raise IndexError(index)
|
||||
|
||||
|
||||
def step(
|
||||
state: RefState, action: int, *, validate: bool = True
|
||||
) -> tuple[RefState, list[float], bool]:
|
||||
if state.done or action < 0 or action >= N_ACTIONS:
|
||||
return clone_state(state), [0.0, 0.0], state.done
|
||||
if validate and not legal_action_mask(state)[action]:
|
||||
return clone_state(state), [0.0, 0.0], state.done
|
||||
|
||||
next_state = clone_state(state)
|
||||
_, reward, done = step_in_place(next_state, action, validate=False)
|
||||
return next_state, reward, done
|
||||
|
||||
|
||||
def step_in_place(
|
||||
state: RefState, action: int, *, validate: bool = True
|
||||
) -> tuple[RefState, list[float], bool]:
|
||||
if state.done or action < 0 or action >= N_ACTIONS:
|
||||
return state, [0.0, 0.0], state.done
|
||||
if validate and not legal_action_mask(state)[action]:
|
||||
return state, [0.0, 0.0], state.done
|
||||
|
||||
player = state.to_move
|
||||
hand_slot, place_type, draw_source = decode_action(action)
|
||||
hand = state.hands[player]
|
||||
card = hand.pop(hand_slot)
|
||||
color = card_color(card)
|
||||
|
||||
state.hand_public[card] = False
|
||||
if place_type == PLAY:
|
||||
state.board[player][color].append(card)
|
||||
if not is_handshake(card):
|
||||
state.col_top[player][color] = rank(card)
|
||||
state.card_loc[card] = LOC_P0_BOARD + player
|
||||
else:
|
||||
state.piles[color].append(card)
|
||||
state.card_loc[card] = LOC_DISCARD
|
||||
|
||||
if draw_source == DRAW_DECK:
|
||||
drawn = state.deck_order[state.draw_ptr]
|
||||
state.draw_ptr += 1
|
||||
public = False
|
||||
else:
|
||||
src = draw_source - 1
|
||||
drawn = state.piles[src].pop()
|
||||
public = True
|
||||
|
||||
insort(hand, drawn)
|
||||
state.card_loc[drawn] = LOC_P0_HAND + player
|
||||
state.hand_public[drawn] = public
|
||||
state.just_discarded = NO_CARD
|
||||
state.step_count += 1
|
||||
state.done = (draw_source == DRAW_DECK and state.draw_ptr >= N_CARDS) or (
|
||||
state.step_count >= MAX_STEPS
|
||||
)
|
||||
state.to_move = 1 - player
|
||||
|
||||
reward = board_score(state) if state.done else [0.0, 0.0]
|
||||
return state, reward, state.done
|
||||
|
||||
|
||||
def can_play_card(state: RefState, player: int, card: int) -> bool:
|
||||
top_rank = state.col_top[player][card_color(card)]
|
||||
if is_handshake(card):
|
||||
return top_rank == 0
|
||||
return rank(card) > top_rank
|
||||
|
||||
|
||||
def board_score(state: RefState) -> list[float]:
|
||||
return [float(sum(score_column(column) for column in player)) for player in state.board]
|
||||
|
||||
|
||||
def score(state: RefState) -> list[float]:
|
||||
return board_score(state)
|
||||
|
||||
|
||||
def score_column(column: list[int]) -> int:
|
||||
if not column:
|
||||
return 0
|
||||
handshakes = sum(1 for card in column if is_handshake(card))
|
||||
rank_sum = sum(rank(card) for card in column if not is_handshake(card))
|
||||
value = (rank_sum - 20) * (1 + handshakes)
|
||||
if len(column) >= 8:
|
||||
value += 20
|
||||
return value
|
||||
|
||||
|
||||
def card_color(card: int) -> int:
|
||||
return int(card) // CARDS_PER_COLOR
|
||||
|
||||
|
||||
def card_slot(card: int) -> int:
|
||||
return int(card) % CARDS_PER_COLOR
|
||||
|
||||
|
||||
def is_handshake(card: int) -> bool:
|
||||
return card_slot(card) < 3
|
||||
|
||||
|
||||
def rank(card: int) -> int:
|
||||
return card_slot(card) - 1
|
||||
|
||||
|
||||
def _can_draw_after_place(state: RefState, card: int, place_type: int, draw_source: int) -> bool:
|
||||
if draw_source == DRAW_DECK:
|
||||
return True
|
||||
src = draw_source - 1
|
||||
same_discard_pile = place_type == DISCARD and card_color(card) == src
|
||||
after_len = len(state.piles[src]) + int(same_discard_pile)
|
||||
if after_len == 0:
|
||||
return False
|
||||
after_top = card if same_discard_pile else state.piles[src][-1]
|
||||
just_discarded = card if place_type == DISCARD else state.just_discarded
|
||||
return after_top != just_discarded
|
||||
|
||||
|
||||
__all__ = [
|
||||
"N_ACTIONS",
|
||||
"RefState",
|
||||
"board_score",
|
||||
"clone_state",
|
||||
"decode_action",
|
||||
"hand_cards",
|
||||
"legal_action_bits",
|
||||
"legal_action_mask",
|
||||
"nth_legal_action",
|
||||
"reset",
|
||||
"reset_from_order",
|
||||
"score",
|
||||
"step",
|
||||
"step_in_place",
|
||||
]
|
||||
@@ -1,148 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""What is each piece of the match stack worth?
|
||||
|
||||
Train one variant per switch at identical compute, then play each against the full
|
||||
stack in duplicate matches -- same three deals, same coins, both seats -- so deal
|
||||
luck cancels and only the policy difference is left.
|
||||
|
||||
The variants do not share an observation shape (dropping the privileged critic or
|
||||
the match features changes the input dims), so this carries its own head-to-head
|
||||
rather than reusing match_eval's, which assumes one architecture.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.match import (
|
||||
match_legal_action_mask,
|
||||
match_reset_from,
|
||||
match_score,
|
||||
match_step,
|
||||
)
|
||||
from lost_cities_jax.match_eval import MATCH_SCAN_STEPS, _wilson, match_bank
|
||||
from lost_cities_jax.match_ppo import (
|
||||
Ablation,
|
||||
MatchActorCritic,
|
||||
_seat_views,
|
||||
create_match_train_state,
|
||||
match_train,
|
||||
)
|
||||
from lost_cities_jax.ppo import load_config, mask_logits, restore_checkpoint
|
||||
|
||||
VARIANTS = [
|
||||
Ablation(privileged_critic=False),
|
||||
Ablation(both_seats=False),
|
||||
Ablation(match_obs=False),
|
||||
]
|
||||
MATCHES = 4096
|
||||
FULL_CHECKPOINT = Path("runs/jax-ppo-match/2026-07-15_030152_match-linear/latest")
|
||||
|
||||
|
||||
def _load(cfg, checkpoint: Path, ablation: Ablation):
|
||||
state = create_match_train_state(cfg, jax.random.PRNGKey(0), ablation)
|
||||
return restore_checkpoint(checkpoint, state).params
|
||||
|
||||
|
||||
def _existing_run(label: str) -> Path | None:
|
||||
runs = sorted(Path("runs/jax-ppo-match").glob(f"*ablate-{label}"))
|
||||
for run in reversed(runs):
|
||||
if (run / "latest").exists():
|
||||
return run
|
||||
return None
|
||||
|
||||
|
||||
def duel(cfg, params_a, ablation_a, params_b, ablation_b, *, matches: int) -> dict:
|
||||
"""A vs B over duplicate matches. Each side sees the world its own way."""
|
||||
|
||||
decks, coins = match_bank(20260719, matches)
|
||||
model = MatchActorCritic(cfg.network.hidden_size, cfg.network.num_layers)
|
||||
seats = jnp.arange(2, dtype=jnp.int32)
|
||||
|
||||
@jax.jit
|
||||
def run(env, a_seat):
|
||||
def body(carry, _):
|
||||
env, _unused = carry
|
||||
to_move = env.round.to_move.astype(jnp.int32)
|
||||
mask = jax.vmap(match_legal_action_mask)(env)
|
||||
|
||||
def act(params, ablation):
|
||||
obs, critic = _seat_views(env, seats, ablation)
|
||||
idx = to_move[None, :, None]
|
||||
seat_obs = jnp.take_along_axis(obs, idx, axis=0)[0]
|
||||
seat_critic = jnp.take_along_axis(critic, idx, axis=0)[0]
|
||||
logits, _ = model.apply(params, seat_obs, seat_critic)
|
||||
return jnp.argmax(mask_logits(logits, mask), axis=-1).astype(jnp.int32)
|
||||
|
||||
action = jnp.where(
|
||||
to_move == a_seat, act(params_a, ablation_a), act(params_b, ablation_b)
|
||||
)
|
||||
env, _, _ = jax.vmap(match_step, in_axes=(0, 0))(env, action)
|
||||
return (env, _unused), None
|
||||
|
||||
(env, _), _ = jax.lax.scan(body, (env, jnp.int32(0)), xs=None, length=MATCH_SCAN_STEPS)
|
||||
return env
|
||||
|
||||
leads = []
|
||||
for seat in (0, 1):
|
||||
env = jax.vmap(match_reset_from)(decks, coins)
|
||||
final = run(env, jnp.full((matches,), seat, dtype=jnp.int32))
|
||||
totals = np.asarray(jax.vmap(match_score)(final))
|
||||
leads.append(totals[:, seat] - totals[:, 1 - seat])
|
||||
|
||||
lead = np.concatenate(leads)
|
||||
games = float(lead.size)
|
||||
wins = float((lead > 0).sum())
|
||||
low, high = _wilson(wins, games)
|
||||
return {
|
||||
"matches": games,
|
||||
"a_win_rate": wins / games,
|
||||
"wilson_low": low,
|
||||
"wilson_high": high,
|
||||
"a_mean_lead": float(lead.mean()),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
full = _load(cfg, FULL_CHECKPOINT, Ablation())
|
||||
|
||||
rows = []
|
||||
for ablation in VARIANTS:
|
||||
label = ablation.label()
|
||||
run_dir = _existing_run(label)
|
||||
if run_dir is None:
|
||||
print(f"\n===== training {label} =====", flush=True)
|
||||
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
cfg.run.experiment_name = f"ablate-{label}"
|
||||
run_dir = match_train(cfg, ablation=ablation)
|
||||
else:
|
||||
print(f"\n===== reusing {run_dir} =====", flush=True)
|
||||
|
||||
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
params = _load(cfg, run_dir / "latest", ablation)
|
||||
# The ablated net plays seat A, the full stack answers. Below 0.5 means
|
||||
# the piece we removed was carrying weight.
|
||||
result = duel(cfg, params, ablation, full, Ablation(), matches=MATCHES)
|
||||
rows.append({"ablation": label, **result})
|
||||
print(f" {label}: win rate vs full = {result['a_win_rate']:.4f}", flush=True)
|
||||
|
||||
print("\n\n==================== ablation ====================")
|
||||
print(f"{'removed':<26}{'win rate vs full':>18}{'95% CI':>22}{'mean lead':>12}")
|
||||
print("-" * 78)
|
||||
for row in rows:
|
||||
ci = f"[{row['wilson_low']:.3f}, {row['wilson_high']:.3f}]"
|
||||
print(
|
||||
f"{row['ablation']:<26}{row['a_win_rate']:>18.4f}{ci:>22}{row['a_mean_lead']:>+12.1f}"
|
||||
)
|
||||
print("\nbelow 0.5 = the removed piece was carrying weight")
|
||||
Path("runs/jax-ppo-match/ablation.json").write_text(json.dumps(rows, indent=2, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,201 +0,0 @@
|
||||
"""altair (single-round, gen a) vs borealis (3-round match, gen b).
|
||||
|
||||
Full 2x2: {single deal, 3-round match} x {win rate + Wilson, mean margin + CI}.
|
||||
|
||||
Both policies act on the SAME MatchState but from their own view:
|
||||
- borealis reads the full MatchState (match_observation + privileged critic).
|
||||
- altair reads only the round in play as a single-round State (observation).
|
||||
|
||||
We play whole matches (duplicated: every deal-triple from both seats). From the
|
||||
same runs we harvest two scoring conventions:
|
||||
- single deal = the round-0 board score, snapshotted the ply round 0 rolls over
|
||||
(carry=0, round_idx=0 there, so it is an honest standalone deal).
|
||||
- 3-round match = match_score(final), the summed total.
|
||||
|
||||
Margins are borealis-minus-altair. Self-play (borealis vs borealis) is run as a
|
||||
harness check: mirrored identical policies must give win rate 0.5 / margin 0.0.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.match import (
|
||||
MatchState,
|
||||
match_legal_action_mask,
|
||||
match_reset_from,
|
||||
match_score,
|
||||
match_step,
|
||||
)
|
||||
from lost_cities_jax.match_eval import MATCH_SCAN_STEPS, _wilson, match_bank
|
||||
from lost_cities_jax.match_obs import match_critic_observation, match_observation
|
||||
from lost_cities_jax.match_ppo import Ablation, MatchActorCritic, create_match_train_state
|
||||
from lost_cities_jax.obs import observation
|
||||
from lost_cities_jax.ppo import (
|
||||
ActorCritic,
|
||||
create_train_state,
|
||||
load_config,
|
||||
mask_logits,
|
||||
restore_checkpoint,
|
||||
)
|
||||
|
||||
ALTAIR_CKPT = Path(
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/latest"
|
||||
)
|
||||
BOREALIS_CKPT = Path("runs/jax-ppo-match/2026-07-15_031529_match-scaled/latest")
|
||||
MATCHES = 4096 # -> 8192 duplicate games per cell
|
||||
SEED = 20260715
|
||||
OUT = Path("runs/jax-ppo-match/altair_vs_borealis.json")
|
||||
|
||||
borealis_cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
altair_cfg = load_config("configs/jax_ppo/balanced.yaml")
|
||||
|
||||
ABL = Ablation() # privileged_critic=True, as borealis was trained
|
||||
borealis_st = restore_checkpoint(
|
||||
BOREALIS_CKPT, create_match_train_state(borealis_cfg, jax.random.PRNGKey(0), ABL)
|
||||
)
|
||||
altair_st = restore_checkpoint(ALTAIR_CKPT, create_train_state(altair_cfg, jax.random.PRNGKey(0)))
|
||||
|
||||
borealis_model = MatchActorCritic(borealis_cfg.network.hidden_size, borealis_cfg.network.num_layers)
|
||||
altair_model = ActorCritic(altair_cfg.network.hidden_size, altair_cfg.network.num_layers)
|
||||
|
||||
|
||||
def _borealis_action(env: MatchState, to_move, mask):
|
||||
obs = jax.vmap(match_observation)(env, to_move)
|
||||
crit = jax.vmap(match_critic_observation)(env, to_move)
|
||||
logits, _ = borealis_model.apply(borealis_st.params, obs, crit)
|
||||
return jnp.argmax(mask_logits(logits, mask), axis=-1).astype(jnp.int32)
|
||||
|
||||
|
||||
def _altair_action(env: MatchState, to_move, mask):
|
||||
obs = jax.vmap(observation)(env.round, to_move)
|
||||
logits, _ = altair_model.apply(altair_st.params, obs)
|
||||
return jnp.argmax(mask_logits(logits, mask), axis=-1).astype(jnp.int32)
|
||||
|
||||
|
||||
def _make_run(action_seat0, action_seat1):
|
||||
"""Build a jitted full-match runner.
|
||||
|
||||
``action_seat0`` is the policy that plays when ``to_move == borealis_seat``
|
||||
(i.e. borealis); ``action_seat1`` is the other policy (altair). Selection is
|
||||
by ``borealis_seat`` so mirrored orientation is a pure seat relabel.
|
||||
"""
|
||||
|
||||
@jax.jit
|
||||
def run(env: MatchState, borealis_seat):
|
||||
def body(carry, _):
|
||||
env, r0_snap = carry
|
||||
to_move = env.round.to_move.astype(jnp.int32)
|
||||
mask = jax.vmap(match_legal_action_mask)(env)
|
||||
|
||||
a0 = action_seat0(env, to_move, mask)
|
||||
a1 = action_seat1(env, to_move, mask)
|
||||
action = jnp.where(to_move == borealis_seat, a0, a1)
|
||||
|
||||
was_r0 = env.round_idx == 0
|
||||
nxt, _, _ = jax.vmap(match_step, in_axes=(0, 0))(env, action)
|
||||
just_finished_r0 = was_r0 & (nxt.round_idx == 1)
|
||||
# nxt.carry == round-0 board score exactly on the roll-over ply.
|
||||
r0_snap = jnp.where(just_finished_r0[:, None], nxt.carry, r0_snap)
|
||||
return (nxt, r0_snap), None
|
||||
|
||||
n = env.done.shape[0]
|
||||
r0_snap = jnp.zeros((n, 2), dtype=jnp.int32)
|
||||
(env, r0_snap), _ = jax.lax.scan(body, (env, r0_snap), xs=None, length=MATCH_SCAN_STEPS)
|
||||
return env, r0_snap
|
||||
|
||||
return run
|
||||
|
||||
|
||||
def _summ(margin: np.ndarray) -> dict:
|
||||
"""margin = borealis - altair, per duplicate game. Positive = borealis wins."""
|
||||
n = int(margin.size)
|
||||
b_wins = int((margin > 0).sum())
|
||||
a_wins = int((margin < 0).sum())
|
||||
ties = int((margin == 0).sum())
|
||||
lo, hi = _wilson(float(b_wins), float(n))
|
||||
std = float(margin.std(ddof=1))
|
||||
sem = std / math.sqrt(n)
|
||||
return {
|
||||
"n_duplicate_games": n,
|
||||
"borealis_wins": b_wins,
|
||||
"altair_wins": a_wins,
|
||||
"ties": ties,
|
||||
"borealis_win_rate": b_wins / n,
|
||||
"wilson_95": [lo, hi],
|
||||
"mean_margin_borealis_minus_altair": float(margin.mean()),
|
||||
"margin_std": std,
|
||||
"margin_sem": sem,
|
||||
"margin_95ci": [float(margin.mean() - 1.96 * sem), float(margin.mean() + 1.96 * sem)],
|
||||
}
|
||||
|
||||
|
||||
def _play(run, decks, coins, borealis_first: bool):
|
||||
"""Duplicate play; returns (single_deal_margins, match_margins).
|
||||
|
||||
``borealis_first`` picks which policy is action_seat0 in the runner. When the
|
||||
two policies are identical (self-play) this must yield perfectly antisymmetric
|
||||
margins -> win rate 0.5, margin 0.
|
||||
"""
|
||||
single, match = [], []
|
||||
for seat in (0, 1):
|
||||
env = jax.vmap(match_reset_from)(decks, coins)
|
||||
b_seat = jnp.full((MATCHES,), seat, dtype=jnp.int32)
|
||||
final, r0 = run(env, b_seat)
|
||||
r0 = np.asarray(r0)
|
||||
totals = np.asarray(jax.vmap(match_score)(final))
|
||||
# borealis is at index ``seat``.
|
||||
single.append(r0[:, seat] - r0[:, 1 - seat])
|
||||
match.append(totals[:, seat] - totals[:, 1 - seat])
|
||||
return np.concatenate(single), np.concatenate(match)
|
||||
|
||||
|
||||
def main():
|
||||
decks, coins = match_bank(SEED, MATCHES)
|
||||
|
||||
# --- Harness check: borealis vs borealis (both seats borealis) ---
|
||||
run_self = _make_run(_borealis_action, _borealis_action)
|
||||
self_single, self_match = _play(run_self, decks, coins, True)
|
||||
self_check = {
|
||||
"single_deal": _summ(self_single),
|
||||
"three_round_match": _summ(self_match),
|
||||
}
|
||||
|
||||
# --- Real comparison: altair vs borealis ---
|
||||
# seat0-slot = borealis (selected when to_move == borealis_seat), seat1 = altair
|
||||
run_av = _make_run(_borealis_action, _altair_action)
|
||||
av_single, av_match = _play(run_av, decks, coins, True)
|
||||
comparison = {
|
||||
"single_deal": _summ(av_single),
|
||||
"three_round_match": _summ(av_match),
|
||||
}
|
||||
|
||||
result = {
|
||||
"meta": {
|
||||
"altair_ckpt": str(ALTAIR_CKPT),
|
||||
"borealis_ckpt": str(BOREALIS_CKPT),
|
||||
"matches_per_orientation": MATCHES,
|
||||
"duplicate_games_per_cell": 2 * MATCHES,
|
||||
"match_scan_steps": MATCH_SCAN_STEPS,
|
||||
"seed": SEED,
|
||||
"privileged_critic": ABL.privileged_critic,
|
||||
"policies": "greedy argmax (deterministic given the deal)",
|
||||
"margin_sign": "borealis total minus altair total",
|
||||
},
|
||||
"harness_check_borealis_vs_borealis": self_check,
|
||||
"altair_vs_borealis": comparison,
|
||||
}
|
||||
|
||||
OUT.parent.mkdir(parents=True, exist_ok=True)
|
||||
OUT.write_text(json.dumps(result, indent=2))
|
||||
print(json.dumps(result, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -37,12 +37,12 @@ def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description="Benchmark Deep CFR traversal inference backends.")
|
||||
parser.add_argument(
|
||||
"--config-local",
|
||||
default="legacy/deep-cfr/configs/default.yaml",
|
||||
default="configs/deep_cfr/default.yaml",
|
||||
help="Config for the local traversal inference backend.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--config-server",
|
||||
default="legacy/deep-cfr/configs/default_server.yaml",
|
||||
default="configs/deep_cfr/default_server.yaml",
|
||||
help="Config for the server traversal inference backend.",
|
||||
)
|
||||
parser.add_argument("--iterations", type=int, default=10)
|
||||
|
||||
@@ -1,518 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from itertools import pairwise
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
from lost_cities_jax.ppo import (
|
||||
evaluate_checkpoint_match,
|
||||
evaluate_checkpoint_vs_static,
|
||||
load_config,
|
||||
)
|
||||
|
||||
DATE = "2026-07-05"
|
||||
ARTIFACT_DIR = Path("/mnt/2tbhdd/coolrl-lost-cities-artifacts/diminishing-returns/2026-07-05")
|
||||
REPORT_PATH = Path(f"docs/reports/diminishing-returns-{DATE}.md")
|
||||
SUMMARY_PATH = Path(f"docs/reports/diminishing-returns-{DATE}-summary.jsonl")
|
||||
GAMES_PER_SEAT = 2000
|
||||
TOTAL_DUPLICATE_GAMES = GAMES_PER_SEAT * 2
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CheckpointSpec:
|
||||
name: str
|
||||
label: str
|
||||
config: str
|
||||
checkpoint: str
|
||||
stage_index: int
|
||||
|
||||
|
||||
CHECKPOINTS = [
|
||||
CheckpointSpec(
|
||||
name="ladder_v2_gate3",
|
||||
label="ladder v2 gate3",
|
||||
config="configs/jax_ppo/ladder-v2-expert.yaml",
|
||||
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/2026-07-05_013223_jax-ppo-ladder-v2-expert/latest",
|
||||
stage_index=0,
|
||||
),
|
||||
CheckpointSpec(
|
||||
name="league_v1_update_250",
|
||||
label="league v1 update 250",
|
||||
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json",
|
||||
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000250",
|
||||
stage_index=1,
|
||||
),
|
||||
CheckpointSpec(
|
||||
name="league_v1_update_500",
|
||||
label="league v1 update 500",
|
||||
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/main_ppo_config.json",
|
||||
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/snapshots/cycle_01_update_000500",
|
||||
stage_index=2,
|
||||
),
|
||||
CheckpointSpec(
|
||||
name="repair_c01_update_500",
|
||||
label="repair c01 update 500",
|
||||
config="/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json",
|
||||
checkpoint="/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/cycle_01_update_000500",
|
||||
stage_index=3,
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
EXPLOITER_ROWS = [
|
||||
{
|
||||
"phase": "ladder_v2",
|
||||
"target": "ladder_v2_gate3",
|
||||
"protocol": "short_random_shaping",
|
||||
"budget": "same-budget phase-1 exploiter",
|
||||
"warmstart": "random",
|
||||
"win_rate": 0.83135,
|
||||
"mean_score_diff": 54.7721,
|
||||
"source": "docs/reports/ladder-v2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "league_v1",
|
||||
"target": "league_v1_update_250",
|
||||
"protocol": "league_cycle_exploiter",
|
||||
"budget": "league v1 cycle exploiter",
|
||||
"warmstart": "random",
|
||||
"win_rate": 0.50225,
|
||||
"mean_score_diff": 0.6818,
|
||||
"source": "docs/reports/league-v1-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "league_v1",
|
||||
"target": "league_v1_update_500",
|
||||
"protocol": "league_cycle_exploiter",
|
||||
"budget": "same exploiter vs final",
|
||||
"warmstart": "random",
|
||||
"win_rate": 0.47095,
|
||||
"mean_score_diff": -2.63775,
|
||||
"source": "docs/reports/league-v1-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "gates_1_2_original",
|
||||
"target": "league_v1_update_500",
|
||||
"protocol": "long_random_shaping",
|
||||
"budget": "1200 updates, shaping anneal",
|
||||
"warmstart": "random",
|
||||
"win_rate": 0.587,
|
||||
"mean_score_diff": 10.95125,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "gates_1_2_original",
|
||||
"target": "league_v1_update_500",
|
||||
"protocol": "warmstart_gate3_no_shaping",
|
||||
"budget": "900 updates, shaping 0",
|
||||
"warmstart": "ladder_v2_gate3",
|
||||
"win_rate": 0.590,
|
||||
"mean_score_diff": 11.38075,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "gates_1_2_original",
|
||||
"target": "league_v1_update_500",
|
||||
"protocol": "replay_exploiter_no_shaping",
|
||||
"budget": "900 updates, shaping 0",
|
||||
"warmstart": "league_v1_cycle_1_exploiter",
|
||||
"win_rate": 0.582,
|
||||
"mean_score_diff": 10.42525,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "repair_c01",
|
||||
"target": "repair_c01_update_500",
|
||||
"protocol": "long_random_shaping",
|
||||
"budget": "1200 updates, shaping anneal",
|
||||
"warmstart": "random",
|
||||
"win_rate": 0.539,
|
||||
"mean_score_diff": 6.27925,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "repair_c01",
|
||||
"target": "repair_c01_update_500",
|
||||
"protocol": "warmstart_gate3_no_shaping",
|
||||
"budget": "900 updates, shaping 0",
|
||||
"warmstart": "ladder_v2_gate3",
|
||||
"win_rate": 0.549,
|
||||
"mean_score_diff": 7.0285,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
{
|
||||
"phase": "repair_c01",
|
||||
"target": "repair_c01_update_500",
|
||||
"protocol": "replay_exploiter_no_shaping",
|
||||
"budget": "900 updates, shaping 0",
|
||||
"warmstart": "league_v1_cycle_1_exploiter",
|
||||
"win_rate": 0.53925,
|
||||
"mean_score_diff": 6.10225,
|
||||
"source": "docs/reports/gates-1-2-2026-07-05-summary.jsonl",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
started = time.perf_counter()
|
||||
ARTIFACT_DIR.mkdir(parents=True, exist_ok=True)
|
||||
REPORT_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
h2h_rows = run_h2h()
|
||||
expert_rows = run_expert_anchor()
|
||||
exploiter_rows = add_exploiter_deltas(EXPLOITER_ROWS)
|
||||
plot_paths = write_plots(h2h_rows, expert_rows, exploiter_rows)
|
||||
report = build_report(h2h_rows, expert_rows, exploiter_rows, plot_paths)
|
||||
REPORT_PATH.write_text(report, encoding="utf-8")
|
||||
write_summary(h2h_rows, expert_rows, exploiter_rows, time.perf_counter() - started)
|
||||
|
||||
|
||||
def run_h2h() -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for previous, current in pairwise(CHECKPOINTS):
|
||||
output = ARTIFACT_DIR / f"h2h_{current.name}_vs_{previous.name}.json"
|
||||
result = evaluate_checkpoint_match(
|
||||
load_config(current.config),
|
||||
current.checkpoint,
|
||||
load_config(previous.config),
|
||||
previous.checkpoint,
|
||||
games=GAMES_PER_SEAT,
|
||||
duplicate=True,
|
||||
output=output,
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"event": "h2h_adjacent",
|
||||
"previous": previous.name,
|
||||
"current": current.name,
|
||||
"previous_label": previous.label,
|
||||
"current_label": current.label,
|
||||
"json": str(output),
|
||||
**summarize_match(result),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def run_expert_anchor() -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for spec in CHECKPOINTS:
|
||||
output = ARTIFACT_DIR / f"expert_anchor_{spec.name}.json"
|
||||
result = evaluate_checkpoint_vs_static(
|
||||
load_config(spec.config),
|
||||
spec.checkpoint,
|
||||
"heuristic_expert",
|
||||
games=GAMES_PER_SEAT,
|
||||
duplicate=True,
|
||||
output=output,
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"event": "expert_anchor",
|
||||
"checkpoint": spec.name,
|
||||
"label": spec.label,
|
||||
"stage_index": spec.stage_index,
|
||||
"json": str(output),
|
||||
**summarize_match(result),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def summarize_match(result: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"games": result["games"],
|
||||
"win_rate": result["win_rate"],
|
||||
"wilson_low": result["wilson_low"],
|
||||
"wilson_high": result["wilson_high"],
|
||||
"mean_score_diff": result["mean_score_diff"],
|
||||
"score_diff_ci95_low": result["score_diff_ci95_low"],
|
||||
"score_diff_ci95_high": result["score_diff_ci95_high"],
|
||||
"score_diff_std": result["score_diff_std"],
|
||||
"mean_game_length": result["mean_game_length"],
|
||||
"max_steps_rate": result["max_steps_rate"],
|
||||
"opened_colors_per_game": result["opened_colors_per_game"],
|
||||
"play_action_rate": result["play_action_rate"],
|
||||
"positive_expeditions_per_game": result["positive_expeditions_per_game"],
|
||||
}
|
||||
|
||||
|
||||
def add_exploiter_deltas(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
enriched = [dict(row) for row in rows]
|
||||
by_protocol: dict[str, list[dict[str, Any]]] = {}
|
||||
for row in enriched:
|
||||
by_protocol.setdefault(row["protocol"], []).append(row)
|
||||
for protocol_rows in by_protocol.values():
|
||||
previous = None
|
||||
for row in protocol_rows:
|
||||
row["delta_from_previous_same_protocol"] = (
|
||||
None if previous is None else row["win_rate"] - previous["win_rate"]
|
||||
)
|
||||
previous = row
|
||||
return enriched
|
||||
|
||||
|
||||
def write_plots(
|
||||
h2h_rows: list[dict[str, Any]],
|
||||
expert_rows: list[dict[str, Any]],
|
||||
exploiter_rows: list[dict[str, Any]],
|
||||
) -> dict[str, str]:
|
||||
paths = {
|
||||
"h2h": REPORT_PATH.parent / f"diminishing-returns-{DATE}-h2h.png",
|
||||
"expert": REPORT_PATH.parent / f"diminishing-returns-{DATE}-expert.png",
|
||||
"exploiter": REPORT_PATH.parent / f"diminishing-returns-{DATE}-exploiter.png",
|
||||
}
|
||||
|
||||
plt.figure(figsize=(7.5, 4.2))
|
||||
x = list(range(len(h2h_rows)))
|
||||
means = [row["mean_score_diff"] for row in h2h_rows]
|
||||
lows = [row["score_diff_ci95_low"] for row in h2h_rows]
|
||||
highs = [row["score_diff_ci95_high"] for row in h2h_rows]
|
||||
plt.errorbar(
|
||||
x,
|
||||
means,
|
||||
yerr=[
|
||||
[m - lo for m, lo in zip(means, lows, strict=True)],
|
||||
[hi - m for m, hi in zip(means, highs, strict=True)],
|
||||
],
|
||||
fmt="o-",
|
||||
capsize=4,
|
||||
)
|
||||
plt.axhline(0, color="black", linewidth=1)
|
||||
plt.xticks(x, [row["current"] for row in h2h_rows], rotation=20, ha="right")
|
||||
plt.ylabel("Mean score diff vs previous")
|
||||
plt.title("Adjacent checkpoint gains")
|
||||
plt.tight_layout()
|
||||
plt.savefig(paths["h2h"], dpi=160)
|
||||
plt.close()
|
||||
|
||||
plt.figure(figsize=(7.5, 4.2))
|
||||
x = [row["stage_index"] for row in expert_rows]
|
||||
means = [row["mean_score_diff"] for row in expert_rows]
|
||||
lows = [row["score_diff_ci95_low"] for row in expert_rows]
|
||||
highs = [row["score_diff_ci95_high"] for row in expert_rows]
|
||||
plt.errorbar(
|
||||
x,
|
||||
means,
|
||||
yerr=[
|
||||
[m - lo for m, lo in zip(means, lows, strict=True)],
|
||||
[hi - m for m, hi in zip(means, highs, strict=True)],
|
||||
],
|
||||
fmt="o-",
|
||||
capsize=4,
|
||||
)
|
||||
plt.axhline(0, color="black", linewidth=1)
|
||||
plt.xticks(x, [row["checkpoint"] for row in expert_rows], rotation=20, ha="right")
|
||||
plt.ylabel("Mean score diff vs heuristic_expert")
|
||||
plt.title("External anchor trajectory")
|
||||
plt.tight_layout()
|
||||
plt.savefig(paths["expert"], dpi=160)
|
||||
plt.close()
|
||||
|
||||
plt.figure(figsize=(8.2, 4.4))
|
||||
protocols = sorted({row["protocol"] for row in exploiter_rows})
|
||||
for protocol in protocols:
|
||||
series = [row for row in exploiter_rows if row["protocol"] == protocol]
|
||||
plt.plot(
|
||||
[row["target"] for row in series],
|
||||
[row["win_rate"] for row in series],
|
||||
"o-",
|
||||
label=protocol,
|
||||
)
|
||||
plt.axhline(0.55, color="black", linewidth=1, linestyle="--", label="0.55 gate")
|
||||
plt.xticks(rotation=20, ha="right")
|
||||
plt.ylabel("Exploiter win rate")
|
||||
plt.title("Exploitability by protocol")
|
||||
plt.legend(fontsize=8)
|
||||
plt.tight_layout()
|
||||
plt.savefig(paths["exploiter"], dpi=160)
|
||||
plt.close()
|
||||
|
||||
return {key: str(path) for key, path in paths.items()}
|
||||
|
||||
|
||||
def build_report(
|
||||
h2h_rows: list[dict[str, Any]],
|
||||
expert_rows: list[dict[str, Any]],
|
||||
exploiter_rows: list[dict[str, Any]],
|
||||
plot_paths: dict[str, str],
|
||||
) -> str:
|
||||
expert_recent = expert_rows[-1]["mean_score_diff"] - expert_rows[-2]["mean_score_diff"]
|
||||
expert_recent_ci_crosses_zero = (
|
||||
expert_rows[-1]["score_diff_ci95_low"] <= expert_rows[-2]["score_diff_ci95_high"]
|
||||
and expert_rows[-2]["score_diff_ci95_low"] <= expert_rows[-1]["score_diff_ci95_high"]
|
||||
)
|
||||
same_protocol_repairs = [
|
||||
row
|
||||
for row in exploiter_rows
|
||||
if row["phase"] == "repair_c01" and row["delta_from_previous_same_protocol"] is not None
|
||||
]
|
||||
repair_deltas = [row["delta_from_previous_same_protocol"] for row in same_protocol_repairs]
|
||||
mean_repair_drop = -sum(repair_deltas) / len(repair_deltas)
|
||||
expected = estimate_expected_improvement(h2h_rows, expert_recent, mean_repair_drop)
|
||||
judgment = (
|
||||
f"추가 학습 1사이클(~1.5시간 GPU)의 기대 개선은 인접 H2H 최근 이득 "
|
||||
f"{h2h_rows[-1]['mean_score_diff']:+.2f}점, expert 앵커 최근 변화 "
|
||||
f"{expert_recent:+.2f}점, 동일 프로토콜 exploiter 평균 피탈률 감소 "
|
||||
f"{mean_repair_drop:.3f}에 따라 대략 {expected} 수준으로 추정되며, "
|
||||
f"수확체감 구간에 진입했다. 근거는 최근 H2H 이득이 첫 리그 전이보다 작고, "
|
||||
f"expert 앵커 성능이 v1 update 250 이후 하락/회복을 반복하며, 보수 후 "
|
||||
f"exploiter 최악값이 0.549로 게이트 바로 아래에 머문다는 세 측정의 일치다."
|
||||
)
|
||||
if not expert_recent_ci_crosses_zero:
|
||||
judgment += " 단, repair가 expert 앵커를 유의하게 회복시킨 점은 별도 긍정 신호다."
|
||||
|
||||
lines = [
|
||||
f"# Diminishing Returns Diagnostic - {DATE}",
|
||||
"",
|
||||
"신규 학습 없이 기존 체크포인트와 평가 롤아웃만 사용했다. 모든 duplicate 평가는 "
|
||||
f"셔플 뱅크 seed 20260704, 2,000쌍({TOTAL_DUPLICATE_GAMES} games) 기준이다.",
|
||||
"",
|
||||
f"Raw artifacts: `{ARTIFACT_DIR}`",
|
||||
"",
|
||||
"## Checkpoints",
|
||||
"",
|
||||
"| Order | Name | Config | Checkpoint |",
|
||||
"| ---: | --- | --- | --- |",
|
||||
]
|
||||
for spec in CHECKPOINTS:
|
||||
lines.append(
|
||||
f"| {spec.stage_index} | `{spec.name}` | `{spec.config}` | `{spec.checkpoint}` |"
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 1. Adjacent Head-to-Head",
|
||||
"",
|
||||
f".name})",
|
||||
"",
|
||||
"| Later | Earlier | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |",
|
||||
"| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for row in h2h_rows:
|
||||
lines.append(match_table_row(row, row["current"], row["previous"]))
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## 2. External Anchor Trajectory",
|
||||
"",
|
||||
f".name})",
|
||||
"",
|
||||
"| Checkpoint | Win rate | Wilson CI | Mean diff | Score CI | Opened colors | Max-step |",
|
||||
"| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for row in expert_rows:
|
||||
lines.append(anchor_table_row(row))
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
f"Recent expert-anchor slope: `{expert_recent:+.4f}` points "
|
||||
f"({expert_rows[-2]['checkpoint']} -> {expert_rows[-1]['checkpoint']}).",
|
||||
"",
|
||||
"## 3. Exploitability Trajectory",
|
||||
"",
|
||||
f".name})",
|
||||
"",
|
||||
"프로토콜이 다른 피탈률은 한 곡선에 섞지 않았다. `Delta`는 같은 프로토콜 안에서만 계산했다.",
|
||||
"",
|
||||
"| Phase | Target | Protocol | Budget | Warm start | Win rate | Delta same protocol | Mean diff | Source |",
|
||||
"| --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |",
|
||||
]
|
||||
)
|
||||
for row in exploiter_rows:
|
||||
delta = row["delta_from_previous_same_protocol"]
|
||||
delta_text = "n/a" if delta is None else f"{delta:+.5f}"
|
||||
source_name = Path(row["source"]).name
|
||||
lines.append(
|
||||
f"| `{row['phase']}` | `{row['target']}` | `{row['protocol']}` | "
|
||||
f"{row['budget']} | `{row['warmstart']}` | {row['win_rate']:.5f} | "
|
||||
f"{delta_text} | {row['mean_score_diff']:+.4f} | [{source_name}]({source_name}) |"
|
||||
)
|
||||
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"## Judgment",
|
||||
"",
|
||||
judgment,
|
||||
"",
|
||||
"## Notes",
|
||||
"",
|
||||
"- H2H와 expert 앵커 평가는 이번 작업에서 새로 실행했다.",
|
||||
"- Exploiter 표는 지금까지 생성된 리포트/JSONL의 기존 측정값만 재정리했다.",
|
||||
"- 이 작업에서는 신규 학습, 봇 수정, 체크포인트 수정이 없었다.",
|
||||
]
|
||||
)
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def match_table_row(row: dict[str, Any], left: str, right: str) -> str:
|
||||
return (
|
||||
f"| `{left}` | `{right}` | {row['win_rate']:.4f} | "
|
||||
f"[{row['wilson_low']:.4f}, {row['wilson_high']:.4f}] | "
|
||||
f"{row['mean_score_diff']:+.4f} | "
|
||||
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] | "
|
||||
f"{row['opened_colors_per_game']:.4f} | {row['max_steps_rate']:.4f} |"
|
||||
)
|
||||
|
||||
|
||||
def anchor_table_row(row: dict[str, Any]) -> str:
|
||||
return (
|
||||
f"| `{row['checkpoint']}` | {row['win_rate']:.4f} | "
|
||||
f"[{row['wilson_low']:.4f}, {row['wilson_high']:.4f}] | "
|
||||
f"{row['mean_score_diff']:+.4f} | "
|
||||
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] | "
|
||||
f"{row['opened_colors_per_game']:.4f} | {row['max_steps_rate']:.4f} |"
|
||||
)
|
||||
|
||||
|
||||
def estimate_expected_improvement(
|
||||
h2h_rows: list[dict[str, Any]],
|
||||
expert_recent: float,
|
||||
mean_repair_drop: float,
|
||||
) -> str:
|
||||
latest_h2h = h2h_rows[-1]["mean_score_diff"]
|
||||
previous_h2h = h2h_rows[-2]["mean_score_diff"]
|
||||
conservative_h2h = max(0.0, min(latest_h2h, previous_h2h))
|
||||
expert_component = max(0.0, expert_recent)
|
||||
return (
|
||||
f"+{conservative_h2h:.1f}~+{max(conservative_h2h, expert_component):.1f}점 "
|
||||
f"또는 피탈률 -{mean_repair_drop:.3f} 내외"
|
||||
)
|
||||
|
||||
|
||||
def write_summary(
|
||||
h2h_rows: list[dict[str, Any]],
|
||||
expert_rows: list[dict[str, Any]],
|
||||
exploiter_rows: list[dict[str, Any]],
|
||||
elapsed_seconds: float,
|
||||
) -> None:
|
||||
rows = []
|
||||
rows.extend(h2h_rows)
|
||||
rows.extend(expert_rows)
|
||||
rows.extend({"event": "exploiter_trajectory", **row} for row in exploiter_rows)
|
||||
rows.append(
|
||||
{
|
||||
"event": "diminishing_returns_complete",
|
||||
"elapsed_seconds": elapsed_seconds,
|
||||
"artifact_dir": str(ARTIFACT_DIR),
|
||||
"report": str(REPORT_PATH),
|
||||
}
|
||||
)
|
||||
SUMMARY_PATH.write_text(
|
||||
"\n".join(json.dumps(row, ensure_ascii=False, sort_keys=True) for row in rows) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,117 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export the actor head of a JAX PPO Orbax checkpoint to ONNX."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.human_play import infer_config_path, load_agent
|
||||
from lost_cities_jax.ppo import load_config
|
||||
from lost_cities_jax.types import N_ACTIONS, OBS_DIM
|
||||
|
||||
|
||||
def build_argparser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--checkpoint", type=Path, required=True)
|
||||
parser.add_argument("--config", type=Path)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
return parser
|
||||
|
||||
|
||||
def export_model(checkpoint: Path, config: Path | None, output: Path) -> None:
|
||||
try:
|
||||
import onnx
|
||||
from onnx import TensorProto, helper, numpy_helper
|
||||
from onnx.reference import ReferenceEvaluator
|
||||
except ImportError as exc:
|
||||
raise SystemExit("onnx is required; run with `uv run --with onnx ...`") from exc
|
||||
|
||||
config_path = infer_config_path(checkpoint) if config is None else config
|
||||
cfg = load_config(config_path)
|
||||
params, flax_model = load_agent(cfg, checkpoint)
|
||||
dense = params["params"]
|
||||
nodes = []
|
||||
initializers = []
|
||||
previous = "obs"
|
||||
|
||||
for index in range(cfg.network.num_layers):
|
||||
layer = dense[f"Dense_{index}"]
|
||||
weight_name = f"dense_{index}.weight"
|
||||
bias_name = f"dense_{index}.bias"
|
||||
linear_name = f"dense_{index}.linear"
|
||||
output_name = f"dense_{index}.relu"
|
||||
initializers.extend(
|
||||
[
|
||||
numpy_helper.from_array(np.asarray(layer["kernel"], dtype=np.float32), weight_name),
|
||||
numpy_helper.from_array(np.asarray(layer["bias"], dtype=np.float32), bias_name),
|
||||
]
|
||||
)
|
||||
nodes.append(helper.make_node("Gemm", [previous, weight_name, bias_name], [linear_name]))
|
||||
nodes.append(helper.make_node("Relu", [linear_name], [output_name]))
|
||||
previous = output_name
|
||||
|
||||
actor = dense[f"Dense_{cfg.network.num_layers}"]
|
||||
initializers.extend(
|
||||
[
|
||||
numpy_helper.from_array(np.asarray(actor["kernel"], dtype=np.float32), "actor.weight"),
|
||||
numpy_helper.from_array(np.asarray(actor["bias"], dtype=np.float32), "actor.bias"),
|
||||
]
|
||||
)
|
||||
nodes.append(helper.make_node("Gemm", [previous, "actor.weight", "actor.bias"], ["logits"]))
|
||||
graph = helper.make_graph(
|
||||
nodes,
|
||||
"coolrl-lost-cities-jax-ppo-actor",
|
||||
[helper.make_tensor_value_info("obs", TensorProto.FLOAT, [None, OBS_DIM])],
|
||||
[helper.make_tensor_value_info("logits", TensorProto.FLOAT, [None, N_ACTIONS])],
|
||||
initializer=initializers,
|
||||
)
|
||||
model = helper.make_model(
|
||||
graph,
|
||||
producer_name="coolrl-lost-cities",
|
||||
opset_imports=[helper.make_opsetid("", 17)],
|
||||
)
|
||||
model.ir_version = 8
|
||||
onnx.checker.check_model(model)
|
||||
sample = np.random.default_rng(20260713).normal(size=(3, OBS_DIM)).astype(np.float32)
|
||||
expected_logits, _ = flax_model.apply(params, jnp.asarray(sample))
|
||||
actual_logits = ReferenceEvaluator(model).run(None, {"obs": sample})[0]
|
||||
np.testing.assert_allclose(actual_logits, np.asarray(expected_logits), rtol=2e-5, atol=2e-5)
|
||||
np.testing.assert_array_equal(
|
||||
np.argmax(actual_logits, axis=1), np.argmax(np.asarray(expected_logits), axis=1)
|
||||
)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
onnx.save(model, output)
|
||||
model_bytes = output.read_bytes()
|
||||
manifest = {
|
||||
"format": "coolrl-lost-cities-jax-ppo-onnx-v1",
|
||||
"model_file": output.name,
|
||||
"model_size_bytes": len(model_bytes),
|
||||
"model_sha256": hashlib.sha256(model_bytes).hexdigest(),
|
||||
"source_checkpoint": checkpoint.name,
|
||||
"source_config": config_path.name,
|
||||
"observation_size": OBS_DIM,
|
||||
"action_size": N_ACTIONS,
|
||||
"hidden_size": cfg.network.hidden_size,
|
||||
"num_layers": cfg.network.num_layers,
|
||||
"dtype": "float32",
|
||||
"validation_max_abs_error": float(
|
||||
np.max(np.abs(actual_logits - np.asarray(expected_logits)))
|
||||
),
|
||||
}
|
||||
output.with_suffix(".json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
print(f"exported {output} ({output.stat().st_size:,} bytes)")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_argparser().parse_args()
|
||||
export_model(args.checkpoint, args.config, args.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,138 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export a match policy's actor trunk to ONNX for the browser.
|
||||
|
||||
Only the actor ships. The critic exists to grade moves during training and never
|
||||
plays, so its trunk -- and the privileged view of the opponent's hand and the deck
|
||||
that feeds it -- is dropped here rather than shipped and then not used. That also
|
||||
means the exported graph physically cannot leak hidden state, which is a stronger
|
||||
guarantee than promising not to call it.
|
||||
|
||||
MatchActorCritic lays the actor out as Dense_0..Dense_{num_layers} exactly as the
|
||||
single-round model does, so the graph construction is the same; only the input
|
||||
width and the checkpoint loader differ.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.match_obs import MATCH_CRITIC_OBS_DIM, MATCH_OBS_DIM
|
||||
from lost_cities_jax.match_ppo import Ablation, MatchActorCritic, create_match_train_state
|
||||
from lost_cities_jax.ppo import load_config, restore_checkpoint
|
||||
from lost_cities_jax.types import N_ACTIONS
|
||||
|
||||
|
||||
def build_argparser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--checkpoint", type=Path, required=True)
|
||||
parser.add_argument("--config", type=Path, default=Path("configs/jax_ppo/match-selfplay.yaml"))
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
parser.add_argument("--codename", required=True, help="see data/models.json")
|
||||
return parser
|
||||
|
||||
|
||||
def export_model(checkpoint: Path, config: Path, output: Path, codename: str) -> None:
|
||||
try:
|
||||
import onnx
|
||||
from onnx import TensorProto, helper, numpy_helper
|
||||
from onnx.reference import ReferenceEvaluator
|
||||
except ImportError as exc:
|
||||
raise SystemExit("onnx is required; run with `uv run --with onnx ...`") from exc
|
||||
|
||||
cfg = load_config(config)
|
||||
state = restore_checkpoint(
|
||||
checkpoint, create_match_train_state(cfg, jax.random.PRNGKey(0), Ablation())
|
||||
)
|
||||
params = state.params
|
||||
dense = params["params"]
|
||||
|
||||
nodes = []
|
||||
initializers = []
|
||||
previous = "obs"
|
||||
for index in range(cfg.network.num_layers):
|
||||
layer = dense[f"Dense_{index}"]
|
||||
weight, bias = f"dense_{index}.weight", f"dense_{index}.bias"
|
||||
initializers.extend(
|
||||
[
|
||||
numpy_helper.from_array(np.asarray(layer["kernel"], dtype=np.float32), weight),
|
||||
numpy_helper.from_array(np.asarray(layer["bias"], dtype=np.float32), bias),
|
||||
]
|
||||
)
|
||||
nodes.append(helper.make_node("Gemm", [previous, weight, bias], [f"dense_{index}.linear"]))
|
||||
nodes.append(helper.make_node("Relu", [f"dense_{index}.linear"], [f"dense_{index}.relu"]))
|
||||
previous = f"dense_{index}.relu"
|
||||
|
||||
actor = dense[f"Dense_{cfg.network.num_layers}"]
|
||||
initializers.extend(
|
||||
[
|
||||
numpy_helper.from_array(np.asarray(actor["kernel"], dtype=np.float32), "actor.weight"),
|
||||
numpy_helper.from_array(np.asarray(actor["bias"], dtype=np.float32), "actor.bias"),
|
||||
]
|
||||
)
|
||||
nodes.append(helper.make_node("Gemm", [previous, "actor.weight", "actor.bias"], ["logits"]))
|
||||
|
||||
graph = helper.make_graph(
|
||||
nodes,
|
||||
f"coolrl-lost-cities-match-actor-{codename}",
|
||||
[helper.make_tensor_value_info("obs", TensorProto.FLOAT, [None, MATCH_OBS_DIM])],
|
||||
[helper.make_tensor_value_info("logits", TensorProto.FLOAT, [None, N_ACTIONS])],
|
||||
initializer=initializers,
|
||||
)
|
||||
model = helper.make_model(
|
||||
graph, producer_name="coolrl-lost-cities", opset_imports=[helper.make_opsetid("", 17)]
|
||||
)
|
||||
model.ir_version = 8
|
||||
onnx.checker.check_model(model)
|
||||
|
||||
# The exported graph has to agree with the trained one, not merely load.
|
||||
rng = np.random.default_rng(20260715)
|
||||
sample = rng.normal(size=(8, MATCH_OBS_DIM)).astype(np.float32)
|
||||
critic_stub = jnp.zeros((8, MATCH_CRITIC_OBS_DIM), dtype=jnp.float32)
|
||||
flax_model = MatchActorCritic(cfg.network.hidden_size, cfg.network.num_layers)
|
||||
expected, _ = flax_model.apply(params, jnp.asarray(sample), critic_stub)
|
||||
actual = ReferenceEvaluator(model).run(None, {"obs": sample})[0]
|
||||
np.testing.assert_allclose(actual, np.asarray(expected), rtol=2e-5, atol=2e-5)
|
||||
np.testing.assert_array_equal(
|
||||
np.argmax(actual, axis=1), np.argmax(np.asarray(expected), axis=1)
|
||||
)
|
||||
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
onnx.save(model, output)
|
||||
model_bytes = output.read_bytes()
|
||||
manifest = {
|
||||
"format": "coolrl-lost-cities-match-onnx-v1",
|
||||
"codename": codename,
|
||||
"model_file": output.name,
|
||||
"model_size_bytes": len(model_bytes),
|
||||
"model_sha256": hashlib.sha256(model_bytes).hexdigest(),
|
||||
"source_checkpoint": str(checkpoint),
|
||||
"source_config": config.name,
|
||||
"observation_size": MATCH_OBS_DIM,
|
||||
"action_size": N_ACTIONS,
|
||||
"hidden_size": cfg.network.hidden_size,
|
||||
"num_layers": cfg.network.num_layers,
|
||||
"dtype": "float32",
|
||||
"validation_max_abs_error": float(np.max(np.abs(actual - np.asarray(expected)))),
|
||||
}
|
||||
output.with_suffix(".json").write_text(json.dumps(manifest, indent=2) + "\n")
|
||||
print(f"exported {codename} -> {output} ({output.stat().st_size:,} bytes)")
|
||||
print(
|
||||
f" sha256 {manifest['model_sha256'][:12]} obs {MATCH_OBS_DIM} max err "
|
||||
f"{manifest['validation_max_abs_error']:.2e}"
|
||||
)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = build_argparser().parse_args()
|
||||
export_model(args.checkpoint, args.config, args.output, args.codename)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,503 +0,0 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import yaml
|
||||
|
||||
from lost_cities_jax.gates import ExploiterSpec
|
||||
from lost_cities_jax.league import run_league
|
||||
from lost_cities_jax.ppo import (
|
||||
evaluate_checkpoint_match,
|
||||
evaluate_checkpoint_vs_static,
|
||||
load_config,
|
||||
train_against_checkpoint,
|
||||
)
|
||||
|
||||
DATE = "2026-07-05"
|
||||
ROOT = Path("/mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05")
|
||||
REPORT_PATH = Path(f"docs/reports/final-cycles-and-human-play-{DATE}.md")
|
||||
SUMMARY_PATH = Path(f"docs/reports/final-cycles-and-human-play-{DATE}-summary.jsonl")
|
||||
|
||||
START_CONFIG = (
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/"
|
||||
"gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/main_ppo_config.json"
|
||||
)
|
||||
START_CHECKPOINT = (
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/gates-1-2/2026-07-05_094238_gates-1-2/"
|
||||
"gate2c/league/2026-07-05_125709_jax-ppo-gates-1-2-repair-c01/snapshots/"
|
||||
"cycle_01_update_000500"
|
||||
)
|
||||
LEAGUE_TEMPLATE = "configs/jax_ppo/league-v1.yaml"
|
||||
GAMES = 2000
|
||||
PASS_THRESHOLD = 0.52
|
||||
GUARD_CI_LOW = 0.0
|
||||
GUARD_MAX_STEPS = 0.02
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Target:
|
||||
name: str
|
||||
config: str
|
||||
checkpoint: str
|
||||
|
||||
|
||||
EXPLOITERS = [
|
||||
ExploiterSpec(
|
||||
name="long_random",
|
||||
config="configs/jax_ppo/gates-1-2-exploiter-long-random.yaml",
|
||||
notes="random init + shaping anneal, 1200 updates",
|
||||
),
|
||||
ExploiterSpec(
|
||||
name="warmstart_gate3",
|
||||
config="configs/jax_ppo/gates-1-2-exploiter-warmstart.yaml",
|
||||
resume=(
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/ladder-v2/"
|
||||
"2026-07-05_013223_jax-ppo-ladder-v2-expert/latest"
|
||||
),
|
||||
notes="ladder v2 gate-3 warm start, shaping disabled, 900 updates",
|
||||
),
|
||||
ExploiterSpec(
|
||||
name="replay_exploiter",
|
||||
config="configs/jax_ppo/gates-1-2-exploiter-replay.yaml",
|
||||
resume=(
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/"
|
||||
"2026-07-05_052325_jax-ppo-league-v1/exploiters/"
|
||||
"2026-07-05_060727_jax-ppo-league-v1-cycle-1-exploiter/latest"
|
||||
),
|
||||
notes="league v1 cycle-1 exploiter warm start, shaping disabled, 900 updates",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
started = time.perf_counter()
|
||||
ROOT.mkdir(parents=True, exist_ok=True)
|
||||
REPORT_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
SUMMARY_PATH.write_text("", encoding="utf-8")
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
current = Target("repair_c01_update_500", START_CONFIG, START_CHECKPOINT)
|
||||
best = current
|
||||
worst_protocol = "warmstart_gate3"
|
||||
extra_pool: list[dict[str, Any]] = []
|
||||
stop_reason = "max_cycles_exhausted"
|
||||
|
||||
for cycle in range(1, 3):
|
||||
previous = current
|
||||
league_config = write_league_config(cycle, previous, worst_protocol, extra_pool)
|
||||
league_dir = run_league(league_config)
|
||||
league_rows = read_jsonl(league_dir / "league_summary.jsonl")
|
||||
completion = latest_row(league_rows, "league_complete")
|
||||
if completion is None:
|
||||
stop_reason = "league_missing_completion"
|
||||
rows.append({"event": "final_cycle_error", "cycle": cycle, "reason": stop_reason})
|
||||
break
|
||||
current = Target(
|
||||
f"final_cycle_{cycle:02d}",
|
||||
str(league_dir / "main_ppo_config.json"),
|
||||
completion["final_checkpoint"],
|
||||
)
|
||||
cycle_row = {
|
||||
"event": "final_cycle_league",
|
||||
"cycle": cycle,
|
||||
"worst_protocol_used": worst_protocol,
|
||||
"league_config": str(league_config),
|
||||
"league_run_dir": str(league_dir),
|
||||
"target_config": current.config,
|
||||
"target_checkpoint": current.checkpoint,
|
||||
}
|
||||
rows.append(cycle_row)
|
||||
append_jsonl(SUMMARY_PATH, cycle_row)
|
||||
|
||||
exploiter_member = exploiter_member_from_league(league_rows, cycle)
|
||||
if exploiter_member is not None:
|
||||
extra_pool.append(exploiter_member)
|
||||
|
||||
battery = run_battery(cycle, current)
|
||||
for row in battery:
|
||||
rows.append(row)
|
||||
append_jsonl(SUMMARY_PATH, row)
|
||||
judgment = battery_judgment(cycle, battery)
|
||||
rows.append(judgment)
|
||||
append_jsonl(SUMMARY_PATH, judgment)
|
||||
|
||||
guard = evaluate_guard(cycle, current)
|
||||
rows.append(guard)
|
||||
append_jsonl(SUMMARY_PATH, guard)
|
||||
|
||||
h2h = evaluate_h2h(cycle, current, previous)
|
||||
rows.append(h2h)
|
||||
append_jsonl(SUMMARY_PATH, h2h)
|
||||
|
||||
best = current
|
||||
if not guard["passed"]:
|
||||
stop_reason = "expert_guard_failed"
|
||||
current = previous
|
||||
best = previous
|
||||
rows.append(
|
||||
{
|
||||
"event": "final_cycle_rollback",
|
||||
"cycle": cycle,
|
||||
"rolled_back_to": previous.name,
|
||||
"rolled_back_checkpoint": previous.checkpoint,
|
||||
"failed_checkpoint": guard["target_checkpoint"],
|
||||
}
|
||||
)
|
||||
append_jsonl(SUMMARY_PATH, rows[-1])
|
||||
break
|
||||
if judgment["worst_win_rate"] <= PASS_THRESHOLD:
|
||||
stop_reason = "success_exploiter_threshold"
|
||||
break
|
||||
if h2h["score_diff_ci95_low"] <= 0.0 <= h2h["score_diff_ci95_high"]:
|
||||
stop_reason = "h2h_stagnation"
|
||||
break
|
||||
worst_protocol = judgment["worst_exploiter"]
|
||||
|
||||
final_candidate = fix_final_candidate(best)
|
||||
final_row = {
|
||||
"event": "final_cycles_complete",
|
||||
"stop_reason": stop_reason,
|
||||
"final_candidate": str(final_candidate),
|
||||
"final_config": best.config,
|
||||
"source_checkpoint": best.checkpoint,
|
||||
"elapsed_seconds": time.perf_counter() - started,
|
||||
}
|
||||
rows.append(final_row)
|
||||
append_jsonl(SUMMARY_PATH, final_row)
|
||||
write_report(rows, final_row)
|
||||
|
||||
|
||||
def write_league_config(
|
||||
cycle: int, target: Target, worst_protocol: str, extra_pool: list[dict[str, Any]]
|
||||
) -> Path:
|
||||
data = yaml.safe_load(Path(LEAGUE_TEMPLATE).read_text(encoding="utf-8")) or {}
|
||||
data["base_config"] = target.config
|
||||
data["warm_start_checkpoint"] = target.checkpoint
|
||||
run = data.setdefault("run", {})
|
||||
run["experiment_name"] = f"jax-ppo-final-cycle-c{cycle:02d}"
|
||||
run["artifact_root"] = str(ROOT / "league")
|
||||
run["seed"] = int(run.get("seed", 20260705)) + 100 + cycle
|
||||
league = data.setdefault("league", {})
|
||||
league["cycles"] = 1
|
||||
league["league_updates_per_cycle"] = 500
|
||||
league["snapshot_interval_updates"] = 500
|
||||
league["success_exploiter_win_rate"] = PASS_THRESHOLD
|
||||
evaluation = data.setdefault("evaluation", {})
|
||||
evaluation["games"] = GAMES
|
||||
evaluation["batch_games"] = 8192
|
||||
exploiter = data.setdefault("exploiter", {})
|
||||
spec = exploiter_spec(worst_protocol)
|
||||
exploiter["config"] = spec.config
|
||||
exploiter["updates"] = load_config(spec.config).run.total_updates
|
||||
if spec.resume:
|
||||
exploiter["resume"] = spec.resume
|
||||
else:
|
||||
exploiter.pop("resume", None)
|
||||
guards = data.setdefault("guards", {})
|
||||
guards["expert_ci_low"] = GUARD_CI_LOW
|
||||
guards["max_steps_rate"] = GUARD_MAX_STEPS
|
||||
cycle_dir = ROOT / f"cycle_{cycle:02d}"
|
||||
tracking = data.setdefault("tracking", {})
|
||||
tracking["tracked_summary_path"] = str(cycle_dir / "league_summary.jsonl")
|
||||
tracking["report_path"] = str(cycle_dir / "league_report.md")
|
||||
anchors = list(data.get("anchors", []))
|
||||
names = {item.get("name") for item in anchors}
|
||||
for member in extra_pool:
|
||||
if member["name"] not in names:
|
||||
anchors.append(member)
|
||||
names.add(member["name"])
|
||||
data["anchors"] = anchors
|
||||
path = cycle_dir / "league_config.yaml"
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(yaml.safe_dump(data, sort_keys=False), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def evaluate_guard(cycle: int, target: Target) -> dict[str, Any]:
|
||||
output = ROOT / f"cycle_{cycle:02d}" / "guard_vs_heuristic_expert.json"
|
||||
result = evaluate_checkpoint_vs_static(
|
||||
load_config(target.config),
|
||||
target.checkpoint,
|
||||
"heuristic_expert",
|
||||
games=GAMES,
|
||||
duplicate=True,
|
||||
output=output,
|
||||
)
|
||||
return {
|
||||
"event": "final_cycle_guard_expert",
|
||||
"cycle": cycle,
|
||||
"target": target.name,
|
||||
"target_config": target.config,
|
||||
"target_checkpoint": target.checkpoint,
|
||||
"json": str(output),
|
||||
"passed": result["score_diff_ci95_low"] > GUARD_CI_LOW
|
||||
and result["max_steps_rate"] <= GUARD_MAX_STEPS,
|
||||
**summary(result),
|
||||
}
|
||||
|
||||
|
||||
def evaluate_h2h(cycle: int, target: Target, previous: Target) -> dict[str, Any]:
|
||||
output = ROOT / f"cycle_{cycle:02d}" / f"h2h_{target.name}_vs_{previous.name}.json"
|
||||
result = evaluate_checkpoint_match(
|
||||
load_config(target.config),
|
||||
target.checkpoint,
|
||||
load_config(previous.config),
|
||||
previous.checkpoint,
|
||||
games=GAMES,
|
||||
duplicate=True,
|
||||
output=output,
|
||||
)
|
||||
return {
|
||||
"event": "final_cycle_h2h",
|
||||
"cycle": cycle,
|
||||
"current": target.name,
|
||||
"previous": previous.name,
|
||||
"current_checkpoint": target.checkpoint,
|
||||
"previous_checkpoint": previous.checkpoint,
|
||||
"json": str(output),
|
||||
**summary(result),
|
||||
}
|
||||
|
||||
|
||||
def run_battery(cycle: int, target: Target) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for spec in EXPLOITERS:
|
||||
target_cfg = load_config(target.config)
|
||||
exploiter_cfg = load_config(spec.config)
|
||||
exploiter_cfg.run.artifact_root = str(ROOT / f"cycle_{cycle:02d}" / "battery")
|
||||
exploiter_cfg.run.experiment_name = f"final-c{cycle:02d}-{spec.name}-exploiter"
|
||||
train_dir = train_against_checkpoint(
|
||||
exploiter_cfg,
|
||||
target_cfg,
|
||||
target.checkpoint,
|
||||
resume=spec.resume,
|
||||
)
|
||||
output = train_dir / f"eval_vs_{target.name}_duplicate.json"
|
||||
result = evaluate_checkpoint_match(
|
||||
exploiter_cfg,
|
||||
train_dir / "latest",
|
||||
target_cfg,
|
||||
target.checkpoint,
|
||||
games=GAMES,
|
||||
duplicate=True,
|
||||
output=output,
|
||||
)
|
||||
rows.append(
|
||||
{
|
||||
"event": "final_cycle_exploiter",
|
||||
"cycle": cycle,
|
||||
"exploiter": spec.name,
|
||||
"target": target.name,
|
||||
"target_config": target.config,
|
||||
"target_checkpoint": target.checkpoint,
|
||||
"train_dir": str(train_dir),
|
||||
"checkpoint": str(train_dir / "latest"),
|
||||
"resume": spec.resume,
|
||||
"notes": spec.notes,
|
||||
"json": str(output),
|
||||
**summary(result),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def battery_judgment(cycle: int, rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
worst = max(rows, key=lambda row: row["win_rate"])
|
||||
return {
|
||||
"event": "final_cycle_battery_judgment",
|
||||
"cycle": cycle,
|
||||
"worst_exploiter": worst["exploiter"],
|
||||
"worst_win_rate": worst["win_rate"],
|
||||
"threshold": PASS_THRESHOLD,
|
||||
"passed": worst["win_rate"] <= PASS_THRESHOLD,
|
||||
}
|
||||
|
||||
|
||||
def exploiter_member_from_league(rows: list[dict[str, Any]], cycle: int) -> dict[str, Any] | None:
|
||||
row = latest_row(rows, "exploiter_eval")
|
||||
if row is None:
|
||||
return None
|
||||
checkpoint = Path(row["exploiter_checkpoint"])
|
||||
return {
|
||||
"name": f"final_cycle_exploiter_c{cycle:02d}",
|
||||
"kind": "checkpoint",
|
||||
"anchor": False,
|
||||
"stalling": False,
|
||||
"exploiter": True,
|
||||
"config": str(checkpoint.parent / "config.json"),
|
||||
"checkpoint": str(checkpoint),
|
||||
"recent_win_rate": 1.0 - float(row["win_rate"]),
|
||||
"created_cycle": cycle,
|
||||
"created_update": 0,
|
||||
}
|
||||
|
||||
|
||||
def fix_final_candidate(target: Target) -> Path:
|
||||
destination = ROOT / "final_candidate"
|
||||
if destination.exists():
|
||||
shutil.rmtree(destination)
|
||||
shutil.copytree(target.checkpoint, destination)
|
||||
(ROOT / "final_candidate_config.txt").write_text(target.config + "\n", encoding="utf-8")
|
||||
shutil.copy2(target.config, ROOT / "main_ppo_config.json")
|
||||
return destination
|
||||
|
||||
|
||||
def write_report(rows: list[dict[str, Any]], final: dict[str, Any]) -> None:
|
||||
guards = [row for row in rows if row.get("event") == "final_cycle_guard_expert"]
|
||||
h2h = [row for row in rows if row.get("event") == "final_cycle_h2h"]
|
||||
exploiters = [row for row in rows if row.get("event") == "final_cycle_exploiter"]
|
||||
judgments = [row for row in rows if row.get("event") == "final_cycle_battery_judgment"]
|
||||
lines = [
|
||||
f"# Final Cycles and Human Play - {DATE}",
|
||||
"",
|
||||
"## Part A - Closing Reinforcement Cycles",
|
||||
"",
|
||||
f"Raw artifacts: `{ROOT}`",
|
||||
f"Stop reason: `{final['stop_reason']}`",
|
||||
f"Final candidate: `{final['final_candidate']}`",
|
||||
f"Final candidate config: `{final['final_config']}`",
|
||||
"",
|
||||
"### Expert Guard",
|
||||
"",
|
||||
"| Cycle | Passed | Win rate | Mean diff | CI low | Opened colors | Max-step |",
|
||||
"| ---: | ---: | ---: | ---: | ---: | ---: | ---: |",
|
||||
]
|
||||
for row in guards:
|
||||
lines.append(match_row(row, leading=[str(row["cycle"]), str(row["passed"])]))
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"### Adjacent H2H",
|
||||
"",
|
||||
"| Cycle | Current | Previous | Win rate | Mean diff | Score CI |",
|
||||
"| ---: | --- | --- | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for row in h2h:
|
||||
lines.append(
|
||||
f"| {row['cycle']} | `{row['current']}` | `{row['previous']}` | "
|
||||
f"{row['win_rate']:.4f} | {row['mean_score_diff']:+.4f} | "
|
||||
f"[{row['score_diff_ci95_low']:+.4f}, {row['score_diff_ci95_high']:+.4f}] |"
|
||||
)
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"### Strengthened Exploiter Battery",
|
||||
"",
|
||||
"| Cycle | Exploiter | Win rate | Mean diff | CI low | Opened colors | Max-step |",
|
||||
"| ---: | --- | ---: | ---: | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for row in exploiters:
|
||||
lines.append(match_row(row, leading=[str(row["cycle"]), f"`{row['exploiter']}`"]))
|
||||
lines.extend(
|
||||
[
|
||||
"",
|
||||
"### Battery Judgment",
|
||||
"",
|
||||
"| Cycle | Worst exploiter | Worst win rate | Threshold | Passed |",
|
||||
"| ---: | --- | ---: | ---: | ---: |",
|
||||
]
|
||||
)
|
||||
for row in judgments:
|
||||
lines.append(
|
||||
f"| {row['cycle']} | `{row['worst_exploiter']}` | {row['worst_win_rate']:.4f} | "
|
||||
f"{row['threshold']:.4f} | {row['passed']} |"
|
||||
)
|
||||
lines.extend(human_play_usage())
|
||||
REPORT_PATH.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def human_play_usage() -> list[str]:
|
||||
return [
|
||||
"",
|
||||
"## Part B - Human Play Interface",
|
||||
"",
|
||||
"Start a single game:",
|
||||
"",
|
||||
"```bash",
|
||||
"uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \\",
|
||||
" --checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \\",
|
||||
" --seat 0",
|
||||
"```",
|
||||
"",
|
||||
"Start a duplicate set with one shared shuffle and swapped seats:",
|
||||
"",
|
||||
"```bash",
|
||||
"uv run --with 'jax[cuda12]' lost-cities-jax-ppo play \\",
|
||||
" --checkpoint /mnt/2tbhdd/coolrl-lost-cities-artifacts/final-cycles/2026-07-05/final_candidate \\",
|
||||
" --seat 0 --duplicate",
|
||||
"```",
|
||||
"",
|
||||
"Summarize logged human games:",
|
||||
"",
|
||||
"```bash",
|
||||
"uv run lost-cities-jax-ppo human-play summarize \\",
|
||||
" --log-dir /mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/",
|
||||
"```",
|
||||
"",
|
||||
"Move syntax: `play R7 draw deck`, `discard G3 draw Y`, or `play RHS draw deck`.",
|
||||
"The renderer shows only the human hand, both boards, all public discard piles, deck count, and current board score differential. Opponent hand and deck order are not rendered.",
|
||||
"Every game is appended to `/mnt/2tbhdd/coolrl-lost-cities-artifacts/human-play/games.jsonl` with deck seed/index, full action list, AI top-3 policy actions/probabilities, value outputs, scoring breakdown, and optional human comment.",
|
||||
]
|
||||
|
||||
|
||||
def match_row(row: dict[str, Any], leading: list[str]) -> str:
|
||||
return (
|
||||
"| " + " | ".join(leading) + f" | {row['win_rate']:.4f} | {row['mean_score_diff']:+.4f} | "
|
||||
f"{row['score_diff_ci95_low']:+.4f} | {row['opened_colors_per_game']:.4f} | "
|
||||
f"{row['max_steps_rate']:.4f} |"
|
||||
)
|
||||
|
||||
|
||||
def summary(result: dict[str, Any]) -> dict[str, Any]:
|
||||
keys = [
|
||||
"games",
|
||||
"wins",
|
||||
"losses",
|
||||
"ties",
|
||||
"win_rate",
|
||||
"wilson_low",
|
||||
"wilson_high",
|
||||
"mean_score_diff",
|
||||
"score_diff_ci95_low",
|
||||
"score_diff_ci95_high",
|
||||
"mean_game_length",
|
||||
"max_steps_rate",
|
||||
"opened_colors_per_game",
|
||||
"play_action_rate",
|
||||
"positive_expeditions_per_game",
|
||||
]
|
||||
return {key: result[key] for key in keys if key in result}
|
||||
|
||||
|
||||
def exploiter_spec(name: str) -> ExploiterSpec:
|
||||
for spec in EXPLOITERS:
|
||||
if spec.name == name:
|
||||
return spec
|
||||
raise ValueError(f"unknown exploiter protocol: {name}")
|
||||
|
||||
|
||||
def latest_row(rows: list[dict[str, Any]], event: str) -> dict[str, Any] | None:
|
||||
return next((row for row in reversed(rows) if row.get("event") == event), None)
|
||||
|
||||
|
||||
def read_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
if not path.exists():
|
||||
return []
|
||||
return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line]
|
||||
|
||||
|
||||
def append_jsonl(path: Path, row: dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open("a", encoding="utf-8") as handle:
|
||||
handle.write(json.dumps(row, sort_keys=True) + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,97 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate match states and their observations so TypeScript can be checked against JAX.
|
||||
|
||||
The two observation builders must agree to the bit. A mismatch does not throw --
|
||||
the ONNX policy consumes a wrong vector quite happily and plays worse for reasons
|
||||
nobody can see. So the port is not trusted; it is checked.
|
||||
|
||||
States are drawn from real random play so the fixture covers the awkward parts:
|
||||
mid-round, both seats to move, past a round roll-over, with a non-zero carry.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.match import MatchState, match_reset_from, match_step
|
||||
from lost_cities_jax.match_obs import MATCH_OBS_DIM, match_observation
|
||||
from lost_cities_jax.opponents import random_legal_action
|
||||
from lost_cities_jax.types import N_CARDS
|
||||
|
||||
OUTPUT = Path(__file__).resolve().parents[1] / "web" / "src" / "game" / "match-parity-fixture.json"
|
||||
N_ROUNDS = 3
|
||||
|
||||
|
||||
def match_json(match: MatchState) -> dict:
|
||||
round_state = match.round
|
||||
return {
|
||||
"round": {
|
||||
"deckOrder": np.asarray(round_state.deck_order).astype(int).tolist(),
|
||||
"drawPtr": int(round_state.draw_ptr),
|
||||
"cardLoc": np.asarray(round_state.card_loc).astype(int).tolist(),
|
||||
"handPublic": np.asarray(round_state.hand_public).astype(bool).tolist(),
|
||||
"colTop": np.asarray(round_state.col_top).astype(int).tolist(),
|
||||
"colHandshakes": np.asarray(round_state.col_hs).astype(int).tolist(),
|
||||
"colLength": np.asarray(round_state.col_len).astype(int).tolist(),
|
||||
"piles": [
|
||||
np.asarray(round_state.pile[color, : int(round_state.pile_len[color])])
|
||||
.astype(int)
|
||||
.tolist()
|
||||
for color in range(5)
|
||||
],
|
||||
"toMove": int(round_state.to_move),
|
||||
"stepCount": int(round_state.step_count),
|
||||
"done": bool(round_state.done),
|
||||
},
|
||||
"deckOrders": np.asarray(match.deck_orders).astype(int).tolist(),
|
||||
"coinFlips": np.asarray(match.coin_flips).astype(int).tolist(),
|
||||
"roundIdx": int(match.round_idx),
|
||||
"carry": np.asarray(match.carry).astype(int).tolist(),
|
||||
"done": bool(match.done),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
rng = np.random.default_rng(20260715)
|
||||
key = jax.random.PRNGKey(7)
|
||||
rows = []
|
||||
|
||||
for match_index in range(6):
|
||||
decks = np.stack([rng.permutation(N_CARDS) for _ in range(N_ROUNDS)])
|
||||
coins = rng.integers(0, 2, size=(N_ROUNDS,))
|
||||
match = match_reset_from(decks.astype(np.int8), coins.astype(np.int8))
|
||||
|
||||
# Sample the opening position and then every 17th ply, which lands in all
|
||||
# three rounds and on both seats without hand-picking anything.
|
||||
ply = 0
|
||||
while not bool(match.done) and ply < 400:
|
||||
if ply % 17 == 0 or ply == 0:
|
||||
for player in (0, 1):
|
||||
obs = np.asarray(match_observation(match, player), dtype=np.float64)
|
||||
assert obs.shape == (MATCH_OBS_DIM,)
|
||||
rows.append(
|
||||
{
|
||||
"match": match_json(match),
|
||||
"player": player,
|
||||
"observation": [round(float(v), 7) for v in obs],
|
||||
}
|
||||
)
|
||||
key, step_key = jax.random.split(key)
|
||||
action = int(random_legal_action(match.round, match.round.to_move, step_key))
|
||||
match, _, _ = match_step(match, action)
|
||||
ply += 1
|
||||
del match_index
|
||||
|
||||
OUTPUT.write_text(
|
||||
json.dumps({"format": "jax-web-match-parity-v1", "obsDim": MATCH_OBS_DIM, "rows": rows})
|
||||
+ "\n"
|
||||
)
|
||||
print(f"wrote {len(rows)} rows -> {OUTPUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,64 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate deterministic JAX states for TypeScript engine parity tests."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.engine import legal_action_mask, reset_from_order, step
|
||||
from lost_cities_jax.obs import observation
|
||||
|
||||
OUTPUT = Path(__file__).resolve().parents[1] / "web" / "src" / "game" / "parity-fixture.json"
|
||||
|
||||
|
||||
def state_json(state) -> dict:
|
||||
return {
|
||||
"deckOrder": np.asarray(state.deck_order).astype(int).tolist(),
|
||||
"drawPtr": int(state.draw_ptr),
|
||||
"cardLoc": np.asarray(state.card_loc).astype(int).tolist(),
|
||||
"handPublic": np.asarray(state.hand_public).astype(bool).tolist(),
|
||||
"colTop": np.asarray(state.col_top).astype(int).tolist(),
|
||||
"colHandshakes": np.asarray(state.col_hs).astype(int).tolist(),
|
||||
"colLength": np.asarray(state.col_len).astype(int).tolist(),
|
||||
"piles": [
|
||||
np.asarray(state.pile[color, : int(state.pile_len[color])]).astype(int).tolist()
|
||||
for color in range(5)
|
||||
],
|
||||
"toMove": int(state.to_move),
|
||||
"stepCount": int(state.step_count),
|
||||
"done": bool(state.done),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
rng = np.random.default_rng(20260713)
|
||||
order = rng.permutation(60).astype(np.int8)
|
||||
state = reset_from_order(jnp.asarray(order))
|
||||
rows = []
|
||||
for index in range(24):
|
||||
mask = np.asarray(legal_action_mask(state), dtype=bool)
|
||||
rows.append(
|
||||
{
|
||||
"index": index,
|
||||
"state": state_json(state),
|
||||
"legalMask": mask.tolist(),
|
||||
"observationP0": np.asarray(observation(state, jnp.int32(0))).tolist(),
|
||||
"observationP1": np.asarray(observation(state, jnp.int32(1))).tolist(),
|
||||
}
|
||||
)
|
||||
legal = np.flatnonzero(mask)
|
||||
action = int(legal[(index * 17 + 3) % len(legal)])
|
||||
rows[-1]["action"] = action
|
||||
state, _, _ = step(state, jnp.int32(action))
|
||||
if bool(state.done):
|
||||
break
|
||||
OUTPUT.write_text(json.dumps({"format": "jax-web-parity-v1", "rows": rows}) + "\n")
|
||||
print(f"wrote {OUTPUT} ({len(rows)} states)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -7,7 +7,14 @@
|
||||
#
|
||||
# Format examples:
|
||||
# scripts/foo.sh # exact path
|
||||
# configs/jax_ppo/model-*.yaml # glob
|
||||
# configs/deep_cfr/model-*.yaml # glob
|
||||
# docs/research/*.md # whole directory
|
||||
#
|
||||
# Comments after `#` are stripped per line. Blank lines ignored.
|
||||
|
||||
# Future config + script described in docs/plans/model_size_experiment.md
|
||||
configs/deep_cfr/model-size-*.yaml
|
||||
scripts/run_model_size_experiment.sh
|
||||
|
||||
# Future config described in docs/plans/torch_compile.md
|
||||
configs/deep_cfr/default_compile.yaml
|
||||
|
||||
@@ -1,124 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Freeze each policy, train an exploiter against it on the same budget, compare.
|
||||
|
||||
Winning the head-to-head says a policy is strong on average. It does not say the
|
||||
policy is hard to beat. This does: whatever the exploiter reaches is a habit the
|
||||
frozen policy could not defend.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.exploit import (
|
||||
match_frozen_policy,
|
||||
single_round_frozen_policy,
|
||||
train_exploiter,
|
||||
)
|
||||
from lost_cities_jax.match import match_reset_from, match_score
|
||||
from lost_cities_jax.match_eval import MATCH_SCAN_STEPS, _wilson, match_bank
|
||||
from lost_cities_jax.match_ppo import Ablation, create_match_train_state
|
||||
from lost_cities_jax.ppo import create_train_state, load_config, restore_checkpoint
|
||||
|
||||
OURS = Path("runs/jax-ppo-match/2026-07-15_031529_match-scaled/latest")
|
||||
LEAGUE = Path(
|
||||
"/mnt/2tbhdd/coolrl-lost-cities-artifacts/league/2026-07-05_052325_jax-ppo-league-v1/latest"
|
||||
)
|
||||
MATCHES = 4096
|
||||
|
||||
|
||||
def _final_score(cfg, exploiter_params, frozen, matches: int) -> dict:
|
||||
"""Play the trained exploiter against the frozen policy, both seats."""
|
||||
import jax.numpy as jnp
|
||||
|
||||
from lost_cities_jax.exploit import match_frozen_policy as _mk
|
||||
from lost_cities_jax.match import match_step
|
||||
|
||||
attacker = _mk(cfg, exploiter_params, Ablation())
|
||||
decks, coins = match_bank(20260722, matches)
|
||||
|
||||
@jax.jit
|
||||
def run(env, a_seat):
|
||||
def body(carry, _):
|
||||
env, _u = carry
|
||||
action = jnp.where(
|
||||
env.round.to_move.astype(jnp.int32) == a_seat,
|
||||
attacker(env, a_seat),
|
||||
frozen(env, 1 - a_seat),
|
||||
)
|
||||
env, _, _ = jax.vmap(match_step, in_axes=(0, 0))(env, action)
|
||||
return (env, _u), None
|
||||
|
||||
(env, _), _ = jax.lax.scan(body, (env, jnp.int32(0)), xs=None, length=MATCH_SCAN_STEPS)
|
||||
return env
|
||||
|
||||
leads = []
|
||||
for seat in (0, 1):
|
||||
env = jax.vmap(match_reset_from)(decks, coins)
|
||||
final = run(env, jnp.full((matches,), seat, dtype=jnp.int32))
|
||||
totals = np.asarray(jax.vmap(match_score)(final))
|
||||
leads.append(totals[:, seat] - totals[:, 1 - seat])
|
||||
|
||||
lead = np.concatenate(leads)
|
||||
games = float(lead.size)
|
||||
wins = float((lead > 0).sum())
|
||||
low, high = _wilson(wins, games)
|
||||
return {
|
||||
"exploiter_win_rate": wins / games,
|
||||
"wilson_low": low,
|
||||
"wilson_high": high,
|
||||
"exploiter_mean_lead": float(lead.mean()),
|
||||
"matches": games,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--updates", type=int, default=250, help="exploiter training updates")
|
||||
parser.add_argument("--batch-games", type=int, default=1024)
|
||||
parser.add_argument("--tag", default="", help="suffix for the run dir, to keep runs apart")
|
||||
args = parser.parse_args()
|
||||
|
||||
match_cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
old_cfg = load_config("configs/jax_ppo/balanced.yaml")
|
||||
|
||||
ours = restore_checkpoint(
|
||||
OURS, create_match_train_state(match_cfg, jax.random.PRNGKey(0), Ablation())
|
||||
).params
|
||||
league = restore_checkpoint(LEAGUE, create_train_state(old_cfg, jax.random.PRNGKey(0))).params
|
||||
|
||||
targets = {
|
||||
"ours (match stack, 131M)": match_frozen_policy(match_cfg, ours, Ablation()),
|
||||
"league (web-deployed)": single_round_frozen_policy(old_cfg, league),
|
||||
}
|
||||
|
||||
rows = []
|
||||
for name, frozen in targets.items():
|
||||
print(f"\n===== training an exploiter against: {name} =====", flush=True)
|
||||
cfg = load_config("configs/jax_ppo/match-selfplay.yaml")
|
||||
cfg.ppo.batch_games = args.batch_games
|
||||
cfg.run.total_updates = args.updates
|
||||
cfg.run.log_every = max(1, args.updates // 10)
|
||||
slug = name.split()[0] + args.tag
|
||||
state = train_exploiter(cfg, frozen, Path(f"runs/jax-ppo-match/exploit-{slug}"))
|
||||
result = _final_score(cfg, state.params, frozen, MATCHES)
|
||||
rows.append({"target": name, **result})
|
||||
print(f" exploiter reached {result['exploiter_win_rate']:.4f} vs {name}", flush=True)
|
||||
|
||||
print("\n\n============ exploitability (same exploiter budget) ============")
|
||||
print(f"{'frozen policy':<28}{'exploiter win rate':>20}{'95% CI':>22}")
|
||||
print("-" * 72)
|
||||
for row in rows:
|
||||
ci = f"[{row['wilson_low']:.3f}, {row['wilson_high']:.3f}]"
|
||||
print(f"{row['target']:<28}{row['exploiter_win_rate']:>20.4f}{ci:>22}")
|
||||
print("\nhigher = the frozen policy had more to farm. 0.5 = nothing found.")
|
||||
Path(f"runs/jax-ppo-match/exploitability{args.tag}.json").write_text(json.dumps(rows, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,7 +1,7 @@
|
||||
"""Profile GPU forward-pass throughput for the Deep CFR trainer network.
|
||||
|
||||
Builds the same DeepCFRMLP that ``DeepCFRTrainer.__init__`` constructs from
|
||||
``legacy/deep-cfr/configs/default.yaml``, then measures average forward-pass time on
|
||||
``configs/deep_cfr/default.yaml``, then measures average forward-pass time on
|
||||
CUDA across a sweep of batch sizes. The goal is to decide whether batched
|
||||
traversal inference (Optimization Priorities #5) is worth implementing.
|
||||
"""
|
||||
@@ -19,7 +19,7 @@ from coolrl_lost_cities.games.classic.deep_cfr.config import load_config
|
||||
from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
CONFIG_PATH = REPO_ROOT / "legacy" / "deep-cfr" / "configs" / "default.yaml"
|
||||
CONFIG_PATH = REPO_ROOT / "configs" / "deep_cfr" / "default.yaml"
|
||||
|
||||
BATCH_SIZES = [1, 4, 16, 64, 256, 1024]
|
||||
WARMUP_ITERS = 10
|
||||
|
||||
@@ -1,334 +0,0 @@
|
||||
"""Render human-readable JAX PPO Lost Cities transcripts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import NamedTuple
|
||||
|
||||
import jax
|
||||
import jax.numpy as jnp
|
||||
import numpy as np
|
||||
|
||||
from lost_cities_jax.engine import (
|
||||
board_score,
|
||||
current_hand_sorted,
|
||||
decode_action,
|
||||
reset_from_order,
|
||||
step,
|
||||
)
|
||||
from lost_cities_jax.opponents import policy_by_name
|
||||
from lost_cities_jax.ppo import checkpoint_policy, load_config, make_shuffle_bank
|
||||
from lost_cities_jax.types import (
|
||||
CARDS_PER_COLOR,
|
||||
DISCARD,
|
||||
DRAW_DECK,
|
||||
LOC_P0_BOARD,
|
||||
MAX_STEPS,
|
||||
N_CARDS,
|
||||
N_COLORS,
|
||||
PLAY,
|
||||
State,
|
||||
)
|
||||
|
||||
COLOR_NAMES = ["red", "green", "white", "blue", "yellow"]
|
||||
STEP_JIT = jax.jit(step)
|
||||
|
||||
|
||||
class PolicySpec(NamedTuple):
|
||||
label: str
|
||||
fn: object
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", required=True)
|
||||
parser.add_argument("--checkpoint", required=True)
|
||||
parser.add_argument("--opponent", default="heuristic_cautious")
|
||||
parser.add_argument("--shuffle-bank-seed", type=int, default=20260704)
|
||||
parser.add_argument("--pairs", type=int, default=10)
|
||||
parser.add_argument("--output", required=True)
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg = load_config(args.config)
|
||||
agent = PolicySpec("gate3_checkpoint", jax.jit(checkpoint_policy(cfg, args.checkpoint)))
|
||||
opponent = PolicySpec(args.opponent, jax.jit(policy_by_name(args.opponent)))
|
||||
orders = make_shuffle_bank(args.shuffle_bank_seed, args.pairs)
|
||||
|
||||
sections = []
|
||||
summaries = []
|
||||
for idx, order in enumerate(orders):
|
||||
sections.append(f"\n\n## Pair {idx:02d} / learner seat 0\n")
|
||||
text, summary = render_game(order, [agent, opponent], agent_seat=0, game_id=idx * 2)
|
||||
sections.append(text)
|
||||
summaries.append(summary)
|
||||
|
||||
sections.append(f"\n\n## Pair {idx:02d} / learner seat 1\n")
|
||||
text, summary = render_game(order, [opponent, agent], agent_seat=1, game_id=idx * 2 + 1)
|
||||
sections.append(text)
|
||||
summaries.append(summary)
|
||||
|
||||
output = render_summary(summaries, args) + "".join(sections)
|
||||
Path(args.output).parent.mkdir(parents=True, exist_ok=True)
|
||||
Path(args.output).write_text(output, encoding="utf-8")
|
||||
|
||||
|
||||
def render_game(
|
||||
deck_order: list[int], policies: list[PolicySpec], *, agent_seat: int, game_id: int
|
||||
) -> tuple[str, dict]:
|
||||
state = reset_from_order(jnp.asarray(deck_order, dtype=jnp.int8))
|
||||
lines = [f"Game {game_id}: P0={policies[0].label}, P1={policies[1].label}"]
|
||||
handshake_events = []
|
||||
cautious_openings = []
|
||||
cautious_useful_discards = []
|
||||
|
||||
while not bool(state.done):
|
||||
player = int(state.to_move)
|
||||
policy = policies[player]
|
||||
key = jax.random.PRNGKey(game_id * 1000 + int(state.step_count))
|
||||
hand = [int(x) for x in current_hand_sorted(state, player)]
|
||||
action = int(policy.fn(state, jnp.asarray(player, dtype=jnp.int32), key))
|
||||
hand_slot, place_type, draw_source = [int(x) for x in decode_action(jnp.asarray(action))]
|
||||
card = hand[hand_slot]
|
||||
drawn = peek_drawn_card(state, place_type, draw_source)
|
||||
before_col_len = int(state.col_len[player, card_color(card)])
|
||||
before_hand = [card_label(x) for x in hand if x < N_CARDS]
|
||||
before_scores = [int(x) for x in board_score(state)]
|
||||
|
||||
if policy.label == "heuristic_cautious":
|
||||
if place_type == PLAY and before_col_len == 0:
|
||||
cautious_openings.append(
|
||||
{
|
||||
"ply": int(state.step_count),
|
||||
"seat": player,
|
||||
"card": card_label(card),
|
||||
"rank": card_rank(card),
|
||||
}
|
||||
)
|
||||
if place_type == DISCARD and can_play_card(state, 1 - player, card):
|
||||
cautious_useful_discards.append(
|
||||
{
|
||||
"ply": int(state.step_count),
|
||||
"seat": player,
|
||||
"card": card_label(card),
|
||||
"opponent": 1 - player,
|
||||
}
|
||||
)
|
||||
|
||||
if place_type == PLAY and is_handshake(card):
|
||||
handshake_events.append(
|
||||
{
|
||||
"ply": int(state.step_count),
|
||||
"seat": player,
|
||||
"policy": policy.label,
|
||||
"card": card_label(card),
|
||||
"hand": before_hand,
|
||||
}
|
||||
)
|
||||
|
||||
next_state, _, _ = STEP_JIT(state, jnp.asarray(action, dtype=jnp.int32))
|
||||
place_text = "PLAY" if place_type == PLAY else "DISCARD"
|
||||
target = "expedition" if place_type == PLAY else "discard"
|
||||
draw_text = "deck" if draw_source == DRAW_DECK else f"{COLOR_NAMES[draw_source - 1]} pile"
|
||||
lines.append(
|
||||
f"Ply {int(state.step_count):03d} P{player} {policy.label}: "
|
||||
f"{place_text} {card_label(card)} to {target}; "
|
||||
f"DRAW {draw_text} -> {card_label(drawn)}; "
|
||||
f"score_before={before_scores}"
|
||||
)
|
||||
lines.extend(render_board(next_state))
|
||||
state = next_state
|
||||
|
||||
final_scores = [int(x) for x in board_score(state)]
|
||||
lines.append(f"Final score: P0={final_scores[0]} P1={final_scores[1]}")
|
||||
lines.extend(render_final_breakdown(state))
|
||||
agent_score = final_scores[agent_seat]
|
||||
opponent_score = final_scores[1 - agent_seat]
|
||||
summary = {
|
||||
"game_id": game_id,
|
||||
"agent_seat": agent_seat,
|
||||
"agent_score_diff": agent_score - opponent_score,
|
||||
"agent_opened_colors": int(jnp.sum(state.col_len[agent_seat] > 0)),
|
||||
"length": int(state.step_count),
|
||||
"max_steps": int(state.step_count) >= MAX_STEPS,
|
||||
"handshake_events": handshake_events,
|
||||
"cautious_openings": cautious_openings,
|
||||
"cautious_useful_discards": cautious_useful_discards,
|
||||
}
|
||||
return "\n".join(lines) + "\n", summary
|
||||
|
||||
|
||||
def render_summary(summaries: list[dict], args: argparse.Namespace) -> str:
|
||||
diffs = np.asarray([item["agent_score_diff"] for item in summaries], dtype=np.float64)
|
||||
opened = np.asarray([item["agent_opened_colors"] for item in summaries], dtype=np.float64)
|
||||
lengths = np.asarray([item["length"] for item in summaries], dtype=np.float64)
|
||||
handshakes = [event for item in summaries for event in item["handshake_events"]]
|
||||
cautious_openings = [event for item in summaries for event in item["cautious_openings"]]
|
||||
useful_discards = [event for item in summaries for event in item["cautious_useful_discards"]]
|
||||
opening_ranks = Counter(str(event["rank"]) for event in cautious_openings)
|
||||
low_openings = [
|
||||
event for event in cautious_openings if event["rank"] not in ("HS",) and event["rank"] < 7
|
||||
]
|
||||
|
||||
lines = [
|
||||
"# Gate-3 Transcript Dump",
|
||||
"",
|
||||
f"config: `{args.config}`",
|
||||
f"checkpoint: `{args.checkpoint}`",
|
||||
f"opponent: `{args.opponent}`",
|
||||
f"duplicate pairs: `{args.pairs}`",
|
||||
"",
|
||||
"## Summary",
|
||||
"",
|
||||
f"games: {len(summaries)}",
|
||||
f"agent_score_diff_mean: {float(np.mean(diffs)):.3f}",
|
||||
f"agent_score_diff_min_max: {int(np.min(diffs))} / {int(np.max(diffs))}",
|
||||
f"agent_opened_colors_mean: {float(np.mean(opened)):.3f}",
|
||||
f"game_length_mean: {float(np.mean(lengths)):.3f}",
|
||||
f"game_length_min_p50_p95_max: {int(np.min(lengths))} / "
|
||||
f"{float(np.quantile(lengths, 0.50)):.1f} / "
|
||||
f"{float(np.quantile(lengths, 0.95)):.1f} / {int(np.max(lengths))}",
|
||||
f"max_steps_rate: {float(np.mean([item['max_steps'] for item in summaries])):.3f}",
|
||||
f"handshake_play_events: {len(handshakes)}",
|
||||
f"cautious_openings: {len(cautious_openings)}",
|
||||
f"cautious_opening_rank_counts: {json.dumps(dict(sorted(opening_ranks.items())))}",
|
||||
f"cautious_low_openings_lt7: {len(low_openings)}",
|
||||
f"cautious_discards_immediately_playable_by_opponent: {len(useful_discards)}",
|
||||
"",
|
||||
"## Handshake Play Contexts",
|
||||
"",
|
||||
]
|
||||
if handshakes:
|
||||
for event in handshakes[:80]:
|
||||
lines.append(
|
||||
f"- game_event ply={event['ply']} seat=P{event['seat']} "
|
||||
f"policy={event['policy']} card={event['card']} hand={event['hand']}"
|
||||
)
|
||||
if len(handshakes) > 80:
|
||||
lines.append(f"- ... {len(handshakes) - 80} more")
|
||||
else:
|
||||
lines.append("- none")
|
||||
|
||||
lines.extend(["", "## Cautious Opening Audit", ""])
|
||||
if cautious_openings:
|
||||
for event in cautious_openings[:80]:
|
||||
lines.append(
|
||||
f"- ply={event['ply']} seat=P{event['seat']} card={event['card']} "
|
||||
f"rank={event['rank']}"
|
||||
)
|
||||
if len(cautious_openings) > 80:
|
||||
lines.append(f"- ... {len(cautious_openings) - 80} more")
|
||||
else:
|
||||
lines.append("- none")
|
||||
|
||||
lines.extend(["", "## Cautious Useful Discard Audit", ""])
|
||||
if useful_discards:
|
||||
for event in useful_discards[:80]:
|
||||
lines.append(
|
||||
f"- ply={event['ply']} seat=P{event['seat']} card={event['card']} "
|
||||
f"opponent=P{event['opponent']}"
|
||||
)
|
||||
if len(useful_discards) > 80:
|
||||
lines.append(f"- ... {len(useful_discards) - 80} more")
|
||||
else:
|
||||
lines.append("- none")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_board(state: State) -> list[str]:
|
||||
scores = [int(x) for x in board_score(state)]
|
||||
lines = [f" Board scores: P0={scores[0]} P1={scores[1]}"]
|
||||
for player in range(2):
|
||||
parts = []
|
||||
for color in range(N_COLORS):
|
||||
cards = board_cards(state, player, color)
|
||||
score = color_score(cards)
|
||||
parts.append(f"{COLOR_NAMES[color]}={format_cards(cards)}({score:+d})")
|
||||
lines.append(f" P{player}: " + " | ".join(parts))
|
||||
return lines
|
||||
|
||||
|
||||
def render_final_breakdown(state: State) -> list[str]:
|
||||
lines = ["Final color breakdown:"]
|
||||
for player in range(2):
|
||||
parts = []
|
||||
for color in range(N_COLORS):
|
||||
cards = board_cards(state, player, color)
|
||||
parts.append(f"{COLOR_NAMES[color]} {format_cards(cards)} => {color_score(cards):+d}")
|
||||
lines.append(f" P{player}: " + "; ".join(parts))
|
||||
return lines
|
||||
|
||||
|
||||
def board_cards(state: State, player: int, color: int) -> list[int]:
|
||||
loc = np.asarray(state.card_loc)
|
||||
board_loc = LOC_P0_BOARD + player
|
||||
start = color * CARDS_PER_COLOR
|
||||
cards = [card for card in range(start, start + CARDS_PER_COLOR) if int(loc[card]) == board_loc]
|
||||
return sorted(
|
||||
cards, key=lambda card: (0 if is_handshake(card) else 1, card_rank_value(card), card)
|
||||
)
|
||||
|
||||
|
||||
def color_score(cards: list[int]) -> int:
|
||||
if not cards:
|
||||
return 0
|
||||
handshakes = sum(1 for card in cards if is_handshake(card))
|
||||
ranks = [card_rank_value(card) for card in cards if not is_handshake(card)]
|
||||
score = (sum(ranks) - 20) * (1 + handshakes)
|
||||
if len(cards) >= 8:
|
||||
score += 20
|
||||
return score
|
||||
|
||||
|
||||
def peek_drawn_card(state: State, place_type: int, draw_source: int) -> int:
|
||||
del place_type
|
||||
if draw_source == DRAW_DECK:
|
||||
return int(state.deck_order[int(state.draw_ptr)])
|
||||
color = draw_source - 1
|
||||
length = int(state.pile_len[color])
|
||||
return int(state.pile[color, length - 1])
|
||||
|
||||
|
||||
def can_play_card(state: State, player: int, card: int) -> bool:
|
||||
top = int(state.col_top[player, card_color(card)])
|
||||
if is_handshake(card):
|
||||
return top == 0
|
||||
return card_rank_value(card) > top
|
||||
|
||||
|
||||
def format_cards(cards: list[int]) -> str:
|
||||
if not cards:
|
||||
return "[]"
|
||||
return "[" + ",".join(card_label(card) for card in cards) + "]"
|
||||
|
||||
|
||||
def card_label(card: int) -> str:
|
||||
if card < 0 or card >= N_CARDS:
|
||||
return "none"
|
||||
color = COLOR_NAMES[card_color(card)][0].upper()
|
||||
if is_handshake(card):
|
||||
return f"{color}HS{card % CARDS_PER_COLOR + 1}"
|
||||
return f"{color}{card_rank_value(card)}"
|
||||
|
||||
|
||||
def card_color(card: int) -> int:
|
||||
return card // CARDS_PER_COLOR
|
||||
|
||||
|
||||
def is_handshake(card: int) -> bool:
|
||||
return card % CARDS_PER_COLOR < 3
|
||||
|
||||
|
||||
def card_rank(card: int) -> int | str:
|
||||
return "HS" if is_handshake(card) else card_rank_value(card)
|
||||
|
||||
|
||||
def card_rank_value(card: int) -> int:
|
||||
return card % CARDS_PER_COLOR - 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,86 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serve the production web client and append completed games to JSONL."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from http import HTTPStatus
|
||||
from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
MAX_RECORD_BYTES = 6_000_000
|
||||
|
||||
# v2 adds the opponent's identity (codename + hash) and the match layer -- three
|
||||
# deals, the coin flips, and which round each move belongs to. v1 is still
|
||||
# accepted because 111 games were recorded under it; they are all altair, which
|
||||
# data/models.json records since v1 has nowhere to say so.
|
||||
SUPPORTED_FORMATS = frozenset({"lost-cities-web-game-v1", "lost-cities-web-game-v2"})
|
||||
|
||||
|
||||
def parse_record(body: bytes) -> dict[str, Any]:
|
||||
if len(body) > MAX_RECORD_BYTES:
|
||||
raise ValueError("record is too large")
|
||||
value = json.loads(body)
|
||||
if not isinstance(value, dict) or value.get("format") not in SUPPORTED_FORMATS:
|
||||
raise ValueError("unsupported game record")
|
||||
if not isinstance(value.get("gameId"), str) or not isinstance(value.get("moves"), list):
|
||||
raise ValueError("invalid game record")
|
||||
return value
|
||||
|
||||
|
||||
def make_handler(dist: Path, output: Path):
|
||||
seen_ids: set[str] = set()
|
||||
if output.exists():
|
||||
for line in output.read_text(encoding="utf-8").splitlines():
|
||||
try:
|
||||
game_id = json.loads(line).get("gameId")
|
||||
if isinstance(game_id, str):
|
||||
seen_ids.add(game_id)
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
continue
|
||||
|
||||
class Handler(SimpleHTTPRequestHandler):
|
||||
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||
super().__init__(*args, directory=str(dist), **kwargs)
|
||||
|
||||
def do_POST(self) -> None: # noqa: N802
|
||||
if self.path != "/api/game-records":
|
||||
self.send_error(HTTPStatus.NOT_FOUND)
|
||||
return
|
||||
try:
|
||||
length = int(self.headers.get("content-length", "0"))
|
||||
record = parse_record(self.rfile.read(length))
|
||||
except (ValueError, json.JSONDecodeError) as error:
|
||||
self.send_error(HTTPStatus.BAD_REQUEST, str(error))
|
||||
return
|
||||
game_id = record["gameId"]
|
||||
if game_id not in seen_ids:
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
with output.open("a", encoding="utf-8") as stream:
|
||||
stream.write(
|
||||
json.dumps(record, ensure_ascii=False, separators=(",", ":")) + "\n"
|
||||
)
|
||||
seen_ids.add(game_id)
|
||||
self.send_response(HTTPStatus.NO_CONTENT)
|
||||
self.end_headers()
|
||||
|
||||
return Handler
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--host", default="0.0.0.0")
|
||||
parser.add_argument("--port", type=int, default=5173)
|
||||
parser.add_argument("--dist", type=Path, default=Path("web/dist"))
|
||||
parser.add_argument("--output", type=Path, default=Path("data/human-play/game-records.jsonl"))
|
||||
args = parser.parse_args()
|
||||
server = ThreadingHTTPServer((args.host, args.port), make_handler(args.dist, args.output))
|
||||
print(f"Serving {args.dist} on http://{args.host}:{args.port}", flush=True)
|
||||
print(f"Writing game records to {args.output}", flush=True)
|
||||
server.serve_forever()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Train the classic three-round agent by self-play, then measure it."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import jax
|
||||
|
||||
from lost_cities_jax.match_eval import carry_probe, match_evaluate
|
||||
from lost_cities_jax.match_ppo import create_match_train_state, match_train
|
||||
from lost_cities_jax.ppo import load_config, restore_checkpoint
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--config", default="configs/jax_ppo/match-selfplay.yaml")
|
||||
parser.add_argument("--eval-only", type=Path, default=None)
|
||||
parser.add_argument("--matches", type=int, default=1024)
|
||||
parser.add_argument("--set", action="append", default=[])
|
||||
args = parser.parse_args()
|
||||
|
||||
cfg = load_config(args.config)
|
||||
for override in args.set:
|
||||
path, _, raw = override.partition("=")
|
||||
section, _, field = path.partition(".")
|
||||
target = getattr(cfg, section)
|
||||
current = getattr(target, field)
|
||||
value = type(current)(raw) if not isinstance(current, bool) else raw == "true"
|
||||
setattr(target, field, value)
|
||||
|
||||
if args.eval_only is None:
|
||||
run_dir = match_train(cfg)
|
||||
checkpoint = run_dir / "latest"
|
||||
else:
|
||||
checkpoint = args.eval_only
|
||||
|
||||
state = create_match_train_state(cfg, jax.random.PRNGKey(0))
|
||||
state = restore_checkpoint(Path(checkpoint), state)
|
||||
|
||||
result = match_evaluate(cfg, state.params, matches=args.matches)
|
||||
print("\n== duplicate match eval (self-play) ==")
|
||||
print(json.dumps(result, indent=2, sort_keys=True))
|
||||
|
||||
print("\n== carry probe: does round-three play react to the deficit? ==")
|
||||
rows = carry_probe(cfg, state.params, matches=args.matches // 2)
|
||||
header = f"{'carry':>7}{'win_rate':>10}{'opened':>9}{'wagers':>9}{'deck_race':>11}{'plies':>8}"
|
||||
print(header)
|
||||
for row in rows:
|
||||
print(
|
||||
f"{row['carry']:>7}{row['win_rate']:>10.3f}{row['opened_colors']:>9.2f}"
|
||||
f"{row['wagers_played']:>9.2f}{row['deck_race_rate']:>11.3f}{row['mean_plies']:>8.0f}"
|
||||
)
|
||||
Path(checkpoint).parent.joinpath("carry_probe.json").write_text(
|
||||
json.dumps({"eval": result, "probe": rows}, indent=2, sort_keys=True)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -57,6 +57,7 @@ cdef class GameState:
|
||||
cdef void _clear(self) noexcept
|
||||
|
||||
cpdef GameState clone(self)
|
||||
cpdef GameState determinize_for_player(self, int player, object rng)
|
||||
cpdef list legal_card_mask(self)
|
||||
cpdef list legal_draw_mask(self)
|
||||
cpdef list legal_mask(self)
|
||||
|
||||
@@ -595,6 +595,29 @@ cdef class GameState:
|
||||
other.terminal = self.terminal
|
||||
return other
|
||||
|
||||
cpdef GameState determinize_for_player(self, int player, object rng):
|
||||
"""Clone and reshuffle hidden opponent hand/deck cards for ``player``."""
|
||||
cdef int p = int(player)
|
||||
cdef int opponent = 1 - p
|
||||
cdef int opponent_hand_len = self.hand_lens[opponent]
|
||||
cdef int unseen_len = opponent_hand_len + self.deck_len
|
||||
cdef int i
|
||||
cdef list unseen = [0] * unseen_len
|
||||
cdef GameState other
|
||||
if p < 0 or p > 1:
|
||||
raise ValueError(f"player must be 0 or 1, got {player}")
|
||||
for i in range(opponent_hand_len):
|
||||
unseen[i] = self.hand_cards[self._hand_index(opponent, i)]
|
||||
for i in range(self.deck_len):
|
||||
unseen[opponent_hand_len + i] = self.deck_cards[i]
|
||||
rng.shuffle(unseen)
|
||||
other = self.clone()
|
||||
for i in range(opponent_hand_len):
|
||||
other.hand_cards[other._hand_index(opponent, i)] = <int>unseen[i]
|
||||
for i in range(self.deck_len):
|
||||
other.deck_cards[i] = <int>unseen[opponent_hand_len + i]
|
||||
return other
|
||||
|
||||
cpdef list legal_card_mask(self):
|
||||
cdef list mask = [False] * (2 * self.hand_size)
|
||||
cdef int slot
|
||||
|
||||
@@ -79,19 +79,6 @@ def train_command(args: argparse.Namespace) -> None:
|
||||
device=config.run.device,
|
||||
tracker=tracker,
|
||||
)
|
||||
if args.resume_from:
|
||||
import torch
|
||||
|
||||
ckpt = torch.load(args.resume_from, map_location=trainer.device, weights_only=False)
|
||||
trainer.network.load_state_dict(ckpt["network"])
|
||||
if "optimizer" in ckpt:
|
||||
trainer.optimizer.load_state_dict(ckpt["optimizer"])
|
||||
print(
|
||||
f"[resume] loaded network + optimizer from {args.resume_from} "
|
||||
f"(prior iteration={ckpt.get('iteration', '?')}); "
|
||||
f"new run starts at iteration 1 with current config",
|
||||
flush=True,
|
||||
)
|
||||
try:
|
||||
trainer.train()
|
||||
finally:
|
||||
@@ -124,31 +111,7 @@ def main(argv: list[str] | None = None) -> None:
|
||||
train.add_argument("--wandb-job-type")
|
||||
train.add_argument("--wandb-tag", action="append", default=[])
|
||||
train.add_argument("--wandb-notes")
|
||||
train.add_argument(
|
||||
"--resume-from",
|
||||
default=None,
|
||||
help="Path to a .pt checkpoint to warm-start network + optimizer state.",
|
||||
)
|
||||
train.set_defaults(func=train_command)
|
||||
|
||||
from .eval_checkpoint import add_eval_args, run_eval
|
||||
|
||||
eval_cmd = subparsers.add_parser(
|
||||
"eval",
|
||||
help="Evaluate a saved checkpoint vs heuristic bots in parallel.",
|
||||
)
|
||||
add_eval_args(eval_cmd)
|
||||
eval_cmd.set_defaults(func=lambda a: run_eval(a))
|
||||
|
||||
from .pretrain import add_pretrain_args, run_pretrain
|
||||
|
||||
pretrain_cmd = subparsers.add_parser(
|
||||
"pretrain",
|
||||
help="Behavior-clone a heuristic bot into the AlphaZero network as a warm start.",
|
||||
)
|
||||
add_pretrain_args(pretrain_cmd)
|
||||
pretrain_cmd.set_defaults(func=lambda a: run_pretrain(a))
|
||||
|
||||
args = parser.parse_args(argv)
|
||||
args.func(args)
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user