From f170fcdcfd7a88a35e0501f28fadb27eb7820c87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=A0=95=EC=8B=9C=EC=9B=90?= Date: Wed, 15 Jul 2026 02:44:04 +0900 Subject: [PATCH] Run the classic self-play trainer and record what the carry probe says Self-play works: matches converge to 146.8 plies (~49 a round) with a 91% deck- race rate, so the stalling that static opponents induced is gone. Duplicate match eval scores 0.4968 with a mean lead of exactly 0.0 -- same deals, same coins, both seats, deal luck cancelling exactly. Success criterion 1 does not pass. The carry probe is close to flat: expeditions opened sit at 5.00 whether the policy is 60 points down or 60 points up. Wager use does move monotonically across all six carry levels, and in the right direction (behind -> more multipliers), but the spread is 0.31 wagers. Two diagnoses, one of which was mine and wrong: - Residual potential shaping was NOT the cause. Annealing it fully to zero left the probe just as flat. - terminal_scale is. At carry -60, tanh((margin - 60)/50) is close to linear over any realistic round margin, and maximising E[tanh] on a linear stretch is just maximising E[margin] -- there is no reason to gamble. Risk-seeking only appears where tanh is sharply convex, which needs a smaller scale. Dropping 50 -> 12 widens the wager spread 0.19 -> 0.31, which is the mechanism showing up. The probe itself is also mis-scaled: at scale 12, tanh(60/12) is 1.0, so +/-60 is a saturated dead zone with no gradient and the policy has learned nothing there. The measurable band is |carry| <~ 2 * terminal_scale, and the probe levels have to be set from the scale rather than fixed. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh --- configs/jax_ppo/match-selfplay.yaml | 50 +++++++++++++++++++ docs/plans/lost-cities-classic-3round.md | 46 ++++++++++++++++++ scripts/train_match.py | 62 ++++++++++++++++++++++++ 3 files changed, 158 insertions(+) create mode 100644 configs/jax_ppo/match-selfplay.yaml create mode 100644 scripts/train_match.py diff --git a/configs/jax_ppo/match-selfplay.yaml b/configs/jax_ppo/match-selfplay.yaml new file mode 100644 index 0000000..4dd42f6 --- /dev/null +++ b/configs/jax_ppo/match-selfplay.yaml @@ -0,0 +1,50 @@ +# Classic Lost Cities: three rounds, self-play. +# +# rollout_steps is deliberately shorter than a match. Early self-play stalls +# rounds badly (untrained matches run ~900 plies), and the env carries across +# updates, so a long match simply spans several rollouts -- the GAE truncation +# bootstrap keeps that unbiased. Sizing the scan to the worst-case match instead +# would blow up the rollout tensors, which are already doubled by training both +# seats and widened by the critic's 681-dim privileged view. +run: + experiment_name: match-selfplay + seed: 20260715 + learner_seat: 0 + total_updates: 300 + log_every: 10 + checkpoint_every: 50 + artifact_root: runs/jax-ppo-match + +opponent: + name: discard_only # unused: self-play + +network: + hidden_size: 512 + num_layers: 3 + +ppo: + batch_games: 512 + rollout_steps: 256 + gamma: 1.0 + gae_lambda: 0.97 # a match is ~3x a round; 0.95 reaches too little of it + clip_epsilon: 0.2 + entropy_coef: 0.01 + value_coef: 0.5 + max_grad_norm: 0.5 + learning_rate: 0.0003 + epochs: 4 + minibatches: 32 + +reward: + terminal_scale: 50.0 + # Counted in learner actions now, not padded scan steps. Held slightly above + # zero: the match-terminal signal alone is one bounded number per ~160 plies. + potential_shaping_initial: 1.0 + potential_shaping_final: 0.05 + potential_shaping_anneal_steps: 20000000 + +evaluation: + games: 2000 + duplicate: true + shuffle_bank_seed: 20260715 + batch_games: 512 diff --git a/docs/plans/lost-cities-classic-3round.md b/docs/plans/lost-cities-classic-3round.md index 5f5cbe4..33e23f7 100644 --- a/docs/plans/lost-cities-classic-3round.md +++ b/docs/plans/lost-cities-classic-3round.md @@ -309,3 +309,49 @@ scale을 낮춰서(예: 50 → 25) 제거한다. **`heuristic_expert`처럼 stalling을 허용하지 않는 anchor**(게임이 자연 길이로 끝남)만 실력 지표로 쓰고, 약한 anchor는 **자연 종료 게임만 필터링해서** 보거나 `max_steps_rate`를 함께 보고할 것. + +--- + +## 실행 결과 (2026-07-15, 커밋 `9ba5a07`) + +### 동작하는 것 + +- **셀프플레이가 stalling을 스스로 제거한다.** 매치가 146.8수(라운드 ≈49수)로 수렴하고 + 덱 레이스 비율 91%. Phase 0a에서 정적 상대가 225수까지 끌던 것과 대조된다. +- **미러 매치 평가가 정확하다.** 셀프플레이 duplicate 승률 0.4968, **평균 lead 정확히 0.0** — + 같은 3딜 + 같은 코인 + 자리 교대가 딜 운을 완전히 상쇄한다. 미완료 0%. +- **전지적 critic이 격리돼 있다.** 특권 입력을 흔들어도 정책 로짓은 비트 단위로 동일하고, + 가치만 움직인다 (테스트로 고정). + +### 성공 기준 1 — **약하게만 충족. 사실상 미달.** + +carry 프로브(3라운드 시작 시 carry 주입)에서: + +| | scale 50 | scale 25 | scale 12 | +|---|---|---|---| +| 악수 스프레드 (carry −60 → +60) | 0.19 | 0.08 | **0.31** | +| 원정 개수 | 5.00 고정 | 4.99 고정 | **5.00 고정** | + +- **악수 사용은 6개 carry 수준 전체에서 단조**로 움직인다 (뒤지면 배수 베팅 ↑). 방향은 맞다. +- 그러나 **원정 개수는 carry와 무관하게 5.00에 붙어 있다.** 굳은 습관이지 조건부 플레이가 아니다. +- 효과 크기가 작다. + +### 확정된 진단 + +1. **shaping 가설은 틀렸다.** `potential_shaping_final=0.0`으로 완전히 꺼도 프로브는 평평했다. +2. **원인은 `terminal_scale`이다.** carry = −60에서 `tanh((margin − 60)/50)`의 인자는 현실적 + 마진 범위에서 **거의 선형**이고, 선형 구간에서 `E[tanh]` 최대화는 `E[margin]` 최대화와 같다 + → 도박할 이유가 없다. 위험 추구는 tanh가 **강하게 볼록한** 구간에서만 나오며 scale을 낮춰야 + 그 구간에 들어간다. scale 50 → 12에서 스프레드가 0.19 → 0.31로 커진 것이 이를 확인한다. +3. **프로브 설계 결함:** scale 12에서 `tanh(60/12) ≈ 1.0`이라 **±60은 포화 = 그래디언트 0**. + 거기서 정책은 학습된 바가 없다. **의미 있는 측정 구간은 `|carry| ≲ 2 × terminal_scale`.** + 프로브 수준을 scale에 맞춰 재설계해야 한다. + +### 다음에 할 일 + +- [ ] **프로브 재설계**: carry 수준을 `terminal_scale`에 맞춰 잡는다 (포화 구간 측정 금지). +- [ ] **학습 중 carry 분포 확인**: 셀프플레이에서 3라운드 진입 시 carry가 실제로 얼마나 + 퍼지는가. 좁으면 정책이 큰 deficit을 본 적이 없다는 뜻이고, 그게 진짜 원인일 수 있다. +- [ ] **"항상 5색 개시"가 정상인지 검증**: 인간 기보에서 AI는 4.81, 인간은 4.19를 열었다. + 5.00 고정은 의심스럽다. 셀프플레이 균형인지, 탐색 붕괴인지 (엔트로피 3.5 → 1.03). +- [ ] scale 스케줄(50 → 12 후기 fine-tune)이 처음부터 12로 학습하는 것보다 나은지 A/B. diff --git a/scripts/train_match.py b/scripts/train_match.py new file mode 100644 index 0000000..e3dbecd --- /dev/null +++ b/scripts/train_match.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""Train the classic three-round agent by self-play, then measure it.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + +import jax + +from lost_cities_jax.match_eval import carry_probe, match_evaluate +from lost_cities_jax.match_ppo import create_match_train_state, match_train +from lost_cities_jax.ppo import load_config, restore_checkpoint + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--config", default="configs/jax_ppo/match-selfplay.yaml") + parser.add_argument("--eval-only", type=Path, default=None) + parser.add_argument("--matches", type=int, default=1024) + parser.add_argument("--set", action="append", default=[]) + args = parser.parse_args() + + cfg = load_config(args.config) + for override in args.set: + path, _, raw = override.partition("=") + section, _, field = path.partition(".") + target = getattr(cfg, section) + current = getattr(target, field) + value = type(current)(raw) if not isinstance(current, bool) else raw == "true" + setattr(target, field, value) + + if args.eval_only is None: + run_dir = match_train(cfg) + checkpoint = run_dir / "latest" + else: + checkpoint = args.eval_only + + state = create_match_train_state(cfg, jax.random.PRNGKey(0)) + state = restore_checkpoint(Path(checkpoint), state) + + result = match_evaluate(cfg, state.params, matches=args.matches) + print("\n== duplicate match eval (self-play) ==") + print(json.dumps(result, indent=2, sort_keys=True)) + + print("\n== carry probe: does round-three play react to the deficit? ==") + rows = carry_probe(cfg, state.params, matches=args.matches // 2) + header = f"{'carry':>7}{'win_rate':>10}{'opened':>9}{'wagers':>9}{'deck_race':>11}{'plies':>8}" + print(header) + for row in rows: + print( + f"{row['carry']:>7}{row['win_rate']:>10.3f}{row['opened_colors']:>9.2f}" + f"{row['wagers_played']:>9.2f}{row['deck_race_rate']:>11.3f}{row['mean_plies']:>8.0f}" + ) + Path(checkpoint).parent.joinpath("carry_probe.json").write_text( + json.dumps({"eval": result, "probe": rows}, indent=2, sort_keys=True) + ) + + +if __name__ == "__main__": + main()