coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Deep CFR evaluation 배칭 및 병렬화 추가

Browse Source
This commit is contained in:
coolguy
2026-05-07 03:42:00 +09:00
parent 54347e1e7f
commit 148be6e9a0
8 changed files with 641 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml
+3
View File
@@ -72,6 +72,9 @@ memory:
evaluation:
eval_every: 5
games: 100
batch_size: 64
device: trainer
num_workers: 4
max_steps: 1000
on_max_steps: score_diff
opponents:
docs/deep-cfr-batched-evaluation-2026-05-07.md
+111
View File
@@ -0,0 +1,111 @@
# Deep CFR Batched Evaluation 2026-05-07
Run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter`
Command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only
```
The config used `evaluation.batch_size: 64` and `evaluation.device: trainer`.
The base run device was CUDA.
## Summary
The run completed one training iteration and evaluated immediately.
| Metric | Value |
| --- | ---: |
| `iteration_seconds` | 21.322356 |
| `evaluation_seconds` | 14.834096 |
| `traversal_seconds` | 3.850308 |
| `advantage_train_seconds` | 1.778991 |
| `strategy_train_seconds` | 0.847232 |
Compared with the previous batch-size-1 CUDA check, evaluation time changed
from about 61.826s to about 14.834s.
## Opponent Timing
| Opponent | elapsed | avg len | policy turns | network | postprocess | encoding | legal mask | opponent act |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `safe_heuristic_strict` | 3.971 | 1000.0 | 50000 | 0.195 | 0.251 | 0.147 | 0.084 | 3.168 |
| `safe_heuristic` | 3.685 | 997.5 | 49876 | 0.199 | 0.257 | 0.148 | 0.084 | 2.868 |
| `safe_heuristic_loose` | 3.330 | 980.8 | 49038 | 0.194 | 0.250 | 0.145 | 0.082 | 2.536 |
| `noisy_safe` | 2.902 | 955.9 | 47784 | 0.195 | 0.250 | 0.144 | 0.079 | 2.109 |
| `random` | 0.797 | 640.4 | 31968 | 0.154 | 0.204 | 0.095 | 0.053 | 0.212 |
| `passive_discard` | 0.138 | 172.2 | 8562 | 0.032 | 0.042 | 0.025 | 0.014 | 0.004 |
## Totals
| Metric | Value |
| --- | ---: |
| `eval_elapsed_seconds` | 14.824448 |
| `eval_policy_turns` | 237228 |
| `eval_policy_network_seconds` | 0.968006 |
| `eval_policy_postprocess_seconds` | 1.253108 |
| `eval_policy_encoding_seconds` | 0.702534 |
| `eval_policy_legal_mask_seconds` | 0.396494 |
| `eval_opponent_act_seconds` | 10.896567 |
| `eval_apply_action_seconds` | 0.068075 |
| `eval_diagnostics_seconds` | 0.060940 |
## Notes
Batched network inference removed the previous batch-size-1 CUDA network
bottleneck. After batching, safe heuristic opponent action time became the
largest remaining eval cost for safe heuristic opponents.
The first batched implementation exposed a postprocess synchronization cost
from per-row entropy calculation. Moving entropy calculation into torch batch
postprocess reduced that cost before this run.
## Opponent Parallel Evaluation
Run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter`
Command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only
```
This run used `evaluation.num_workers: 4`, `evaluation.batch_size: 64`, and
`evaluation.device: trainer` on CUDA.
| Metric | Batched sequential | Batched opponent-parallel |
| --- | ---: | ---: |
| `iteration_seconds` | 21.322356 | 12.854383 |
| `evaluation_seconds` | 14.834096 | 6.420402 |
| `traversal_seconds` | 3.850308 | 3.852627 |
| `advantage_train_seconds` | 1.778991 | 1.746623 |
| `strategy_train_seconds` | 0.847232 | 0.819998 |
Opponent elapsed values from the parallel run:
| Opponent | elapsed | opponent act | network | avg len |
| --- | ---: | ---: | ---: | ---: |
| `safe_heuristic_strict` | 4.121 | 3.225 | 0.196 | 1000.0 |
| `safe_heuristic` | 4.029 | 2.863 | 0.281 | 997.5 |
| `safe_heuristic_loose` | 3.807 | 2.631 | 0.286 | 981.6 |
| `noisy_safe` | 3.026 | 2.141 | 0.211 | 951.7 |
| `random` | 1.130 | 0.212 | 0.253 | 639.0 |
| `passive_discard` | 0.417 | 0.004 | 0.114 | 172.2 |
Parallel eval reduced measured eval wall time from about 14.83s to about
6.42s, roughly `2.31x` faster for this one-iteration profile.
docs/deep-cfr-evaluation-profile-2026-05-07.md
+89
View File
@@ -62,3 +62,92 @@ still smaller than policy network time.
`apply_action_seconds`, `diagnostics_seconds`, and `final_scoring_seconds` were
small in this run.
## CPU vs CUDA Evaluation Check
This check compared `--device cpu` and `--device cuda` on the same base
configuration after the evaluation breakdown metrics were available. The base
configuration was:
`configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml`
Both runs used one training iteration and ran evaluation on that iteration.
The base configuration's evaluation settings were kept at 100 games per
opponent and the six configured opponents.
CPU run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter`
CPU command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only \
--device cpu
```
CUDA run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter`
CUDA command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only \
--device cuda
```
Top-level timing:
| Metric | CPU | CUDA | CUDA / CPU |
| --- | ---: | ---: | ---: |
| `iteration_seconds` | 50.969 | 72.501 | 1.42 |
| `evaluation_seconds` | 38.920 | 61.826 | 1.59 |
| `traversal_seconds` | 6.343 | 6.409 | 1.01 |
| `advantage_train_seconds` | 4.054 | 2.884 | 0.71 |
| `strategy_train_seconds` | 1.643 | 1.369 | 0.83 |
Opponent elapsed timing:
| Opponent | CPU elapsed | CUDA elapsed | CUDA / CPU |
| --- | ---: | ---: | ---: |
| `random` | 4.038 | 7.220 | 1.79 |
| `passive_discard` | 0.990 | 1.773 | 1.79 |
| `safe_heuristic` | 8.701 | 13.518 | 1.55 |
| `safe_heuristic_loose` | 8.334 | 13.145 | 1.58 |
| `safe_heuristic_strict` | 9.097 | 13.882 | 1.53 |
| `noisy_safe` | 7.751 | 12.281 | 1.58 |
Opponent breakdown for the CPU run:
| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `random` | 4.038 | 32170 | 2.401 | 0.075 ms | 0.586 | 0.169 | 0.114 | 0.361 | 644.4 |
| `passive_discard` | 0.990 | 8562 | 0.633 | 0.074 ms | 0.155 | 0.044 | 0.034 | 0.010 | 172.2 |
| `safe_heuristic` | 8.701 | 49756 | 3.650 | 0.073 ms | 0.903 | 0.253 | 0.211 | 3.068 | 995.1 |
| `safe_heuristic_loose` | 8.334 | 48958 | 3.592 | 0.073 ms | 0.886 | 0.248 | 0.207 | 2.798 | 979.2 |
| `safe_heuristic_strict` | 9.097 | 50000 | 3.678 | 0.074 ms | 0.910 | 0.254 | 0.214 | 3.420 | 1000.0 |
| `noisy_safe` | 7.751 | 47276 | 3.501 | 0.074 ms | 0.864 | 0.244 | 0.195 | 2.354 | 945.7 |
| Total | 38.911 | 236722 | 17.456 | 0.074 ms | 4.304 | 1.212 | 0.975 | 12.011 | |
Opponent breakdown for the CUDA run:
| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `random` | 7.220 | 32096 | 5.337 | 0.166 ms | 0.716 | 0.179 | 0.129 | 0.384 | 642.9 |
| `passive_discard` | 1.773 | 8562 | 1.364 | 0.159 ms | 0.185 | 0.047 | 0.039 | 0.012 | 172.2 |
| `safe_heuristic` | 13.518 | 49876 | 8.053 | 0.161 ms | 1.097 | 0.274 | 0.239 | 3.139 | 997.5 |
| `safe_heuristic_loose` | 13.145 | 49078 | 7.967 | 0.162 ms | 1.080 | 0.271 | 0.236 | 2.890 | 981.6 |
| `safe_heuristic_strict` | 13.882 | 50000 | 8.034 | 0.161 ms | 1.096 | 0.274 | 0.239 | 3.524 | 1000.0 |
| `noisy_safe` | 12.281 | 47266 | 7.650 | 0.162 ms | 1.044 | 0.262 | 0.219 | 2.421 | 945.5 |
| Total | 61.819 | 236878 | 38.406 | 0.162 ms | 5.218 | 1.307 | 1.101 | 12.370 | |
docs/deep-cfr-legacy-runtime-comparison-2026-05-07.md
+99
View File
@@ -0,0 +1,99 @@
# Deep CFR Legacy Runtime Comparison 2026-05-07
This note records a runtime summary from the older `../coolrl` Lost Cities
Deep CFR implementation and compares it with the current profiling runs in
this repository.
## Legacy Run Summary
The older run completed metrics through iteration 387. The process had stopped
before completing iteration 388.
Total elapsed time through iteration 387:
`5879.17s`, or about `1h 37m 59s`.
| Segment | Mean | Median | Note |
| --- | ---: | ---: | --- |
| All iterations | 15.19s/iter | 11.44s | Includes eval iterations |
| Non-eval iterations | 11.51s/iter | 11.32s | Normal training iteration |
| Eval iterations | 30.00s/iter | 29.31s | Eval every 5 iterations |
| Evaluation only | 18.61s/eval | 18.04s | Early evals were slower |
| Traversal | 7.16s/iter | 6.98s | 140 traversals/iter |
| Advantage train | 2.81s/iter | 2.78s | Player 0 + player 1 |
| Strategy train | 1.44s/iter | 1.44s | |
| Overall throughput | 5387 nodes/s | 5437 nodes/s | |
| Traversal throughput | 19.8 traversals/s | 20.1 traversals/s | |
Recent 50 iteration window from that run:
| Segment | Mean |
| --- | ---: |
| All iterations | 15.57s/iter |
| Non-eval iterations | 12.21s/iter |
| Recent 20 evals, eval only | 17.31s/eval |
| Recent 20 eval iterations | 29.30s/iter |
Evaluation ran every 5 iterations: 5, 10, 15, ..., 385.
The practical legacy cadence was roughly:
`4 normal iterations + 1 eval iteration ~= 75s per 5 iterations`.
## Current Repo Reference Points
From `docs/deep-cfr-profile-advantage-memory-split-2026-05-07.md`:
| Segment | Current mean |
| --- | ---: |
| Non-eval iterations | 5.832958s/iter |
| Traversal | 3.160975s/iter |
| Advantage train | 1.742895s/iter |
| Strategy train | 0.912324s/iter |
From `docs/deep-cfr-batched-evaluation-2026-05-07.md`:
| Segment | Current value |
| --- | ---: |
| Batched CUDA evaluation | 14.834096s/eval |
| Batched CUDA 1-iter wall time with eval | 21.322356s |
| Batched opponent-parallel CUDA evaluation | 6.420402s/eval |
| Batched opponent-parallel CUDA 1-iter wall time with eval | 12.854383s |
## Rough Comparison
Normal training iterations improved from about `11.51s` to about `5.83s`,
roughly `1.97x` faster.
Traversal improved from about `7.16s` to about `3.16s`, roughly `2.27x`
faster.
Evaluation improved from about `18.61s` to about `14.83s`, roughly `1.25x`
faster for the measured batched CUDA profile. With opponent-parallel eval, the
measured eval time was about `6.42s`, roughly `2.90x` faster than the legacy
eval-only average.
Using the simple cadence model:
Legacy:
`4 * 11.51 + (11.51 + 18.61) = 76.16s per 5 iterations`
Current batched sequential:
`4 * 5.83 + (5.83 + 14.83) = 43.98s per 5 iterations`
Current batched opponent-parallel:
`4 * 5.83 + (5.83 + 6.42) = 35.57s per 5 iterations`
That implies about `1.73x` faster eval-included wall time for the batched
sequential rough comparison, and about `2.14x` faster for the batched
opponent-parallel rough comparison.
## Caveat
The legacy numbers came from a long run through iteration 387. The current
numbers are from targeted profiling runs. The comparison is useful for order of
magnitude and bottleneck direction, not as a strict benchmark under identical
runtime conditions.
src/coolrl_lost_cities/games/classic/deep_cfr/config.py
+20
View File
@@ -232,6 +232,9 @@ class EvaluationConfig(StrictModel):
opponents: tuple[str, ...] = ("random",)
max_steps: int = 10_000
on_max_steps: str = "score_diff"
batch_size: int = 64
device: str = "trainer"
num_workers: int = 4
@field_validator("on_max_steps")
@classmethod
@@ -241,6 +244,23 @@ class EvaluationConfig(StrictModel):
raise ValueError("must be 'score_diff', 'loss', or 'draw'")
return token
@field_validator("device")
@classmethod
def _validate_device(cls, value: str) -> str:
token = value.strip().lower()
if token not in {"trainer", "auto", "cpu", "cuda"}:
raise ValueError("must be 'trainer', 'auto', 'cpu', or 'cuda'")
return token
def resolved_batch_size(self) -> int:
return max(1, int(self.batch_size))
def resolved_num_workers(self, opponent_count: int | None = None) -> int:
workers = max(1, int(self.num_workers))
if opponent_count is not None:
workers = min(workers, max(1, int(opponent_count)))
return workers
class DeepCFRConfig(StrictModel):
run: RunConfig = Field(default_factory=RunConfig)
src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py
+156 -14
View File
@@ -159,6 +159,17 @@ class PolicyEvalDiagnostics:
return data
@dataclass
class _EvalGame:
state: GameState
policies: list[LostCitiesPolicy]
policy_player: int
diagnostics: PolicyEvalDiagnostics
first_open_recoverable_by_color: dict[int, float]
steps: int = 0
done: bool = False
class StrategyNetPolicy(LostCitiesPolicy):
def __init__(
self,
@@ -176,6 +187,51 @@ class StrategyNetPolicy(LostCitiesPolicy):
self.encoding = encoding
self.runtime = EvalRuntimeCounters()
def select_actions_batch(self, states: list[GameState]) -> list[tuple[int, float]]:
if not states:
return []
started = time.perf_counter()
self.runtime.policy_turns += len(states)
legal_started = time.perf_counter()
legal_masks = [np.asarray(state.unified_legal_mask(), dtype=bool) for state in states]
legal_actions_list = [np.flatnonzero(legal) for legal in legal_masks]
self.runtime.policy_legal_mask_seconds += time.perf_counter() - legal_started
if any(len(legal_actions) == 0 for legal_actions in legal_actions_list):
raise RuntimeError("no legal action available")
encoding_started = time.perf_counter()
infos = [encode_info_state(state, state.current_player, self.encoding) for state in states]
self.runtime.policy_encoding_seconds += time.perf_counter() - encoding_started
network_started = time.perf_counter()
with torch.inference_mode():
x = torch.as_tensor(np.stack(infos), dtype=torch.float32, device=self.device)
logits = self.strategy_network(x)
self.runtime.policy_network_seconds += time.perf_counter() - network_started
postprocess_started = time.perf_counter()
legal_tensor = torch.as_tensor(np.stack(legal_masks), dtype=torch.bool, device=self.device)
masked = logits.masked_fill(~legal_tensor, torch.finfo(torch.float32).min)
probs_tensor = torch.softmax(masked, dim=-1).masked_fill(~legal_tensor, 0.0)
entropy_tensor = -(probs_tensor * probs_tensor.clamp_min(1.0e-12).log()).sum(dim=-1)
if self.sample:
probs_np = probs_tensor.detach().cpu().numpy()
unified_actions = [
int(self.rng.choice(legal_actions, p=probs_np[index][legal_actions]))
for index, legal_actions in enumerate(legal_actions_list)
]
else:
unified_actions = [int(value) for value in torch.argmax(masked, dim=-1).cpu()]
entropies = [float(value) for value in entropy_tensor.cpu()]
actions = [
state.from_unified_action(unified)
for state, unified in zip(states, unified_actions, strict=True)
]
self.runtime.policy_postprocess_seconds += time.perf_counter() - postprocess_started
self.runtime.policy_select_seconds += time.perf_counter() - started
return list(zip(actions, entropies, strict=True))
def action_distribution(self, state: GameState) -> tuple[np.ndarray, np.ndarray]:
started = time.perf_counter()
legal = np.asarray(state.unified_legal_mask(), dtype=bool)
@@ -236,6 +292,7 @@ def evaluate_strategy_network(
device: torch.device | str = "cpu",
max_steps: int = 10_000,
encoding: EncodingConfig | None = None,
batch_size: int = 64,
) -> dict[str, float | int]:
strategy_network.eval()
return _evaluate_strategy_network_with_diagnostics(
@@ -247,6 +304,7 @@ def evaluate_strategy_network(
device=device,
max_steps=max_steps,
encoding=encoding,
batch_size=batch_size,
)
@@ -260,35 +318,119 @@ def _evaluate_strategy_network_with_diagnostics(
device: torch.device | str,
max_steps: int,
encoding: EncodingConfig | None,
batch_size: int,
) -> dict[str, float | int]:
if games <= 0:
raise ValueError(f"games must be positive, got {games}")
diagnostics = PolicyEvalDiagnostics()
policy = StrategyNetPolicy(
strategy_network,
device=device,
seed=seed * 2,
encoding=encoding,
)
started = time.perf_counter()
active_games: list[_EvalGame] = []
for index in range(games):
game_seed = seed + index
swap = index % 2 == 1
policy_player = 1 if swap else 0
policy = StrategyNetPolicy(
strategy_network,
device=device,
seed=game_seed * 2 + policy_player,
encoding=encoding,
)
opponent_policy = build_bot(opponent, seed=game_seed * 2 + (1 - policy_player))
policies = [opponent_policy, policy] if swap else [policy, opponent_policy]
game_diag = _evaluate_one_game(
policies,
policy_player,
config,
seed=game_seed,
max_steps=max_steps,
active_games.append(
_EvalGame(
state=GameState.new_game(config, seed=game_seed),
policies=policies,
policy_player=policy_player,
diagnostics=PolicyEvalDiagnostics(games=1),
first_open_recoverable_by_color={},
)
game_diag.runtime.accumulate(policy.runtime)
_accumulate_game_diagnostics(diagnostics, game_diag)
)
while active_games:
pending_policy_games: list[_EvalGame] = []
next_active_games: list[_EvalGame] = []
for game in active_games:
if _finalize_if_done(game, max_steps=max_steps):
_accumulate_game_diagnostics(diagnostics, game.diagnostics)
continue
state = game.state
if state.current_player == game.policy_player and isinstance(
game.policies[state.current_player], StrategyNetPolicy
):
pending_policy_games.append(game)
else:
_advance_opponent_turn(game)
if _finalize_if_done(game, max_steps=max_steps):
_accumulate_game_diagnostics(diagnostics, game.diagnostics)
else:
next_active_games.append(game)
for start in range(0, len(pending_policy_games), max(1, int(batch_size))):
chunk = pending_policy_games[start : start + max(1, int(batch_size))]
actions = policy.select_actions_batch([game.state for game in chunk])
for game, (action, entropy) in zip(chunk, actions, strict=True):
_advance_policy_turn(game, action, entropy)
if _finalize_if_done(game, max_steps=max_steps):
_accumulate_game_diagnostics(diagnostics, game.diagnostics)
else:
next_active_games.append(game)
active_games = next_active_games
diagnostics.runtime.accumulate(policy.runtime)
return diagnostics.to_dict(time.perf_counter() - started)
def _finalize_if_done(game: _EvalGame, *, max_steps: int) -> bool:
if game.done:
return True
if not game.state.terminal and game.steps < max_steps:
return False
timed_out = not game.state.terminal
if timed_out:
game.steps = max_steps
final_started = time.perf_counter()
_record_final_game_state(
game.diagnostics,
game.state,
game.policy_player,
game.steps,
timed_out,
game.first_open_recoverable_by_color,
)
game.diagnostics.runtime.final_scoring_seconds += time.perf_counter() - final_started
game.done = True
return True
def _advance_policy_turn(game: _EvalGame, action: int, entropy: float) -> None:
game.diagnostics.entropies.append(entropy)
diagnostics_started = time.perf_counter()
_record_policy_action(
game.diagnostics,
game.state,
action,
game.first_open_recoverable_by_color,
)
game.diagnostics.runtime.diagnostics_seconds += time.perf_counter() - diagnostics_started
apply_started = time.perf_counter()
game.state.apply_action(action)
game.diagnostics.runtime.apply_action_seconds += time.perf_counter() - apply_started
game.steps += 1
def _advance_opponent_turn(game: _EvalGame) -> None:
state = game.state
game.diagnostics.runtime.opponent_turns += 1
opponent_started = time.perf_counter()
action = game.policies[state.current_player].act(state)
game.diagnostics.runtime.opponent_act_seconds += time.perf_counter() - opponent_started
apply_started = time.perf_counter()
state.apply_action(action)
game.diagnostics.runtime.apply_action_seconds += time.perf_counter() - apply_started
game.steps += 1
def _evaluate_one_game(
policies: list[LostCitiesPolicy],
policy_player: int,
src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py
+113 -3
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import copy
import json
import multiprocessing as mp
import time
@@ -15,7 +16,11 @@ from coolrl_lost_cities.games.classic.deep_cfr.checkpoints import (
load_checkpoint,
save_checkpoint,
)
from coolrl_lost_cities.games.classic.deep_cfr.config import DeepCFRConfig
from coolrl_lost_cities.games.classic.deep_cfr.config import (
DeepCFRConfig,
EncodingConfig,
NetworkConfig,
)
from coolrl_lost_cities.games.classic.deep_cfr.encoding import input_dim
from coolrl_lost_cities.games.classic.deep_cfr.evaluate import evaluate_strategy_network
from coolrl_lost_cities.games.classic.deep_cfr.memory import ReservoirMemory, TrainingSample
@@ -42,6 +47,44 @@ def _resolve_torch_device(device: str) -> torch.device:
return torch.device(token)
@dataclass(frozen=True)
class EvaluationWorkerJob:
opponent: str
strategy_state_dict: dict[str, torch.Tensor]
game_config: dict
network_config: dict
encoding_config: dict
input_dim: int
action_size: int
games: int
seed: int
device: str
max_steps: int
batch_size: int
def run_evaluation_worker(job: EvaluationWorkerJob) -> tuple[str, dict[str, float | int]]:
device = _resolve_torch_device(job.device)
game_config = LostCitiesConfig(**job.game_config)
network_config = NetworkConfig.model_validate(job.network_config)
encoding_config = EncodingConfig.model_validate(job.encoding_config)
network = DeepCFRMLP.from_config(job.input_dim, job.action_size, network_config).to(device)
network.load_state_dict(job.strategy_state_dict)
network.eval()
result = evaluate_strategy_network(
network,
game_config,
games=job.games,
seed=job.seed,
opponent=job.opponent,
device=device,
max_steps=job.max_steps,
encoding=encoding_config,
batch_size=job.batch_size,
)
return job.opponent, result
@dataclass(frozen=True)
class IterationMetrics:
iteration: int
@@ -540,21 +583,88 @@ class DeepCFRTrainer:
):
return {}
results: dict[str, float | int] = {}
eval_device = self._evaluation_device()
if self.config.evaluation.resolved_num_workers(len(self.config.evaluation.opponents)) > 1:
return self._evaluate_parallel(iteration, eval_device)
eval_network = self._evaluation_network(eval_device)
for opponent in self.config.evaluation.opponents:
result = evaluate_strategy_network(
self.strategy_network,
eval_network,
self.game_config,
games=self.config.evaluation.games,
seed=self.config.run.seed + iteration * 1000,
opponent=opponent,
device=self.device,
device=eval_device,
max_steps=self.config.evaluation.max_steps,
encoding=self.config.encoding,
batch_size=self.config.evaluation.resolved_batch_size(),
)
for key, value in result.items():
results[f"eval_{opponent}_{key}"] = value
return results
def _evaluate_parallel(
self,
iteration: int,
eval_device: torch.device,
) -> dict[str, float | int]:
opponents = self.config.evaluation.opponents
max_workers = self.config.evaluation.resolved_num_workers(len(opponents))
self.tracker.log_event(
f"Evaluation multiprocessing enabled iteration={iteration} "
f"effective_workers={max_workers} opponents={len(opponents)} "
f"batch_size={self.config.evaluation.resolved_batch_size()} device={eval_device}"
)
state_dict = self._strategy_state_dict_cpu()
jobs = [
EvaluationWorkerJob(
opponent=opponent,
strategy_state_dict=state_dict,
game_config=self.game_config.to_snapshot(),
network_config=self.config.network.model_dump(mode="python"),
encoding_config=self.config.encoding.model_dump(mode="python"),
input_dim=self.input_dim,
action_size=self.action_size,
games=self.config.evaluation.games,
seed=self.config.run.seed + iteration * 1000,
device=str(eval_device),
max_steps=self.config.evaluation.max_steps,
batch_size=self.config.evaluation.resolved_batch_size(),
)
for opponent in opponents
]
results: dict[str, float | int] = {}
with ProcessPoolExecutor(
max_workers=max_workers,
mp_context=mp.get_context("spawn"),
) as executor:
futures = [executor.submit(run_evaluation_worker, job) for job in jobs]
for future in as_completed(futures):
opponent, result = future.result()
for key, value in result.items():
results[f"eval_{opponent}_{key}"] = value
return results
def _evaluation_device(self) -> torch.device:
token = self.config.evaluation.device
if token == "trainer":
return self.device
if token == "auto":
return torch.device("cuda" if torch.cuda.is_available() else "cpu")
if token == "cuda" and not torch.cuda.is_available():
raise RuntimeError("evaluation.device=cuda requested but CUDA is unavailable")
return torch.device(token)
def _evaluation_network(self, device: torch.device) -> torch.nn.Module:
if device == self.device:
return self.strategy_network
return copy.deepcopy(self.strategy_network).to(device).eval()
def _strategy_state_dict_cpu(self) -> dict[str, torch.Tensor]:
return {
key: value.detach().cpu() for key, value in self.strategy_network.state_dict().items()
}
def _train_advantage_networks(self) -> float:
losses: list[float] = []
for player, (network, memory) in enumerate(
tests/games/classic/test_deep_cfr_trainer.py
+50
View File
@@ -19,7 +19,9 @@ from coolrl_lost_cities.games.classic.deep_cfr.cli import (
_with_overrides,
)
from coolrl_lost_cities.games.classic.deep_cfr.config import DeepCFRConfig, load_config
from coolrl_lost_cities.games.classic.deep_cfr.evaluate import evaluate_strategy_network
from coolrl_lost_cities.games.classic.deep_cfr.memory import ReservoirMemory, TrainingSample
from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP
from coolrl_lost_cities.games.classic.deep_cfr.trainer import DeepCFRTrainer
@@ -59,6 +61,9 @@ def test_deep_cfr_loads_mapped_legacy_reproduction_config() -> None:
assert config.optimization.weight_decay == 0.0001
assert config.optimization.grad_clip == 1.0
assert config.evaluation.on_max_steps == "score_diff"
assert config.evaluation.resolved_batch_size() == 64
assert config.evaluation.device == "trainer"
assert config.evaluation.resolved_num_workers() == 4
assert config.checkpoint.save_iteration_interval == 10
assert (
config.checkpoint.directory == "runs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability"
@@ -128,6 +133,51 @@ def test_deep_cfr_train_cli_checkpoint_save_overrides() -> None:
assert overridden.checkpoint.save_iteration_interval == 1
def test_deep_cfr_batched_evaluation_matches_batch_size_one() -> None:
config = _deep_cfr_config(
{
"network": {"hidden_size": 16},
"encoding": {"derived_playability": True, "slot_aware_playability": True},
}
)
game_config = LostCitiesConfig(seed=123)
state = GameState.new_game(game_config, seed=123)
network = DeepCFRMLP.from_config(
input_dim(state, config.encoding),
game_config.action_size,
config.network,
)
network.eval()
kwargs = {
"strategy_network": network,
"config": game_config,
"games": 8,
"seed": 55,
"opponent": "random",
"device": "cpu",
"max_steps": 200,
"encoding": config.encoding,
}
batch_one = evaluate_strategy_network(**kwargs, batch_size=1)
batched = evaluate_strategy_network(**kwargs, batch_size=64)
for key in [
"games",
"wins0",
"wins1",
"draws",
"avg_score0",
"avg_score1",
"avg_score_diff0",
"avg_game_length",
"max_step_timeouts",
"play_action_rate",
]:
assert batched[key] == batch_one[key]
assert np.isclose(batched["policy_entropy"], batch_one["policy_entropy"])
def test_deep_cfr_resume_latest_resolution_uses_config_checkpoint_dir(tmp_path) -> None:
config = _deep_cfr_config({"checkpoint": {"directory": str(tmp_path)}})
latest = tmp_path / "latest.pt"