diff --git a/configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml b/configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml index 2e9b3e6..109eb89 100644 --- a/configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml +++ b/configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml @@ -72,6 +72,9 @@ memory: evaluation: eval_every: 5 games: 100 + batch_size: 64 + device: trainer + num_workers: 4 max_steps: 1000 on_max_steps: score_diff opponents: diff --git a/docs/deep-cfr-batched-evaluation-2026-05-07.md b/docs/deep-cfr-batched-evaluation-2026-05-07.md new file mode 100644 index 0000000..3b72376 --- /dev/null +++ b/docs/deep-cfr-batched-evaluation-2026-05-07.md @@ -0,0 +1,111 @@ +# Deep CFR Batched Evaluation 2026-05-07 + +Run directory: + +`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter` + +Command: + +```bash +uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \ + --config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \ + --checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter \ + --max-iterations 1 \ + --eval-every 1 \ + --save-latest-only +``` + +The config used `evaluation.batch_size: 64` and `evaluation.device: trainer`. +The base run device was CUDA. + +## Summary + +The run completed one training iteration and evaluated immediately. + +| Metric | Value | +| --- | ---: | +| `iteration_seconds` | 21.322356 | +| `evaluation_seconds` | 14.834096 | +| `traversal_seconds` | 3.850308 | +| `advantage_train_seconds` | 1.778991 | +| `strategy_train_seconds` | 0.847232 | + +Compared with the previous batch-size-1 CUDA check, evaluation time changed +from about 61.826s to about 14.834s. + +## Opponent Timing + +| Opponent | elapsed | avg len | policy turns | network | postprocess | encoding | legal mask | opponent act | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `safe_heuristic_strict` | 3.971 | 1000.0 | 50000 | 0.195 | 0.251 | 0.147 | 0.084 | 3.168 | +| `safe_heuristic` | 3.685 | 997.5 | 49876 | 0.199 | 0.257 | 0.148 | 0.084 | 2.868 | +| `safe_heuristic_loose` | 3.330 | 980.8 | 49038 | 0.194 | 0.250 | 0.145 | 0.082 | 2.536 | +| `noisy_safe` | 2.902 | 955.9 | 47784 | 0.195 | 0.250 | 0.144 | 0.079 | 2.109 | +| `random` | 0.797 | 640.4 | 31968 | 0.154 | 0.204 | 0.095 | 0.053 | 0.212 | +| `passive_discard` | 0.138 | 172.2 | 8562 | 0.032 | 0.042 | 0.025 | 0.014 | 0.004 | + +## Totals + +| Metric | Value | +| --- | ---: | +| `eval_elapsed_seconds` | 14.824448 | +| `eval_policy_turns` | 237228 | +| `eval_policy_network_seconds` | 0.968006 | +| `eval_policy_postprocess_seconds` | 1.253108 | +| `eval_policy_encoding_seconds` | 0.702534 | +| `eval_policy_legal_mask_seconds` | 0.396494 | +| `eval_opponent_act_seconds` | 10.896567 | +| `eval_apply_action_seconds` | 0.068075 | +| `eval_diagnostics_seconds` | 0.060940 | + +## Notes + +Batched network inference removed the previous batch-size-1 CUDA network +bottleneck. After batching, safe heuristic opponent action time became the +largest remaining eval cost for safe heuristic opponents. + +The first batched implementation exposed a postprocess synchronization cost +from per-row entropy calculation. Moving entropy calculation into torch batch +postprocess reduced that cost before this run. + +## Opponent Parallel Evaluation + +Run directory: + +`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter` + +Command: + +```bash +uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \ + --config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \ + --checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter \ + --max-iterations 1 \ + --eval-every 1 \ + --save-latest-only +``` + +This run used `evaluation.num_workers: 4`, `evaluation.batch_size: 64`, and +`evaluation.device: trainer` on CUDA. + +| Metric | Batched sequential | Batched opponent-parallel | +| --- | ---: | ---: | +| `iteration_seconds` | 21.322356 | 12.854383 | +| `evaluation_seconds` | 14.834096 | 6.420402 | +| `traversal_seconds` | 3.850308 | 3.852627 | +| `advantage_train_seconds` | 1.778991 | 1.746623 | +| `strategy_train_seconds` | 0.847232 | 0.819998 | + +Opponent elapsed values from the parallel run: + +| Opponent | elapsed | opponent act | network | avg len | +| --- | ---: | ---: | ---: | ---: | +| `safe_heuristic_strict` | 4.121 | 3.225 | 0.196 | 1000.0 | +| `safe_heuristic` | 4.029 | 2.863 | 0.281 | 997.5 | +| `safe_heuristic_loose` | 3.807 | 2.631 | 0.286 | 981.6 | +| `noisy_safe` | 3.026 | 2.141 | 0.211 | 951.7 | +| `random` | 1.130 | 0.212 | 0.253 | 639.0 | +| `passive_discard` | 0.417 | 0.004 | 0.114 | 172.2 | + +Parallel eval reduced measured eval wall time from about 14.83s to about +6.42s, roughly `2.31x` faster for this one-iteration profile. diff --git a/docs/deep-cfr-evaluation-profile-2026-05-07.md b/docs/deep-cfr-evaluation-profile-2026-05-07.md index 0495c69..5dd6519 100644 --- a/docs/deep-cfr-evaluation-profile-2026-05-07.md +++ b/docs/deep-cfr-evaluation-profile-2026-05-07.md @@ -62,3 +62,92 @@ still smaller than policy network time. `apply_action_seconds`, `diagnostics_seconds`, and `final_scoring_seconds` were small in this run. + +## CPU vs CUDA Evaluation Check + +This check compared `--device cpu` and `--device cuda` on the same base +configuration after the evaluation breakdown metrics were available. The base +configuration was: + +`configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml` + +Both runs used one training iteration and ran evaluation on that iteration. +The base configuration's evaluation settings were kept at 100 games per +opponent and the six configured opponents. + +CPU run directory: + +`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter` + +CPU command: + +```bash +uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \ + --config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \ + --checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter \ + --max-iterations 1 \ + --eval-every 1 \ + --save-latest-only \ + --device cpu +``` + +CUDA run directory: + +`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter` + +CUDA command: + +```bash +uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \ + --config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \ + --checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter \ + --max-iterations 1 \ + --eval-every 1 \ + --save-latest-only \ + --device cuda +``` + +Top-level timing: + +| Metric | CPU | CUDA | CUDA / CPU | +| --- | ---: | ---: | ---: | +| `iteration_seconds` | 50.969 | 72.501 | 1.42 | +| `evaluation_seconds` | 38.920 | 61.826 | 1.59 | +| `traversal_seconds` | 6.343 | 6.409 | 1.01 | +| `advantage_train_seconds` | 4.054 | 2.884 | 0.71 | +| `strategy_train_seconds` | 1.643 | 1.369 | 0.83 | + +Opponent elapsed timing: + +| Opponent | CPU elapsed | CUDA elapsed | CUDA / CPU | +| --- | ---: | ---: | ---: | +| `random` | 4.038 | 7.220 | 1.79 | +| `passive_discard` | 0.990 | 1.773 | 1.79 | +| `safe_heuristic` | 8.701 | 13.518 | 1.55 | +| `safe_heuristic_loose` | 8.334 | 13.145 | 1.58 | +| `safe_heuristic_strict` | 9.097 | 13.882 | 1.53 | +| `noisy_safe` | 7.751 | 12.281 | 1.58 | + +Opponent breakdown for the CPU run: + +| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `random` | 4.038 | 32170 | 2.401 | 0.075 ms | 0.586 | 0.169 | 0.114 | 0.361 | 644.4 | +| `passive_discard` | 0.990 | 8562 | 0.633 | 0.074 ms | 0.155 | 0.044 | 0.034 | 0.010 | 172.2 | +| `safe_heuristic` | 8.701 | 49756 | 3.650 | 0.073 ms | 0.903 | 0.253 | 0.211 | 3.068 | 995.1 | +| `safe_heuristic_loose` | 8.334 | 48958 | 3.592 | 0.073 ms | 0.886 | 0.248 | 0.207 | 2.798 | 979.2 | +| `safe_heuristic_strict` | 9.097 | 50000 | 3.678 | 0.074 ms | 0.910 | 0.254 | 0.214 | 3.420 | 1000.0 | +| `noisy_safe` | 7.751 | 47276 | 3.501 | 0.074 ms | 0.864 | 0.244 | 0.195 | 2.354 | 945.7 | +| Total | 38.911 | 236722 | 17.456 | 0.074 ms | 4.304 | 1.212 | 0.975 | 12.011 | | + +Opponent breakdown for the CUDA run: + +| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| `random` | 7.220 | 32096 | 5.337 | 0.166 ms | 0.716 | 0.179 | 0.129 | 0.384 | 642.9 | +| `passive_discard` | 1.773 | 8562 | 1.364 | 0.159 ms | 0.185 | 0.047 | 0.039 | 0.012 | 172.2 | +| `safe_heuristic` | 13.518 | 49876 | 8.053 | 0.161 ms | 1.097 | 0.274 | 0.239 | 3.139 | 997.5 | +| `safe_heuristic_loose` | 13.145 | 49078 | 7.967 | 0.162 ms | 1.080 | 0.271 | 0.236 | 2.890 | 981.6 | +| `safe_heuristic_strict` | 13.882 | 50000 | 8.034 | 0.161 ms | 1.096 | 0.274 | 0.239 | 3.524 | 1000.0 | +| `noisy_safe` | 12.281 | 47266 | 7.650 | 0.162 ms | 1.044 | 0.262 | 0.219 | 2.421 | 945.5 | +| Total | 61.819 | 236878 | 38.406 | 0.162 ms | 5.218 | 1.307 | 1.101 | 12.370 | | diff --git a/docs/deep-cfr-legacy-runtime-comparison-2026-05-07.md b/docs/deep-cfr-legacy-runtime-comparison-2026-05-07.md new file mode 100644 index 0000000..a94762a --- /dev/null +++ b/docs/deep-cfr-legacy-runtime-comparison-2026-05-07.md @@ -0,0 +1,99 @@ +# Deep CFR Legacy Runtime Comparison 2026-05-07 + +This note records a runtime summary from the older `../coolrl` Lost Cities +Deep CFR implementation and compares it with the current profiling runs in +this repository. + +## Legacy Run Summary + +The older run completed metrics through iteration 387. The process had stopped +before completing iteration 388. + +Total elapsed time through iteration 387: + +`5879.17s`, or about `1h 37m 59s`. + +| Segment | Mean | Median | Note | +| --- | ---: | ---: | --- | +| All iterations | 15.19s/iter | 11.44s | Includes eval iterations | +| Non-eval iterations | 11.51s/iter | 11.32s | Normal training iteration | +| Eval iterations | 30.00s/iter | 29.31s | Eval every 5 iterations | +| Evaluation only | 18.61s/eval | 18.04s | Early evals were slower | +| Traversal | 7.16s/iter | 6.98s | 140 traversals/iter | +| Advantage train | 2.81s/iter | 2.78s | Player 0 + player 1 | +| Strategy train | 1.44s/iter | 1.44s | | +| Overall throughput | 5387 nodes/s | 5437 nodes/s | | +| Traversal throughput | 19.8 traversals/s | 20.1 traversals/s | | + +Recent 50 iteration window from that run: + +| Segment | Mean | +| --- | ---: | +| All iterations | 15.57s/iter | +| Non-eval iterations | 12.21s/iter | +| Recent 20 evals, eval only | 17.31s/eval | +| Recent 20 eval iterations | 29.30s/iter | + +Evaluation ran every 5 iterations: 5, 10, 15, ..., 385. + +The practical legacy cadence was roughly: + +`4 normal iterations + 1 eval iteration ~= 75s per 5 iterations`. + +## Current Repo Reference Points + +From `docs/deep-cfr-profile-advantage-memory-split-2026-05-07.md`: + +| Segment | Current mean | +| --- | ---: | +| Non-eval iterations | 5.832958s/iter | +| Traversal | 3.160975s/iter | +| Advantage train | 1.742895s/iter | +| Strategy train | 0.912324s/iter | + +From `docs/deep-cfr-batched-evaluation-2026-05-07.md`: + +| Segment | Current value | +| --- | ---: | +| Batched CUDA evaluation | 14.834096s/eval | +| Batched CUDA 1-iter wall time with eval | 21.322356s | +| Batched opponent-parallel CUDA evaluation | 6.420402s/eval | +| Batched opponent-parallel CUDA 1-iter wall time with eval | 12.854383s | + +## Rough Comparison + +Normal training iterations improved from about `11.51s` to about `5.83s`, +roughly `1.97x` faster. + +Traversal improved from about `7.16s` to about `3.16s`, roughly `2.27x` +faster. + +Evaluation improved from about `18.61s` to about `14.83s`, roughly `1.25x` +faster for the measured batched CUDA profile. With opponent-parallel eval, the +measured eval time was about `6.42s`, roughly `2.90x` faster than the legacy +eval-only average. + +Using the simple cadence model: + +Legacy: + +`4 * 11.51 + (11.51 + 18.61) = 76.16s per 5 iterations` + +Current batched sequential: + +`4 * 5.83 + (5.83 + 14.83) = 43.98s per 5 iterations` + +Current batched opponent-parallel: + +`4 * 5.83 + (5.83 + 6.42) = 35.57s per 5 iterations` + +That implies about `1.73x` faster eval-included wall time for the batched +sequential rough comparison, and about `2.14x` faster for the batched +opponent-parallel rough comparison. + +## Caveat + +The legacy numbers came from a long run through iteration 387. The current +numbers are from targeted profiling runs. The comparison is useful for order of +magnitude and bottleneck direction, not as a strict benchmark under identical +runtime conditions. diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/config.py b/src/coolrl_lost_cities/games/classic/deep_cfr/config.py index 04bce04..1e17c58 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/config.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/config.py @@ -232,6 +232,9 @@ class EvaluationConfig(StrictModel): opponents: tuple[str, ...] = ("random",) max_steps: int = 10_000 on_max_steps: str = "score_diff" + batch_size: int = 64 + device: str = "trainer" + num_workers: int = 4 @field_validator("on_max_steps") @classmethod @@ -241,6 +244,23 @@ class EvaluationConfig(StrictModel): raise ValueError("must be 'score_diff', 'loss', or 'draw'") return token + @field_validator("device") + @classmethod + def _validate_device(cls, value: str) -> str: + token = value.strip().lower() + if token not in {"trainer", "auto", "cpu", "cuda"}: + raise ValueError("must be 'trainer', 'auto', 'cpu', or 'cuda'") + return token + + def resolved_batch_size(self) -> int: + return max(1, int(self.batch_size)) + + def resolved_num_workers(self, opponent_count: int | None = None) -> int: + workers = max(1, int(self.num_workers)) + if opponent_count is not None: + workers = min(workers, max(1, int(opponent_count))) + return workers + class DeepCFRConfig(StrictModel): run: RunConfig = Field(default_factory=RunConfig) diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py b/src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py index fd6b811..93c4c95 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/evaluate.py @@ -159,6 +159,17 @@ class PolicyEvalDiagnostics: return data +@dataclass +class _EvalGame: + state: GameState + policies: list[LostCitiesPolicy] + policy_player: int + diagnostics: PolicyEvalDiagnostics + first_open_recoverable_by_color: dict[int, float] + steps: int = 0 + done: bool = False + + class StrategyNetPolicy(LostCitiesPolicy): def __init__( self, @@ -176,6 +187,51 @@ class StrategyNetPolicy(LostCitiesPolicy): self.encoding = encoding self.runtime = EvalRuntimeCounters() + def select_actions_batch(self, states: list[GameState]) -> list[tuple[int, float]]: + if not states: + return [] + started = time.perf_counter() + self.runtime.policy_turns += len(states) + + legal_started = time.perf_counter() + legal_masks = [np.asarray(state.unified_legal_mask(), dtype=bool) for state in states] + legal_actions_list = [np.flatnonzero(legal) for legal in legal_masks] + self.runtime.policy_legal_mask_seconds += time.perf_counter() - legal_started + if any(len(legal_actions) == 0 for legal_actions in legal_actions_list): + raise RuntimeError("no legal action available") + + encoding_started = time.perf_counter() + infos = [encode_info_state(state, state.current_player, self.encoding) for state in states] + self.runtime.policy_encoding_seconds += time.perf_counter() - encoding_started + + network_started = time.perf_counter() + with torch.inference_mode(): + x = torch.as_tensor(np.stack(infos), dtype=torch.float32, device=self.device) + logits = self.strategy_network(x) + self.runtime.policy_network_seconds += time.perf_counter() - network_started + + postprocess_started = time.perf_counter() + legal_tensor = torch.as_tensor(np.stack(legal_masks), dtype=torch.bool, device=self.device) + masked = logits.masked_fill(~legal_tensor, torch.finfo(torch.float32).min) + probs_tensor = torch.softmax(masked, dim=-1).masked_fill(~legal_tensor, 0.0) + entropy_tensor = -(probs_tensor * probs_tensor.clamp_min(1.0e-12).log()).sum(dim=-1) + if self.sample: + probs_np = probs_tensor.detach().cpu().numpy() + unified_actions = [ + int(self.rng.choice(legal_actions, p=probs_np[index][legal_actions])) + for index, legal_actions in enumerate(legal_actions_list) + ] + else: + unified_actions = [int(value) for value in torch.argmax(masked, dim=-1).cpu()] + entropies = [float(value) for value in entropy_tensor.cpu()] + actions = [ + state.from_unified_action(unified) + for state, unified in zip(states, unified_actions, strict=True) + ] + self.runtime.policy_postprocess_seconds += time.perf_counter() - postprocess_started + self.runtime.policy_select_seconds += time.perf_counter() - started + return list(zip(actions, entropies, strict=True)) + def action_distribution(self, state: GameState) -> tuple[np.ndarray, np.ndarray]: started = time.perf_counter() legal = np.asarray(state.unified_legal_mask(), dtype=bool) @@ -236,6 +292,7 @@ def evaluate_strategy_network( device: torch.device | str = "cpu", max_steps: int = 10_000, encoding: EncodingConfig | None = None, + batch_size: int = 64, ) -> dict[str, float | int]: strategy_network.eval() return _evaluate_strategy_network_with_diagnostics( @@ -247,6 +304,7 @@ def evaluate_strategy_network( device=device, max_steps=max_steps, encoding=encoding, + batch_size=batch_size, ) @@ -260,35 +318,119 @@ def _evaluate_strategy_network_with_diagnostics( device: torch.device | str, max_steps: int, encoding: EncodingConfig | None, + batch_size: int, ) -> dict[str, float | int]: if games <= 0: raise ValueError(f"games must be positive, got {games}") diagnostics = PolicyEvalDiagnostics() + policy = StrategyNetPolicy( + strategy_network, + device=device, + seed=seed * 2, + encoding=encoding, + ) started = time.perf_counter() + active_games: list[_EvalGame] = [] for index in range(games): game_seed = seed + index swap = index % 2 == 1 policy_player = 1 if swap else 0 - policy = StrategyNetPolicy( - strategy_network, - device=device, - seed=game_seed * 2 + policy_player, - encoding=encoding, - ) opponent_policy = build_bot(opponent, seed=game_seed * 2 + (1 - policy_player)) policies = [opponent_policy, policy] if swap else [policy, opponent_policy] - game_diag = _evaluate_one_game( - policies, - policy_player, - config, - seed=game_seed, - max_steps=max_steps, + active_games.append( + _EvalGame( + state=GameState.new_game(config, seed=game_seed), + policies=policies, + policy_player=policy_player, + diagnostics=PolicyEvalDiagnostics(games=1), + first_open_recoverable_by_color={}, + ) ) - game_diag.runtime.accumulate(policy.runtime) - _accumulate_game_diagnostics(diagnostics, game_diag) + + while active_games: + pending_policy_games: list[_EvalGame] = [] + next_active_games: list[_EvalGame] = [] + for game in active_games: + if _finalize_if_done(game, max_steps=max_steps): + _accumulate_game_diagnostics(diagnostics, game.diagnostics) + continue + state = game.state + if state.current_player == game.policy_player and isinstance( + game.policies[state.current_player], StrategyNetPolicy + ): + pending_policy_games.append(game) + else: + _advance_opponent_turn(game) + if _finalize_if_done(game, max_steps=max_steps): + _accumulate_game_diagnostics(diagnostics, game.diagnostics) + else: + next_active_games.append(game) + + for start in range(0, len(pending_policy_games), max(1, int(batch_size))): + chunk = pending_policy_games[start : start + max(1, int(batch_size))] + actions = policy.select_actions_batch([game.state for game in chunk]) + for game, (action, entropy) in zip(chunk, actions, strict=True): + _advance_policy_turn(game, action, entropy) + if _finalize_if_done(game, max_steps=max_steps): + _accumulate_game_diagnostics(diagnostics, game.diagnostics) + else: + next_active_games.append(game) + + active_games = next_active_games + diagnostics.runtime.accumulate(policy.runtime) return diagnostics.to_dict(time.perf_counter() - started) +def _finalize_if_done(game: _EvalGame, *, max_steps: int) -> bool: + if game.done: + return True + if not game.state.terminal and game.steps < max_steps: + return False + timed_out = not game.state.terminal + if timed_out: + game.steps = max_steps + final_started = time.perf_counter() + _record_final_game_state( + game.diagnostics, + game.state, + game.policy_player, + game.steps, + timed_out, + game.first_open_recoverable_by_color, + ) + game.diagnostics.runtime.final_scoring_seconds += time.perf_counter() - final_started + game.done = True + return True + + +def _advance_policy_turn(game: _EvalGame, action: int, entropy: float) -> None: + game.diagnostics.entropies.append(entropy) + diagnostics_started = time.perf_counter() + _record_policy_action( + game.diagnostics, + game.state, + action, + game.first_open_recoverable_by_color, + ) + game.diagnostics.runtime.diagnostics_seconds += time.perf_counter() - diagnostics_started + apply_started = time.perf_counter() + game.state.apply_action(action) + game.diagnostics.runtime.apply_action_seconds += time.perf_counter() - apply_started + game.steps += 1 + + +def _advance_opponent_turn(game: _EvalGame) -> None: + state = game.state + game.diagnostics.runtime.opponent_turns += 1 + opponent_started = time.perf_counter() + action = game.policies[state.current_player].act(state) + game.diagnostics.runtime.opponent_act_seconds += time.perf_counter() - opponent_started + apply_started = time.perf_counter() + state.apply_action(action) + game.diagnostics.runtime.apply_action_seconds += time.perf_counter() - apply_started + game.steps += 1 + + def _evaluate_one_game( policies: list[LostCitiesPolicy], policy_player: int, diff --git a/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py b/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py index 8a46c6f..4808498 100644 --- a/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py +++ b/src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py @@ -1,5 +1,6 @@ from __future__ import annotations +import copy import json import multiprocessing as mp import time @@ -15,7 +16,11 @@ from coolrl_lost_cities.games.classic.deep_cfr.checkpoints import ( load_checkpoint, save_checkpoint, ) -from coolrl_lost_cities.games.classic.deep_cfr.config import DeepCFRConfig +from coolrl_lost_cities.games.classic.deep_cfr.config import ( + DeepCFRConfig, + EncodingConfig, + NetworkConfig, +) from coolrl_lost_cities.games.classic.deep_cfr.encoding import input_dim from coolrl_lost_cities.games.classic.deep_cfr.evaluate import evaluate_strategy_network from coolrl_lost_cities.games.classic.deep_cfr.memory import ReservoirMemory, TrainingSample @@ -42,6 +47,44 @@ def _resolve_torch_device(device: str) -> torch.device: return torch.device(token) +@dataclass(frozen=True) +class EvaluationWorkerJob: + opponent: str + strategy_state_dict: dict[str, torch.Tensor] + game_config: dict + network_config: dict + encoding_config: dict + input_dim: int + action_size: int + games: int + seed: int + device: str + max_steps: int + batch_size: int + + +def run_evaluation_worker(job: EvaluationWorkerJob) -> tuple[str, dict[str, float | int]]: + device = _resolve_torch_device(job.device) + game_config = LostCitiesConfig(**job.game_config) + network_config = NetworkConfig.model_validate(job.network_config) + encoding_config = EncodingConfig.model_validate(job.encoding_config) + network = DeepCFRMLP.from_config(job.input_dim, job.action_size, network_config).to(device) + network.load_state_dict(job.strategy_state_dict) + network.eval() + result = evaluate_strategy_network( + network, + game_config, + games=job.games, + seed=job.seed, + opponent=job.opponent, + device=device, + max_steps=job.max_steps, + encoding=encoding_config, + batch_size=job.batch_size, + ) + return job.opponent, result + + @dataclass(frozen=True) class IterationMetrics: iteration: int @@ -540,21 +583,88 @@ class DeepCFRTrainer: ): return {} results: dict[str, float | int] = {} + eval_device = self._evaluation_device() + if self.config.evaluation.resolved_num_workers(len(self.config.evaluation.opponents)) > 1: + return self._evaluate_parallel(iteration, eval_device) + eval_network = self._evaluation_network(eval_device) for opponent in self.config.evaluation.opponents: result = evaluate_strategy_network( - self.strategy_network, + eval_network, self.game_config, games=self.config.evaluation.games, seed=self.config.run.seed + iteration * 1000, opponent=opponent, - device=self.device, + device=eval_device, max_steps=self.config.evaluation.max_steps, encoding=self.config.encoding, + batch_size=self.config.evaluation.resolved_batch_size(), ) for key, value in result.items(): results[f"eval_{opponent}_{key}"] = value return results + def _evaluate_parallel( + self, + iteration: int, + eval_device: torch.device, + ) -> dict[str, float | int]: + opponents = self.config.evaluation.opponents + max_workers = self.config.evaluation.resolved_num_workers(len(opponents)) + self.tracker.log_event( + f"Evaluation multiprocessing enabled iteration={iteration} " + f"effective_workers={max_workers} opponents={len(opponents)} " + f"batch_size={self.config.evaluation.resolved_batch_size()} device={eval_device}" + ) + state_dict = self._strategy_state_dict_cpu() + jobs = [ + EvaluationWorkerJob( + opponent=opponent, + strategy_state_dict=state_dict, + game_config=self.game_config.to_snapshot(), + network_config=self.config.network.model_dump(mode="python"), + encoding_config=self.config.encoding.model_dump(mode="python"), + input_dim=self.input_dim, + action_size=self.action_size, + games=self.config.evaluation.games, + seed=self.config.run.seed + iteration * 1000, + device=str(eval_device), + max_steps=self.config.evaluation.max_steps, + batch_size=self.config.evaluation.resolved_batch_size(), + ) + for opponent in opponents + ] + results: dict[str, float | int] = {} + with ProcessPoolExecutor( + max_workers=max_workers, + mp_context=mp.get_context("spawn"), + ) as executor: + futures = [executor.submit(run_evaluation_worker, job) for job in jobs] + for future in as_completed(futures): + opponent, result = future.result() + for key, value in result.items(): + results[f"eval_{opponent}_{key}"] = value + return results + + def _evaluation_device(self) -> torch.device: + token = self.config.evaluation.device + if token == "trainer": + return self.device + if token == "auto": + return torch.device("cuda" if torch.cuda.is_available() else "cpu") + if token == "cuda" and not torch.cuda.is_available(): + raise RuntimeError("evaluation.device=cuda requested but CUDA is unavailable") + return torch.device(token) + + def _evaluation_network(self, device: torch.device) -> torch.nn.Module: + if device == self.device: + return self.strategy_network + return copy.deepcopy(self.strategy_network).to(device).eval() + + def _strategy_state_dict_cpu(self) -> dict[str, torch.Tensor]: + return { + key: value.detach().cpu() for key, value in self.strategy_network.state_dict().items() + } + def _train_advantage_networks(self) -> float: losses: list[float] = [] for player, (network, memory) in enumerate( diff --git a/tests/games/classic/test_deep_cfr_trainer.py b/tests/games/classic/test_deep_cfr_trainer.py index 241d57d..704150e 100644 --- a/tests/games/classic/test_deep_cfr_trainer.py +++ b/tests/games/classic/test_deep_cfr_trainer.py @@ -19,7 +19,9 @@ from coolrl_lost_cities.games.classic.deep_cfr.cli import ( _with_overrides, ) from coolrl_lost_cities.games.classic.deep_cfr.config import DeepCFRConfig, load_config +from coolrl_lost_cities.games.classic.deep_cfr.evaluate import evaluate_strategy_network from coolrl_lost_cities.games.classic.deep_cfr.memory import ReservoirMemory, TrainingSample +from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP from coolrl_lost_cities.games.classic.deep_cfr.trainer import DeepCFRTrainer @@ -59,6 +61,9 @@ def test_deep_cfr_loads_mapped_legacy_reproduction_config() -> None: assert config.optimization.weight_decay == 0.0001 assert config.optimization.grad_clip == 1.0 assert config.evaluation.on_max_steps == "score_diff" + assert config.evaluation.resolved_batch_size() == 64 + assert config.evaluation.device == "trainer" + assert config.evaluation.resolved_num_workers() == 4 assert config.checkpoint.save_iteration_interval == 10 assert ( config.checkpoint.directory == "runs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability" @@ -128,6 +133,51 @@ def test_deep_cfr_train_cli_checkpoint_save_overrides() -> None: assert overridden.checkpoint.save_iteration_interval == 1 +def test_deep_cfr_batched_evaluation_matches_batch_size_one() -> None: + config = _deep_cfr_config( + { + "network": {"hidden_size": 16}, + "encoding": {"derived_playability": True, "slot_aware_playability": True}, + } + ) + game_config = LostCitiesConfig(seed=123) + state = GameState.new_game(game_config, seed=123) + network = DeepCFRMLP.from_config( + input_dim(state, config.encoding), + game_config.action_size, + config.network, + ) + network.eval() + kwargs = { + "strategy_network": network, + "config": game_config, + "games": 8, + "seed": 55, + "opponent": "random", + "device": "cpu", + "max_steps": 200, + "encoding": config.encoding, + } + + batch_one = evaluate_strategy_network(**kwargs, batch_size=1) + batched = evaluate_strategy_network(**kwargs, batch_size=64) + + for key in [ + "games", + "wins0", + "wins1", + "draws", + "avg_score0", + "avg_score1", + "avg_score_diff0", + "avg_game_length", + "max_step_timeouts", + "play_action_rate", + ]: + assert batched[key] == batch_one[key] + assert np.isclose(batched["policy_entropy"], batch_one["policy_entropy"]) + + def test_deep_cfr_resume_latest_resolution_uses_config_checkpoint_dir(tmp_path) -> None: config = _deep_cfr_config({"checkpoint": {"directory": str(tmp_path)}}) latest = tmp_path / "latest.pt"