coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Deep CFR evaluation 배칭 및 병렬화 추가

Browse Source
This commit is contained in:
coolguy
2026-05-07 03:42:00 +09:00
parent 54347e1e7f
commit 148be6e9a0
8 changed files with 641 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deep-cfr-batched-evaluation-2026-05-07.md
+111
View File
@@ -0,0 +1,111 @@
# Deep CFR Batched Evaluation 2026-05-07
Run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter`
Command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_032944_deep_cfr_batched_eval_cuda_postprocess_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only
```
The config used `evaluation.batch_size: 64` and `evaluation.device: trainer`.
The base run device was CUDA.
## Summary
The run completed one training iteration and evaluated immediately.
| Metric | Value |
| --- | ---: |
| `iteration_seconds` | 21.322356 |
| `evaluation_seconds` | 14.834096 |
| `traversal_seconds` | 3.850308 |
| `advantage_train_seconds` | 1.778991 |
| `strategy_train_seconds` | 0.847232 |
Compared with the previous batch-size-1 CUDA check, evaluation time changed
from about 61.826s to about 14.834s.
## Opponent Timing
| Opponent | elapsed | avg len | policy turns | network | postprocess | encoding | legal mask | opponent act |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `safe_heuristic_strict` | 3.971 | 1000.0 | 50000 | 0.195 | 0.251 | 0.147 | 0.084 | 3.168 |
| `safe_heuristic` | 3.685 | 997.5 | 49876 | 0.199 | 0.257 | 0.148 | 0.084 | 2.868 |
| `safe_heuristic_loose` | 3.330 | 980.8 | 49038 | 0.194 | 0.250 | 0.145 | 0.082 | 2.536 |
| `noisy_safe` | 2.902 | 955.9 | 47784 | 0.195 | 0.250 | 0.144 | 0.079 | 2.109 |
| `random` | 0.797 | 640.4 | 31968 | 0.154 | 0.204 | 0.095 | 0.053 | 0.212 |
| `passive_discard` | 0.138 | 172.2 | 8562 | 0.032 | 0.042 | 0.025 | 0.014 | 0.004 |
## Totals
| Metric | Value |
| --- | ---: |
| `eval_elapsed_seconds` | 14.824448 |
| `eval_policy_turns` | 237228 |
| `eval_policy_network_seconds` | 0.968006 |
| `eval_policy_postprocess_seconds` | 1.253108 |
| `eval_policy_encoding_seconds` | 0.702534 |
| `eval_policy_legal_mask_seconds` | 0.396494 |
| `eval_opponent_act_seconds` | 10.896567 |
| `eval_apply_action_seconds` | 0.068075 |
| `eval_diagnostics_seconds` | 0.060940 |
## Notes
Batched network inference removed the previous batch-size-1 CUDA network
bottleneck. After batching, safe heuristic opponent action time became the
largest remaining eval cost for safe heuristic opponents.
The first batched implementation exposed a postprocess synchronization cost
from per-row entropy calculation. Moving entropy calculation into torch batch
postprocess reduced that cost before this run.
## Opponent Parallel Evaluation
Run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter`
Command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_033925_deep_cfr_batched_eval_parallel_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only
```
This run used `evaluation.num_workers: 4`, `evaluation.batch_size: 64`, and
`evaluation.device: trainer` on CUDA.
| Metric | Batched sequential | Batched opponent-parallel |
| --- | ---: | ---: |
| `iteration_seconds` | 21.322356 | 12.854383 |
| `evaluation_seconds` | 14.834096 | 6.420402 |
| `traversal_seconds` | 3.850308 | 3.852627 |
| `advantage_train_seconds` | 1.778991 | 1.746623 |
| `strategy_train_seconds` | 0.847232 | 0.819998 |
Opponent elapsed values from the parallel run:
| Opponent | elapsed | opponent act | network | avg len |
| --- | ---: | ---: | ---: | ---: |
| `safe_heuristic_strict` | 4.121 | 3.225 | 0.196 | 1000.0 |
| `safe_heuristic` | 4.029 | 2.863 | 0.281 | 997.5 |
| `safe_heuristic_loose` | 3.807 | 2.631 | 0.286 | 981.6 |
| `noisy_safe` | 3.026 | 2.141 | 0.211 | 951.7 |
| `random` | 1.130 | 0.212 | 0.253 | 639.0 |
| `passive_discard` | 0.417 | 0.004 | 0.114 | 172.2 |
Parallel eval reduced measured eval wall time from about 14.83s to about
6.42s, roughly `2.31x` faster for this one-iteration profile.
docs/deep-cfr-evaluation-profile-2026-05-07.md
+89
View File
@@ -62,3 +62,92 @@ still smaller than policy network time.
`apply_action_seconds`, `diagnostics_seconds`, and `final_scoring_seconds` were
small in this run.
## CPU vs CUDA Evaluation Check
This check compared `--device cpu` and `--device cuda` on the same base
configuration after the evaluation breakdown metrics were available. The base
configuration was:
`configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml`
Both runs used one training iteration and ran evaluation on that iteration.
The base configuration's evaluation settings were kept at 100 games per
opponent and the six configured opponents.
CPU run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter`
CPU command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cpu_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only \
--device cpu
```
CUDA run directory:
`/mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter`
CUDA command:
```bash
uv run python -m coolrl_lost_cities.games.classic.deep_cfr.cli train \
--config configs/deep_cfr/deep_cfr_selfplay_full_depth_slot_playability.yaml \
--checkpoint-dir /mnt/2tbhdd/coolrl-lost-cities-runs/2026-05-07_eval_device_cuda_1iter \
--max-iterations 1 \
--eval-every 1 \
--save-latest-only \
--device cuda
```
Top-level timing:
| Metric | CPU | CUDA | CUDA / CPU |
| --- | ---: | ---: | ---: |
| `iteration_seconds` | 50.969 | 72.501 | 1.42 |
| `evaluation_seconds` | 38.920 | 61.826 | 1.59 |
| `traversal_seconds` | 6.343 | 6.409 | 1.01 |
| `advantage_train_seconds` | 4.054 | 2.884 | 0.71 |
| `strategy_train_seconds` | 1.643 | 1.369 | 0.83 |
Opponent elapsed timing:
| Opponent | CPU elapsed | CUDA elapsed | CUDA / CPU |
| --- | ---: | ---: | ---: |
| `random` | 4.038 | 7.220 | 1.79 |
| `passive_discard` | 0.990 | 1.773 | 1.79 |
| `safe_heuristic` | 8.701 | 13.518 | 1.55 |
| `safe_heuristic_loose` | 8.334 | 13.145 | 1.58 |
| `safe_heuristic_strict` | 9.097 | 13.882 | 1.53 |
| `noisy_safe` | 7.751 | 12.281 | 1.58 |
Opponent breakdown for the CPU run:
| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `random` | 4.038 | 32170 | 2.401 | 0.075 ms | 0.586 | 0.169 | 0.114 | 0.361 | 644.4 |
| `passive_discard` | 0.990 | 8562 | 0.633 | 0.074 ms | 0.155 | 0.044 | 0.034 | 0.010 | 172.2 |
| `safe_heuristic` | 8.701 | 49756 | 3.650 | 0.073 ms | 0.903 | 0.253 | 0.211 | 3.068 | 995.1 |
| `safe_heuristic_loose` | 8.334 | 48958 | 3.592 | 0.073 ms | 0.886 | 0.248 | 0.207 | 2.798 | 979.2 |
| `safe_heuristic_strict` | 9.097 | 50000 | 3.678 | 0.074 ms | 0.910 | 0.254 | 0.214 | 3.420 | 1000.0 |
| `noisy_safe` | 7.751 | 47276 | 3.501 | 0.074 ms | 0.864 | 0.244 | 0.195 | 2.354 | 945.7 |
| Total | 38.911 | 236722 | 17.456 | 0.074 ms | 4.304 | 1.212 | 0.975 | 12.011 | |
Opponent breakdown for the CUDA run:
| Opponent | elapsed | policy turns | network | network / turn | postprocess | encoding | legal mask | opponent act | avg len |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| `random` | 7.220 | 32096 | 5.337 | 0.166 ms | 0.716 | 0.179 | 0.129 | 0.384 | 642.9 |
| `passive_discard` | 1.773 | 8562 | 1.364 | 0.159 ms | 0.185 | 0.047 | 0.039 | 0.012 | 172.2 |
| `safe_heuristic` | 13.518 | 49876 | 8.053 | 0.161 ms | 1.097 | 0.274 | 0.239 | 3.139 | 997.5 |
| `safe_heuristic_loose` | 13.145 | 49078 | 7.967 | 0.162 ms | 1.080 | 0.271 | 0.236 | 2.890 | 981.6 |
| `safe_heuristic_strict` | 13.882 | 50000 | 8.034 | 0.161 ms | 1.096 | 0.274 | 0.239 | 3.524 | 1000.0 |
| `noisy_safe` | 12.281 | 47266 | 7.650 | 0.162 ms | 1.044 | 0.262 | 0.219 | 2.421 | 945.5 |
| Total | 61.819 | 236878 | 38.406 | 0.162 ms | 5.218 | 1.307 | 1.101 | 12.370 | |
docs/deep-cfr-legacy-runtime-comparison-2026-05-07.md
+99
View File
@@ -0,0 +1,99 @@
# Deep CFR Legacy Runtime Comparison 2026-05-07
This note records a runtime summary from the older `../coolrl` Lost Cities
Deep CFR implementation and compares it with the current profiling runs in
this repository.
## Legacy Run Summary
The older run completed metrics through iteration 387. The process had stopped
before completing iteration 388.
Total elapsed time through iteration 387:
`5879.17s`, or about `1h 37m 59s`.
| Segment | Mean | Median | Note |
| --- | ---: | ---: | --- |
| All iterations | 15.19s/iter | 11.44s | Includes eval iterations |
| Non-eval iterations | 11.51s/iter | 11.32s | Normal training iteration |
| Eval iterations | 30.00s/iter | 29.31s | Eval every 5 iterations |
| Evaluation only | 18.61s/eval | 18.04s | Early evals were slower |
| Traversal | 7.16s/iter | 6.98s | 140 traversals/iter |
| Advantage train | 2.81s/iter | 2.78s | Player 0 + player 1 |
| Strategy train | 1.44s/iter | 1.44s | |
| Overall throughput | 5387 nodes/s | 5437 nodes/s | |
| Traversal throughput | 19.8 traversals/s | 20.1 traversals/s | |
Recent 50 iteration window from that run:
| Segment | Mean |
| --- | ---: |
| All iterations | 15.57s/iter |
| Non-eval iterations | 12.21s/iter |
| Recent 20 evals, eval only | 17.31s/eval |
| Recent 20 eval iterations | 29.30s/iter |
Evaluation ran every 5 iterations: 5, 10, 15, ..., 385.
The practical legacy cadence was roughly:
`4 normal iterations + 1 eval iteration ~= 75s per 5 iterations`.
## Current Repo Reference Points
From `docs/deep-cfr-profile-advantage-memory-split-2026-05-07.md`:
| Segment | Current mean |
| --- | ---: |
| Non-eval iterations | 5.832958s/iter |
| Traversal | 3.160975s/iter |
| Advantage train | 1.742895s/iter |
| Strategy train | 0.912324s/iter |
From `docs/deep-cfr-batched-evaluation-2026-05-07.md`:
| Segment | Current value |
| --- | ---: |
| Batched CUDA evaluation | 14.834096s/eval |
| Batched CUDA 1-iter wall time with eval | 21.322356s |
| Batched opponent-parallel CUDA evaluation | 6.420402s/eval |
| Batched opponent-parallel CUDA 1-iter wall time with eval | 12.854383s |
## Rough Comparison
Normal training iterations improved from about `11.51s` to about `5.83s`,
roughly `1.97x` faster.
Traversal improved from about `7.16s` to about `3.16s`, roughly `2.27x`
faster.
Evaluation improved from about `18.61s` to about `14.83s`, roughly `1.25x`
faster for the measured batched CUDA profile. With opponent-parallel eval, the
measured eval time was about `6.42s`, roughly `2.90x` faster than the legacy
eval-only average.
Using the simple cadence model:
Legacy:
`4 * 11.51 + (11.51 + 18.61) = 76.16s per 5 iterations`
Current batched sequential:
`4 * 5.83 + (5.83 + 14.83) = 43.98s per 5 iterations`
Current batched opponent-parallel:
`4 * 5.83 + (5.83 + 6.42) = 35.57s per 5 iterations`
That implies about `1.73x` faster eval-included wall time for the batched
sequential rough comparison, and about `2.14x` faster for the batched
opponent-parallel rough comparison.
## Caveat
The legacy numbers came from a long run through iteration 387. The current
numbers are from targeted profiling runs. The comparison is useful for order of
magnitude and bottleneck direction, not as a strict benchmark under identical
runtime conditions.