coolguy/coorl-lost-cities
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Measure traversal policy boundary cost

Browse Source 
Add a microbench that separates game/encoding overhead, single-request policy boundary overhead, and batched PyTorch forward lower bounds. Record CPU/CUDA results and link the finding from the Julia port evaluation.

Co-Authored-By: Codex <codex@openai.com>
This commit is contained in:
coolguy
2026-05-07 22:02:46 +09:00
co-authored by Codex
parent 2429d1e210
commit 7c20d53103
6 changed files with 1101 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/performance.md
+50
View File
@@ -700,6 +700,56 @@ either:
Both require restructuring traversal. Option A's "additive, no traversal Both require restructuring traversal. Option A's "additive, no traversal
changes" property turned out to also mean "cannot drive the batch size up." changes" property turned out to also mean "cannot drive the batch size up."
### Clarifying the traversal bottleneck: sync policy boundary, not SIMD
The tempting shorthand is "Python/GIL prevents traversal from using SIMD or
threads." The more precise diagnosis is narrower:
- Lost Cities game mechanics are already mostly Cython C-level operations.
`legal_actions`, action push/pop, and cached scoring are not Python list
walks on the hot path.
- The traversal recursion is Cython, but it synchronously crosses back into
Python/PyTorch at every policy-needed state: encode a single info state,
run one-row PyTorch forward, copy logits back to CPU/Numpy, then continue
recursion.
- This boundary makes every traversal worker **sync-blocking**. With
`num_workers=8`, the inference server can see at most eight in-flight
requests before per-network splitting, no matter how large `max_batch` is.
- GIL-free threading would help only after the same path is made
`nogil`-clean or after traversal is restructured so policy calls can be
batched. Simply "using SIMD" does not address the one-row policy boundary.
So the actionable bottleneck is **policy-call scheduling shape**, not scalar
game-rule arithmetic. The highest-leverage experiment is Option B:
per-worker interleaved traversal, where one worker advances many traversals,
suspends each at a policy request, batches those requests, and resumes the
corresponding continuations.
Microbench evidence (2026-05-07, `configs/deep_cfr/default.yaml`,
`experiments/traversal_policy_boundary/bench_policy_boundary.py`):
| Device | Component | median μs/call | p95 μs/call |
| --- | --- | ---: | ---: |
| CPU | encode + legal | 3.10 | 3.81 |
| CPU | push + pop | 0.15 | 0.22 |
| CPU | policy boundary bs=1 | 111.50 | 125.46 |
| CPU | torch forward bs=64 | 12.84 | 13.14 |
| CUDA | policy boundary bs=1 | 181.30 | 194.77 |
| CUDA | torch forward bs=64 | 2.55 | 2.75 |
This confirms the bottleneck is not Cython game-rule scalar work. The
single-request policy boundary is ~36× larger than encode+legal on CPU, while
CUDA bs=64 forward is ~71× cheaper than the current CUDA bs=1 boundary.
Expected upside is bounded by the fraction of traversal currently spent at
policy calls. Moving realized GPU forward from the current ~4-8 row regime
(~12-20μs/state) to bs=64 (~1.46μs/state) is an ~8-14× improvement on the
forward component, but not on game recursion, sample creation, or replay
writes. For the observed `local` traversal around 10-13s/iter, a realistic
first target is roughly **1.5-3× traversal speedup** if Option B reaches the
bs=64 regime without adding comparable scheduler overhead. Larger claims need
a prototype because traversal has substantial non-forward work.
### Why deferring A (not deleting) is the right call ### Why deferring A (not deleting) is the right call
- The plumbing (server process, shared-memory client, weight sync, config - The plumbing (server process, shared-memory client, weight sync, config
docs/research/julia_port_evaluation.md
+3
View File
@@ -226,6 +226,9 @@ Recommended next path: stay on Python/Cython and pursue Option B
Julia experiment could still be considered later for traversal-only Julia experiment could still be considered later for traversal-only
logic, but it would need an explicit hybrid plan that keeps PyTorch for logic, but it would need an explicit hybrid plan that keeps PyTorch for
networks and separately proves PythonCall/PyCall overhead is acceptable. networks and separately proves PythonCall/PyCall overhead is acceptable.
For the traversal bottleneck clarification and Option B speedup envelope,
see `docs/performance.md` "Clarifying the traversal bottleneck: sync
policy boundary, not SIMD."
Criterion 5 remains unrun because criterion 4 already blocks the full Criterion 5 remains unrun because criterion 4 already blocks the full
port decision. port decision.
experiments/traversal_policy_boundary/README.md
+48
View File
@@ -0,0 +1,48 @@
# Traversal Policy Boundary Microbench
Purpose: separate the current traversal hot path into game/encoding overhead,
single-request policy boundary overhead, and batched forward lower bounds.
Run:
```bash
uv run python experiments/traversal_policy_boundary/bench_policy_boundary.py \
--traversals 32 \
--runs 5 \
--warmup 1 \
--corpus-size 512 \
--component-repeats 4 \
--forward-repeats 32 \
--device cpu
```
CUDA spot check:
```bash
uv run python experiments/traversal_policy_boundary/bench_policy_boundary.py \
--traversals 8 \
--runs 3 \
--warmup 1 \
--corpus-size 512 \
--component-repeats 2 \
--forward-repeats 32 \
--device cuda \
--output experiments/traversal_policy_boundary/results_cuda.json
```
2026-05-07 results, `configs/deep_cfr/default.yaml`, RTX 3090 host:
| Device | Component | Median us/call | p95 us/call |
| --- | --- | ---: | ---: |
| CPU | encode + legal | 3.10 | 3.81 |
| CPU | push + pop | 0.15 | 0.22 |
| CPU | policy boundary bs=1 | 111.50 | 125.46 |
| CPU | torch forward bs=64 | 12.84 | 13.14 |
| CUDA | encode + legal | 3.16 | 3.88 |
| CUDA | push + pop | 0.16 | 0.25 |
| CUDA | policy boundary bs=1 | 181.30 | 194.77 |
| CUDA | torch forward bs=64 | 2.55 | 2.75 |
Interpretation: the game mechanics and state encoding are not the dominant
cost. The current one-row policy boundary dominates, and CUDA only becomes
attractive once requests are actually batched.
experiments/traversal_policy_boundary/bench_policy_boundary.py
+401
View File
@@ -0,0 +1,401 @@
from __future__ import annotations
import argparse
import json
import random
import statistics
import time
from pathlib import Path
from typing import Any
import numpy as np
import torch
from coolrl_lost_cities.games.classic.deep_cfr.encoding import encode_info_state, input_dim
from coolrl_lost_cities.games.classic.deep_cfr.traversal import run_cython_traversal_batch
from coolrl_lost_cities.games.classic.game import GameState
from coolrl_lost_cities.games.classic.deep_cfr.config import load_config
from coolrl_lost_cities.games.classic.deep_cfr.networks import DeepCFRMLP
def _timer(device: torch.device | None = None) -> float:
if device is not None and device.type == "cuda":
torch.cuda.synchronize(device)
return time.perf_counter()
def _median(values: list[float]) -> float:
return float(statistics.median(values))
def _quantile(values: list[float], q: float) -> float:
ordered = sorted(values)
if not ordered:
return 0.0
idx = min(len(ordered) - 1, max(0, round((len(ordered) - 1) * q)))
return float(ordered[idx])
def _build_networks(cfg: Any, input_dim_value: int, action_size: int, device: torch.device):
networks = [
DeepCFRMLP.from_config(input_dim_value, action_size, cfg.network).to(device).eval()
for _ in range(2)
]
strategy = DeepCFRMLP.from_config(input_dim_value, action_size, cfg.network).to(device).eval()
return networks, strategy
def _make_corpus(cfg: Any, count: int, seed: int) -> list[GameState]:
rng = random.Random(seed)
game_config = cfg.rules.to_lost_cities_config(seed=seed)
states: list[GameState] = []
state = GameState.new_game(game_config, seed=seed)
while len(states) < count:
if state.terminal:
state = GameState.new_game(game_config, seed=rng.randrange(1, 2**31))
continue
states.append(state.clone())
legal = state.unified_legal_actions()
if not legal:
state = GameState.new_game(game_config, seed=rng.randrange(1, 2**31))
continue
state.push_unified_action(int(legal[rng.randrange(len(legal))]))
return states
def _regret_matching(values: np.ndarray, legal: list[int], epsilon: float) -> np.ndarray:
policy = np.zeros_like(values, dtype=np.float32)
positives = [max(float(values[action]), 0.0) for action in legal]
positive_sum = sum(positives)
if positive_sum <= epsilon:
uniform = 1.0 / max(1, len(legal))
for action in legal:
policy[action] = uniform
return policy
for action, positive in zip(legal, positives, strict=True):
policy[action] = positive / positive_sum
return policy
def bench_full_traversal(
cfg: Any,
*,
traversals: int,
runs: int,
warmup: int,
device: torch.device,
input_dim_value: int,
action_size: int,
) -> dict[str, Any]:
timings: list[float] = []
stat_rows: list[dict[str, Any]] = []
seed_base = int(cfg.run.seed)
game_config = cfg.rules.to_lost_cities_config(seed=seed_base)
for run_idx in range(runs + warmup):
networks, strategy = _build_networks(cfg, input_dim_value, action_size, device)
seeds = [seed_base + run_idx * 100_000 + idx for idx in range(traversals)]
start = _timer(device)
stats, advantage_samples, strategy_samples = run_cython_traversal_batch(
networks,
game_config,
seeds,
0,
1,
device=device,
action_size=action_size,
strategy_network=strategy,
encoding=cfg.encoding,
epsilon=cfg.traversal.regret_matching_epsilon,
strategy_sample_interval=cfg.traversal.strategy_sample_interval,
store_strategy_on_traverser_nodes=cfg.traversal.store_strategy_on_traverser_nodes,
store_strategy_on_opponent_nodes=cfg.traversal.store_strategy_on_opponent_nodes,
max_depth=cfg.traversal.max_depth,
max_nodes=cfg.traversal.max_nodes_per_traversal,
sampling_mode=cfg.traversal.sampling_mode,
outcome_sampling_epsilon=cfg.traversal.outcome_sampling_epsilon,
outcome_sampling_value_clip=cfg.traversal.outcome_sampling_value_clip,
outcome_unsampled_regret=cfg.traversal.outcome_unsampled_regret,
cutoff_value_mode=cfg.traversal.cutoff_value_mode,
cutoff_rollouts=cfg.traversal.cutoff_rollouts,
cutoff_rollout_policy=cfg.traversal.cutoff_rollout_policy,
cutoff_rollout_max_steps=cfg.traversal.cutoff_rollout_max_steps,
opponent_policy=cfg.traversal.opponent_policy,
all_negative_fallback=cfg.regret_matching.all_negative_fallback,
league_advantage_networks=None,
self_play_anchor_probability=cfg.self_play.anchor_probability,
self_play_current_weight=cfg.self_play.current_weight,
self_play_recent_weight=cfg.self_play.recent_weight,
self_play_older_weight=cfg.self_play.older_weight,
self_play_anchor_weight=cfg.self_play.anchor_weight,
self_play_recent_window=cfg.self_play.recent_window,
endpoint_depth_bucket_width=cfg.traversal.endpoint_depth_bucket_width,
endpoint_depth_bucket_max=cfg.traversal.endpoint_depth_bucket_max,
seed=seed_base + run_idx,
)
elapsed = _timer(device) - start
if run_idx >= warmup:
stats_dict = stats.to_dict()
stats_dict["advantage_samples_returned"] = len(advantage_samples)
stats_dict["strategy_samples_returned"] = len(strategy_samples)
timings.append(elapsed)
stat_rows.append(stats_dict)
del networks, strategy
policy_calls = [int(row["traversal_regret_matching_decisions"]) for row in stat_rows]
nodes = [int(row["traversal_nodes"]) for row in stat_rows]
return {
"seconds": timings,
"median_seconds": _median(timings),
"median_nodes": _median([float(value) for value in nodes]),
"median_policy_calls": _median([float(value) for value in policy_calls]),
"median_us_per_node": _median(
[sec * 1_000_000.0 / max(1, node) for sec, node in zip(timings, nodes, strict=True)]
),
"median_us_per_policy_call": _median(
[
sec * 1_000_000.0 / max(1, calls)
for sec, calls in zip(timings, policy_calls, strict=True)
]
),
"stats": stat_rows,
}
def bench_encode_legal(corpus: list[GameState], cfg: Any, repeats: int) -> dict[str, Any]:
durations: list[float] = []
counts: list[int] = []
total = 0.0
for _ in range(repeats):
for state in corpus:
start = time.perf_counter()
_ = encode_info_state(state, state.current_player, cfg.encoding)
legal = state.unified_legal_actions()
total += len(legal)
durations.append(time.perf_counter() - start)
counts.append(len(legal))
return {
"median_us": _median(durations) * 1_000_000.0,
"p95_us": _quantile(durations, 0.95) * 1_000_000.0,
"mean_legal_actions": float(statistics.mean(counts)),
"checksum": total,
}
def bench_push_pop(corpus: list[GameState], repeats: int) -> dict[str, Any]:
durations: list[float] = []
total = 0
for _ in range(repeats):
for state in corpus:
legal = state.unified_legal_actions()
if not legal:
continue
action = int(legal[total % len(legal)])
start = time.perf_counter()
state.push_unified_action(action)
state.pop_action()
durations.append(time.perf_counter() - start)
total += action
return {
"median_us": _median(durations) * 1_000_000.0,
"p95_us": _quantile(durations, 0.95) * 1_000_000.0,
"checksum": total,
}
def bench_policy_boundary(
corpus: list[GameState],
cfg: Any,
network: torch.nn.Module,
device: torch.device,
action_size: int,
repeats: int,
) -> dict[str, Any]:
durations: list[float] = []
checksum = 0.0
with torch.inference_mode():
for _ in range(repeats):
for state in corpus:
start = _timer(device)
info_state = encode_info_state(state, state.current_player, cfg.encoding)
legal = state.unified_legal_actions()
x = torch.as_tensor(info_state, dtype=torch.float32, device=device).unsqueeze(0)
advantages = network(x).squeeze(0).detach().cpu().numpy().astype(np.float32)
policy = _regret_matching(
advantages,
legal,
float(cfg.traversal.regret_matching_epsilon),
)
checksum += float(policy.sum()) + float(advantages[0])
durations.append(_timer(device) - start)
return {
"median_us": _median(durations) * 1_000_000.0,
"p95_us": _quantile(durations, 0.95) * 1_000_000.0,
"checksum": checksum,
}
def bench_torch_forward(
corpus: list[GameState],
cfg: Any,
network: torch.nn.Module,
device: torch.device,
batch_sizes: list[int],
repeats: int,
) -> dict[str, Any]:
states = np.stack(
[encode_info_state(state, state.current_player, cfg.encoding) for state in corpus]
).astype(np.float32)
results: dict[str, Any] = {}
with torch.inference_mode():
for batch_size in batch_sizes:
durations: list[float] = []
checksum = 0.0
for _ in range(repeats):
for offset in range(0, len(states), batch_size):
chunk = states[offset : offset + batch_size]
if len(chunk) != batch_size:
continue
x = torch.as_tensor(chunk, dtype=torch.float32, device=device)
start = _timer(device)
out = network(x)
checksum += float(out.detach().sum().cpu())
durations.append((_timer(device) - start) / batch_size)
if not durations:
continue
results[str(batch_size)] = {
"median_us_per_call": _median(durations) * 1_000_000.0,
"p95_us_per_call": _quantile(durations, 0.95) * 1_000_000.0,
"checksum": checksum,
}
return results
def _print_table(result: dict[str, Any]) -> None:
full = result["full_traversal"]
encode = result["encode_legal"]
push_pop = result["push_pop"]
boundary = result["policy_boundary_bs1"]
forward = result["torch_forward"]
print("Traversal policy-boundary microbench")
print(
f"config={result['config']} device={result['device']} torch_threads={result['torch_threads']}"
)
print(
f"full traversal: {full['median_seconds']:.3f}s, "
f"policy_calls={full['median_policy_calls']:.0f}, nodes={full['median_nodes']:.0f}"
)
print()
print("component median us/call p95 us/call")
print(
f"encode+legal {encode['median_us']:9.2f} {encode['p95_us']:9.2f}"
)
print(
f"push+pop {push_pop['median_us']:9.2f} {push_pop['p95_us']:9.2f}"
)
print(
f"policy boundary bs=1 {boundary['median_us']:9.2f} {boundary['p95_us']:9.2f}"
)
for batch_size, row in forward.items():
print(
f"torch forward bs={batch_size:<3} "
f"{row['median_us_per_call']:9.2f} {row['p95_us_per_call']:9.2f}"
)
print()
print("derived")
print(f"full traversal us/node {full['median_us_per_node']:.2f}")
print(f"full traversal us/policy_call {full['median_us_per_policy_call']:.2f}")
if boundary["median_us"] > 0:
print(
"full/policy-boundary ratio "
f"{full['median_us_per_policy_call'] / boundary['median_us']:.2f}x"
)
bs64 = forward.get("64")
if bs64 and bs64["median_us_per_call"] > 0:
print(
"policy bs=1 vs forward bs=64 "
f"{boundary['median_us'] / bs64['median_us_per_call']:.2f}x"
)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--config", default="configs/deep_cfr/default.yaml")
parser.add_argument("--device", default="cpu")
parser.add_argument("--traversals", type=int, default=32)
parser.add_argument("--runs", type=int, default=5)
parser.add_argument("--warmup", type=int, default=1)
parser.add_argument("--corpus-size", type=int, default=512)
parser.add_argument("--component-repeats", type=int, default=4)
parser.add_argument("--forward-repeats", type=int, default=64)
parser.add_argument("--torch-threads", type=int, default=1)
parser.add_argument("--output", default="")
args = parser.parse_args()
torch.set_num_threads(max(1, int(args.torch_threads)))
cfg = load_config(args.config)
device = torch.device(args.device)
if device.type == "cuda" and not torch.cuda.is_available():
raise RuntimeError("--device cuda requested but CUDA is not available")
probe = GameState.new_game(
cfg.rules.to_lost_cities_config(seed=cfg.run.seed), seed=cfg.run.seed
)
input_dim_value = input_dim(probe, cfg.encoding)
action_size = 2 * probe.config.hand_size + 1 + probe.config.n_colors
networks, _strategy = _build_networks(cfg, input_dim_value, action_size, device)
corpus = _make_corpus(cfg, args.corpus_size, int(cfg.run.seed))
# Run a tiny warmup through the component path before timed loops.
_ = encode_info_state(corpus[0], corpus[0].current_player, cfg.encoding)
with torch.inference_mode():
_ = networks[0](torch.as_tensor(_, dtype=torch.float32, device=device).unsqueeze(0))
result = {
"config": args.config,
"device": str(device),
"torch_threads": torch.get_num_threads(),
"traversals": args.traversals,
"runs": args.runs,
"warmup": args.warmup,
"corpus_size": args.corpus_size,
"input_dim": input_dim_value,
"action_size": action_size,
"full_traversal": bench_full_traversal(
cfg,
traversals=args.traversals,
runs=args.runs,
warmup=args.warmup,
device=device,
input_dim_value=input_dim_value,
action_size=action_size,
),
"encode_legal": bench_encode_legal(corpus, cfg, args.component_repeats),
"push_pop": bench_push_pop(corpus, args.component_repeats),
"policy_boundary_bs1": bench_policy_boundary(
corpus,
cfg,
networks[0],
device,
action_size,
args.component_repeats,
),
"torch_forward": bench_torch_forward(
corpus,
cfg,
networks[0],
device,
[1, 4, 8, 64, 256],
args.forward_repeats,
),
}
_print_table(result)
output = Path(args.output) if args.output else Path(__file__).with_name("results.json")
output.write_text(json.dumps(result, indent=2), encoding="utf-8")
print(f"\nwrote {output}")
if __name__ == "__main__":
main()
experiments/traversal_policy_boundary/results.json
+359
View File
@@ -0,0 +1,359 @@
{
"config": "configs/deep_cfr/default.yaml",
"device": "cpu",
"torch_threads": 1,
"traversals": 32,
"runs": 5,
"warmup": 1,
"corpus_size": 512,
"input_dim": 365,
"action_size": 22,
"full_traversal": {
"seconds": [
1.4774325820035301,
1.3015301960112993,
0.8726662800181657,
1.3711390359967481,
1.4644392740156036
],
"median_seconds": 1.3711390359967481,
"median_nodes": 12960.0,
"median_policy_calls": 6470.0,
"median_us_per_node": 107.45790918191044,
"median_us_per_policy_call": 215.2712861414653,
"stats": [
{
"traversal_nodes": 13710,
"traversal_terminals": 32,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 560,
"traversal_advantage_samples": 6842,
"traversal_strategy_samples": 6842,
"traversal_sampled_actions": 6842,
"traversal_regret_matching_decisions": 6842,
"traversal_regret_fallback_count": 512,
"traversal_regret_fallback_rate": 0.07483192049108447,
"traversal_regret_fallback_avg_depth": 91.796875,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 208,
"traversal_regret_fallback_action_draw_pile": 304,
"traversal_regret_fallback_legal_actions_mean": 2.046875,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 1.046875,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 3.384765625,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 13678,
"traversal_endpoints": 32,
"traversal_avg_endpoint_depth": 427.4375,
"traversal_endpoint_depth_bucket_400_499": 15,
"traversal_endpoint_depth_bucket_500_599": 5,
"traversal_endpoint_depth_bucket_300_399": 12,
"traversal_regret_fallback_depth_bucket_200_249": 26,
"traversal_regret_fallback_depth_bucket_150_199": 53,
"traversal_regret_fallback_depth_bucket_50_99": 109,
"traversal_regret_fallback_depth_bucket_0_49": 212,
"traversal_regret_fallback_depth_bucket_400_plus": 3,
"traversal_regret_fallback_depth_bucket_250_299": 18,
"traversal_regret_fallback_depth_bucket_100_149": 82,
"traversal_regret_fallback_depth_bucket_300_349": 8,
"traversal_regret_fallback_depth_bucket_350_399": 1,
"traversal_regret_fallback_opened_colors_count_5": 152,
"traversal_regret_fallback_opened_colors_count_4": 128,
"traversal_regret_fallback_opened_colors_count_3": 98,
"traversal_regret_fallback_opened_colors_count_2": 64,
"traversal_regret_fallback_opened_colors_count_1": 39,
"traversal_regret_fallback_opened_colors_count_0": 31,
"advantage_samples_returned": 6842,
"strategy_samples_returned": 6842
},
{
"traversal_nodes": 12112,
"traversal_terminals": 32,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 504,
"traversal_advantage_samples": 6046,
"traversal_strategy_samples": 6046,
"traversal_sampled_actions": 6046,
"traversal_regret_matching_decisions": 6046,
"traversal_regret_fallback_count": 680,
"traversal_regret_fallback_rate": 0.11247105524313596,
"traversal_regret_fallback_avg_depth": 98.55294117647058,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 366,
"traversal_regret_fallback_action_draw_pile": 314,
"traversal_regret_fallback_legal_actions_mean": 2.6470588235294117,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 1.6470588235294117,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 4.067647058823529,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 12080,
"traversal_endpoints": 32,
"traversal_avg_endpoint_depth": 377.5,
"traversal_endpoint_depth_bucket_400_499": 11,
"traversal_endpoint_depth_bucket_200_299": 3,
"traversal_endpoint_depth_bucket_300_399": 17,
"traversal_endpoint_depth_bucket_500_599": 1,
"traversal_regret_fallback_depth_bucket_300_349": 17,
"traversal_regret_fallback_depth_bucket_200_249": 46,
"traversal_regret_fallback_depth_bucket_150_199": 74,
"traversal_regret_fallback_depth_bucket_100_149": 108,
"traversal_regret_fallback_depth_bucket_50_99": 147,
"traversal_regret_fallback_depth_bucket_0_49": 258,
"traversal_regret_fallback_depth_bucket_250_299": 24,
"traversal_regret_fallback_depth_bucket_350_399": 6,
"traversal_regret_fallback_opened_colors_count_5": 404,
"traversal_regret_fallback_opened_colors_count_4": 107,
"traversal_regret_fallback_opened_colors_count_3": 57,
"traversal_regret_fallback_opened_colors_count_2": 56,
"traversal_regret_fallback_opened_colors_count_1": 35,
"traversal_regret_fallback_opened_colors_count_0": 21,
"advantage_samples_returned": 6046,
"strategy_samples_returned": 6046
},
{
"traversal_nodes": 8094,
"traversal_terminals": 32,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 342,
"traversal_advantage_samples": 4048,
"traversal_strategy_samples": 4048,
"traversal_sampled_actions": 4048,
"traversal_regret_matching_decisions": 4048,
"traversal_regret_fallback_count": 0,
"traversal_regret_fallback_rate": 0.0,
"traversal_regret_fallback_avg_depth": 0.0,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 0,
"traversal_regret_fallback_action_draw_pile": 0,
"traversal_regret_fallback_legal_actions_mean": 0.0,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 0.0,
"traversal_regret_fallback_legal_draw_pile_mean": 0.0,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 0.0,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 8062,
"traversal_endpoints": 32,
"traversal_avg_endpoint_depth": 251.9375,
"traversal_endpoint_depth_bucket_200_299": 28,
"traversal_endpoint_depth_bucket_300_399": 3,
"traversal_endpoint_depth_bucket_100_199": 1,
"advantage_samples_returned": 4048,
"strategy_samples_returned": 4048
},
{
"traversal_nodes": 12960,
"traversal_terminals": 32,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 598,
"traversal_advantage_samples": 6470,
"traversal_strategy_samples": 6470,
"traversal_sampled_actions": 6470,
"traversal_regret_matching_decisions": 6470,
"traversal_regret_fallback_count": 971,
"traversal_regret_fallback_rate": 0.1500772797527048,
"traversal_regret_fallback_avg_depth": 121.39546858908342,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 372,
"traversal_regret_fallback_action_draw_pile": 599,
"traversal_regret_fallback_legal_actions_mean": 2.818743563336766,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 1.8187435633367661,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 3.9742533470648818,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 12928,
"traversal_endpoints": 32,
"traversal_avg_endpoint_depth": 404.0,
"traversal_endpoint_depth_bucket_300_399": 16,
"traversal_endpoint_depth_bucket_500_599": 3,
"traversal_endpoint_depth_bucket_400_499": 12,
"traversal_endpoint_depth_bucket_200_299": 1,
"traversal_regret_fallback_depth_bucket_300_349": 45,
"traversal_regret_fallback_depth_bucket_200_249": 98,
"traversal_regret_fallback_depth_bucket_150_199": 98,
"traversal_regret_fallback_depth_bucket_100_149": 149,
"traversal_regret_fallback_depth_bucket_50_99": 194,
"traversal_regret_fallback_depth_bucket_0_49": 306,
"traversal_regret_fallback_depth_bucket_250_299": 60,
"traversal_regret_fallback_depth_bucket_350_399": 16,
"traversal_regret_fallback_depth_bucket_400_plus": 5,
"traversal_regret_fallback_opened_colors_count_5": 503,
"traversal_regret_fallback_opened_colors_count_4": 213,
"traversal_regret_fallback_opened_colors_count_2": 83,
"traversal_regret_fallback_opened_colors_count_1": 62,
"traversal_regret_fallback_opened_colors_count_3": 88,
"traversal_regret_fallback_opened_colors_count_0": 22,
"advantage_samples_returned": 6470,
"strategy_samples_returned": 6470
},
{
"traversal_nodes": 13732,
"traversal_terminals": 32,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 576,
"traversal_advantage_samples": 6852,
"traversal_strategy_samples": 6852,
"traversal_sampled_actions": 6852,
"traversal_regret_matching_decisions": 6852,
"traversal_regret_fallback_count": 199,
"traversal_regret_fallback_rate": 0.029042615294804435,
"traversal_regret_fallback_avg_depth": 49.54271356783919,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 180,
"traversal_regret_fallback_action_draw_pile": 19,
"traversal_regret_fallback_legal_actions_mean": 1.5577889447236182,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 0.5577889447236181,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 2.4673366834170856,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 13700,
"traversal_endpoints": 32,
"traversal_avg_endpoint_depth": 428.125,
"traversal_endpoint_depth_bucket_400_499": 12,
"traversal_endpoint_depth_bucket_300_399": 12,
"traversal_endpoint_depth_bucket_200_299": 2,
"traversal_endpoint_depth_bucket_500_599": 6,
"traversal_regret_fallback_depth_bucket_100_149": 18,
"traversal_regret_fallback_depth_bucket_0_49": 125,
"traversal_regret_fallback_depth_bucket_50_99": 46,
"traversal_regret_fallback_depth_bucket_150_199": 8,
"traversal_regret_fallback_depth_bucket_250_299": 1,
"traversal_regret_fallback_depth_bucket_200_249": 1,
"traversal_regret_fallback_opened_colors_count_4": 38,
"traversal_regret_fallback_opened_colors_count_3": 36,
"traversal_regret_fallback_opened_colors_count_2": 34,
"traversal_regret_fallback_opened_colors_count_1": 43,
"traversal_regret_fallback_opened_colors_count_0": 24,
"traversal_regret_fallback_opened_colors_count_5": 24,
"advantage_samples_returned": 6852,
"strategy_samples_returned": 6852
}
]
},
"encode_legal": {
"median_us": 3.096007276326418,
"p95_us": 3.8069847505539656,
"mean_legal_actions": 6.955078125,
"checksum": 14244.0
},
"push_pop": {
"median_us": 0.1510197762399912,
"p95_us": 0.22101448848843575,
"checksum": 26295
},
"policy_boundary_bs1": {
"median_us": 111.50000500492752,
"p95_us": 125.45599020086229,
"checksum": 2128.840651668608
},
"torch_forward": {
"1": {
"median_us_per_call": 88.96699000615627,
"p95_us_per_call": 97.41401299834251,
"checksum": 1910.6115314364433
},
"4": {
"median_us_per_call": 42.4988720624242,
"p95_us_per_call": 46.12924385583028,
"checksum": 1910.611463546753
},
"8": {
"median_us_per_call": 26.491186872590333,
"p95_us_per_call": 28.330872737569734,
"checksum": 1910.611445426941
},
"64": {
"median_us_per_call": 12.843968761444557,
"p95_us_per_call": 13.137109363015043,
"checksum": 1910.6114654541016
},
"256": {
"median_us_per_call": 11.324570266424416,
"p95_us_per_call": 11.40932033649733,
"checksum": 1910.6114501953125
}
}
}
experiments/traversal_policy_boundary/results_cuda.json
+240
View File
@@ -0,0 +1,240 @@
{
"config": "configs/deep_cfr/default.yaml",
"device": "cuda",
"torch_threads": 1,
"traversals": 8,
"runs": 3,
"warmup": 1,
"corpus_size": 512,
"input_dim": 365,
"action_size": 22,
"full_traversal": {
"seconds": [
0.3639091220102273,
0.623861116997432,
0.5022158679785207
],
"median_seconds": 0.5022158679785207,
"median_nodes": 3010.0,
"median_policy_calls": 1502.0,
"median_us_per_node": 163.2289683405107,
"median_us_per_policy_call": 326.97123532360166,
"stats": [
{
"traversal_nodes": 2256,
"traversal_terminals": 8,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 398,
"traversal_advantage_samples": 1126,
"traversal_strategy_samples": 1126,
"traversal_sampled_actions": 1126,
"traversal_regret_matching_decisions": 1126,
"traversal_regret_fallback_count": 35,
"traversal_regret_fallback_rate": 0.03108348134991119,
"traversal_regret_fallback_avg_depth": 113.8,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 19,
"traversal_regret_fallback_action_draw_pile": 16,
"traversal_regret_fallback_legal_actions_mean": 3.057142857142857,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 2.057142857142857,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 4.057142857142857,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 2248,
"traversal_endpoints": 8,
"traversal_avg_endpoint_depth": 281.0,
"traversal_endpoint_depth_bucket_300_399": 2,
"traversal_endpoint_depth_bucket_200_299": 6,
"traversal_regret_fallback_depth_bucket_150_199": 7,
"traversal_regret_fallback_depth_bucket_50_99": 5,
"traversal_regret_fallback_depth_bucket_0_49": 11,
"traversal_regret_fallback_depth_bucket_350_399": 1,
"traversal_regret_fallback_depth_bucket_200_249": 4,
"traversal_regret_fallback_depth_bucket_100_149": 7,
"traversal_regret_fallback_opened_colors_count_5": 24,
"traversal_regret_fallback_opened_colors_count_0": 3,
"traversal_regret_fallback_opened_colors_count_4": 4,
"traversal_regret_fallback_opened_colors_count_1": 3,
"traversal_regret_fallback_opened_colors_count_3": 1,
"advantage_samples_returned": 1126,
"strategy_samples_returned": 1126
},
{
"traversal_nodes": 3822,
"traversal_terminals": 8,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 560,
"traversal_advantage_samples": 1908,
"traversal_strategy_samples": 1908,
"traversal_sampled_actions": 1908,
"traversal_regret_matching_decisions": 1908,
"traversal_regret_fallback_count": 96,
"traversal_regret_fallback_rate": 0.050314465408805034,
"traversal_regret_fallback_avg_depth": 64.83333333333333,
"traversal_regret_fallback_action_play_existing": 0,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 0,
"traversal_regret_fallback_action_draw_deck": 47,
"traversal_regret_fallback_action_draw_pile": 49,
"traversal_regret_fallback_legal_actions_mean": 1.90625,
"traversal_regret_fallback_legal_play_existing_mean": 0.0,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 0.0,
"traversal_regret_fallback_legal_draw_deck_mean": 1.0,
"traversal_regret_fallback_legal_draw_pile_mean": 0.90625,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 3.25,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 3814,
"traversal_endpoints": 8,
"traversal_avg_endpoint_depth": 476.75,
"traversal_endpoint_depth_bucket_400_499": 4,
"traversal_endpoint_depth_bucket_300_399": 1,
"traversal_endpoint_depth_bucket_500_599": 3,
"traversal_regret_fallback_depth_bucket_300_349": 1,
"traversal_regret_fallback_depth_bucket_200_249": 4,
"traversal_regret_fallback_depth_bucket_150_199": 4,
"traversal_regret_fallback_depth_bucket_50_99": 24,
"traversal_regret_fallback_depth_bucket_0_49": 50,
"traversal_regret_fallback_depth_bucket_100_149": 13,
"traversal_regret_fallback_opened_colors_count_5": 24,
"traversal_regret_fallback_opened_colors_count_4": 22,
"traversal_regret_fallback_opened_colors_count_3": 20,
"traversal_regret_fallback_opened_colors_count_1": 14,
"traversal_regret_fallback_opened_colors_count_2": 15,
"traversal_regret_fallback_opened_colors_count_0": 1,
"advantage_samples_returned": 1908,
"strategy_samples_returned": 1908
},
{
"traversal_nodes": 3010,
"traversal_terminals": 8,
"traversal_depth_cutoffs": 0,
"traversal_node_limit_cutoffs": 0,
"traversal_max_depth_reached": 532,
"traversal_advantage_samples": 1502,
"traversal_strategy_samples": 1502,
"traversal_sampled_actions": 1502,
"traversal_regret_matching_decisions": 1502,
"traversal_regret_fallback_count": 72,
"traversal_regret_fallback_rate": 0.047936085219707054,
"traversal_regret_fallback_avg_depth": 246.29166666666666,
"traversal_regret_fallback_action_play_existing": 1,
"traversal_regret_fallback_action_open_new": 0,
"traversal_regret_fallback_action_discard": 54,
"traversal_regret_fallback_action_draw_deck": 17,
"traversal_regret_fallback_action_draw_pile": 0,
"traversal_regret_fallback_legal_actions_mean": 6.375,
"traversal_regret_fallback_legal_play_existing_mean": 0.027777777777777776,
"traversal_regret_fallback_legal_open_new_mean": 0.0,
"traversal_regret_fallback_legal_discard_mean": 6.111111111111111,
"traversal_regret_fallback_legal_draw_deck_mean": 0.2361111111111111,
"traversal_regret_fallback_legal_draw_pile_mean": 0.0,
"traversal_regret_fallback_open_new_available_rate": 0.0,
"traversal_regret_fallback_open_new_selected": 0,
"traversal_regret_fallback_open_new_selected_rate": 0.0,
"traversal_regret_fallback_open_new_selection_over_availability": 0.0,
"traversal_regret_fallback_avg_opened_colors_before_action": 4.638888888888889,
"traversal_regret_fallback_argmax_tie_count": 0,
"traversal_regret_fallback_argmax_tie_rate": 0.0,
"traversal_regret_fallback_argmax_tie_size_mean": 0.0,
"traversal_regret_fallback_argmax_full_tie_count": 0,
"traversal_regret_fallback_argmax_full_tie_rate": 0.0,
"traversal_cutoff_rollouts": 0,
"traversal_cutoff_rollout_steps": 0,
"traversal_cutoff_rollout_timeouts": 0,
"traversal_endpoint_depth_sum": 3002,
"traversal_endpoints": 8,
"traversal_avg_endpoint_depth": 375.25,
"traversal_endpoint_depth_bucket_300_399": 7,
"traversal_endpoint_depth_bucket_500_599": 1,
"traversal_regret_fallback_depth_bucket_300_349": 11,
"traversal_regret_fallback_depth_bucket_250_299": 18,
"traversal_regret_fallback_depth_bucket_150_199": 5,
"traversal_regret_fallback_depth_bucket_0_49": 17,
"traversal_regret_fallback_depth_bucket_200_249": 6,
"traversal_regret_fallback_depth_bucket_400_plus": 11,
"traversal_regret_fallback_depth_bucket_350_399": 4,
"traversal_regret_fallback_opened_colors_count_5": 62,
"traversal_regret_fallback_opened_colors_count_2": 2,
"traversal_regret_fallback_opened_colors_count_3": 2,
"traversal_regret_fallback_opened_colors_count_4": 3,
"traversal_regret_fallback_opened_colors_count_1": 2,
"traversal_regret_fallback_opened_colors_count_0": 1,
"advantage_samples_returned": 1502,
"strategy_samples_returned": 1502
}
]
},
"encode_legal": {
"median_us": 3.156019374728203,
"p95_us": 3.877998096868396,
"mean_legal_actions": 6.955078125,
"checksum": 7122.0
},
"push_pop": {
"median_us": 0.16100239008665085,
"p95_us": 0.2500019036233425,
"checksum": 13048
},
"policy_boundary_bs1": {
"median_us": 181.29700038116425,
"p95_us": 194.76699526421726,
"checksum": 998.0805744677782
},
"torch_forward": {
"1": {
"median_us_per_call": 141.6669983882457,
"p95_us_per_call": 156.93597379140556,
"checksum": 850.3550980091095
},
"4": {
"median_us_per_call": 34.81799649307504,
"p95_us_per_call": 37.939003959763795,
"checksum": 850.355094909668
},
"8": {
"median_us_per_call": 19.62381247722078,
"p95_us_per_call": 21.306375856511295,
"checksum": 850.3550970554352
},
"64": {
"median_us_per_call": 2.5487893253739458,
"p95_us_per_call": 2.75081220024731,
"checksum": 850.3550720214844
},
"256": {
"median_us_per_call": 0.6167675792312366,
"p95_us_per_call": 0.7250391718116589,
"checksum": 850.3551635742188
}
}
}