Add Julia CFR-shape toy benchmark + Julia port evaluation thread
experiments/julia_cfr_toy/ ports a synthetic CFR external-sampling traversal to both Julia and Cython for direct head-to-head measurement of the actual project hot-path pattern (recursive tree + mutable regret state + branch-heavy legal-action logic). Headline result (2026-05-07, single run): Julia ~1.9× faster than Cython on this pattern, 0 MB allocation, 0% GC time. Root regret parity ε ≤ 1e-9. The GC-pause concern that was the main argument against Julia adoption did not materialize. Cython's 21.3 MB allocation suggests its implementation can be tightened, so the honest gap window is roughly 1.3×–1.9×. Multi-thread scaling (bench_cfr_threaded.jl): 2.44× wall-clock at 8T but only 31% efficiency — inconclusive, likely a toy-size artifact (per-thread workload too small to amortize dispatch). A heavier per-thread workload sweep is the remaining decisive test. docs/research/julia_port_evaluation.md captures this evidence alongside the earlier safe-heuristic single-thread parity result and lists the remaining decision criteria (multi-thread scaling with heavier workload, Flux.jl+CUDA.jl coverage, real-game-state slice). Do not commit to porting until multi-thread scaling is conclusively settled. ideas.md gets a second Active Research Threads pointer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 4.7
@@ -0,0 +1,3 @@
|
||||
bench_cfr.c
|
||||
bench_cfr*.so
|
||||
build/
|
||||
@@ -0,0 +1,79 @@
|
||||
# Julia CFR Toy Benchmark
|
||||
|
||||
This experiment compares Julia and Cython on a CFR-shaped hot path: recursive
|
||||
tree traversal, mutable regret state, branch-heavy legal-action logic, and
|
||||
counterfactual updates. The toy tree is not a game.
|
||||
|
||||
Run Julia only:
|
||||
|
||||
```bash
|
||||
tools/julia/current/bin/julia experiments/julia_cfr_toy/bench_cfr.jl
|
||||
```
|
||||
|
||||
Run Cython plus Julia/Cython parity and comparison:
|
||||
|
||||
```bash
|
||||
uv run python experiments/julia_cfr_toy/bench_cfr_runner.py
|
||||
```
|
||||
|
||||
Run Julia thread-local scaling:
|
||||
|
||||
```bash
|
||||
tools/julia/current/bin/julia --threads=8 experiments/julia_cfr_toy/bench_cfr_threaded.jl
|
||||
```
|
||||
|
||||
The Python runner builds `bench_cfr.pyx` in place when needed, runs the Julia
|
||||
benchmark in JSON mode, then aborts if the final root regret vectors differ by
|
||||
more than `1e-9`.
|
||||
|
||||
Interpretation:
|
||||
|
||||
- `ratio < 1.0`: Julia is faster than Cython for this toy pattern.
|
||||
- `ratio > 1.0`: Julia is slower than Cython for this toy pattern.
|
||||
- Julia `gc share` near zero means mutable-state recursion is not creating
|
||||
meaningful garbage in this benchmark.
|
||||
|
||||
## Results (2026-05-07)
|
||||
|
||||
Single run on the host's Julia 1.11.9 + Cython build. 100 iterations × 1000 traversals.
|
||||
|
||||
| Lang | iter mean (ms) | total (s) | alloc (MB) | gc time (s) | gc share |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Julia | 0.21 | 0.02 | 0.0 | 0.00 | 0.0% |
|
||||
| Cython | 0.41 | 0.04 | 21.3 | n/a | - |
|
||||
| ratio | 0.53× | - | - | - | - |
|
||||
|
||||
Root regret parity verified to ε ≤ 1e-9.
|
||||
|
||||
**Interpretation:** Julia ~1.9× faster than this Cython implementation on the
|
||||
CFR-shape hot path (recursive traversal + mutable regret state +
|
||||
branch-heavy legal-action logic). Zero allocation, zero GC time on the
|
||||
Julia side — escape analysis eliminates heap traffic when the code is
|
||||
type-stable. The GC-pause concern that has been the main argument
|
||||
against Julia adoption is not realized in this pattern.
|
||||
|
||||
**Caveats:**
|
||||
- Cython 21.3 MB alloc suggests room for a more aggressively typed
|
||||
implementation (memoryviews end-to-end). A best-effort Cython could
|
||||
narrow the gap to roughly 1.3×–1.9×.
|
||||
- Toy is not a game. Real Lost Cities CFR has larger state, replay
|
||||
buffer interactions, and an existing Cython implementation already
|
||||
optimized over time.
|
||||
- Single seed, single run. Variance unmeasured.
|
||||
## Thread Scaling (2026-05-07)
|
||||
|
||||
Thread-local trees, same 100 iterations × 1000 traversals total work. Each
|
||||
thread processes its own chunk, then root regrets are reduced. Each threaded
|
||||
case is checked against a sequential run with the same chunking.
|
||||
|
||||
| threads | iter ms | total s | μs/trav | alloc MB | gc s | gc share | speedup | efficiency |
|
||||
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| 1 | 0.219 | 0.0219 | 0.219 | 0.000 | 0.000 | 0.0% | 1.00× | 100% |
|
||||
| 2 | 0.129 | 0.0129 | 0.129 | 0.004 | 0.000 | 0.0% | 1.69× | 85% |
|
||||
| 4 | 0.096 | 0.0096 | 0.096 | 0.004 | 0.000 | 0.0% | 2.28× | 57% |
|
||||
| 8 | 0.090 | 0.0090 | 0.090 | 0.004 | 0.000 | 0.0% | 2.44× | 31% |
|
||||
|
||||
**Interpretation:** 8T efficiency 31% — contention/dispatch overhead dominates
|
||||
at this tiny per-thread workload. Julia still improves absolute throughput, but
|
||||
this is not near-linear scaling. The next scaling test should increase per-thread
|
||||
work before treating this as a hard limit.
|
||||
@@ -0,0 +1,255 @@
|
||||
using Printf
|
||||
|
||||
const MAX_DEPTH = 10
|
||||
const BRANCHING = 4
|
||||
const HALF_DEPTH = MAX_DEPTH ÷ 2
|
||||
const TRAVERSALS_PER_ITER = 1000
|
||||
const MEASURE_ITERATIONS = 100
|
||||
const SEED = UInt64(0x00000000013579bd)
|
||||
const NUM_INTERNAL_NODES = (BRANCHING^MAX_DEPTH - 1) ÷ (BRANCHING - 1)
|
||||
const MASK64 = typemax(UInt64)
|
||||
const TWO_POW_53 = 9007199254740992.0
|
||||
|
||||
mutable struct CFRTree
|
||||
regret::Vector{Float64}
|
||||
end
|
||||
|
||||
function CFRTree()
|
||||
return CFRTree(zeros(Float64, NUM_INTERNAL_NODES * BRANCHING))
|
||||
end
|
||||
|
||||
@inline function regret_index(node_id::Int, action::Int)::Int
|
||||
return (node_id - 1) * BRANCHING + action
|
||||
end
|
||||
|
||||
@inline function splitmix64(x::UInt64)::UInt64
|
||||
x += UInt64(0x9e3779b97f4a7c15)
|
||||
x = (x ⊻ (x >> 30)) * UInt64(0xbf58476d1ce4e5b9)
|
||||
x = (x ⊻ (x >> 27)) * UInt64(0x94d049bb133111eb)
|
||||
return x ⊻ (x >> 31)
|
||||
end
|
||||
|
||||
@inline function hash_key(node_id::Int, depth::Int, action::Int, traversal::Int)::UInt64
|
||||
x = SEED
|
||||
x ⊻= UInt64(node_id) * UInt64(0xd6e8feb86659fd93)
|
||||
x ⊻= UInt64(depth + 1) * UInt64(0xa5a3564e27f886d9)
|
||||
x ⊻= UInt64(action + 11) * UInt64(0x9e3779b185ebca87)
|
||||
x ⊻= UInt64(traversal + 17) * UInt64(0xc2b2ae3d27d4eb4f)
|
||||
return splitmix64(x)
|
||||
end
|
||||
|
||||
@inline function unit_value(key::UInt64)::Float64
|
||||
bits = key >> 11
|
||||
return Float64(bits) / TWO_POW_53
|
||||
end
|
||||
|
||||
@inline function terminal_value(node_id::Int, depth::Int, action::Int, traversal::Int)::Float64
|
||||
return 2.0 * unit_value(hash_key(node_id, depth, action, traversal)) - 1.0
|
||||
end
|
||||
|
||||
@inline function is_legal(depth::Int, action::Int)::Bool
|
||||
return depth < HALF_DEPTH || action != BRANCHING
|
||||
end
|
||||
|
||||
function traverse!(tree::CFRTree, node_id::Int, depth::Int, traversal::Int)::Float64
|
||||
positive_sum = 0.0
|
||||
legal_count = 0
|
||||
strategy1 = 0.0
|
||||
strategy2 = 0.0
|
||||
strategy3 = 0.0
|
||||
strategy4 = 0.0
|
||||
cf1 = 0.0
|
||||
cf2 = 0.0
|
||||
cf3 = 0.0
|
||||
cf4 = 0.0
|
||||
|
||||
@inbounds for action in 1:BRANCHING
|
||||
if is_legal(depth, action)
|
||||
legal_count += 1
|
||||
positive = max(tree.regret[regret_index(node_id, action)], 0.0)
|
||||
positive_sum += positive
|
||||
end
|
||||
end
|
||||
|
||||
if positive_sum > 0.0
|
||||
@inbounds begin
|
||||
strategy1 = is_legal(depth, 1) ? max(tree.regret[regret_index(node_id, 1)], 0.0) / positive_sum : 0.0
|
||||
strategy2 = is_legal(depth, 2) ? max(tree.regret[regret_index(node_id, 2)], 0.0) / positive_sum : 0.0
|
||||
strategy3 = is_legal(depth, 3) ? max(tree.regret[regret_index(node_id, 3)], 0.0) / positive_sum : 0.0
|
||||
strategy4 = is_legal(depth, 4) ? max(tree.regret[regret_index(node_id, 4)], 0.0) / positive_sum : 0.0
|
||||
end
|
||||
else
|
||||
uniform = 1.0 / legal_count
|
||||
strategy1 = is_legal(depth, 1) ? uniform : 0.0
|
||||
strategy2 = is_legal(depth, 2) ? uniform : 0.0
|
||||
strategy3 = is_legal(depth, 3) ? uniform : 0.0
|
||||
strategy4 = is_legal(depth, 4) ? uniform : 0.0
|
||||
end
|
||||
|
||||
r = unit_value(hash_key(node_id, depth, 97, traversal))
|
||||
cumulative = 0.0
|
||||
sampled_action = 1
|
||||
cumulative += strategy1
|
||||
if r <= cumulative
|
||||
sampled_action = 1
|
||||
else
|
||||
cumulative += strategy2
|
||||
if r <= cumulative
|
||||
sampled_action = 2
|
||||
else
|
||||
cumulative += strategy3
|
||||
if r <= cumulative
|
||||
sampled_action = 3
|
||||
else
|
||||
sampled_action = 4
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
sampled_value = 0.0
|
||||
child_id = (node_id - 1) * BRANCHING + sampled_action + 1
|
||||
if depth + 1 >= MAX_DEPTH
|
||||
sampled_value = terminal_value(node_id, depth, sampled_action, traversal)
|
||||
else
|
||||
sampled_value = traverse!(tree, child_id, depth + 1, traversal)
|
||||
end
|
||||
|
||||
cf1 = if !is_legal(depth, 1)
|
||||
0.0
|
||||
elseif sampled_action == 1
|
||||
sampled_value
|
||||
else
|
||||
terminal_value(node_id, depth, 1, traversal)
|
||||
end
|
||||
cf2 = if !is_legal(depth, 2)
|
||||
0.0
|
||||
elseif sampled_action == 2
|
||||
sampled_value
|
||||
else
|
||||
terminal_value(node_id, depth, 2, traversal)
|
||||
end
|
||||
cf3 = if !is_legal(depth, 3)
|
||||
0.0
|
||||
elseif sampled_action == 3
|
||||
sampled_value
|
||||
else
|
||||
terminal_value(node_id, depth, 3, traversal)
|
||||
end
|
||||
cf4 = if !is_legal(depth, 4)
|
||||
0.0
|
||||
elseif sampled_action == 4
|
||||
sampled_value
|
||||
else
|
||||
terminal_value(node_id, depth, 4, traversal)
|
||||
end
|
||||
|
||||
@inbounds begin
|
||||
if is_legal(depth, 1)
|
||||
tree.regret[regret_index(node_id, 1)] += cf1 - sampled_value
|
||||
end
|
||||
if is_legal(depth, 2)
|
||||
tree.regret[regret_index(node_id, 2)] += cf2 - sampled_value
|
||||
end
|
||||
if is_legal(depth, 3)
|
||||
tree.regret[regret_index(node_id, 3)] += cf3 - sampled_value
|
||||
end
|
||||
if is_legal(depth, 4)
|
||||
tree.regret[regret_index(node_id, 4)] += cf4 - sampled_value
|
||||
end
|
||||
end
|
||||
|
||||
return strategy1 * cf1 + strategy2 * cf2 + strategy3 * cf3 + strategy4 * cf4
|
||||
end
|
||||
|
||||
function run_iteration!(tree::CFRTree, iteration::Int)
|
||||
base = (iteration - 1) * TRAVERSALS_PER_ITER
|
||||
value = 0.0
|
||||
for offset in 1:TRAVERSALS_PER_ITER
|
||||
value += traverse!(tree, 1, 0, base + offset)
|
||||
end
|
||||
return value
|
||||
end
|
||||
|
||||
function run_benchmark()
|
||||
warmup_tree = CFRTree()
|
||||
run_iteration!(warmup_tree, 1)
|
||||
|
||||
GC.gc()
|
||||
tree = CFRTree()
|
||||
gc_before = Base.gc_num()
|
||||
elapsed_ref = Ref(0.0)
|
||||
allocated = @allocated begin
|
||||
elapsed_ref[] = @elapsed begin
|
||||
for iteration in 1:MEASURE_ITERATIONS
|
||||
run_iteration!(tree, iteration)
|
||||
end
|
||||
end
|
||||
end
|
||||
gc_after = Base.gc_num()
|
||||
gc_time = (gc_after.total_time - gc_before.total_time) / 1e9
|
||||
total = elapsed_ref[]
|
||||
alloc_mb = allocated / 1024.0 / 1024.0
|
||||
iter_ms = total * 1000.0 / MEASURE_ITERATIONS
|
||||
traversal_us = total * 1_000_000.0 / (MEASURE_ITERATIONS * TRAVERSALS_PER_ITER)
|
||||
root_regret = [
|
||||
tree.regret[regret_index(1, 1)],
|
||||
tree.regret[regret_index(1, 2)],
|
||||
tree.regret[regret_index(1, 3)],
|
||||
tree.regret[regret_index(1, 4)],
|
||||
]
|
||||
return Dict(
|
||||
"lang" => "Julia",
|
||||
"iterations" => MEASURE_ITERATIONS,
|
||||
"traversals_per_iter" => TRAVERSALS_PER_ITER,
|
||||
"total_s" => total,
|
||||
"iter_ms" => iter_ms,
|
||||
"traversal_us" => traversal_us,
|
||||
"alloc_mb" => alloc_mb,
|
||||
"gc_time_s" => gc_time,
|
||||
"gc_share" => total > 0.0 ? gc_time / total : 0.0,
|
||||
"root_regret" => root_regret,
|
||||
)
|
||||
end
|
||||
|
||||
function print_json(result)
|
||||
@printf(
|
||||
"{\"lang\":\"Julia\",\"iterations\":%d,\"traversals_per_iter\":%d,\"total_s\":%.17g,\"iter_ms\":%.17g,\"traversal_us\":%.17g,\"alloc_mb\":%.17g,\"gc_time_s\":%.17g,\"gc_share\":%.17g,\"root_regret\":[%.17g,%.17g,%.17g,%.17g]}\n",
|
||||
result["iterations"],
|
||||
result["traversals_per_iter"],
|
||||
result["total_s"],
|
||||
result["iter_ms"],
|
||||
result["traversal_us"],
|
||||
result["alloc_mb"],
|
||||
result["gc_time_s"],
|
||||
result["gc_share"],
|
||||
result["root_regret"]...,
|
||||
)
|
||||
end
|
||||
|
||||
function print_table(result)
|
||||
println("Lang iter mean (ms) total (s) alloc (MB) gc time (s) gc share")
|
||||
@printf(
|
||||
"%-8s %14.2f %10.2f %11.1f %12.2f %8.1f%%\n",
|
||||
result["lang"],
|
||||
result["iter_ms"],
|
||||
result["total_s"],
|
||||
result["alloc_mb"],
|
||||
result["gc_time_s"],
|
||||
result["gc_share"] * 100.0,
|
||||
)
|
||||
@printf("mean traversal: %.2f μs\n", result["traversal_us"])
|
||||
@printf("root regret: [%.12f, %.12f, %.12f, %.12f]\n", result["root_regret"]...)
|
||||
end
|
||||
|
||||
function main()
|
||||
result = run_benchmark()
|
||||
if "--json" in ARGS
|
||||
print_json(result)
|
||||
else
|
||||
print_table(result)
|
||||
end
|
||||
end
|
||||
|
||||
if abspath(PROGRAM_FILE) == @__FILE__
|
||||
main()
|
||||
end
|
||||
@@ -0,0 +1,156 @@
|
||||
# cython: boundscheck=False, wraparound=False, initializedcheck=False, cdivision=True, language_level=3
|
||||
|
||||
import numpy as np
|
||||
cimport numpy as cnp
|
||||
|
||||
ctypedef unsigned long long uint64_t
|
||||
|
||||
cdef int MAX_DEPTH = 10
|
||||
cdef int BRANCHING = 4
|
||||
cdef int HALF_DEPTH = 5
|
||||
cdef int TRAVERSALS_PER_ITER = 1000
|
||||
cdef int MEASURE_ITERATIONS = 100
|
||||
cdef int NUM_INTERNAL_NODES = 349525
|
||||
cdef uint64_t SEED = <uint64_t>0x00000000013579bd
|
||||
cdef double TWO_POW_53 = 9007199254740992.0
|
||||
|
||||
|
||||
cdef inline uint64_t splitmix64(uint64_t x) noexcept nogil:
|
||||
x = x + <uint64_t>0x9e3779b97f4a7c15
|
||||
x = (x ^ (x >> 30)) * <uint64_t>0xbf58476d1ce4e5b9
|
||||
x = (x ^ (x >> 27)) * <uint64_t>0x94d049bb133111eb
|
||||
return x ^ (x >> 31)
|
||||
|
||||
|
||||
cdef inline uint64_t hash_key(int node_id, int depth, int action, int traversal) noexcept nogil:
|
||||
cdef uint64_t x = SEED
|
||||
x = x ^ (<uint64_t>node_id * <uint64_t>0xd6e8feb86659fd93)
|
||||
x = x ^ (<uint64_t>(depth + 1) * <uint64_t>0xa5a3564e27f886d9)
|
||||
x = x ^ (<uint64_t>(action + 11) * <uint64_t>0x9e3779b185ebca87)
|
||||
x = x ^ (<uint64_t>(traversal + 17) * <uint64_t>0xc2b2ae3d27d4eb4f)
|
||||
return splitmix64(x)
|
||||
|
||||
|
||||
cdef inline double unit_value(uint64_t key) noexcept nogil:
|
||||
return <double>(key >> 11) / TWO_POW_53
|
||||
|
||||
|
||||
cdef inline double terminal_value(int node_id, int depth, int action, int traversal) noexcept nogil:
|
||||
return 2.0 * unit_value(hash_key(node_id, depth, action, traversal)) - 1.0
|
||||
|
||||
|
||||
cdef inline bint is_legal(int depth, int action) noexcept nogil:
|
||||
return depth < HALF_DEPTH or action != BRANCHING
|
||||
|
||||
|
||||
cdef class CFRTree:
|
||||
cdef cnp.ndarray regret_arr
|
||||
cdef double[:, ::1] regret
|
||||
|
||||
def __cinit__(self):
|
||||
self.regret_arr = np.zeros((NUM_INTERNAL_NODES, BRANCHING), dtype=np.float64)
|
||||
self.regret = self.regret_arr
|
||||
|
||||
cdef double traverse(self, int node_id, int depth, int traversal) noexcept nogil:
|
||||
cdef double positive_sum = 0.0
|
||||
cdef int legal_count = 0
|
||||
cdef int action
|
||||
cdef int sampled_action = 1
|
||||
cdef int child_id
|
||||
cdef double positive
|
||||
cdef double uniform
|
||||
cdef double cumulative = 0.0
|
||||
cdef double r
|
||||
cdef double sampled_value
|
||||
cdef double expected = 0.0
|
||||
cdef double strategy[4]
|
||||
cdef double counterfactual[4]
|
||||
|
||||
for action in range(1, BRANCHING + 1):
|
||||
strategy[action - 1] = 0.0
|
||||
counterfactual[action - 1] = 0.0
|
||||
if is_legal(depth, action):
|
||||
legal_count += 1
|
||||
positive = self.regret[node_id - 1, action - 1]
|
||||
if positive < 0.0:
|
||||
positive = 0.0
|
||||
positive_sum += positive
|
||||
|
||||
if positive_sum > 0.0:
|
||||
for action in range(1, BRANCHING + 1):
|
||||
if is_legal(depth, action):
|
||||
positive = self.regret[node_id - 1, action - 1]
|
||||
if positive < 0.0:
|
||||
positive = 0.0
|
||||
strategy[action - 1] = positive / positive_sum
|
||||
else:
|
||||
uniform = 1.0 / legal_count
|
||||
for action in range(1, BRANCHING + 1):
|
||||
if is_legal(depth, action):
|
||||
strategy[action - 1] = uniform
|
||||
|
||||
r = unit_value(hash_key(node_id, depth, 97, traversal))
|
||||
for action in range(1, BRANCHING + 1):
|
||||
cumulative += strategy[action - 1]
|
||||
if r <= cumulative:
|
||||
sampled_action = action
|
||||
break
|
||||
|
||||
child_id = (node_id - 1) * BRANCHING + sampled_action + 1
|
||||
if depth + 1 >= MAX_DEPTH:
|
||||
sampled_value = terminal_value(node_id, depth, sampled_action, traversal)
|
||||
else:
|
||||
sampled_value = self.traverse(child_id, depth + 1, traversal)
|
||||
|
||||
for action in range(1, BRANCHING + 1):
|
||||
if is_legal(depth, action):
|
||||
if action == sampled_action:
|
||||
counterfactual[action - 1] = sampled_value
|
||||
else:
|
||||
counterfactual[action - 1] = terminal_value(node_id, depth, action, traversal)
|
||||
|
||||
for action in range(1, BRANCHING + 1):
|
||||
if is_legal(depth, action):
|
||||
self.regret[node_id - 1, action - 1] += counterfactual[action - 1] - sampled_value
|
||||
|
||||
for action in range(1, BRANCHING + 1):
|
||||
expected += strategy[action - 1] * counterfactual[action - 1]
|
||||
return expected
|
||||
|
||||
cdef double run_iteration(self, int iteration) noexcept nogil:
|
||||
cdef int base = (iteration - 1) * TRAVERSALS_PER_ITER
|
||||
cdef int offset
|
||||
cdef double value = 0.0
|
||||
for offset in range(1, TRAVERSALS_PER_ITER + 1):
|
||||
value += self.traverse(1, 0, base + offset)
|
||||
return value
|
||||
|
||||
cpdef run_iterations(self, int iterations):
|
||||
cdef int iteration
|
||||
cdef double total_value = 0.0
|
||||
with nogil:
|
||||
for iteration in range(1, iterations + 1):
|
||||
total_value += self.run_iteration(iteration)
|
||||
return total_value
|
||||
|
||||
cpdef root_regret(self):
|
||||
return [
|
||||
float(self.regret[0, 0]),
|
||||
float(self.regret[0, 1]),
|
||||
float(self.regret[0, 2]),
|
||||
float(self.regret[0, 3]),
|
||||
]
|
||||
|
||||
|
||||
def run_benchmark():
|
||||
warmup_tree = CFRTree()
|
||||
warmup_tree.run_iterations(1)
|
||||
|
||||
tree = CFRTree()
|
||||
tree.run_iterations(MEASURE_ITERATIONS)
|
||||
return {
|
||||
"lang": "Cython",
|
||||
"iterations": MEASURE_ITERATIONS,
|
||||
"traversals_per_iter": TRAVERSALS_PER_ITER,
|
||||
"root_regret": tree.root_regret(),
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import gc
|
||||
import importlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import tracemalloc
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
REPO_ROOT = ROOT.parents[1]
|
||||
ITERATIONS = 100
|
||||
TRAVERSALS_PER_ITER = 1000
|
||||
EPSILON = 1e-9
|
||||
|
||||
|
||||
def _build_extension() -> None:
|
||||
subprocess.run(
|
||||
[sys.executable, "bench_cfr_setup.py", "build_ext", "--inplace"],
|
||||
cwd=ROOT,
|
||||
check=True,
|
||||
)
|
||||
|
||||
|
||||
def _import_extension():
|
||||
sys.path.insert(0, str(ROOT))
|
||||
source_mtime = (ROOT / "bench_cfr.pyx").stat().st_mtime
|
||||
extensions = list(ROOT.glob("bench_cfr*.so"))
|
||||
if not extensions or max(path.stat().st_mtime for path in extensions) < source_mtime:
|
||||
_build_extension()
|
||||
try:
|
||||
return importlib.import_module("bench_cfr")
|
||||
except ImportError:
|
||||
_build_extension()
|
||||
importlib.invalidate_caches()
|
||||
return importlib.import_module("bench_cfr")
|
||||
|
||||
|
||||
def _julia_executable() -> str:
|
||||
local = REPO_ROOT / "tools" / "julia" / "current" / "bin" / "julia"
|
||||
if local.exists():
|
||||
return str(local)
|
||||
return "julia"
|
||||
|
||||
|
||||
def _run_julia() -> dict[str, Any]:
|
||||
proc = subprocess.run(
|
||||
[_julia_executable(), str(ROOT / "bench_cfr.jl"), "--json"],
|
||||
cwd=REPO_ROOT,
|
||||
check=True,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
return json.loads(proc.stdout)
|
||||
|
||||
|
||||
def _run_cython() -> dict[str, Any]:
|
||||
bench_cfr = _import_extension()
|
||||
gc.collect()
|
||||
tracemalloc.start()
|
||||
start = time.perf_counter()
|
||||
result = bench_cfr.run_benchmark()
|
||||
total_s = time.perf_counter() - start
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
|
||||
result["total_s"] = total_s
|
||||
result["iter_ms"] = total_s * 1000.0 / ITERATIONS
|
||||
result["traversal_us"] = total_s * 1_000_000.0 / (ITERATIONS * TRAVERSALS_PER_ITER)
|
||||
result["alloc_mb"] = peak / 1024.0 / 1024.0
|
||||
result["gc_time_s"] = None
|
||||
result["gc_share"] = None
|
||||
return result
|
||||
|
||||
|
||||
def _assert_equivalent(julia: dict[str, Any], cython: dict[str, Any]) -> None:
|
||||
diffs = [
|
||||
abs(float(j_value) - float(c_value))
|
||||
for j_value, c_value in zip(julia["root_regret"], cython["root_regret"], strict=True)
|
||||
]
|
||||
max_diff = max(diffs)
|
||||
if max_diff > EPSILON:
|
||||
raise SystemExit(
|
||||
"root regret mismatch: "
|
||||
f"max_diff={max_diff:.3e} "
|
||||
f"julia={julia['root_regret']} cython={cython['root_regret']}"
|
||||
)
|
||||
|
||||
|
||||
def _print_table(julia: dict[str, Any], cython: dict[str, Any]) -> None:
|
||||
ratio = julia["iter_ms"] / cython["iter_ms"]
|
||||
gc_share = julia["gc_share"] * 100.0
|
||||
print("Lang iter mean (ms) total (s) alloc (MB) gc time (s) gc share")
|
||||
print(
|
||||
f"{'Julia':<8} {julia['iter_ms']:14.2f} {julia['total_s']:10.2f} "
|
||||
f"{julia['alloc_mb']:11.1f} {julia['gc_time_s']:12.2f} {gc_share:8.1f}%"
|
||||
)
|
||||
print(
|
||||
f"{'Cython':<8} {cython['iter_ms']:14.2f} {cython['total_s']:10.2f} "
|
||||
f"{cython['alloc_mb']:11.1f} {'n/a':>12} {'-':>8}"
|
||||
)
|
||||
print(f"{'ratio':<8} {ratio:13.2f}x {'-':>10} {'-':>11} {'-':>12} {'-':>8}")
|
||||
|
||||
if ratio < 0.95:
|
||||
verdict = "Julia faster"
|
||||
elif ratio > 1.05:
|
||||
verdict = "Julia slower"
|
||||
else:
|
||||
verdict = "rough parity"
|
||||
gc_note = "GC negligible" if gc_share < 5.0 else "GC visible"
|
||||
print(f"Julia/Cython iter ratio {ratio:.2f}x — {verdict}, {gc_note} ({gc_share:.1f}% GC)")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
os.environ.setdefault("JULIA_NUM_THREADS", "1")
|
||||
julia = _run_julia()
|
||||
cython = _run_cython()
|
||||
_assert_equivalent(julia, cython)
|
||||
_print_table(julia, cython)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,23 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
from Cython.Build import cythonize
|
||||
from setuptools import Extension, setup
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
setup(
|
||||
name="bench_cfr",
|
||||
ext_modules=cythonize(
|
||||
[
|
||||
Extension(
|
||||
"bench_cfr",
|
||||
[str(ROOT / "bench_cfr.pyx")],
|
||||
include_dirs=[np.get_include()],
|
||||
)
|
||||
],
|
||||
compiler_directives={"language_level": "3"},
|
||||
),
|
||||
)
|
||||
@@ -0,0 +1,166 @@
|
||||
using Printf
|
||||
using Statistics
|
||||
using Base.Threads
|
||||
|
||||
include("bench_cfr.jl")
|
||||
|
||||
const THREAD_GRID = (1, 2, 4, 8)
|
||||
const RUNS_PER_CASE = 5
|
||||
const ROOT_ACTIONS = 4
|
||||
|
||||
function iteration_ranges(chunks::Int)::Vector{UnitRange{Int}}
|
||||
ranges = Vector{UnitRange{Int}}(undef, chunks)
|
||||
base = MEASURE_ITERATIONS ÷ chunks
|
||||
extra = MEASURE_ITERATIONS % chunks
|
||||
start = 1
|
||||
for chunk in 1:chunks
|
||||
width = base + (chunk <= extra ? 1 : 0)
|
||||
stop = start + width - 1
|
||||
ranges[chunk] = start:stop
|
||||
start = stop + 1
|
||||
end
|
||||
return ranges
|
||||
end
|
||||
|
||||
function reset_trees!(trees::Vector{CFRTree}, chunks::Int)
|
||||
for chunk in 1:chunks
|
||||
fill!(trees[chunk].regret, 0.0)
|
||||
end
|
||||
end
|
||||
|
||||
function reduce_root!(out::Vector{Float64}, trees::Vector{CFRTree}, chunks::Int)
|
||||
fill!(out, 0.0)
|
||||
@inbounds for chunk in 1:chunks
|
||||
tree = trees[chunk]
|
||||
out[1] += tree.regret[regret_index(1, 1)]
|
||||
out[2] += tree.regret[regret_index(1, 2)]
|
||||
out[3] += tree.regret[regret_index(1, 3)]
|
||||
out[4] += tree.regret[regret_index(1, 4)]
|
||||
end
|
||||
end
|
||||
|
||||
function run_chunk!(tree::CFRTree, range::UnitRange{Int})
|
||||
value = 0.0
|
||||
for iteration in range
|
||||
value += run_iteration!(tree, iteration)
|
||||
end
|
||||
return value
|
||||
end
|
||||
|
||||
function run_case!(
|
||||
out::Vector{Float64},
|
||||
trees::Vector{CFRTree},
|
||||
ranges::Vector{UnitRange{Int}},
|
||||
chunks::Int;
|
||||
threaded::Bool,
|
||||
)
|
||||
reset_trees!(trees, chunks)
|
||||
if threaded && chunks > 1
|
||||
@threads for chunk in 1:chunks
|
||||
run_chunk!(trees[chunk], ranges[chunk])
|
||||
end
|
||||
else
|
||||
for chunk in 1:chunks
|
||||
run_chunk!(trees[chunk], ranges[chunk])
|
||||
end
|
||||
end
|
||||
reduce_root!(out, trees, chunks)
|
||||
end
|
||||
|
||||
function assert_close(label::AbstractString, actual::Vector{Float64}, expected::Vector{Float64})
|
||||
max_diff = maximum(abs.(actual .- expected))
|
||||
if max_diff > 1e-9
|
||||
error("$label parity failed: max_diff=$max_diff actual=$actual expected=$expected")
|
||||
end
|
||||
end
|
||||
|
||||
function measure_case(chunks::Int)
|
||||
ranges = iteration_ranges(chunks)
|
||||
reference_trees = [CFRTree() for _ in 1:chunks]
|
||||
measured_trees = [CFRTree() for _ in 1:chunks]
|
||||
reference = zeros(Float64, ROOT_ACTIONS)
|
||||
actual = zeros(Float64, ROOT_ACTIONS)
|
||||
|
||||
run_case!(reference, reference_trees, ranges, chunks; threaded=false)
|
||||
run_case!(actual, measured_trees, ranges, chunks; threaded=chunks > 1)
|
||||
assert_close("warmup $(chunks)T", actual, reference)
|
||||
|
||||
times = Vector{Float64}(undef, RUNS_PER_CASE)
|
||||
allocs = Vector{Int}(undef, RUNS_PER_CASE)
|
||||
gc_times = Vector{Float64}(undef, RUNS_PER_CASE)
|
||||
for run in 1:RUNS_PER_CASE
|
||||
GC.gc()
|
||||
before = Base.gc_num()
|
||||
elapsed_ref = Ref(0.0)
|
||||
allocated = @allocated begin
|
||||
elapsed_ref[] = @elapsed begin
|
||||
run_case!(actual, measured_trees, ranges, chunks; threaded=chunks > 1)
|
||||
end
|
||||
end
|
||||
after = Base.gc_num()
|
||||
assert_close("run $(run) $(chunks)T", actual, reference)
|
||||
times[run] = elapsed_ref[]
|
||||
allocs[run] = allocated
|
||||
gc_times[run] = (after.total_time - before.total_time) / 1e9
|
||||
end
|
||||
|
||||
total_s = median(times)
|
||||
alloc_mb = median(allocs) / 1024.0 / 1024.0
|
||||
gc_time_s = median(gc_times)
|
||||
return Dict(
|
||||
"threads" => chunks,
|
||||
"total_s" => total_s,
|
||||
"iter_ms" => total_s * 1000.0 / MEASURE_ITERATIONS,
|
||||
"traversal_us" => total_s * 1_000_000.0 / (MEASURE_ITERATIONS * TRAVERSALS_PER_ITER),
|
||||
"alloc_mb" => alloc_mb,
|
||||
"gc_time_s" => gc_time_s,
|
||||
"gc_share" => total_s > 0.0 ? gc_time_s / total_s : 0.0,
|
||||
"root_regret" => copy(reference),
|
||||
)
|
||||
end
|
||||
|
||||
function scaling_label(efficiency::Float64)::String
|
||||
if efficiency >= 0.80
|
||||
return "near-linear"
|
||||
elseif efficiency >= 0.50
|
||||
return "partial scale"
|
||||
end
|
||||
return "contention"
|
||||
end
|
||||
|
||||
function main()
|
||||
available = Threads.nthreads()
|
||||
grid = [threads for threads in THREAD_GRID if threads <= available]
|
||||
if isempty(grid)
|
||||
error("no thread counts available")
|
||||
end
|
||||
|
||||
results = [measure_case(threads) for threads in grid]
|
||||
one_thread_s = results[1]["total_s"]
|
||||
|
||||
println("threads iter_ms total_s μs/trav alloc_MB gc_s gc_share speedup efficiency")
|
||||
for result in results
|
||||
threads = result["threads"]
|
||||
speedup = one_thread_s / result["total_s"]
|
||||
efficiency = speedup / threads
|
||||
@printf(
|
||||
"%-7d %8.3f %8.4f %8.3f %9.3f %5.3f %8.1f%% %8.2f× %9.0f%%\n",
|
||||
threads,
|
||||
result["iter_ms"],
|
||||
result["total_s"],
|
||||
result["traversal_us"],
|
||||
result["alloc_mb"],
|
||||
result["gc_time_s"],
|
||||
result["gc_share"] * 100.0,
|
||||
speedup,
|
||||
efficiency * 100.0,
|
||||
)
|
||||
end
|
||||
|
||||
last = results[end]
|
||||
speedup = one_thread_s / last["total_s"]
|
||||
efficiency = speedup / last["threads"]
|
||||
@printf("%dT efficiency %.0f%% — %s\n", last["threads"], efficiency * 100.0, scaling_label(efficiency))
|
||||
end
|
||||
|
||||
main()
|
||||
Reference in New Issue
Block a user