Add explicit pass/fail thresholds for Julia port criteria

Decision-in-advance for criteria 3 (multi-thread scaling), 4 (Flux/CUDA
MLP forward), and 5 (real-game-state slice). Each criterion lists PASS
/ PARTIAL / FAIL bands with concrete numerical thresholds, plus a
decision rule that maps {3, 4, 5} outcomes to a single action: port,
hybrid (Julia traversal + PyTorch networks), or stay on Python/Cython
and pursue Option B instead.

Cost-of-being-wrong asymmetry stated explicitly: port is months,
staying is zero work, so the GO bar is deliberately above 50% and the
STAY bar is permissive.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-07 21:12:10 +09:00
co-authored by Claude Opus 4.7
parent 100f498cf1
commit 14dbf9c804
+68
View File
@@ -99,6 +99,74 @@ the decisive PASS threshold for the threading criterion.
fixed corpus of game states, compare to current Cython. Criterion 5.
Not started.
## Pass/fail thresholds (decided in advance)
These are explicit so that the moment a measurement lands, the decision
is automatic. No re-deliberation, no "let's discuss it." If a result
sits on a boundary, treat it as the pessimistic side.
**Criterion 3 (multi-thread scaling)** — already measured on the toy:
- PASS: 8T efficiency ≥ 80%.
- PARTIAL: 5080%. (Current toy result: 59%.)
- FAIL: < 50%.
Re-measure on the real-game-state slice (criterion 5) when that lands.
The toy figure is informative but not authoritative; real workloads have
more compute per traversal and may scale better.
**Criterion 4 (Flux.jl + CUDA.jl MLP forward)**:
- PASS: bs=64 forward time within ±20% of PyTorch on the same GPU and
same `DeepCFRMLP` shape (input_dim=365, hidden=512, layers=3,
output_dim=22). Both bs=1 and bs=256 must also be within ±30% (single-
state and large-batch matter for traversal-time and eval-time
respectively).
- FAIL: any of the three batch sizes regresses by more than the band
above. Single PASS at one batch size is not sufficient.
Methodology: 100-iter timing with 10-iter warm-up, `inference_mode`
equivalent on both sides. Identical weights (export from PyTorch, load
into Flux). Compare per-state latency (μs/state).
**Criterion 5 (real-game-state slice)**:
- Scope: a Julia port of a single representative slice of the Lost
Cities engine — at minimum `legal_actions` + `apply_action` + scoring
for end-of-game, against a fixed corpus of ≥1000 game states exported
from the current Cython engine.
- Action-equivalence requirement: byte-identical `legal_actions` set
and post-action state for every corpus entry. ε = 0; this is a
correctness gate, not a numerical one.
- PASS: single-thread per-state cost within ±25% of current Cython, AND
re-running the heavy threaded benchmark on this slice (1/2/4/8T)
yields 8T efficiency ≥ 75%. Both conditions required.
- FAIL: single-thread regresses > 50% vs Cython, OR 8T efficiency
< 60% on the real slice.
- BORDERLINE: between FAIL and PASS — flag in the doc, do not start a
port; consider whether a different scoping (port traversal recursion
only, leave game engine in Cython) clears the threshold instead.
## Decision rule
After all of criteria 3 (re-measured on real slice), 4, 5 land:
- All three PASS → start a port plan. Deferred Option A re-enable, AMP,
compile, TensorRT all get re-evaluated under the new runtime.
- Criterion 5 PASS but criterion 4 FAIL → consider a hybrid: keep
PyTorch for networks via PythonCall.jl/PyCall, port traversal to
Julia. Re-evaluate IPC/FFI cost separately.
- Any criterion FAIL beyond the borderline → **stay on Python/Cython,
pursue Option B (per-worker interleaved traversal)** as the
GIL-escape path instead. Document the FAIL result, close this thread.
- Criterion 5 BORDERLINE → reduce port scope (recursion-only) and
re-test.
Cost-of-being-wrong asymmetry: a port is months of work; staying is
zero work but caps us at the current ceiling. So the bar to GO is
deliberately set above 50%; the bar to STAY is permissive. This is on
purpose.
## Decision posture
Promising but not enough to justify a port yet. The completed benchmarks