Add explicit pass/fail thresholds for Julia port criteria

Decision-in-advance for criteria 3 (multi-thread scaling), 4 (Flux/CUDA
MLP forward), and 5 (real-game-state slice). Each criterion lists PASS
/ PARTIAL / FAIL bands with concrete numerical thresholds, plus a
decision rule that maps {3, 4, 5} outcomes to a single action: port,
hybrid (Julia traversal + PyTorch networks), or stay on Python/Cython
and pursue Option B instead.

Cost-of-being-wrong asymmetry stated explicitly: port is months,
staying is zero work, so the GO bar is deliberately above 50% and the
STAY bar is permissive.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-07 21:12:10 +09:00
co-authored by Claude Opus 4.7
parent 100f498cf1
commit 14dbf9c804
+68
View File
@@ -99,6 +99,74 @@ the decisive PASS threshold for the threading criterion.
fixed corpus of game states, compare to current Cython. Criterion 5. fixed corpus of game states, compare to current Cython. Criterion 5.
Not started. Not started.
## Pass/fail thresholds (decided in advance)
These are explicit so that the moment a measurement lands, the decision
is automatic. No re-deliberation, no "let's discuss it." If a result
sits on a boundary, treat it as the pessimistic side.
**Criterion 3 (multi-thread scaling)** — already measured on the toy:
- PASS: 8T efficiency ≥ 80%.
- PARTIAL: 5080%. (Current toy result: 59%.)
- FAIL: < 50%.
Re-measure on the real-game-state slice (criterion 5) when that lands.
The toy figure is informative but not authoritative; real workloads have
more compute per traversal and may scale better.
**Criterion 4 (Flux.jl + CUDA.jl MLP forward)**:
- PASS: bs=64 forward time within ±20% of PyTorch on the same GPU and
same `DeepCFRMLP` shape (input_dim=365, hidden=512, layers=3,
output_dim=22). Both bs=1 and bs=256 must also be within ±30% (single-
state and large-batch matter for traversal-time and eval-time
respectively).
- FAIL: any of the three batch sizes regresses by more than the band
above. Single PASS at one batch size is not sufficient.
Methodology: 100-iter timing with 10-iter warm-up, `inference_mode`
equivalent on both sides. Identical weights (export from PyTorch, load
into Flux). Compare per-state latency (μs/state).
**Criterion 5 (real-game-state slice)**:
- Scope: a Julia port of a single representative slice of the Lost
Cities engine — at minimum `legal_actions` + `apply_action` + scoring
for end-of-game, against a fixed corpus of ≥1000 game states exported
from the current Cython engine.
- Action-equivalence requirement: byte-identical `legal_actions` set
and post-action state for every corpus entry. ε = 0; this is a
correctness gate, not a numerical one.
- PASS: single-thread per-state cost within ±25% of current Cython, AND
re-running the heavy threaded benchmark on this slice (1/2/4/8T)
yields 8T efficiency ≥ 75%. Both conditions required.
- FAIL: single-thread regresses > 50% vs Cython, OR 8T efficiency
< 60% on the real slice.
- BORDERLINE: between FAIL and PASS — flag in the doc, do not start a
port; consider whether a different scoping (port traversal recursion
only, leave game engine in Cython) clears the threshold instead.
## Decision rule
After all of criteria 3 (re-measured on real slice), 4, 5 land:
- All three PASS → start a port plan. Deferred Option A re-enable, AMP,
compile, TensorRT all get re-evaluated under the new runtime.
- Criterion 5 PASS but criterion 4 FAIL → consider a hybrid: keep
PyTorch for networks via PythonCall.jl/PyCall, port traversal to
Julia. Re-evaluate IPC/FFI cost separately.
- Any criterion FAIL beyond the borderline → **stay on Python/Cython,
pursue Option B (per-worker interleaved traversal)** as the
GIL-escape path instead. Document the FAIL result, close this thread.
- Criterion 5 BORDERLINE → reduce port scope (recursion-only) and
re-test.
Cost-of-being-wrong asymmetry: a port is months of work; staying is
zero work but caps us at the current ceiling. So the bar to GO is
deliberately set above 50%; the bar to STAY is permissive. This is on
purpose.
## Decision posture ## Decision posture
Promising but not enough to justify a port yet. The completed benchmarks Promising but not enough to justify a port yet. The completed benchmarks