Document run comparison protocol: sequential, single seed, shared tag

Default is one baseline + one treatment, sequential, same seed, with a
shared --wandb-tag hypothesis label for W&B Compare Runs filtering.
Multi-seed only on explicit request; never run two trainings on the same
GPU.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-07 16:56:17 +09:00
co-authored by Claude Opus 4.7
parent 59b8f24b91
commit b561fa8457
+15
View File
@@ -325,6 +325,21 @@ keywords, repeatable). Otherwise use them however you like. Just avoid:
to compare against). Long analyses go in `docs/` and are linked from
notes; don't paste them in.
### Comparing two runs
Default is **sequential, single seed**. Run baseline first, then the
treatment with exactly one config change, both with the same `run.seed`.
Tag both with a shared hypothesis tag (e.g. `--wandb-tag lr-bump`) so
they show up together in W&B's Compare Runs view.
Do **not** run multiple seeds per condition unless explicitly asked —
that doubles or quintuples wall-clock and isn't the default protocol.
Single-seed comparison is enough to surface a signal; multi-seed is a
follow-up to confirm it.
Do **not** run two trainings in parallel on the same GPU — VRAM/SM
contention slows both unevenly and breaks the comparison.
## Notes For Future Agents
- Prefer `rg`/`rg --files` for search.