From b561fa845798f2ba7f66e84cd7be058836c4c814 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=A0=95=EC=8B=9C=EC=9B=90?= Date: Thu, 7 May 2026 16:56:17 +0900 Subject: [PATCH] Document run comparison protocol: sequential, single seed, shared tag Default is one baseline + one treatment, sequential, same seed, with a shared --wandb-tag hypothesis label for W&B Compare Runs filtering. Multi-seed only on explicit request; never run two trainings on the same GPU. Co-Authored-By: Claude Opus 4.7 (1M context) --- AGENTS.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 6e51829..99c9569 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -325,6 +325,21 @@ keywords, repeatable). Otherwise use them however you like. Just avoid: to compare against). Long analyses go in `docs/` and are linked from notes; don't paste them in. +### Comparing two runs + +Default is **sequential, single seed**. Run baseline first, then the +treatment with exactly one config change, both with the same `run.seed`. +Tag both with a shared hypothesis tag (e.g. `--wandb-tag lr-bump`) so +they show up together in W&B's Compare Runs view. + +Do **not** run multiple seeds per condition unless explicitly asked — +that doubles or quintuples wall-clock and isn't the default protocol. +Single-seed comparison is enough to surface a signal; multi-seed is a +follow-up to confirm it. + +Do **not** run two trainings in parallel on the same GPU — VRAM/SM +contention slows both unevenly and breaks the comparison. + ## Notes For Future Agents - Prefer `rg`/`rg --files` for search.