Pay the match total densely and drop the tanh terminal reward

The user asked why we were not simply maximising the three-round total, and they
were right. Each ply now pays the points by which it moved the running match
difference; at gamma=1 that telescopes to the final total, so the objective is
exactly the rulebook's, handed out every ply instead of once 150 plies later.

Three measurements killed the tanh design:

- Rounds are independent (corr(m1,m2)=0.004, corr(m1+m2,m3)=0.05), so a reward
  linear in the total decomposes the match into three independent rounds and
  carry enters the objective nowhere. The only coupling, the start-player rule,
  is worth +0.73 +/- 0.84 points -- indistinguishable from zero.
- Risk attitude, the one thing tanh buys, is worthless here. A policy made to
  gamble when it trails by 20 entering round three *loses* to a greedy clone over
  6144 duplicate matches (0.482); gambling only at -40 breaks even (0.498). A
  marginal wager buys about +1.7 sigma for -2 to -3 expected points. Ceiling on
  the whole carry-conditioning idea: under one win-rate point.
- Head to head over 10,000 duplicate matches at equal compute, the linear reward
  *beats* tanh(total/12): 0.5859 (CI 0.576-0.596), +20.3 points. Dropping it is
  not merely free, it is better -- not because of risk, but because tanh hands a
  ~150-ply match one saturated +/-1 and leaves all credit assignment to the critic.

The flat carry probe was not exploration collapse: sampled play still opens 5.00
expeditions, entropy settles at 1.36 nats (3.9 effective actions), and the critic
reads carry cleanly (round-three values run -0.87 to +0.86, monotone). The signal
was there; there was nothing to buy with it.

Criterion 1 (a monotone carry response) comes off the gate accordingly -- the
optimal response barely exists in this game. carry stays in the observation: it
costs nothing and the start-player rule keys off it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XBQKgvBbxbheiTF1AVy1Sh

tests/lost_cities_jax/test_match_ppo.py

@@ -74,7 +74,7 @@ def test_privileged_input_cannot_move_the_policy_logits():
 # --- the match reward ------------------------------------------------------
 
 
-def test_reward_is_zero_sum_and_only_paid_at_the_end_of_the_match():
+def test_reward_is_zero_sum():
     cfg = _cfg()
     state = create_match_train_state(cfg, jax.random.PRNGKey(3))
     env = jax.jit(jax.vmap(match_reset))(
@@ -82,37 +82,37 @@ def test_reward_is_zero_sum_and_only_paid_at_the_end_of_the_match():
     )
 
     rollout = make_match_rollout_fn(cfg)
-    # Shaping off, so any non-zero reward must be a terminal one.
-    _, transitions, _, metrics = rollout(state, env, jax.random.PRNGKey(5), jnp.asarray(0.0))
+    _, transitions, _, metrics = rollout(state, env, jax.random.PRNGKey(5), jnp.asarray(1.0))
 
     reward = np.asarray(transitions.reward)  # (T, 2 * batch)
     seat0, seat1 = reward[:, : cfg.ppo.batch_games], reward[:, cfg.ppo.batch_games :]
     assert np.allclose(seat0, -seat1)
-
-    done = np.asarray(transitions.done)[:, : cfg.ppo.batch_games]
-    paid = seat0 != 0.0
-    assert np.array_equal(paid, done & paid)  # never paid on a non-terminal ply
     assert float(metrics["matches_completed"]) > 0
-    assert np.abs(seat0[paid]).max() <= 1.0  # tanh-bounded
 
 
-def test_shaping_tracks_the_running_match_total():
-    """Phi is carry + board diff, so shaping must follow the total, not the round."""
+def test_the_dense_reward_telescopes_to_the_match_total():
+    """The whole justification for paying every ply: at gamma=1 the sum is the total.
+
+    That is what makes "score more across three rounds" the objective, rather than
+    some shaped proxy for it.
+    """
     match = match_reset(jax.random.PRNGKey(6))
     key = jax.random.PRNGKey(7)
+    scale = 50.0
 
-    prev = np.asarray(match_score(match))
-    for _ in range(200):
-        if bool(match.done):
-            break
+    paid = 0.0
+    while not bool(match.done):
         key, step_key = jax.random.split(key)
         action = random_legal_action(match.round, match.round.to_move, step_key)
+        before = np.asarray(match_score(match))
         match, _, _ = match_step(match, action)
-        total = np.asarray(match_score(match))
-        # The total never resets when a round rolls over; it only accumulates.
-        assert total.shape == (2,)
-        prev = total
-    assert prev.shape == (2,)
+        after = np.asarray(match_score(match))
+        paid += ((after[0] - after[1]) - (before[0] - before[1])) / scale
+
+    final = np.asarray(match_score(match))
+    total_lead = (final[0] - final[1]) / scale
+    # Every point banked along the way, and nothing else.
+    assert paid == pytest.approx(total_lead, abs=1e-4)
 
 
 # --- both seats ------------------------------------------------------------