coorl-lost-cities/docs/research/opponent-policy-network-divergence.md

Opponent Policy: Network vs. Self-Play League

Last verified: 2026-05-07, commit ad0be89

Source: docs/archive/deep-cfr-opponent-policy-network-divergence-2026-05-07.md

Question

Why does traversal.opponent_policy: network lead to policy collapse, and what makes self_play_league (the default) stable?

Short answer: using the currently training network as its own traversal opponent violates the stationarity assumption that Deep CFR's convergence proof rests on. The opponent policy must be fixed (or drawn from a fixed distribution) within a training iteration; feeding a moving target to the advantage estimator produces non-stationary regret signals that compound into divergence. A snapshot pool supplies that fixed diversity.

Code reference

The opponent policy mode is selected in src/coolrl_lost_cities/games/classic/deep_cfr/traversal.pyx during opponent node evaluation. The trainer wires the policy source in src/coolrl_lost_cities/games/classic/deep_cfr/trainer.py, which reads traversal.opponent_policy from config (src/coolrl_lost_cities/games/classic/deep_cfr/config.py).

The snapshot pool that backs self_play_league is managed through src/coolrl_lost_cities/games/classic/deep_cfr/checkpoints.py and the self_play.max_snapshots / self_play.current_weight / self_play.recent_weight / self_play.older_weight config knobs.

Divergence mechanism

Deep CFR's convergence guarantee is that the average strategy (maintained by the strategy network) converges toward a Nash equilibrium as regret sums accumulate over many iterations. This holds when the opponent at each traversal node plays a policy that is either (a) fixed or (b) drawn i.i.d. from a stationary distribution — the classic external-sampling assumption.

opponent_policy: network breaks this in four compounding ways:

1. Moving target. Every iteration updates the network weights, so the opponent's policy distribution shifts between iterations. Advantage samples stored in the replay buffer were measured under different opponent policies and cannot be treated as samples from the same distribution. The advantage network learns a target that keeps moving underneath it.

2. Echo chamber. The traverser and its opponent share the same network, so whatever weaknesses the traverser has are invisible to the opponent. States that would expose those weaknesses (e.g., a patient Safe Heuristic-style opponent that never over-opens) are never generated during traversal. Regret signals for responding to such opponents never appear.

3. No-regret guarantee breaks. External-sampling MCCFR's unbiased regret estimate requires the opponent to sample from a fixed strategy. When the opponent is the network-in-training, the estimator is biased in a time-varying way. The no-regret property that drives average-strategy convergence no longer holds.

4. Strategy mode collapse. Self-play between identical agents tends to converge to a deterministic-like Nash approximation even when the true Nash is mixed. In an imperfect-information game like Lost Cities, that collapsed strategy is exploitable by any opponent outside the narrow equilibrium.

Observed behavior

Two controlled experiments (512x3 and 1024x4 hidden size / layers) both reached a performance peak early and then diverged:

• 512x3: peak at iteration 15 (~85% win rate vs. Random), then rapid collapse by iteration 30 to below-random performance, stable there through iteration 363.
• 1024x4: larger capacity delayed collapse — plateau held from roughly iteration 30 to 95, with a best win rate of 13% against Safe Heuristic at iteration 85 — but divergence was ultimately the same.

A directly comparable run with self_play_league (512x3 architecture, otherwise identical hyperparameters) reached a similar early peak, then continued improving through iteration 350 with a 72% win rate vs. Random — a 38 percentage-point gap against the collapsed network run at the same iteration.

The key insight from the comparison: the early peak is similar regardless of opponent policy, because the initial regret signal is useful for both. The divergence is entirely post-peak, driven by the stationarity violation.

Why self_play_league is stable

With self_play.max_snapshots > 0, the opponent at each traversal is sampled from a pool of past checkpoints. Each snapshot is a fixed policy at the moment it was saved. The traversal therefore draws its opponent from a stationary distribution (the snapshot pool), satisfying the external-sampling assumption. Diversity across snapshots ensures the traverser encounters a range of opponent styles, preventing echo-chamber collapse.

The weighted bucket scheme (current_weight, recent_weight, older_weight) controls how much the pool emphasizes recent vs. historical policies, letting practitioners tune recency without sacrificing the stationarity guarantee.

Practical implication

• Do not use opponent_policy: network for extended training runs. It can look promising in the first 10–20 iterations, which makes it easy to misinterpret short pilots as success.
• If network-opponent runs are conducted (e.g., to examine early dynamics), enable short save_iteration_interval and retain checkpoints from the plateau phase — divergence is irreversible once started, and final checkpoints are useless.
• self_play_league with max_snapshots ≥ 10 is the stable default.
• opponent_policy: average_strategy (using the running average-strategy network as the opponent) is a theoretically interesting alternative — it more closely mirrors the CFR proof — but has not been run at scale in this repo as of ad0be89.

References

• Brown, Lerer, Gross, Sandholm. Deep Counterfactual Regret Minimization. ICML 2019. (Section 4, convergence requirements for the strategy network.)
• Lanctot et al. Monte Carlo Sampling for Regret Minimization in Extensive Games. NeurIPS 2009. (External-sampling stationarity assumption.)