Seed Variance in Deep RL

Train a deep RL agent twice, changing nothing but the random seed, and you get two different policies with two different performances. Everyone knows this. Almost nobody's evaluation protocol reflects it.

Report one run and you have reported a sample. Report the best of five and you have reported a sample and a selection bias. The only honest version is to retrain the whole pipeline many times and describe the distribution you get back, which is what the execution study does, at 100 runs per configuration.

Two things called "risk" that are not the same thing

The word collapses two genuinely different quantities, and the conflation is everywhere in the literature:

Across-seed dispersion. The standard deviation of the per-run mean cost, across the 100 runs. It answers: if I train this architecture once, how far from typical am I likely to land? This is a property of the training procedure.

Within-policy tail risk. CVaR-95 computed inside a single run, over its evaluation episodes, then averaged. It answers: given the policy I ended up with, how bad can one trade be? This is a property of the policy.

They are different axes and they do not have to move together. In our results they moved in opposite directions: the configuration with the tightest across-seed dispersion had the worst per-run tails. A paper that reports only "risk" can accidentally claim credit for one while quietly worsening the other.

Pairing is what makes it inference

Sharing seeds across configurations turns a pile of separate summary statistics into paired comparisons: the same seed, the same evaluation episodes, one architectural difference. That's what licenses paired bootstraps, permutation tests, and McNemar, and it costs nothing but discipline in how you index your runs.

The failure mode this catches

Some seeds don't produce a mediocre policy. They produce a degenerate one. In our execution setup a meaningful fraction of baseline runs converged on never trading at all, taking the forced terminal liquidation every time. Averaged into a mean, that's an unremarkable smear. Counted as an event rate, it's the most interesting thing in the distribution.

Which raises the question of what you count. A run-level collapse rate and an episode-level one can tell different stories about the same data, and if your architecture aggregates several policies under the hood, the level you choose to count at is a modelling decision rather than a reporting detail. Pick it before you look.

Underpowering hurts here in a specific way too: a decomposition run at a convenient sample size can give a clean, coherent attribution that reverses outright at full power.