Specification Artifacts
When your finding turns out to be a property of your config file rather than the world.
An empirical result is always a statement about a specification: this reward, this exploration schedule, this replay buffer, this preprocessing, this data window. The sentence people write down is almost always broader than that: "method X doesn't beat baseline Y here". The gap between those two statements is where a great deal of published work quietly lives.
The only way to close it is unglamorous. Take every defect and every questionable choice in the pipeline, fix each one alone, and re-run the whole campaign.
Why "alone" is doing the work
Bundling corrections is the natural instinct and it destroys the information you were trying to get. Fixes interact, sometimes violently. Two changes that each improve things in isolation can be worse together than either was by itself, because they're compensating for each other in ways nobody designed. If you only ever run baseline and everything fixed, you cannot tell which of those worlds you're in.
That's an argument for a factorial design over an ablation list, and for spending compute on the boring cells.
Underpowered decomposition lies confidently
A decomposition is a stack of comparisons, and each one inherits all the seed noise of the runs beneath it. Run it at a sample size chosen for convenience and you don't get a noisy answer. You get a clean-looking answer that points the wrong way, with no signal that anything is wrong.
I've watched the attribution in an analysis reverse between a small pilot and the full campaign. Not weaken. Reverse. The pilot's story was coherent, plausible and wrong, and nothing inside it flagged the problem.
Record the hypotheses that died
The mechanism you propose for an effect is a claim, and it is usually the first thing to be falsified, well before the effect itself. My own record on this project is roughly one for three: I've argued for mechanisms that the data then contradicted, sometimes reversing their sign entirely.
Writing the dead hypothesis down, with why it was wrong, costs a paragraph. Not writing it down costs somebody, often me six months later, the time to re-derive it from scratch.
The general shape
A result you can't decompose isn't a finding yet, it's an observation about one configuration. Isolate the changes, seed-match every comparison, state which family your p-values belong to, and be precise about which sentence your data actually supports. Most of the execution research I do is this, rather than modelling.