Scientists Are Misusing Statistics When Analyzing Ecological Simulation Models

We identified a troubling trend in how researchers analyze computer simulation models. When examining a simulation study of rocky reef ecosystems, We found that researchers had run 24,000 computer simulations to test whether different predator behaviors affected their model results, producing a p-value of 10^-15. However, the original researchers admitted this result was meaningless because of their huge sample size and ignored their own statistical test.

White and colleagues wanted to understand why so many ecologists were using statistical significance tests on computer simulation results when these tests seemed inappropriate. They examined the ecological literature and found multiple examples where researchers were applying t-tests and ANOVAs to simulation outputs, revealing a fundamental misunderstanding of statistical methodology.

The core problem is that when running computer simulations, researchers can control exactly how many replications to perform. Statistical power becomes meaningless when replication is essentially unlimited. More importantly, different model parameters will produce different results by design - that's why researchers test them. The null hypothesis of 'no difference' is known to be false from the beginning.

"We identified a troubling trend in how researchers analyze computer simulation models."

In the rocky reef study examined, 24,000 runs produced an F-statistic of 67.5 with a p-value of 10^-15. But this statistical significance is meaningless because researchers could achieve any desired p-value simply by running more simulations.

This matters because simulation models are increasingly central to ecological research and management decisions. When researchers focus on p-values instead of effect sizes, they might conclude that small, biologically meaningless differences are important simply because they're statistically significant with thousands of simulation runs. Conversely, they might dismiss large, ecologically important effects that don't reach arbitrary significance thresholds.

The solution isn't to abandon quantitative analysis of models, but to focus on what really matters: the magnitude of differences and their ecological significance. Instead of asking 'is this difference statistically significant?' researchers should ask 'how big is this difference and does it matter ecologically?' This requires defining beforehand what magnitude of change would constitute meaningful ecological effects, similar to focusing on 'biological significance' rather than statistical significance.

Citation

White, J. Wilson; Rassweiler, Andrew; Samhouri, Jameal F.; Stier, Adrian C.; White, Crow (2014). Ecologists should not use statistical significance tests to interpret simulation model results. Oikos.

Read the full paper

This paper is Open Access.