Threats to statistical conclusion validity
| Threat | Description |
| Low statistical power (see step 3) | The sample size is not adequate to detect an effect. |
| Violated assumptions of statistical tests (see step 6) | The data violate assumptions needed for the test, such as normality. |
| Fishing and error rates | Repeated tests of the same data (eg, multiple comparisons) increase chances of errors in conclusions. |
| Unreliability of measures | Error in measurement or instruments can artificially inflate or decrease apparent relationships among variables. |
| Restricted range | Statistics can be biased by limited outcome values (eg, high/low only) or floor or ceiling effects in which participants scores are clustered around high or low values. |
| Unreliability of treatment implementation | In experiments, unstandardised or inconsistent implementation affects conclusions about correlation. |
| Extraneous variance in an experiment | The setting of a study can introduce error. |
| Heterogeneity of units | As participants differ within conditions, standard deviation can increase and introduce error, making it harder to detect effects. |
| Inaccurate effect size estimation | Outliers or incorrect effect size calculations (eg, a continuous measure for a dichotomous dependent variable) can skew measures of effect. |