Table 4

Threats to statistical conclusion validity

Low statistical power (see step 3)The sample size is not adequate to detect an effect.
Violated assumptions of statistical tests (see step 6)The data violate assumptions needed for the test, such as normality.
Fishing and error ratesRepeated tests of the same data (eg, multiple comparisons) increase chances of errors in conclusions.
Unreliability of measuresError in measurement or instruments can artificially inflate or decrease apparent relationships among variables.
Restricted rangeStatistics can be biased by limited outcome values (eg, high/low only) or floor or ceiling effects in which participants scores are clustered around high or low values.
Unreliability of treatment implementationIn experiments, unstandardised or inconsistent implementation affects conclusions about correlation.
Extraneous variance in an experimentThe setting of a study can introduce error.
Heterogeneity of unitsAs participants differ within conditions, standard deviation can increase and introduce error, making it harder to detect effects.
Inaccurate effect size estimationOutliers or incorrect effect size calculations (eg, a continuous measure for a dichotomous dependent variable) can skew measures of effect.