In this article:
What is statistical significance?
Statistical significance is a way to determine if the differences or patterns we observe in data are likely to be real or just due to chance.
We compare what we've observed to what we'd expect if there was no actual difference. If what we've seen is very unlikely to occur by chance, we say it's statistically significant, indicating that there's likely something real going on.
Why is 99% our default significance threshold?
Relative Insight sets a 99th percentile significance threshold (p < 0.01) as the default because it helps to avoid making mistakes where we think something is significant when it's not.
By being more cautious with our threshold, it makes our conclusions stronger and more reliable. The minimum accepted significance threshold for academic, pharmaceutical, and other studies is the 95th percentile (p < 0.05). Adding an extra level of rigor means that the differences you see are far less likely to happen by chance.
What other significance thresholds are available?
Industry and academic standards recognize four general levels of significance thresholds:
Medium: 95th percentile (p < 0.05)
Default: 99th percentile (p < 0.01)
High: 99.9th percentile (p < 0.001)
Very High: 99.99th percentile (p < 0.0001)
While the 99th percentile works for most cases, there are situations where other levels are more appropriate:
95th percentile (p < 0.05): Common in situations with small data sizes, such as medical trials, where it's important to detect even small effects.
99.99th percentile (p < 0.0001): Useful when there are a large number of results and you want to focus only on the very strongest results, such as in high-stakes decision-making or when resources are limited.
In addition, Relative Insight allows you to turn off the significance threshold (less than 95%) altogether. While this is not generally recommended, you may want to consider this option if your comparison reveals no differences.
Adjusting the significance threshold in your analysis
To change the significance threshold in your analysis, follow these steps:
Navigate to the Explore tab of your analysis.
From the top menu, select the Threshold icon
Using the slider, adjust your significance accordingly. Each time, you'll see a new set of results
Remember, choosing the right significance threshold depends on the specific context and goals of your analysis.