What is Sample Ratio Mismatch?
Sample Ratio Mismatch (SRM) happens when the number of visitors landing in each variant of your test doesn't match the traffic split you configured — for example, you set a 50/50 split but end up with something like 53/47 in sessions or pageviews. A small amount of variance is normal, but a persistent or growing mismatch can signal that something is skewing your results, and it's worth understanding before you trust a "winner."
Why does my confidence level swing early in a test?
Early in a test, small differences in a handful of conversions can produce a misleadingly high confidence score. Shogun requires a minimum of 20 successful conversions per variant before it begins reporting confidence — this reduces, but doesn't eliminate, the chance of an early result looking more decisive than it is. As more data comes in, that confidence can and often does move, sometimes significantly.
What to do if you see SRM or a large confidence swing
Reach out to our support team in-app with the test name — we can pull the result history and take a look.