When you run an experiment in Shogun A/B Testing, you might notice that the numbers in your Shogun dashboard don't exactly match what you see in Shopify or Google Analytics 4 (GA4).

Some differences are expected because each platform tracks and attributes shopper activity differently. Shopify and GA4 can still provide useful points of comparison, but how closely the numbers should align depends on whether you're comparing the same group of shoppers.

In Shogun, the population reported for an experiment can be affected by:

Understanding these factors is important before deciding whether a difference between Shogun and another analytics platform is expected or needs further investigation.

Shogun experiment analytics are specific to shoppers participating in an experiment. That doesn't always match the population represented in all sessions reported by Shopify or GA4.

For some experiment types, the populations can be very similar. For others, shoppers only become eligible once they reach a particular page or experience, which means only part of your overall store traffic can participate. Other experiment settings can narrow that population further.

Because of this, the first question to ask when comparing analytics isn't "Why don't these numbers match?" It's "Are these reports measuring the same shoppers?"

Different experiment types can become eligible at different points in the shopper journey.

Experiments such as theme, price, product details, and checkout tests can be eligible broadly across the store experience. As a result, Shogun and Shopify will often observe broadly similar session populations for these tests, assuming no other experiment settings restrict participation. Shopify can therefore provide a useful point of comparison for these experiments.

Other experiment types, such as page, template, and URL redirect tests, depend on the shopper encountering a particular tested surface. A shopper might browse your store for some time before becoming eligible for one of these experiments, or they may never encounter the tested surface at all. This can make overall Shopify traffic a less direct comparison.

This is particularly important when using Shopify or GA4 reports filtered by landing page. A landing page tells you where a shopper's session started, but experiment participation can happen later. For example:

Google → Homepage → Collection → Experiment page → Checkout

The shopper's landing page is the homepage, but they can still become eligible for and participate in an experiment when they later visit the experiment page. A Shopify or GA4 report filtered to sessions that landed on the experiment page wouldn't include this shopper, even though Shogun may correctly report them as an experiment participant.

A landing-page report therefore doesn't necessarily represent everyone who encountered an experiment during their session.

This distinction is especially important for URL redirect experiments. In a URL redirect test, the shopper encounters the original URL. Shogun can then dispatch the experiment and, depending on their variant, redirect them to an alternate URL. For example:

Homepage → Original experiment URL → Redirected to Variant B URL → Checkout

The shopper participated in Variant B. However:

Because of this, comparing Variant B sessions in Shogun with sessions that landed on the Variant B URL in Shopify or GA4 isn't an equivalent comparison.

Shogun may report more participating sessions for a URL redirect experiment than a Shopify report filtered to sessions that landed on the original experiment URL. This doesn't necessarily mean Shogun is duplicating or overcounting sessions. Consider three shoppers:

All three can participate in the Shogun experiment. However, a Shopify report filtered to sessions whose landing page was the original experiment URL would only include Shopper 1. Shoppers 2 and 3 entered the experiment after their sessions had already started elsewhere.

In this situation, the Shopify landing-page report can still provide useful context, but it represents only a partial view of the population that can participate in the Shogun experiment.

A URL redirect test can either redirect an assigned shopper only on their first visit, or every time they reach the original URL.

When the redirect only happens on the first visit, both URLs stay reachable afterwards. A shopper assigned to Variant B can come back later through an abandoned-cart email, a saved link, an ad, or your site's own navigation, and land on the original URL without being redirected. For example:

Shogun credits this order to Variant B, because that's the variant the shopper was assigned to. A Shopify report by product or page would count it under the original. So a Shopify product or page report can show more sales for the original, and fewer for Variant B, than Shogun does, even when both are working correctly.

If you want each shopper to only ever see their assigned URL, set the test to redirect every time. If your Variant B URL is a separate product, you can also hide it from your collections and search, so shoppers only reach it through the test.

A URL redirect test only enters shoppers who reach the exact path of the original URL. Language prefixes, such as /de/, are handled for you.

Shopify can serve the same product at more than one path. For example, a product at /products/hoodie may also be reachable at /collections/sale/products/hoodie. A shopper who reaches the product through a different path sees the original page but isn't entered into the test. Their visit and any order won't appear in Shogun, but Shopify still counts the sale under the original product.

If your theme or apps link to products through collection paths, the original can look stronger in Shopify than in Shogun. Check how your store links to the original URL before comparing the two.

If you're running multiple Shogun experiments at the same time, this can also affect how many shoppers participate in each one. A shopper is only assigned to one experiment at a time, so being potentially eligible for an experiment doesn't guarantee the shopper will ultimately participate. How shoppers are allocated across concurrent experiments depends on your distribution method.

With Greedy distribution, experiments that are eligible broadly across the shopper journey, such as price, product, checkout, and theme tests, can take the shopper when they land on the store. That assignment then sticks. Experiments that only become eligible once the shopper reaches a particular surface, such as page, template, and URL redirect tests, may therefore receive fewer participants when other experiments are running. For example:

Shopper lands → assigned to an eligible price test → later visits a page with a URL redirect test

By the time the shopper reaches the URL redirect test, they're already participating in another experiment. So even though they visited the URL being tested, you shouldn't necessarily expect them to appear as a participant in that URL redirect experiment.

With Even distribution, shoppers are allocated across concurrent experiments according to their traffic allocation. However, being allocated to an experiment doesn't guarantee that the shopper will ultimately encounter the experience being tested. For example, a shopper could be allocated to a page experiment but never visit that page during their session, so they won't produce an exposure for that experiment.

This is another reason why overall Shopify session totals shouldn't automatically be expected to match the sessions or exposures reported for an individual Shogun experiment.

If you've configured an audience for an experiment, only shoppers who meet those audience conditions are eligible to participate. Comparing the resulting Shogun experiment sessions with all store sessions in Shopify wouldn't be an equivalent comparison, because Shopify's total includes shoppers who were never eligible for the experiment.

Before comparing Shogun with Shopify or GA4, always check whether an audience is configured and, where possible, make sure your external report represents a similar population.

Being allocated or assigned to an experiment doesn't always mean a shopper will actually encounter the experience being tested. This is most noticeable with experiments that depend on the shopper reaching a particular page or surface. For example:

Shopper allocated to Page Experiment → browses Homepage → Collection → leaves store

If the shopper never visits the tested page, they never encounter the experiment experience. This can become especially important when multiple experiments are running concurrently. When comparing analytics, consider not only which shoppers could be allocated to an experiment, but which shoppers could actually become exposed to the experience being tested.

It's important to know that consent to tracking isn't handled the same way everywhere. Depending on where a shopper is browsing from, privacy rules work in one of two ways:

Shogun doesn't make this decision itself. Shopify manages consent for your store through its Customer Privacy settings, and Shogun follows the consent signal Shopify provides for each shopper. Shopify's own analytics and your ad pixels follow the same signal.

What this means for your test results:

To make sure consenting shoppers are tracked, show a cookie banner in the regions that need one:

Shoppers who decline will still be missing from Shogun's numbers, which is expected.

An order appearing in Shopify doesn't necessarily mean it should also appear as an order attributed to a particular Shogun experiment. The shopper needs to have participated in that experiment, and the purchase needs to be attributable to their experiment session and variant. Some Shopify orders may come from shoppers who:

Tracking factors such as ad blockers, consent settings, site performance, and competing scripts can also create differences between analytics platforms. For these reasons, total Shopify orders shouldn't automatically be expected to equal the orders attributed to a specific Shogun experiment.

Conversion rate is the number of converting sessions ÷ the number of sessions included in the calculation. If Shogun and Shopify are measuring different groups of shoppers, their conversion rates can differ even when both systems are working correctly. For example, comparing:

isn't necessarily an apples-to-apples comparison. Before comparing conversion rates, first establish whether the underlying populations are comparable.

Shopify can be a valuable comparison source when the populations represented by both systems should be broadly similar. For example, a theme test that applies broadly across the storefront may include a large proportion of the same sessions Shopify observes, so Shopify's overall session data can be a strong point of comparison.

You still shouldn't necessarily expect an exact match. Differences can occur because of ad blockers, consent settings, session definitions, tracking behavior, and other technical factors.

For experiments whose participation depends on a shopper encountering a particular page or surface, Shopify may only provide a partial comparison, and concurrent experiments and audiences can further change which shoppers are eligible. The key question is therefore: Should these two reports represent approximately the same population?

Before comparing numbers between platforms, check the following:

Only after establishing these factors should you use the size of a discrepancy to determine whether something looks unexpected.

Some differences between Shogun, Shopify, and GA4 are expected, but there isn't a single percentage threshold that determines whether a discrepancy is normal. A difference can be caused by:

That doesn't mean you should dismiss every discrepancy as a difference between analytics platforms. If you've accounted for these factors and you're still seeing a consistent or unexplained difference between populations that should be similar, please contact Shogun Support. Our team can investigate whether your experiment is being dispatched, tracked, and attributed as expected, and whether the difference needs further investigation.

Shogun metrics update about every minute. This includes sessions, clicks, and orders. The confidence level and winning variant update about every 15 minutes.

Orders can sometimes feel like they take longer to appear because they need to be tied back to the shopper's experiment session and variant before appearing in your dashboard. If a shopper completes a purchase that can be attributed to an experiment, you can generally expect it to appear within a few minutes.