Plan Availability
Experiments is available for customers on Free, Growth, and Enterprise plans. Usage is measured in Monthly Experiment Users (MEU): the number of distinct users exposed to an experiment during a calendar month. A user exposed to several experiments in one month counts as one MEU, up to 30 experiments — past that, see how MEU is calculated. MEU and experiment counts apply across your whole organization unless noted. See our pricing page for more details.
Every plan includes Feature Flags alongside Experiments, with limits on active flags and cohort targeting that vary by plan. See Feature Flags plan availability for details.
Reaching Your MEU Allowance
What happens when you reach your allowance depends on your plan. On a Free plan, Experiments pauses: you can’t create, edit, or view experiments until your usage resets at the start of the next calendar month, or you upgrade. Your feature flags keep serving and routing traffic, so what your users see doesn’t change — but any experiment you’re running stops producing new results for the rest of that month. On a Growth plan, Experiments keeps working and usage beyond your included or purchased MEU is billed at an overage rate. Purchasing a larger MEU tier raises the amount included before overage applies. Track your usage under Organization Settings > Plan Details & Billing.Why Experiment?
Experimentation helps you make data-driven product decisions by measuring the real impact of changes on user behavior. Mixpanel is an ideal place to run experiments because all your product analytics data is already here, providing you with comprehensive insights into how changes affect your users’ journey.Prerequisites
Before getting started with experiments:- Exposure Event Tracking: Implement your experimentation events
- Baseline Metrics: Ensure that Mixpanel is already tracking your key metrics
Overview & Workflow
Experiment Process
Plan → Setup & Launch → Monitor → Interpret Results → Make Decisions- Plan: Define hypothesis, success metrics, and test parameters
- Setup & Launch: Configure experiment settings and begin exposure
- Monitor: Track experiment progress and data collection
- Interpret Results: Analyze statistical significance and lift
- Make Decisions: Choose whether to ship, iterate, or abandon changes
Plan Your Experiment
Before creating an experiment report, ensure you have:- A clear hypothesis about what change will improve which metric
- Defined primary success metrics (and secondary/guardrail metrics)
- Estimated sample size and test duration requirements
- Proper exposure event tracking implemented
Setup & Launch Your Experiment
Step 1: Select an Experiment
Click ‘New Experiment’ from the Experiment report menu and select your experiment. Both experiments run through Mixpanel Feature Flags and experiments tracked via exposure events are detected and populated in the dropdown. To keep the dropdown focused on experiments you’re likely to care about, we hide Mixpanel Feature Flag experiments once their flag is archived, and we hide exposure-event experiments that haven’t received an exposure event in the last 30 days. To analyze an exposure-event experiment beyond that window, you can hard-code the experiment name by typing$experiment:Experiment name as a filter or breakdown property, where Experiment name is the value of whichever property you configured as the experiment name on your exposure events.
Two kinds of experiments can be analyzed in the experiment report: experiments run through Mixpanel Feature Flags with the Experiment type, and experiments tracked via exposure events (i.e.,
$experiment_started). Read more on how to track experiments here.Step 2: Choose the ‘Control’ Variant
Select the ‘Variant’ that represents your control. All your other variant(s) will be compared to the control, i.e, how much better they perform compared to the control variant.Step 3: Choose Success Metrics
Choose the primary metrics of success for the experiment. You can choose from either saved Mixpanel metrics or create a new metric leveraging the query panel. You can also add secondary metrics and guardrail metrics as required.Step 4: Select the Test Duration
Enter either the sample size (the number of users to be exposed to the experiment) or the minimum number of days you want the experiment to run. This will determine the test duration. Once the sample size or days are complete, you can conclusively read the experiment results and make a decision.Step 5: Confirm other Default Configurations
Mixpanel has set default automatic configurations, seen below. If required, please modify them as needed for the experiment- Experiment Model type: Sequential. You can also choose Frequentist or Bayesian; see Experiment Model Types for how they differ.
- Confidence Threshold: 95%. On a Bayesian experiment this setting is labeled Decision Threshold and sets the minimum Chance to Beat Control required to highlight a variant as a winner.
- Experiment Start Date: Date of the first user exposed to the experiment
The Confidence Threshold and the Bayesian Decision Threshold are the same underlying setting: if you switch an experiment’s model type, your chosen number carries over and is reinterpreted for the new model.
Experiment JSON Parameters
Each variant can optionally carry a value in addition to its name — either a string or a JSON object. The variant name is what your code switches on to render the right experience. The value is configuration data the SDK returns alongside the assignment, so you can remotely control parameters of your app or site (copy, colors, thresholds, layout configs, pricing, etc.) without a code deploy. For example, a variant namedtreatment might carry a JSON object value like:
Changing Settings Mid-Experiment
Mixpanel analyzes your experiment on the fly, so you can adjust settings at any time if needed. If you notice a configuration mistake - like selecting Frequentist when you meant Sequential or Bayesian - feel free to correct it, even if the experiment has been running for weeks. The one thing to avoid is changing settings because the current results aren’t what you hoped for. If you adjust the confidence level, metrics, or duration in search of a significant result, you’re more likely to find something that looks meaningful but is actually noise. When you roll out that change, it may not deliver the improvement you expected. If you want to test a different configuration, restart the experiment instead. This resets the data and ensures your analysis starts fresh.Reopening a Decided Experiment
Once you’ve made a decision on an experiment, you can reopen it in place instead of duplicating it. On the decided banner, use the Change Decision dropdown:- Edit Decision: Revise the decision you recorded (for example, change which variant you shipped or update the notes) while keeping the experiment decided.
- Undo Decision: Revert the experiment to its concluded, pre-decision state so the decision can be revisited.
For Mixpanel feature-flag experiments, undoing a decision does not automatically re-enable the feature flag. If you need the flag serving its variants again, re-enable it separately from the Feature Flags page.
Monitor Your Experiment
Once your experiment is running, you can track its progress in the Experiments dashboard. Monitor key indicators:- Sample Size Progress: Track how many users have been exposed
- Data Quality: Ensure exposure events are being tracked correctly
- Guardrail Metrics: Watch for any negative impacts on important metrics
- External Factors: Note any external events that might affect results
Interpret Your Results
The Experiments report identifies meaningful differences between the Control and Variant groups. Every metric has two key attributes, and which ones you see depends on your experiment model type. Frequentist and Sequential models:- p-value: this shows if the variants’ delta impact vs the control is statistically significant
- lift: the variants’ delta impact on the metric vs control
- Chance to Beat Control: the probability that the variant’s true metric value is better than the control’s
- lift: the variants’ delta impact on the metric vs control
- Positive differences, where the variant value is in the target direction compared to the control, are highlighted in green
- Negative differences, where the variant value is opposite the target direction compared to the control, are highlighted in red
- Statistically insignificant results remain gray
Hover over a value in a metric row to see more detail behind the numbers and to view the metric in Insights.
How do you read statistical significance?
Statistical significance (p-value) helps you determine whether your experiment results are likely to hold true for the full rollout, giving you confidence in your decisions. On Bayesian experiments, the equivalent signal is the Chance to Beat Control, described under Bayesian below.
How Mixpanel Calculates Results
How Mixpanel evaluates your experiment depends on two things: the experiment model type you selected, and the type of each metric. The Frequentist and Sequential models compute p-values and confidence intervals; the Bayesian model computes a probability that each variant beats control.Frequentist and Sequential
Metric Types and Their Distributions: Mixpanel categorizes metrics into four types, each using different statistical distributions:-
Total Events Metrics: Use normal distribution of per-user event counts
- Examples: Total purchases, total page views
- Variance is calculated directly from the distribution of per-user event counts
-
Total Sessions Metrics: Use Poisson distribution
- Variance equals the mean (characteristic of Poisson distributions)
-
Rate Metrics (Conversion rates, Retention rates): Use Bernoulli distribution
- Examples: Signup conversion rate, checkout completion rate, 7-day retention
- Models binary outcomes (did/didn’t convert) across your user base
-
Value Metrics (Averages, Sums of properties): Use normal distribution approximation
- Examples: Average order value, total revenue, average session duration
- Calculates variance using sample statistics
- Calculate group rates for control and treatment
- Estimate variance using the appropriate distribution
- Compute standard error from variance and sample size
- Calculate Z-score measuring how many standard errors apart the groups are
- Derive p-value from Z-score using normal distribution
- Uses continuous monitoring with adjusted significance thresholds with mSPRT method
- Allows for early stopping when significance is reached
- More conservative calculations to account for multiple testing
- Uses traditional hypothesis testing with fixed sample sizes
- Formula: Max Significance Level (p-value) = [1-CI]/2 where CI = Confidence Interval
- p ≤ 0.025: results are statistically significant for this metric, i.e, you can be 95% confident in the lift seen if the change is rolled out to all users.
- p > 0.025: results are not statistically significant for this metric, i.e, you cannot be very confident in the results if the change is rolled out broadly.
Example: E-commerce Checkout Experiment
To illustrate how these calculations work in practice, let’s walk through a concrete example. Scenario: Testing a new checkout UI on an e-commerce site with 20 users (10 control, 10 treatment). Results:- Control group: 5 users converted (50% conversion rate), average cart size $60
- Treatment group: 6 users converted (60% conversion rate), average cart size $67
- Group rates: Control = 0.5, Treatment = 0.6
- Variance calculation: Control = 0.5 × (1-0.5) = 0.25, Treatment = 0.6 × (1-0.6) = 0.24
- Standard error: Combined SE = √((0.25/10) + (0.24/10)) = 0.221
- Z-score: (0.6 - 0.5) / 0.221 = 0.45
- P-value: ~0.65 (not statistically significant)
- Group means: Control = $60, Treatment = $67
- Variance calculation: Uses the sample variance of cart values in each group
- Standard error: Calculated from combined variance and sample sizes
- Z-score and p-value: Computed using the same Z-test framework
Bayesian
The Bayesian model answers a different question than the Frequentist and Sequential models. Instead of asking “how surprising would this data be if the treatment had no effect?” (a p-value), it estimates a distribution of plausible values for each variant’s metric and reports how likely it is that the variant is actually better than control. That probability is surfaced as the Chance to Beat Control column. Priors: Bayesian analysis combines the data you observed with a prior, an assumption about what the metric’s value is likely to be before looking at the results. Mixpanel uses weakly informative priors that are automatically scaled to your experiment’s own data. The prior is centered on the pooled metric value across control and the variant being compared, weighted by each one’s exposures; its spread comes from the pooled variance of that same data; and its strength is a single pseudo-observation. That last point is what makes it weakly informative. The prior is strong enough to keep results stable when a variant has very few users, and negligible once real data accumulates: at the thousands of exposures a typical experiment reaches, it has effectively no influence on the outcome. You don’t have to supply a prior, and the prior won’t push results toward a value you didn’t choose. It is not a fixed uniform prior, and it is not derived from control alone. Models by metric type:Total and session metrics are modeled as normally distributed per-user counts, on the same per-user scale used for averages, rather than as Poisson counts. This differs from the Frequentist path, which applies a Poisson variance on the totals scale.
- 97%: above the threshold, highlighted as a winner.
- 60%: the variant is more likely better than not, but well short of the threshold. You don’t yet have enough data to act.
- 2%: below the 5% lower bound, highlighted as a loser. The variant is very likely worse than control.
- The interval is a highest-density interval, the narrowest range containing 95% of the plausible values. Many Bayesian A/B tools instead report an equal-tailed interval, which trims 2.5% from each end and can be wider.
- It is computed on relative lift (percentage difference from control), not on the absolute difference between the two groups.
- Because of this, the interval is generally asymmetric around the reported lift, rather than “lift ± some margin.”
How do you read lift?
Lift is always calculated using exposures (users assigned to a variant) as the denominator — not the denominator you may see in other tabs like Totals or Chart. For funnel metrics, this means the lift denominator differs from the funnel conversion rate denominator. See the FAQ “Why does the lift in Results disagree with the conversion rate in Totals or Chart?” for a worked example.
$experiment_started event for that variant):
Total Events Metrics:
- Numerator: Total event count in the variant
- Denominator: Number of users exposed to the variant
- Group Rate: Total count ÷ Number of users exposed
- Variance: Calculated directly from the per-user event counts, treated as a continuous variable
- Example: If the treatment group has 150 total purchases from 100 exposed users, the group rate = 1.5 purchases per user
- Numerator: Total session count in the variant
- Denominator: Number of users exposed to the variant
- Group Rate: Total count ÷ Number of users exposed
- Variance: Equal to the mean (Poisson distribution property)
- Numerator: Number of users who converted (or were retained) in the variant
- Denominator: Number of users exposed to the variant
- Group Rate: Conversions ÷ Exposures
- Variance: Calculated using Bernoulli distribution: p × (1-p)
- Example: If 25 out of 100 exposed users convert, group rate = 0.25 (25%). Note that this denominator is exposures — not funnel entrants. This is different from the funnel conversion rate you see in the Totals or Chart tabs, which uses funnel entrants as the denominator.
- Numerator: Sum of property values across all users in the variant
- Denominator: Number of users exposed to the variant
- Group Rate: Sum of property values ÷ Number of users exposed
- Variance: Calculated from the distribution of individual property values
- Example: If the treatment group spent $5,000 total from 100 users, the group rate = $50 average per exposed user
Metrics with a Group Identifier: When you change a metric’s group identifier from Users to a group key (e.g., Company), exposures for that metric are counted at the group level. The number of exposed groups is then used for all of that metric’s calculations — normalizing group rates, computing variance, and determining statistical significance. For example, if a metric is set to aggregate by Company and 200 companies have been exposed to the treatment, the group rate is calculated as the total metric value divided by those 200 exposed companies, not by the number of individual users.
Revenue per User = Total Revenue ÷ Unique Users, Mixpanel uses propagation of uncertainty to estimate variance. This combines the variances of the component metrics (Total Revenue and Unique Users) to calculate the overall metric’s statistical significance. The system assumes metrics in formulas are uncorrelated for these calculations.
Segment your results
Segmentation lets you break your experiment results down by a property (such as country, platform, plan type, or device) to see how the treatment effect varies across groups of users. This is useful for spotting cases where a change helps one segment but hurts another, which an aggregate result can hide. How to use it: Use the breakdown picker above the results table to choose a property to segment by. Mixpanel then computes a separate result (lift, p-value, and confidence) for each value of that property and lays them out in a table, so you can compare the variants within every segment. Each segment row reads exactly like the overall results: green for a significant positive difference, red for a significant negative one, and gray for differences that aren’t statistically significant. This breakdown happens natively inside the Experiment report and is distinct from clicking Analyze on a metric, which opens a separate Insights report (see Diagnosing experiments further).The risk of post-hoc segmentation
Breaking results down after an experiment has run is a form of post-hoc analysis, and it carries a real statistical risk. Every additional segment you inspect is another chance to see a “significant” result that is actually just noise. This is the same multiple-comparisons problem described under Bonferroni and Benjamini-Hochberg, but it’s especially easy to fall into when you slice the same results many different ways looking for something that “worked.” This practice is often called p-hacking: if you test enough segments, you will eventually find one with a low p-value purely by chance. Reporting that segment as a win, and constructing a story around why that particular group responded, can lead you to ship a change based on a false positive.Why Mixpanel corrects segmented results by default
Because segmenting multiplies the number of comparisons being made, Mixpanel corrects segmented results by default. When you don’t already have a Multiple Testing Correction enabled, adding a breakdown turns on the Benjamini-Hochberg correction across its segments. If you already have Bonferroni enabled, Mixpanel keeps it rather than switching to Benjamini-Hochberg, since Bonferroni is the stricter check. Either way the correction is recommended but optional, and you can turn it off from the results if you want to see the uncorrected numbers. Benjamini-Hochberg controls the false discovery rate, the share of your flagged winners that are actually false positives. It does this by making the significance threshold stricter as you add more segments, so a segment still highlighted as a winner after correction is more trustworthy. Segments that only looked significant because you tested many of them are filtered out. See the Benjamini-Hochberg Correction section for a worked example of how the procedure ranks and thresholds p-values. This correction operates on the values in your breakdown, which is a different axis from the Multiple Testing Correction you apply across metrics and variants. That means adding more breakdown values doesn’t inflate your false positive rate.When do we say the Experiment is ready to review?
Once the ‘Test Duration’ setup during configuration is complete, we show a banner that says “Experiment is ready to review”. Test Duration can be either of two options:- Sample size to be exposed
- Number of days you’d like to run the experiment
Diagnosing experiments further in regular Mixpanel reports
Click ‘Analyze’ on a metric to dive deeper into the results. This will open a normal Mixpanel insights report for the time range being analyzed with the experiment breakdown applied. This allows you to view users, view replays, or apply additional breakdowns to further analyze the results. You can also add the experiment breakdowns and filters directly in a report via the Experiments tab in the query builder. This lets you do on-the-fly analysis with the experiment groups. Under the hood, the experiment breakdown and filter work the same as the Experiment report.Multiple Environments
Multiple Environments lets you treat separate Mixpanel projects as development, staging, and production environments for the same experiment or feature flag. Build and iterate on an experiment (and its underlying flag) in one project, then push it to another project when you’re ready — without recreating it by hand. Linked experiments and flags stay connected so you can continue pushing updates over time.Other Environments Selector
The “Other Environments” selector appears under the Configuration tab of an experiment, inside the More section. This panel lets you manage an experiment’s links to copies of itself in other projects.Pushing changes
Pushing changes copies the full configuration of the source (original) experiment to the linked destination experiment in the other project. Any changes that exist only on the destination experiment will be overwritten.Other Environments Menu
This menu lists the experiments in other projects that are linked to the current experiment. From here, you can navigate to a linked experiment, push the source experiment’s current configuration to it, or remove the connection.Duplicate To Project
To duplicate an experiment to another project, open the Other Environments menu and select Duplicate To Project. In the dialog, choose the destination project. This flow is for duplicating across projects only — to duplicate within the same project, use the existing Duplicate option within the experiment’s overflow (...) menu in the top-right of the experiment page.
Once duplicated, the new experiment is linked to the original and can be managed from the “Other Environments” menu.
Feature Flags
If an experiment is linked to a Mixpanel Feature Flag, the underlying flag cannot be duplicated on its own from the Feature Flags page — use the experiment’s Duplicate flow to copy the experiment and flag together. If your experiment uses a third-party flag instead, you can duplicate the experiment directly; no flag duplication step is involved. When duplicating an experiment that is linked to a Mixpanel Flag to another project, you will be prompted to confirm whether you also want to duplicate the linked flag. If you do, the new experiment will be linked to the new flag, and the new flag will also appear in the “Other Environments” menu of the original flag. When pushing changes, if both the source and destination experiments are linked to flags that are also linked (i.e., you chose to duplicate the flag as well), you can choose to push any changes from the source flag to the destination project’s flag at the same time.Permissions
To duplicate an experiment to another project, you need permission to create and edit experiments in both the source and destination projects. The required permission level in the destination project is the same as what’s needed to manually create a new experiment there — governed by that project’s roles and custom roles.What to expect after duplicating
When an experiment is duplicated to another project, time-based settings (such as the experiment period and day counts) reset to the day the duplicate was created — the same behavior as duplicating within a project. If your experiment targets a cohort, Mixpanel assumes the event properties referenced by that cohort exist in the destination project. It does not validate property availability at copy time, so verify that your destination project collects the same events and properties before running the experiment there.Removing a connection
Removing an inter-project connection from the “Other Environments” menu unlinks the two experiments — it does not delete the experiment in the destination project. The copied experiment remains in the destination project and continues to function independently; it simply loses its link to the original.Flag limits across projects: The same feature flag key used across multiple projects counts once against your plan’s active flag limit.