set.seed(123)
x <- t(replicate(1e6, {
theta <- runif(1, 0, 1)
rbinom(2, 1, theta)
}))
p_1_1 <- mean((x[, 1] == 1) & (x[, 2] == 1))
p_1_1
#> [1] 0.332823Earlier this year, I worked on our cumulative impact feature; an implementation of the hierarchical normal normal model to enable customers to estimate the cumulative impact of past experiments while adjusting for the Winner’s Curse.
In my opinion, the largest difficulty with using this feature is going to be defining an exchangeable set of experiments, and so I thought I would collect my thoughts (for all to see) on exchangeability in AB tests so that I can be prepared to answer inevitable questions on the topic.
What is Exchangeability?
A sequence of random variables \(\theta_1, \cdots, \theta_K\) is said to be exchangeable if their joint distribution, \(p\), is invariant to permutations of the indices:
\[ p(\theta_1, \cdots, \theta_K)= p(\theta_{\pi(1)}, \cdots, \theta_{\pi(K)}). \]
Practically, exchangeability means that if I shuffled the treatment effects among the experiments, you would have no reason, a priori, to prefer one assignment of effects to experiments over another. In other words, knowing the identity of experiments tells you nothing about which treatment effect it should have.
It may be easier to see what exchangeability is not. Suppose \(K=5\), and \(\theta_k\) is the true relative lift in revenue per user from experiment \(k\). Further suppose the treatment effects are ordered from smallest to largest, so that
\[ \theta_1 < \theta_2 < \theta_3 < \theta_4 < \theta_5. \]
Now suppose I assign those effects to experiments as follows:
| Experiment | Treatment Effect |
|---|---|
| Pricing Experiment | \(\theta_1\) |
| Checkout Experiment | \(\theta_2\) |
| Site Redesign | \(\theta_3\) |
| Copy Change Experiment | \(\theta_4\) |
| Button Color Experiment | \(\theta_5\) |
If it is surprising to you that a button-color or copy-change experiment in truth produces a larger effect on revenue than a pricing or checkout experiment, then the experiment labels are carrying information about the plausible sizes of the treatment effects. In that case, treating the effects as exchangeable is probably not defensible.
Put differently, exchangeability requires that, before seeing the experimental results, you would be just as comfortable with the assignment above as with one in which \(\theta_5\) belonged to the pricing experiment and \(\theta_1\) belonged to the button-color experiment.
What this example demonstrates is that exchangeability is more than a formal invariance to permutations of indices. It represents a genuine a priori indifference about how treatment effects are assigned to experiments. This is not to say that you know nothing about the experiments. Rather, it says that whatever you do know about them does not give you reason to believe that one experiment should systematically receive a larger or smaller treatment effect than another. Equivalently, if you were to hypothetically reshuffle the treatment effects among the experiments, each reassignment would remain just as plausible under your prior model as the one in front of you.
Gelman and co-authors describe exchangeability as follows in Bayesian Data Analysis 3ed:
If no information – other than the data \(y\) – is available to distinguish any of the \(\theta_j\)’s from any of the others, and no ordering or grouping of the parameters can be made, one must assume symmetry among the parameters in their prior distribution.
The symmetry point is, in a way, referring to the definition of exchangeability in terms of permutations (group theorists will know). What I find useful about this phrasing is that it hands us two concrete things to look for rather than an abstract invariance to check. Can the experiments be put in a meaningful order? Can they be sorted into groups? Either one is a way of saying the labels are informative, and either one is enough to make us think twice. That gives us a useful check we can use to determine how justifiable exchangeability may be. If you find that you can group experiment results, exchangeability may not be justifiable – but more on that later.
It may be useful to state here that exchangeability is different from being independent and identically distributed (IID). In fact,
\[\mbox{IID} \Rightarrow \mbox{Exchangeability} \>,\] however
\[\mbox{Exchangeability} \nRightarrow \mbox{IID}\]
For an example of a sequence of exchangeable but non IID random variables, see the section below.
Example: \(\mbox{Exchangeability} \nRightarrow \mbox{IID}\)
Let \(\Theta \sim \mbox{Uniform}(0, 1)\) and let \(X_1, X_2 \mid \Theta \overset{\mbox{IID}}{\sim} \mbox{Bernoulli}(\Theta)\).
Conditional on \(\Theta\), the \(X_1, X_2\) are IID by construction. However, if we look only at their joint distribution having integrated out \(\Theta\) (that is \(X_1, X_2\) and not \(X_1, X_2 \mid \Theta\)) then we can show they are not IID.
It is straightforward to show \(P(X_1=1)=P(X_2=1)=E[\Theta]=\frac{1}{2}\): Let \(f(\Theta) = 1\) be the probability density of \(\Theta\). Therefore
\[ \begin{align}P(X_j=1) = E_\Theta[P(X_j=1 \mid \Theta)] &= \int_{0}^1 P(X_j=1\mid \Theta=u)f(u) \, du \\ &= \int_0^1 u \, du \\ &= \dfrac{1}{2} \end{align} \]
If \(X_1, X_2\) were independent, then \(P(X_1=1, X_2=1) = P(X_1=1)P(X_2=1) = \frac{1}{4}\). Alas, this is false and quite easy to simulate in R
In truth
\[\begin{align}P(X_1=1, X_2=1) &= \int_0^1 u^2 f(u) \, du \\ &= \dfrac{1}{3}\end{align}\]
Which is pretty close to our simulated estimate. So we’ve shown these random variables are not IID, but are they exchangeable? Their joint mass function is
\[ p(X_1=x_1, X_2=x_2) = \int_0^1 u^{x_1 + x_2} (1-u)^{2 - (x_1 + x_2)} f(u) \,du \>. \] The mass function relies only on the sum \(x_1 + x_2\), and since sums commute the joint mass function is invariant to permutations of the indices, hence these are exchangeable.
Why Is Exchangeability Important for Cumulative Impact?
To begin, let’s write down the model for cumulative impact. Let \(\theta\) be a true treatment effect in an AB test, \(\hat \theta\) be an estimate of \(\theta\), and let \(s\) be the standard error for said estimate. In AB testing, the distribution of \(\hat \theta \mid \theta\) is justified via the Central Limit Theorem, and it is typical to assume \(s\) is measured with enough precision to be assumed known. We use empirical Bayes to fit
\[\begin{align} \hat \theta_k \mid \theta_k &\sim \mbox{Normal}(\theta_k, s_k^2) \\ \theta_k \mid \mu, \tau &\sim \mbox{Normal}(\mu, \tau^2) \\ \end{align}\]Conditional on \(\mu\) and \(\tau\), the treatment effects \(\theta_1,\cdots,\theta_K\) are IID. When \(\mu\) and \(\tau\) are unknown, the \(\theta_k\) are no longer independent after we account for our uncertainty about those shared parameters. They do, however, remain exchangeable. Permuting the experiment indices leaves their joint distribution unchanged.
Cumulative impact is intended to adjust for the bias one would incur when naively adding up treatment effects from winning experiments – i.e. the bias from the Winner’s Curse. The model powering cumulative impact achieves this through partial pooling: the estimate for each \(\theta_k\) is pulled toward the shared mean \(\mu\), allowing information from the collection of experiments to inform inference about the true effect of each individual experiment.
The amount of shrinkage depends on both \(s_k\) and \(\tau\). Experiments with noisier estimates are generally shrunk more heavily toward \(\mu\), while experiments estimated with greater precision are allowed to remain closer to their observed \(\hat\theta_k\).
Exchangeability is what licenses this sharing of information across experiments. If experiment identity itself tells us something about the plausible size of \(\theta_k\), then treating every experiment symmetrically under the same \(\mu\) and \(\tau\) may be inappropriate. Exchangeability is therefore a vital assumption of the model underlying the feature.
However, we don’t ever get to see \(\theta\), only their estimates. In what sense can we assess the plausibility of the assumption of exchangeability if we only have access to the noisy \(\hat \theta\) and not the \(\theta\)? This gives us our first takeaway: exchangeability is less an assumption that can be validated from the data and more a judgement about whether experiment identity contains information about the treatment effect, after accounting for the information we believe matters.
Making that judgement is partly an art. The remainder of this post discusses several common situations in which exchangeability may be more or less defensible, moving from stronger to weaker forms of the assumption.
What Cost Do We Pay For Non-Exchangeable Experiments?
Mathematically, a collection of random variables is either exchangeable or it is not. What is a matter of degree is how defensible we believe the exchangeability assumption to be.
In practice, we may be willing to treat a collection of treatment effects as exchangeable even when there is a little nagging voice in our heads raising arguments to the contrary. In such cases, what costs do we pay if that judgement is wrong?
The main cost is bad pooling.
The empirical Bayes model allows experiments to provide information about one another by shrinking their estimates toward a shared \(\mu\). If the experiments are not exchangeable, that shared mean may be an inappropriate shrinkage target. Some experiments may be shrunk too much, others not enough.
Similarly, \(\tau\) is intended to describe heterogeneity in treatment effects among experiments we are willing to treat symmetrically. If systematic differences between experiments are ignored, then \(\tau\) may end up absorbing structure that should have been modeled explicitly. Both problems ultimately affect the cumulative impact estimate.
How fast and loose we are willing to play with the exchangeability assumption can therefore depend partly on how important the resulting cumulative impact estimate is. If we need a ballpark number, we may tolerate a weaker argument for exchangeability. If we need a highly defensible estimate, then the composition of the experiment set becomes much more important (personally, I might even suggest a different method altogether, such as a holdout).
Small Note on Terminology
Going forward, I’m going to use the term “Treatment Class” to refer to the comparability of two treatments in terms of potential scope, effectiveness, and mechanism. For example, a copy change and a site re-design are in different treatment classes – different scope, effectiveness, and mechanism with which to modulate some metric.
Situation 1: Same Team, Same Metric, Same Treatment Class, Same Time
Consider a (perhaps extreme) scenario in which a single team runs several experiments in quick succession (perhaps on the scale of weeks), on the same surface, targeting the same metric, in which treatments are of the same treatment class.
Suppose this team is targeting revenue per user as their primary metric, and they only run experiments on the checkout page. Their list of 5 experiments may look like
| Experiment | Treatment Effect |
|---|---|
| Add “Free Shipping” copy beneath Place Order button | \(\theta_1\) |
| Add “Secure checkout” copy beneath Place Order button | \(\theta_2\) |
| Add estimated delivery date beside order total | \(\theta_3\) |
| Add “No hidden fees” copy beside order total | \(\theta_4\) |
| Add “Cancel anytime before shipment” copy beneath Place Order button | \(\theta_5\) |
All treatments are of the same treatment class, so there is no reason to suspect that the experiments identity should matter in relation to the effects they produce – that is to say, the permutation of the effects and the experiment labels seems fine, given what we know about the experiment. Exchangeability here is a highly defensible claim in my personal opinion.
This is a purposefully extreme claim. Let’s weaken it slightly, by making it more realistic.
Situation 2: Same Team, Same Metric, Same Treatment Class, Different Times
“Quick succession” is carrying a lot of weight in the previous section. In reality, a sequence of experiments can happen over months. The reality is that economic factors and competitive landscapes can impact treatment sizes.
It may be the case that the “Free shipping” was run during a particularly turbulent economic time, making customers cost sensitive. This may undercut the justification for treating these experiments as exchangeable – clearly, information about the experiments makes certain permutations of the indices less plausible than others.
A more likely threat to exchangeability is “picking low hanging fruit” in experimentation. Consider a team which chooses treatments to implement based on their likelihood to improve the product. The choosing is not the issue here, rather choosing the treatments most likely to win and saving the treatments which may not produce lift for later is the issue. In this case, the marginal lift – the lift produced by the next experiment – is expected to be smaller in magnitude than the ones that came before it. Perhaps the list of 5 experiments looks like the following:
| Treatment | Treatment Effect |
|---|---|
| Add prominent “Free Shipping” messaging near checkout CTA | \(\theta_1\) |
| Show estimated delivery date before purchase | \(\theta_2\) |
| Add “Secure checkout” reassurance beside payment fields | \(\theta_3\) |
| Clarify return policy beneath Place Order button | \(\theta_4\) |
| Slightly revise checkout confirmation copy | \(\theta_5\) |
Again, we see that experiment identity makes certain permutations less plausible than others. The important part is not that the estimated effects must satisfy \(\hat \theta_5 \lt \hat\theta_4 \lt\hat \theta_3 \lt \hat\theta_2 \lt\hat \theta_1\) but that before seeing the results, the team expects earlier experiments to have larger effects because it deliberately prioritized the most promising ideas first.
There are ways to account for this, and we can assume that the experiments are conditionally exchangeable – conditional on time. This would lead to an altogether different model than we posited above, and so how to deal with these cases is, for now, out of scope.
Situation 3: Same Team, Same Metric, Different Treatment Class, Same Times
Suppose the same checkout team runs 5 experiments over the course of a few weeks, all targeting revenue per user. What has changed is the nature of the treatments themselves. Their experiments may look something like the table below.
| Treatment | Treatment Effect |
|---|---|
| Reduce shipping fees by 20% | \(\theta_1\) |
| Remove a step from the checkout flow | \(\theta_2\) |
| Add estimated delivery date beside order total | \(\theta_3\) |
| Add “Secure checkout” copy beside payment fields | \(\theta_4\) |
| Change the color of the Place Order button | \(\theta_5\) |
A 20% reduction in shipping fees has a very different scope and mechanism than changing the color of a button. Similarly, removing an entire step from checkout plausibly has more opportunity to affect purchasing behavior than adding a small piece of reassurance copy.
Again, the important claim is not that the true effects must satisfy
\[ \theta_5 \lt \theta_4 \lt \theta_3 \lt \theta_2 \lt \theta_1. \]
The button-color experiment may very well produce the largest lift. Rather, before seeing any results, I would probably not be indifferent to assigning a very large treatment effect to the shipping-fee experiment versus the button-color experiment. Some permutations simply seem more plausible than others given what we know about the treatments.
Unlike the previous section, this issue cannot be resolved simply by restricting attention to a narrow period of time. The threat to exchangeability is embedded in the treatments themselves. If we want to retain an exchangeability assumption, we would need to account for treatment class in some way – for example, by only comparing experiments whose treatments are sufficiently similar along these dimensions.
How similar is “sufficiently similar” is, of course, a judgement call. There is no bright line separating one treatment class from another. But this is exactly the type of judgement exchangeability forces us to make: given everything we know about these experiments before seeing their results, are we genuinely indifferent to how the treatment effects are assigned to the experiment labels?
Situation 4: Same Team, Same Metric, Same Treatment Class, Same Times, Different Metric Roles
So far, we have implicitly assumed that the metric whose treatment effects we are pooling plays roughly the same role across experiments. This is another assumption that can quietly fail.
Experiments generally contain more than one metric. There may be a primary metric which the treatment is explicitly intended to move, several secondary metrics which help us understand how the treatment works, and guardrail metrics which we monitor largely to make sure that the treatment does not cause harm elsewhere.
Suppose a team runs five experiments over the course of a few weeks and we are interested in their treatment effects on revenue per user. Perhaps the experiments look like
| Treatment | Role of Revenue Per User | Treatment Effect |
|---|---|---|
| Reduce friction in checkout flow | Primary Metric | \(\theta_1\) |
| Add “Free Shipping” messaging near checkout CTA | Primary Metric | \(\theta_2\) |
| Improve search-result relevance | Secondary Metric | \(\theta_3\) |
| Increase engagement with product recommendations | Secondary Metric | \(\theta_4\) |
| Reduce page-load latency on the home page | Guardrail Metric | \(\theta_5\) |
Every \(\theta_k\) is measuring the same quantity: the relative lift in revenue per user. Even so, there is a good reason to argue against exchangeability in this case.
The first two treatments were selected specifically because the team believed they had an opportunity to move revenue. Revenue is the target of the treatment. In the latter experiments, revenue may simply be something we monitor to understand downstream consequences, or to make sure that an otherwise desirable product change does not accidentally hurt the business.
That distinction is information about the plausible size of \(\theta_k\), and we have it before seeing any results. An experiment which was conceived, prioritized, and designed specifically to increase revenue is not on the same footing as one whose objective was to reduce latency and which carries revenue along merely as a guardrail.
Again, this does not mean that every experiment explicitly targeting revenue will produce a larger effect than every experiment which does not. A latency improvement might unexpectedly produce a substantial revenue lift, while a checkout experiment designed to increase revenue may do absolutely nothing.
The claim is instead that, a priori, we would not be indifferent to the assignment. Handed a large revenue effect, we would sooner attach it to the checkout experiment than to the latency experiment, and that preference is exactly what undermines a claim of exchangeability. The role revenue plays is information the experiment label is carrying about \(\theta_k\) – a grouping of the parameters, in Gelman’s terms.
This gives us a relatively simple restriction when constructing an exchangeable set of experiments: if we are trying to learn the distribution of treatment effects on some metric, we should probably focus primarily on experiments for which that metric was itself a primary metric (you could do also focus secondary metrics or guardrails, but I imagine focusing on primary metrics might be more useful).
There is another closely related issue. Even among experiments for which the same metric is primary, the treatment may operate at very different points in the user journey.
Suppose revenue per user is the primary metric in each of the following experiments:
| Treatment | Product Surface | Treatment Effect |
|---|---|---|
| Reduce payment failures at checkout | Checkout | \(\theta_1\) |
| Remove a step from checkout | Checkout | \(\theta_2\) |
| Improve product-page recommendations | Product Page | \(\theta_3\) |
| Improve search ranking | Search | \(\theta_4\) |
| Change home-page merchandising | Home Page | \(\theta_5\) |
The further up in the funnel an experiment takes place, the more things generally need to happen between the treatment and the eventual realization of revenue.
A checkout treatment acts on users who are already very close to purchasing. Improving search ranking, on the other hand, may first need to change what a user sees, then what they click, then whether they find a product appealing, then whether they add it to their cart, and finally whether they complete the purchase. Each additional step provides another opportunity for an effect to attenuate.
For that reason, knowing where in the funnel a treatment acts tells us something about the revenue effect we should find plausible. Treatments higher in the funnel have more places to lose their effect before it reaches revenue, even when revenue is nominally the primary metric for both experiments.
Once again, then, experiment identity makes certain permutations less plausible than others; funnel position even supplies an ordering. If we know that one experiment directly changes the checkout flow and another makes a relatively small change to the home page, we would not be indifferent to swapping their revenue effects.
Perhaps the more general point is that sharing a metric is not enough. We should also ask whether that metric occupies approximately the same role in each experiment, and whether the treatments have a comparable opportunity to affect it.
If the metric is merely observed rather than targeted, or if some treatments are substantially further removed from it in the funnel, then the experiment label is again telling us something we would rather the model knew about.
Situation 5: Different Teams, Same Metric, Same Treatment Class, Same Times
Finally, suppose we allow experiments from different teams into the exchangeable population.
Imagine several teams running experiments over roughly the same period, all targeting revenue per user and all running treatments which appear to belong to approximately the same treatment class.
Perhaps the experiments look like
| Team | Treatment | Treatment Effect |
|---|---|---|
| Checkout | Add reassurance copy beside payment fields | \(\theta_1\) |
| Pricing | Add reassurance copy beside annual-plan pricing | \(\theta_2\) |
| Growth | Add reassurance copy beside signup CTA | \(\theta_3\) |
| Billing | Add reassurance copy during plan upgrade flow | \(\theta_4\) |
| Marketplace | Add reassurance copy before purchase confirmation | \(\theta_5\) |
At first glance this looks fairly reasonable. The experiments happen at similar times, target the same metric, and involve treatments of roughly similar scope.
However, the team label may carry quite a lot of information about the plausible size of the treatment effect.
Different teams generally operate on different product surfaces, at different points in the user journey, and on different populations of users. A checkout team is interacting with users who have already demonstrated substantial purchase intent. A growth team may be operating much earlier in the funnel. A billing team may mostly interact with existing paying customers.
Consequently, even if the metric is the same, the opportunity to move that metric can differ systematically between teams.
There may also be differences in how teams select experiments.
One team may run many speculative experiments, testing ideas cheaply and accepting that most will fail. Another may only run an experiment after extensive research has convinced them that the treatment is likely to work. Some teams may operate on relatively immature surfaces with lots of obvious opportunities for improvement, while others may own products which have already undergone years of optimization.
All of these are things we may know about a team before a single result comes in, and all of them bear on which effects we would find plausible for that team’s experiments. Team identity therefore acts as a proxy for a collection of potentially important characteristics: surface, user population, funnel position, experimentation philosophy, product maturity, and perhaps many others. Team is a grouping in the most literal sense.
This does not mean that experiments from different teams can never be exchangeable. Two teams may operate on very similar surfaces, target similar populations, and have very similar processes for selecting treatments. In such a case, pooling their experiments may be perfectly reasonable. The important question is what information is contained in the team label.
If knowing the team systematically changes what treatment effects we find plausible, then global exchangeability across teams is difficult to justify.
One response would be to suppose that experiments are exchangeable conditional on team. For example, we might imagine something like
\[ \theta_k \sim \mbox{Normal}(\mu_k, \tau_k^2). \]
A hierarchical model could go further and allow those team-specific distributions to themselves share information. This would let us borrow strength across teams without making the much stronger claim that an experiment from the checkout team is directly exchangeable with an experiment from the growth team. That is, of course, a different model from the simple normal-normal model considered here, and hence out of scope.
For our purposes, the important point is simply that crossing team boundaries introduces another potential source of information about the treatment effect, and therefore another reason to question exchangeability.
A/A Tests
Before constructing any exchangeable set of experiments, there is one class of experiments which should simply be removed: A/A tests.
In an A/A test, both experimental groups intentionally receive the same treatment. Assuming that the experiment is implemented correctly, the true treatment effect is therefore
\[ \theta_k = 0. \]
If we know that an experiment is an A/A test, then its identity tells us essentially everything we need to know about its treatment effect. It is therefore clearly not exchangeable with a collection of genuine A/B tests whose true treatment effects are unknown. The experiment label contains information about the treatment effect by construction.
For cumulative impact, known A/A tests should therefore be excluded from the population used to estimate \(\mu\) and \(\tau\).
So What Experiments Should We Treat As Exchangeable?
If you’ve come to this point in the post, you may wonder if exchangeability is ever reasonable to claim. Experiments do happen across time, do happen across teams, surfaces areas, and points in the funnel. What use is cumulative impact if exchangeability is a bar too high to cross?
Remember, exchangeability is ultimately a judgement about the information contained in the experiment labels. You may be willing to trade off a bit of precision in your results for some usefulness in the feature. Cumulative impact is not intended to be a one-stop-shop and replace other methods, like holdouts or constructing your big-ass-heirarchcal Bayesian model to pool experiments nested within teams. Rather, it is intended to be another tool in your toolbox, in which you need to make a judgment about time and place for use and how much weight to place in results. This post just arms you with some information with which to improve your judgement.