Often, it’s hard to design high quality experiments in business settings. Say you operate a regional chain of smoothie restaurants in 8 metro areas, with multiple locations in each metro area. You’re interested in understanding the price elasticity of demand for your smoothies, and want to test how much a 10% price decrease changes the quantity of smoothies demanded.
How do you run an experiment in this case?
One option is to randomize at the store level. However, this can lead to interference between locations, especially in dense metro areas where customers may choose whichever store offers the lower price. This would undermine the integrity of the test.
An alternative is to randomize at the metro level, assigning all stores in half of the metro areas to implement the price reduction. But this comes with trade-offs. If the experiment ultimately shows the 10% discount isn’t cost-effective, reversing course means raising prices for a large segment of your customer base, which may be alienating and frustrating. It may also make the test more expensive than your business is willing to absorb.
A more practical approach might be to select a single pilot city at random. This helps avoid interference and reduces cost and customer disruption. However, with only eight viable metro areas to choose from, there’s a real risk of selecting a market that doesn’t reflect broader customer behavior, which would limit the generalizability of your results.
Enter synthetic controls
Fortunately, there’s a well-established solution: synthetic controls for experimental design. This method was developed to estimate the causal effect of an intervention when there’s no obvious comparison group for a clean, “apples to apples” analysis. More recently, economists realized that it could be taken one step further, and used to identify an ideal comparison group for an “apples to apples” analysis. It’s commonly used in evaluating marketing campaigns, pricing tests, and even clinical trial data.
Here’s a high-level overview of how it works:
Start by selecting one or two metro areas that closely resemble your target market and test the pricing promotion there. Average their sales performance to construct what’s called a synthetic treated unit. Next, identify several other cities whose combined characteristics represent your typical market conditions, and use them to build a synthetic control. The combined outcomes in these other cities, during the period in which you test the pricing promotion, serve as your counterfactual baseline. The estimated effect of the intervention is simply the difference in outcomes between the treated unit and the synthetic control over time.
Why isn’t everyone doing this?
It sounds simple, and conceptually, it is. But each step in the process involves technical detail. Constructing a valid synthetic control requires careful weighting of the comparison metro areas and the treated unit, proper handling of pre-treatment trends, and rigorous estimation of confidence intervals for the effect size. Just as importantly, this method isn’t always the right fit. Depending on your context, another experimental or quasi-experimental design might be more appropriate.
If you’re a decision-maker who could benefit from the kind of insights experiments can provide, chances are you’re also very busy, and don’t have time to dive into the technical work required to do this well. That’s where King Street Economics can help. We bring deep expertise in statistics and experimental design, so you can focus on your business while still getting clear, credible answers to complex questions.
If you’re considering an experiment or want to explore your options, reach out to schedule a free consultation at [email protected].
