From the course: Marketing Attribution and Mix Modeling

Unlock the full course today

Join today to access over 24,200 courses taught by industry experts.

Bandit testing: Optimizing for results over accuracy

Bandit testing: Optimizing for results over accuracy

From the course: Marketing Attribution and Mix Modeling

Start my 1-month free trial Buy for my team

Bandit testing: Optimizing for results over accuracy

“

- [Instructor] A/B test to the most accurate way to tell what's working, but they can take a lot of time. While they're running, you're giving at least 50% of your traffic to the worst performing variation. For example, in this A/B test version A and version B have been sharing equally, even though version B was the eventual loser. So we've shown the worst variation to a lot of our user base and potentially hurt the conversion rate. A bandit test fixes that. It decides a little bit quicker than an A/B test because it gives more traffic to any variation that has an early lead. In this case version B was shown to a lot less users over the lifetime of the experiment because version A took that early lead. Now it's great to have that extra speed in decision-making, but it does come at the cost of accuracy. You're likely to have a lot more false positives and false negatives. A false positive would be if we declared that the…

Contents