How do you design a clear and compelling A/B test report?

Before you even launch, you should have a clearly laid out template for documenting what you’re testing. This includes details on the background and insight (what's the opportunity and why you’re testing what you’re testing), the hypothesis, primary and guardrail metrics, and test variant and implementation details (e.g. audience targeting). What you then don’t want to overlook is including a section to report on your findings.The questions you want answered here are: did the test end conclusive, winner/loser? Was the hypothesis proven or disproven? Performance of variants and the effect (e.g. +4% CVR%), any insights for why you saw the result, and next steps. From here you can repurpose details for presenting with visuals and story.

The big challenge with SMART tests for me is the narrow view. I have found a lot of value in discarding the restrictions of the SMART goals and instead running broad tests. While one person might test a red CTA button vs a green one and measure success on CTR, I would rather we test 5 of our brand colors for the CTA and also moving the CTA higher up the page. My 10 variations would give a much better view of whether color in general has an impact, or if the location matters more, in a much shorter total time vs running each variation as its own SMART test. At the end of the day, data points drive better hypotheses, so artificial constrains in generating data will only slow progress.

It all starts with understanding the background of what we already have in terms of designs & actual implementation, understanding the overall success stats around it, & basing that defining the A/B test goal & hypothesis. For example, in my current project my client's vision is go carbon neutral & we wanted to make EVs more prominent & something which is easy to navigate on the website (And what's better way than to have it placed somewhere on the Global Navigation of the website), so that people can navigate easily towards those products. Using this information & existing data around what the user flow is towards such products on the website, what is the overall conversion, and so on, we defined the goal & hypothesis of our test.

Formulate a hypothesis that aligns with the goal, outlining the expected impact of the changes being tested. This sets the stage for a focused and actionable test that will yield valuable insights for decision-making Next, determine the key metrics that will be used to evaluate the test. Identify relevant KPIs that directly tie back to the goal and hypothesis, ensuring that they provide meaningful and actionable insights. These metrics will serve as the foundation for measuring success and informing future strategies based on the outcomes of the A/B test When creating the report articulate these steps concisely to ensure clarity and make it compelling by highlighting how each element supports decision-making and drives actionable insights

One of the most important aspects I often see neglected in A/B testing is ensuring that the goals and hypotheses are aligned with the company's strategy. Diverging from the company's direction can lead to a lack of global impact and reduce the overall effectiveness of the program. This misalignment can also demotivate stakeholders, as they may feel disconnected from the company's objectives. Therefore, ensuring that your hypotheses are in line with the company's goals is crucial for the success of the test.

From experience, especially when starting a new experimentation program, it's best to keep things simple. Alway run an A/A test for data validation, and run a simple A/B test (copy change, button colour, etc.) just to get started. Once you've ironed out any points of friction in the setup process, it's wise to focused on goals & objectives for the next few experiments. Continuously optimise your optimisation process.

To me, hypothesis formulation is very important as a hypothesis is a proposed explanation or solution to a problem. I would not suggest following hypotheses created by others for your business problems because each business is unique, and its strategies should be tailored accordingly. However, there are essential elements you can use to make winning hypotheses - Look at changing customers during test, either positively or negatively. - Focus on deriving customer learning from tests. Understanding big picture is key. For example, during a recent marketing experiment, Alex Rodriguez, a digital strategist, found that tweaking the CTA copy from 'Join our newsletter' to 'Join my newsletter' contributed to a 60% increase in subscription rates

Start your A/B test report by clearly stating the test’s goal and hypothesis. The goal should articulate what you aimed to achieve, such as improving click-through rates or increasing user engagement. Your hypothesis should predict the outcome of the test, such as "Changing the CTA button color will lead to a 10% increase in clicks." Clearly defining these elements provides context for your test and helps readers understand the purpose and expectations behind the experiment.

Being able to formulate a testable hypothesis is a *key determinant* of A/B test success. Because: 1 - It enables the team to understand the initiative: What are we trying to achieve? 2 - It nudges the team to think cause-effect: why would we think it to have the expected outcome? 3 - It empowers the team to think about success: How can we achieved our goal? The framework "IF, THEN, BECAUSE" can be very useful to create hypothesis: IF we switch to ML-based recommendations on the homepage, THEN the homepage users will click on the recommendations, BECAUSE it will provide more relevant content that users are more interested in. It takes time and teamwork to create strong hypotheses. However hypotheses are key to create business impact.

Select a single primary metric as a part of planning the test. Also ensure there is clear methodology to report on that metrics as well as baseline and preferably historical trend data. One may select some secondary metrics as well as guard rail metrics to ensure that over optimization in one area is not negatively impacting other business metric(s). However, if one does not have a single primary metric for the test, they may pick and choose a metric from a list with most favorable lift to conclude a test prematurely or incorrectly. Selecting a single measurable primary metric will help drive sound decision making process for successful test conclusion.

Know your audience. Leadership may not need to know intricacies of the test execution (testing tool is irrelevant) compared to the results, learnings, and next steps. However, if you are discussing test results among product owners, you may need more technical details for that audience.

The methodology and setup of an A/B test should be clearly outlined in the report to provide transparency and ensure replicability. This involves detailing the specific changes tested (e.g., variants A and B), the duration of the test, and the criteria for inclusion in the experiment. Additionally, it's important to describe how users were randomly assigned to each variant and any potential factors that could impact the results. Include information about any control variables or external factors that were controlled for, as well as any limitations or assumptions made during the test setup. To create a clear and compelling report, present this information in a structured manner.

The mistake often made is making reports either too complicated or adding unnecessary details. If the report is too complex, people won't understand it, and if it includes too many irrelevant details, it becomes overwhelming. For a small company, create accessible reports with a clear hypothesis framework, like the one on the Henkan & Partners website. In large companies, tailor reports to different audiences. I recommend a one-pager with key test information, a detailed report for those involved in the test needing in-depth insights, and a two- or three-slide deck for presenting the experiment to top management.

For multivariate testing, A/B/C... Testing, I randomize combinations. If we test 5 attributes with 4 options each, we will have 625 unique combinations. Statistical significance for that many would be a challenge. Instead, we use (ex. 50) random combinations. Each of the 4 options of each attribute shows up in ~12 versions, and we reach statistical significance with 92% fewer impressions. Each combination is labeled based on its attributes and values for simple parsing & analysis later. (ex: greenCTA_topCTA_informal_noHeader_OceanPic)

For A/B testing methodology and setup: Objective Definition: Clearly state what you aim to improve (e.g., conversion rate, engagement). Hypothesis Creation: Formulate a hypothesis based on insights or assumptions (e.g., changing the call-to-action button color will increase clicks). Variable Selection: Identify the element(s) to test (e.g., button color: red vs. blue). Control and Variation: Establish a control group (original version) and one or more variations with the change applied. Audience Segmentation: Divide your audience randomly to ensure unbiased results. Sample Size Determination: Calculate the sample size needed for statistical significance. Test Duration Success Metrics Data Collection & Analysis Ensure Technical Setup

In describing your A/B test, clarity and transparency are key. Like effective communication, your methodology should be straightforward and replicable. Detail your design, implementation, and tools with the intent to educate and inspire. Share how you split your audience, mirroring how leaders distribute tasks - fairly and effectively. Duration and reach of the test are like setting goals, they should be specific and measurable. Use visuals like diagrams, enhancing understanding as stories do in presentations. Sharing knowledge empowers others.

For A/B test, one can use "Baseball Card" to document the entire experiment which would cover your design, the test background, hypothesis, success metrics, & results. Other tools we use for A/B test is Adobe Target, Adobe Launch, & GA. Dividing the test audience depends on the test type. For our purpose we used a 50-50 split & the time frame also varies test wise till the time one reaches a statistically significant sample (For our case it was 4 weeks).

For my A/B test in real world industry: - To run A/B test: I have to create two different versions of one piece of content, with changes to a single variable. - Tool and platforms: Google Optimize, Adobe Target - Split traffic or audience between control and the variation equally and randomly. Then, determining sample size is needed, but the size will also vary depending on A/B testing tool, and the type of A/B test I am running. If I am A/B testing an email, I must send an A/B test to a subset of my list large enough to achieve statistically significant results. - Then, I have to monitor the results during the test regularly. If the results are inconclusive or unexpected, I will consider running the test for a longer duration.

Detail the design of the A/B test, including the control (A) and variation (B) along with specific changes made to the variant. Explain how traffic was allocated between the two groups and outline the duration of the testing period to ensure statistical validity. Include information on the sample size and demographic considerations to provide context for the data collected. A well-explained methodology allows for transparency and reproducibility of the test.

Detail the methodology and setup of your A/B test to provide transparency and reproducibility. Explain how you divided your audience, the variations tested, and any specific parameters used, such as sample size and duration. For example, describe if you tested two versions of a landing page, with one featuring a new headline and the other using the original headline, and mention how you controlled for variables to ensure valid results. This information helps readers evaluate the robustness of your test and understand how you reached your conclusions.

Designing a clear and compelling A/B test report involves succinctly presenting your findings. In my experience as a CTO, the effectiveness of a report hinges on how well it communicates the core results and their implications. Utilize visual aids like charts and graphs to outline differences between control and variant groups. It's crucial to annotate these visuals with clear, concise captions that explain statistical significance and potential anomalies. Additionally, explaining the statistical methods in plain language ensures that stakeholders without a technical background can grasp the conclusions and their impact on decision-making. This approach fosters a deeper understanding and confidence in the test outcomes.

Presenting A/B test results is like storytelling. It's not just about data, but the narrative behind it. Compare control and variation with clarity, highlighting key differences and trends. Acknowledge errors or anomalies as lessons, much like in leadership. Confidence in results reflects your understanding, tempered with an awareness of limitations. Use visual aids for clarity, making complex data accessible. Your statistical methods are tools, but your interpretation in plain language is what truly informs and inspires action. Remember, data tells a story, make it compelling.

Pour partager les résultats, je montre d'abord l'original vs le variant en expliquant l'hypothèse testée et le changement unique effectué. Ensuite, je montre le CTA ou le KPI choisi et le taux d'amélioration du variant. Je précise également le nombre de visiteurs uniques, ainsi que le taux de fiabilité.

Present A/B test results with clarity. Summarize key metrics, visualize differences, and discuss statistical analysis and outliers. Contextualize results within the test's goals and hypothesis. Provide actionable insights for decision-making. Avoid technical jargon for broader stakeholder understanding.

When presenting A/B test results, it is important to communicate the entire chain of process, from the insight gathered, to a hypothesis framed based off of it, to the actual visuals of control and variation and then clear presentation of the performance against the main defined goal that determines the recommendation. Based on my experience have both the longer thought process explanation in a couple of slides, as well as a concise version, needed i.e. for the annual learning summary. Both have value and different purposes.

-Automate the reporting and publish it for broader users for broader data democracy. -Analyze and document the sample size and MDE reached and what time period you selected for the analysis -Before concluding the test based on primary metric, verify there are no negative effects on guardrail metrics.

Use clear and informative visuals like graphs and tables to illustrate the performance metrics of both variants, comparing conversion rates, engagement levels, and other relevant indicators. Provide a statistical analysis of the results, including significance testing, to assess the validity of the findings. Summarize the data clearly, focusing on key metrics that align with the defined goals. An organized presentation of results enables stakeholders to quickly grasp the impact of the changes.

Let's look at this specific use case for improving click-through rate. - A statistical method: Two-sample t-test to compare the click-through rates of two groups: the control group (Group A) and the variant group (Group B). The t-test helps me determine if the observed differences in click-through rates are likely due to a real effect or if they could have occurred by random chance. - Interpretation: Let's say we found the following results: -- Control Group (Group A): Average CTR: 10% -- Variant Group (Group B): Average CTR: 12% In this case, the relative change is a 20% increase in the click-through rate. As the p-value is less than 0.05, we can say it's likely a real improvement and not just due to random variability.

Going into detail and adding graphs etc. is a great idea, but in my experience, it is really important to have an overview slide or management summary with the most important information. This contains an image of the winning variant, the change in metrics, the hypothesis and a brief explanation of the test result. To be honest, the c-level won't read your whole report and thats ok since their objective is to get an overview and not understanding it in detail. By adding a management summary, we serve those who want to get a quick overview only.

Invest enough time in a proper template where the management summary is the most important page/slide. If you have a clear management summary, which totally clarifies what has been tested, which results it gave and what you conclude from it, you'll definitely get the return on time invested from it. Remember, don't try to squeeze in too much in that summary, but keep it concise. The key message should be in the management summary, other key elements in the rest of the report of max 4-5 pages/slides. Further details in the appendix as it might be interesting, but for a lot of people too much details, most of the time. Last but not least, test your reporting template with less involved people and use their feedback for improvements.

Analyze A/B test results to derive insights and recommendations. Identify trends, patterns, and significant findings. Utilize data to suggest improvements or actions for optimizing performance. Communicate these insights clearly and concisely to stakeholders for informed decision-making.

Proving incrementality is technically impossible, which is why we use statistical analysis. Be specific. "We found the XYZ audience segment clicks our CTA 5.1% more often (95% confidence range of -1% to +15%) when we put the CTA directly below the header image AND shrink the header image to 50% of standard height (75px vs 150px). No other audience segments responded positively to this combination of changes. We recommend a 20% mix of this new layout for the XYZ audience for the summer ABC campaign, which should drive an incremental $27.3mm (-$5.5mm to +$81.9mm)."

This is a crucial step for the A/B test that you are running because if the wider audience is not clear of what the test was intended to & what are the results, then it is a failed test. For the A/B test that we ran on the Global Navigation of the client's website, we learned that there was a drastic uplift in number of users navigating to the EV segment & checking the product, thereby making our test a success. The benefit we got from a 50-50 split in audience was that we got a fair understanding of user interest since equal number of people were exposed to both the experiences. As part of next steps: Implement the success variation, Meeting accessibility standards, Page Performance testing, Retesting the results to understand impact.

Interpret the results to derive actionable insights, explaining how the outcomes support or refute your original hypothesis. Based on the analysis, offer detailed recommendations on whether to implement the changes, conduct further testing, or explore alternative strategies. Highlight the implications for business objectives and future marketing activities. Comprehensive insights empower decision-makers to take informed actions that align with organizational goals.

Summarize the insights gained from your analysis and offer actionable recommendations based on the results. Discuss what the data reveals about the effectiveness of the tested changes and how these findings can inform future decisions. For example, if the new design improved user engagement, recommend implementing the change across the platform. Provide practical next steps, such as further testing or adjustments, to guide readers in applying the insights to their own projects.

Encourage further exploration by highlighting any limitations encountered during the test which may affect the overall findings. Discuss potential confounding variables or external factors that could influence user behavior during the testing period. Suggest areas for additional research or subsequent A/B tests that could provide deeper insights. This forward-looking perspective fosters a culture of continuous improvement and learning.

-Document the test in knowledge base with visual copy of both variations. This documentation will help maintain database of learnings and double down on what worked and prevent retesting same ideas repeatedly. In large organizations with multiple teams running various tests, sharing of knowledge accelerates the learning curve. -Delete the underperforming variation (not just turn off traffic to it) to maintain clean coding practice.

Make it matter to your stakeholders. Put it in their language. Speak about your results in terms of their goals. Start with a high-level summary following other advice on this article, and then ask your stakeholders what they would like to hear more about. Don't forget to talk about next steps. Periodically review your rubric for selecting A/B tests from a backlog with your stakeholders to make sure you're prioritizing the experiments your stakeholders want.

Don't forget to share your reports with the relevant stakeholders so that they can learn from them too! Share an email newsletter with a round-up of recently completed A/B tests or publish an article on your company's intranet every now and then. This is a great way to ensure that your organization can actively participate in the experimentation & optimization process.

In addition to the main report sections, consider including a discussion on the limitations of your A/B test and any external factors that might have influenced the results. Address potential areas for improvement in future tests and provide suggestions for additional experimentation. For example, if the test was limited to a specific demographic, recommend testing the changes with a broader audience to validate results. This comprehensive approach ensures your report is thorough and provides valuable context for interpreting the findings.

Always test one change at a time to get clear results. Also, think about the timing of your test—some changes might perform better during specific seasons. Additionally, highlight the importance of ongoing testing and iteration. A/B testing is not just a one-time effort; it’s about continually improving and learning over time.