A/B testing is a methodology for comparing two versions against each other to determine which one performs better. It is essentially an experiment where two or more variants are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal.
We recommend creating and following a framework for A/B testing. This framework consists of a few questions you should answer before you test. After the test you should revisit your framework, add the results and prepare for the next iteration or a repetition of your test.
- What are you testing? Examples: Subject Line, Whole Email, Landing Page Variant, Number of Form Fields. Please only test one thing at a time.
- What are you expecting? Example: I expect the variant email that I test against my control version to create a 10% increase in click rate.
- What’s your sample size and what’s your random split? Example: 10.000 contacts and a 50/50
- When do you I collect the data? Example: One week after test release.
- What’s the result? Example: Email A had a click rate of 3%, Email B of 4%
- What can you conclude for further tests? Example: I should use text-only emails more often.
- When will you repeat the test? Example: In three months. It’s widely recommended to repeat specific tests from time to time to make sure they are not outlier results.
Document and share results with anyone who might be interested. Keep also in mind that short-term results might not align with long-term results. For example: If you compare 5 to 10 form fields on a form, it might be so that the shorter form creates more conversions, but the longer form creates the more meaningful ones.
Marketo offers two options to execute A/B Testing. The “Email Program” – a program type and sending mechanism specifically designed to test – and “Random Sample” – a powerful Smart Campaign Flow Step that lets you split audiences into smaller chunks for short or long term observation.
Marketing Automation with Like Reply
Like Reply is a specialized expert in digital communication with offices in Munich, Milan and Paris. We support companies in the introduction and development of automated communication as well as in the systematic, effective management of leads and customer relationships. Contact us without obligation and find out how we can take the use of Marketing Automation to the next level together.