Mobile app A/B testing is the practice of using A/B testing to test different experiences within mobile apps. App users are randomly bucketed into different segments, and each is shown a different experience. After enough users go through the experiment, it is possible to tell with statistical confidence which version of the app leads to higher conversions.
For example, an app developer might have a mobile game that includes in-app purchases for extra levels. The app developer might want to test out different banners for the levels to see which one is clicked on the most.
By running an A/B test, they will be able to determine, based on their actual users, which banner performs best. Once the test reaches statistical significance, they can roll out the winning variation to all their users and improve the overall conversion rate.
The benefits to running A/B tests on mobile apps are similar to those running A/B tests on the web. A/B testing allows you to test out different hypotheses for experiences within your app and make changes to your app experience based on actual data instead of gut instinct. A/B testing tools allow you to determine with statistical confidence what the impact of changes you make to your app will have, and measure exactly how great that impact will be.
By continually running tests on your mobile app, you will constantly be able to improve the experience of your app and roll out new features that you know will improve your conversion metrics. It's the reason why top companies such as Facebook, Amazon and Google constantly run A/B tests in their mobile apps. A/B testing is indispensable for experience optimization and conversion rate optimization and personalization on mobile devices.
Another benefit of using a mobile A/B testing such as Optimizely is that it allows you to make changes directly to an app that has been published to Apple's app store, without having to go through the app review process. This allows you to iterate faster on changes and make fixes and updates to live apps.
The power of mobile app A/B testing is captured in this case study of Secret Escapes, a travel app for iOS. In the experiment, Secret Escapes used Optimizely to test the impact of removing the option of skipping sign-in to the app. What they found was that making the sign-in mandatory resulted in a higher average lifetime value for the user, and improved the ratio of user LTV to customer acquisition costs.
Every app is different, and there are countless changes that can be tested on mobile apps. Key areas to focus on when conducting A/B testing are user onboarding, user retention, user engagement rate and for apps that feature transactions, every step along the sales funnel to maximize conversions.
Examples of things to test include the number of forms required to sign up for an account, the frequency of push notifications, the text used in calls-to-action (CTA), new features, and ad placement within your user interface.
If you already have analytics tooling in place, digging into your data and what the end-to-end user experience is like is also a great place to start generating experimentation opportunities.
For more ideas, see our blog post on 50 Mobile App Development Tips.
Optimizely is the leading digital experience platform for A/B testing on desktop and on mobile devices. Optimizely is simple to install with open source SDKs for Swift, Objective-C and Java that work for running tests on Android and iPhone apps.
Optimizely allows you to wrap new features in feature flags (aka feature toggles), which allows you to rollout new features to specific audiences or a segment of your audience to see how it performs, with real-time, enterprise level statistics that give you confidence in your test results. What's more, you can easily roll back any bugs without the need for a code deploy or app-store review-- simply toggle the feature on and off.
If you want to improve the experience of your iOS or Android app, improve conversions, and make data-driven product changes, start testing with Optimizely today!