A memo to the lone rangers of experimentation

Are you a lone ranger of experimentation?   

• The sole person at your company who uses A/B testing & CRO tools such as Optimizely, Adobe Test & Target, VWO or Google Optimize? 
• OR do people think of you as: 

• a jack-of-all-trades regarding project management, user experience, marketing analytics and content management? 
• a “technologist”, even though you work in marketing? 

If you answered yes to the above, you’re mostly responsible for trying to get the most out of your testing platform, with limited help, to drive impact to your business across your digital properties.  

The challenge

Wearing lots of hats is nothing new, people across all disciplines experience this. However, as the sole person driving experimentation internally, there is a lot to juggle. Most likely, you are providing support to various business owners throughout the company: product teams, revenue management, sales, marketing, operations, etc. as it relates to launching tests. There is likely no shortage of ideas on how you can help them or what they want to test.  That’s a good thing.   

But there are opportunity gaps that nobody is owning. It might be in improving the user experience of the checkout flow on your app, making improvements to your deployment processes, or surfacing customer service knowledge on your site, so that customers are happier and more likely to convert. It can also include gaps on the operational side of testing such as program management, backlog creation and curation, test performance analysis and reporting, etc. It’s hard to enforce processes across all these functions when you are a Lone Ranger. 

Do you feel you have the drive internally to tackle the opportunities that you see?    

Are you trapped as a solo practitioner because there is no budget to expand or team members available to support you? 

Does your manager share your frustration but not understand the nuts and bolts of what you do? 

Is your organization unable to grasp experimentation's bottom-line impact on its business (e.g., conversions, retention, etc.), and what is required for you to make it happen? 

Here are some ideas on how you can help your company and hopefully your career throughout this journey! 

Strategies for taking experimentation to the next level 

Redefine your space 

Is your role defined (formally or informally) as responsible for conversion rate optimization? Start using the more comprehensive term “experimentation” instead. This frees you to apply your skills to a broader set of KPI’s. If you are not yet working with customer satisfaction metrics, this may be a good place to start.  Hard metrics like the number of inbound requests and softer metrics like NPS are great candidates for experimentation.

Align with IT 

Are you leveraging your experimentation platform to test new technical releases?   Do project managers and/or product leaders know who you are? See if you can find an ally who isn’t afraid to test new releases when they are introduced.  This can be a big cultural shift for a lot of organizations. Not everyone is open to learning that the new feature they just delivered doesn’t deliver the benefit that was expected.  Proving the value of testing to the IT team can unlock resources, including budget. 

Line up resources  

Experimentation requires a combination of several diverse skills: data analysis, project management, creative design, technology and content creation. It’s a lot to pull together if you don’t have a dedicated budget. Many Lone Rangers we’ve met are able to handle a portion of these roles themselves, but nobody is good at everything. Do you have internal resources that may have the bandwidth? A talented creative designer may be able to give you versions for creative testing as part of their existing scope of work. Does your company already have an agency with some of these skills on retainer? Will a business owner in your organization free up a small budget for a test that they care about?  Are there former employees who now work as consultants you can leverage to get work done quickly? 

Use incremental lift to claim virgin territory in your performance appraisal 

Most companies do not have specific financial goals attributed to experimentation. Develop solid KPI’s that show the incremental revenue lift associated with experiments. This may not be easy. For example, in the car rental business, the ecommerce experience captures reservations, which may or may not lead to revenue. In our tests we captured reservation numbers for each segment of an A/B test. Then once the rentals were complete, a couple of months later, we compared the conversion rates and amount spent. This methodology took a while to develop but gave us more confidence in the financial impact of our experiments. Every industry will have its own quirks, and even within the same company, you will find differences in what people believe is the actual impact.  

Use the performance of your experiments over the past year to create a revenue baseline. Get your manager to aim for and buy into the methodology. Now add 20% to your baseline and ask your manager to make this revenue part of your goals for next year. In your appraisal, include an estimate of the resources you need to achieve the goal. 

Evangelize 

It’s surprising how much time it takes to socialize new capabilities. Share the results of each test, whether good, bad or inconclusive, with everyone who works with you.  Lobby to have your results included in internal newsletters -- chances are, your HR team is always desperate for content to send out. Ask for meetings with leaders in the departments that care about the KPI’s you are addressing and use the opportunity to both share results and build up your backlog of new ideas to test. 

Ultimately, you do not want to remain a lone ranger of experimentation forever. Your job will be more rewarding if you can find ways to influence more direct business impact (e.g., revenue, customer loyalty, etc.) while helping the rest of your organization understand the importance of experimentation through measurable results. This is a recipe for success in elevating yourself and showcasing the value you bring to your business.  

For more ideas on how to take your experimentation career to a new level, send us an email at [email protected].