Content Intel—episode 13: experimentation benchmarks

Transcript:

Laura Dolan (00:03):

Hi everyone, and welcome back to the Content Intel Podcast, brought to you by Optimizely. I am Laura Dolan, your host.

Laura Dolan (00:10):

Today, I am joined by our very own VP of Strategy and Value Consulting, Hazjier Pourkhalkhali, who will be walking us through some experimentation benchmarks. Thank you for coming on, Hazjier. Welcome to the podcast.

Hazjier Pourkhalkhali (00:23):

Thanks, Laura. Nice to be here.

Laura Dolan (00:24):

How's everything going with you?

Hazjier Pourkhalkhali (00:25):

Really good. Beautiful start to the year. We're almost out of lockdown over here in Amsterdam, so I've been excited about that.

Laura Dolan (00:31):

Oh, that's always good.

Hazjier Pourkhalkhali (00:32):

But, yeah. Always good. Always good, indeed.

Laura Dolan (00:35):

Yes. Well, let's start out by you telling us a little bit about your role here at Optimizely and how your journey has been.

Hazjier Pourkhalkhali (00:40):

Gladly, yeah. So my journey at Optimizely has been a long one. I joined the company back in the middle of 2014-

Laura Dolan (00:47):

Wow.

Hazjier Pourkhalkhali (00:47):

... as a young strategy consultant. We were working directly with our customers. And at the time, Optimizely was still a very young startup and customers would, for $1,000 a month, have access to this free strategy consultant resource, which was me at the time in EMEA.

Hazjier Pourkhalkhali (01:03):

I remember sitting there and working with 10 or 20 different companies every week and just advising them on how they should experiment. It was such a change in my career when I first joined the company, but also one of the most liberating that I'd had. Because all of a sudden, I had 10 or 20 different clients, constantly advising them what they should be testing, what they should be changing and learning from all of them. It was one of the most creative bursts of my career.

Hazjier Pourkhalkhali (01:23):

Needless to say, I fell in love with experimentation, and over the years, I ended up leading our consulting team. I ended up becoming head of strategy for the business, leading pricing, value consulting, being chief of staff, just really rotating all across the business. Even was a product manager at some point.

Hazjier Pourkhalkhali (01:39):

But yeah, nowadays, I head up our strategy and value team. That's really focusing on whenever we're working with major companies, giving them a real perspective on what it takes to be successful and what's realistic. Because there's oftentimes, miss-set expectations in this space. People don't know what to ask for, or how to set a good benchmark for their own performance.

Hazjier Pourkhalkhali (01:55):

At the same time, taking our data to make sure that we are giving people the best possible advice for conducting research and we're giving people real insights.

Hazjier Pourkhalkhali (02:02):

I think as Optimizely, it's the largest digital laboratory in the world, that's also our responsibility to this profession because we have that position and that opportunity.

Laura Dolan (02:10):

A hundred percent. What was it about experimentation that made you fall in love with it?

Hazjier Pourkhalkhali (02:14):

Well, way back in the day, I was a management consultant at McKinsey & Company, and it was one of those places where I learned so much about the business world, how to communicate, how to build strategies, but I was always pushed to be conservative because that was the mantra of management consulting, which is you make highly certain decisions, you don't put people into risky of a path, and you make sure that they're ultimately everything is well taken care of and low risk. And that also is a bit limiting on your ability to really experiment, be creative and try new things.

Hazjier Pourkhalkhali (02:42):

And when I had 10 or 20 different customers, it was kind of a relief in some ways, because many experiments are going to fail regardless of what you do. The vast majority of tests are losing experiments. And when you accept that and you realize, "Well, you know what, not every idea I'm going to come up with is worthwhile," and you don't stress about it anymore.

Hazjier Pourkhalkhali (02:59):

It's in a way it's one of the most liberating feelings because you just realize I can try a lot of things. I can learn from them. I can experiment. I can get better at my crafts. And I just really love that feedback loop, because I think oftentimes in the business profession, people can be very flowery. People can be very much managing other people's expectations or perceptions of them, and in experimentation, it's just like right there, clear as daylight where you can just see, "Wow, tried that idea, that did not work. This idea sucks. I really need to change my tax on this." And that was something I loved because it just led to this period of lots of creativity, risk taking, failure, and growth.

Laura Dolan (03:37):

I love that. It really takes the pressure off and that's something that has actually changed my perspective since I started working at Optimizely as well is never stop improving. And I love that mantra. I've actually adopted it in my personal life as well as my professional life. And you can try something, you could fail, and then you can go back to the drawing board and it's still going to be okay. You're still working on the project or blog or video or even podcast in this case. And in your case, working with customers and just educating them, it sounds really rewarding.

Hazjier Pourkhalkhali (04:09):

It's extremely rewarding. And it's also fun. It's because every time you work with someone, you can have a real impact on our business. And you know with certainty. You can literally go back to the date and say, "We did this. We changed this company's trajectory. We had that increase their revenues, their customer signups," or whatever metric was important to them. And that's actually one of the things that I really like because sometimes there's vagueness in how we think about our past achievements and accomplishments and testing is sometimes painfully clear.

Hazjier Pourkhalkhali (04:36):

We're just like, "Wow, you did not have an effect here. You need to step it up." And at the same time, it's also just liberating because you just kind of realize stepping it up is not about getting rid of risk. It's not about preventing myself from failing. These failures are going to happen regardless. Stepping it up is being bold enough and daring enough to keep at it despite all the failures. And that, to me, is something really beautiful that I really enjoyed about experimentation.

Laura Dolan (04:59):

What are some experimentation benchmarks you're working on right now? Can you give some examples of how you're helping customers currently?

Hazjier Pourkhalkhali (05:05):

Gladly. So data's been one of my biggest passion projects at Optimizely. It began actually very early on because I remember when I first got on the experimentation scene, I didn't really have much of a name or any clout back then. And I would hear all these speakers on the conference server talking about this advice that it sounded to me like it was made up and people always felt, "Well, when you hear an experimentation guru tell you some advice, it must be researched. It must be thought out." But I remember a lot of experimentation gurus were saying at the time, for example, A/B tests should be small. They should be highly specific. You should know exactly what you're changing and why. There's kind of this notion that the ideal A/B test was minimal. It was extremely isolated. And there was just one variable that you changed.

Hazjier Pourkhalkhali (05:46):

You changed, for example, the color of a button, the font size, you made some small tweak and knew exactly what that color change did. You know exactly what that font change did. And I realized that a lot of this advice probably wasn't researched because none of these people would have the purview. They only had a couple of customers. They only had their own experiences to rely on. And so what I started to do is to really look across all of our customers. And we have over a thousand companies that have worked with us. We have a data set of hundreds of thousands of experiments, and that gives us a chance to really ask ourselves, "Well, maybe that advice works for one company, but does it for everyone?" So the first things I started to really look into is, "Well, how often do experiments win?" And winning in this case means, for those of you who are very deep in the field, it's really having a statistically significant uplift, above a threshold of 90% on a primary metric.

Hazjier Pourkhalkhali (06:32):

And what that really means for our customers is when you are testing, are you driving improvement in your business? Are you ultimately able to increase the metric that you care about? And what I started to see in the benchmarks was there was a lot of best practice advice that people were told in the industry that actually counteract their likelihood of being successful. And it was actually forcing them down the wrong path. And so I can give a couple examples of these. Two of the really obvious ones are a lot of companies have historically learned experimentation them from classical physics. And if you think about the experiments you were taught in high school chem or physics, they're always very exact tests where some scientists discovered some new universal concert or some law that people have been able to build on for decades and centuries.

Hazjier Pourkhalkhali (07:12):

But in the business world, things change all the time. You can have a winning experiment this week and three years from now, it's not what your customers need anymore, because the smartphones that are accessing your website with have evolved, and they've got very different capabilities and expectations now, or the market has changed and people have very different needs. And so when people are so focused on the minutia, they start losing track of the bigger picture and they stop being able to keep up with it. And so what a lot of the data showed us was the companies who are most successful making big, bold bets, they're trying a lot of ideas side by side. They were really affecting the user experience. And the companies who were successful with incremental tests, the really tiny minute updates, tended to be massive enterprises who can measure the smallest of changes.

Hazjier Pourkhalkhali (07:50):

A good example of this is Google. Google actually once tested 41 different shades of the color blue. This is famous. They called Google's 50 shades of blue experiments. And because they figured out the ideal shade of blue to highlight a URL that you would underline whenever you were searching for results on Google, and they got slightly more clicks, that was worth hundreds of millions of dollars to them. But most companies around the world don't have that kind of traffic or capability. So what I really wanted to understand was, if you are a more medium sized business, if you only have visitors that are in the hundreds of thousands or in the low millions, below 10 million, what is going to make you successful? What do you need to succeed? And what does that mean for your strategy? And so a lot of the benchmarks we work on are really researching, how do companies get better at experimentation?

Hazjier Pourkhalkhali (08:32):

And there are a couple powerhouse programs around the world that the Facebooks and Googles of this world who are really phenomenal at testing, have incredible data, resource technology to be really successful in that domain. But there are a lot of companies who are getting started. They don't know where to go to. And a lot of the advice they're getting might be right for someone else, but it's not right for them. And we wanted to really change that field.

Hazjier Pourkhalkhali (08:51):

So a lot of what I do with benchmarks nowadays is to research, what are the best practices that make your customer successful? How should you think about your testing program? What are the returns that are realistic? How do you set the right expectations with stakeholders? How do you make sure you all set the business case when it comes to getting? For example, more engineering resources, which is oftentimes a sticking point for a lot of programs we work with. And then lastly doing academic collaborations. And so part of my fortune's also been partner with a lot of universities on academic research and really getting insights from a lot of professors in business schools and economics programs on what it takes to evolve experimentation and make sure that the next generation of practitioners have the right tooling, enablement, onboarding, and really support to be successful in that role is do things right.

Laura Dolan (09:30):

That's incredible. Sounds like you were so in the weeds with this, and I could just hear the passion in your voice, it's awesome. It's awesome to hear about it and just to see just the slightest change, like what that can do for a company, it's really encouraging. So it also sounds like a lot of personalization goes into this as well.

Hazjier Pourkhalkhali (09:47):

Absolutely. One of the big things is of course we have to adjust our feedback based off the person receiving it and we have to make them better. And one of the funny things is when I was thinking about this is, people who work in the CRO profession, who are experimenters or product people who have gotten an experimentation focus, they're oftentimes running experiments on their customers, but they're rarely running experiments on themselves, asking themselves if I think experimentally about how to improve my job and my profession in my way of working, what would I do differently? How would I change the way that I go about my day? And that was for me this responsibility where I finally saw, you know what? We are the biggest testing company on earth. We are one of the very few people who do this research, try out these ideas and experiments and actually see what makes the profession as a whole better because no job is finished.

Hazjier Pourkhalkhali (10:31):

The way people practice product five to 10 years ago is very different than today. And the way people experiment 10 years from now will be completely different from how we do it now. So why don't we take the time to really ask ourselves, how do we want to shape this profession? What's going to be right for companies around the world? What's going to be right for people practicing this, who are taking a big, bold leap in their careers as they try on a new profession? And how do we practice what we preach and not just experiment on websites, but experiment on experimenters and help them understand what really moves the needle, what really makes a difference in their lives.

Laura Dolan (11:01):

So have you encountered any challenges along the way when it comes to experimentation?

Hazjier Pourkhalkhali (11:07):

I've encountered a lot of challenges when it came to experimentation. I think first off, as a practitioner, one of the biggest challenges that I've encountered repeatedly was the lack of high quality data on what our customers were doing. And I think this is something I see almost every time I work with a company, because I know from experience and from research how important having access to good analytics is and making programs successful. But it was always painful. Every time I'd work with a new customer, I'd be like, "Okay, well, what analytics tools are using?" And then especially back then, it was a lot of Google analytics and Adobe. And then the second you got in into that platform, you realize they weren't tracking anything correctly, their data was out of sync. They had no meaningful dashboards or reports set up. And even though companies claim to be data driven, the second you started to look at the data you realized, "Well, this is just barely usable."

Hazjier Pourkhalkhali (11:54):

And so for me, one of the biggest challenges in experimentation is experimentation is a high maturity practice. You first have to be a data trusting organization before you can be data driven. Because if you don't trust the data, then how can you rely on that for your biggest and scariest decisions? And only then can you be experimental. And so the question for me oftentimes is when I work with companies, "Well, how do I, number one, get them to understand they need to invest in their data quality?" Because testing requires research. You need to take the time to understand what's happening with your customers. And then secondarily, how do I provide to them the data they need to be successful and they're advocating for the programs, for their teams when they're setting their goals? And I'll give an example. So a lot of the ways executives historically run departments really run counter to what is going to create an environment to be successful at testing.

Hazjier Pourkhalkhali (12:42):

One of these is just timeline pressure. A lot of times when you've got a boss and they're frustrated, they want more performance out of people, they start to set unrealistic deadlines because they want to put people into a little bit of a pressure cracker, they want to put people in this mode of, "Okay, you have to deliver by a certain date." And the problem is when you do that with an experimentation program, people start to take shortcuts in their work. And so a test might need, for example, four weeks of data. And all of a sudden people are trying to call it after a week and a half because they promised your boss an answer in a shorter timeframe. And so you might think from a classic management perspective, I'm doing the right things, I'm pushing people on timeline, but in practice, you're actually sabotaging their work because the practice that might work well on the factory line are not going to translate well in a scientific laboratory.

Hazjier Pourkhalkhali (13:25):

Or alternatively I've seen leaders who came in and they said, "Well, you know what? I expect a team to run 50 experiments per year." But then they weren't resourcing them. And they just kind of felt "Well, since I can't give them more resources, how about I increase their target every year, force them to run more experiments?" But then what these teams would do was run worse experiments.

Hazjier Pourkhalkhali (13:41):

So they went from having these really deep, insightful ideas about how to change the website towards optimizing the minutiae of text on a particular form field where you might do that once or twice, but after a while, there's not a whole lot more opportunity to improve there. And so these tests start to become just very fake and they're aimed for vanity metrics. And so I really wanted to make sure that other people don't face these same challenges, that we give them the data so they understand, here's what it takes to make your team successful. And the best facts they may use successful as a leader of a new testing team are not going to be the same ones that make you successful today. And I think that's been one of the big challenges I face myself working with customers, but I'm also trying to prevent other people from having to deal with just by giving people a better understanding of what it takes to really be successful in this profession.

Laura Dolan (14:23):

Sounds like a quality over quantity issue.

Hazjier Pourkhalkhali (14:25):

A hundred percent. Testing is straight up a profession that is a function of quality more than quantity.

Laura Dolan (14:31):

So what kind of experiments do you have coming up? What are you working on for the rest 2022?

Hazjier Pourkhalkhali (14:37):

Well, I'm not running as many experiments on our customer's websites nowadays, though our team is doing quite a bit of amazing work. But I've been asking myself, "Well, how do I take that knowledge from experimentation and make our company more successful?" And so one of the experiments that I'm working on right now is how do we use our data to know which companies we should be working with? Like I shared earlier, experimentation is the high maturity ambition. You have to be data trusting, then you can be data driven and then lastly, you can be experiment driven.

Hazjier Pourkhalkhali (15:04):

And so what I've been working on is, how do we take the data that we know about companies to start to estimate how ready they are for experimentation by using machine learning and some artificial intelligence tools to really tell us, "Okay, based off the technologies that other businesses are using, based off the way their websites are structured, in what direction would this tell us they are when it comes to their testing maturity?" And then using that data to change the behavior of our people in the fields who are going out there and talking to these companies and who are trying to ultimately sign them on as a new business that we can make sure that the people who get started with us are well set up for that success and they're ready for this maturity shift.

Hazjier Pourkhalkhali (15:37):

And so that's probably the biggest experiment I'm working on right now, which is how do we use the data that we have about the companies that we work with to get better understanding who is ready for the shift right now, who's going to be successful to make this leap and to make sure that when they are getting started with experimentation, all the conditions are set for them to be able to have a really successful career.

Laura Dolan (15:55):

Oh, that was a lot of information, Hazjier. Thank you so much. I'm excited. I am pumped for experimentation. The more I learn about it, the more I embrace it and just love the concept behind it. So it's great to be part of this organization that values it so much.

Hazjier Pourkhalkhali (16:10):

Absolutely. Yeah. It's a real game changer I have to say. And I think you were saying this earlier, experimentation really makes you really being open to change is really one of these values we have with the company. I remember way early when I started, we had this t-shirt that said "Optimize Your Life". And it always stuck with me. I was just like, fundamentally, all life is an experiment-

Laura Dolan (16:31):

It is. It really is. It does change your perspective. It leaves everything open ended. And like I said, it eases the pressure and you can just keep on improving. Nothing is set in stone. You could just keep going back and changing it if you have to. And the slightest change looks like an improvement and it just makes you feel better. And even if you go back and see it didn't work, it's like, "Well, let's just try again."

Hazjier Pourkhalkhali (16:53):

And now you know, and I think that's also the beauty of it. It is fine to mess up, but it's like, you have to embrace the process of change and embrace it even though you don't know what the future looks like, you'll get there and it'll be fine. You don't have to have everything planned out. But like you mentioned, even when things go wrong, at least now you know and you've got more certainty you can move on and you can focus your efforts where it needs to be.

Laura Dolan (17:11):

Yep. It builds that confidence and that's invaluable in itself.

Hazjier Pourkhalkhali (17:16):

Absolutely.

Laura Dolan (17:17):

Awesome Hazjier. Well, thank you so much for taking the time to come on here today. I know you are a very busy person, so I will let you get back to it and just keep on improving the business and lives for our customers. I'm sure they appreciate everything that you do.

Hazjier Pourkhalkhali (17:31):

Thanks, Laura. Thanks for having me on.

Laura Dolan (17:32):

Absolutely. Thank you all so much for tuning into this episode of Content Intel. I am Laura Dolan and I will see you next time.

Laura Dolan (17:40):

Thank you for listening to this episode of Content Intel. If you'd like to check out more episodes or learn more about how we can take your business to the next level by using our content, commerce, or optimization tools, please visit our website at optimizely.com, or you can contact us directly using the link at the bottom of this podcast blog to hear more about how our products will help you unlock your digital potential.