The Optimizely Podcast - episode 25: Fun with Feature Flags

Transcript:  

Hey everyone! Before we jump into today’s episode, I want to remind you that registration for Opticon 2022, Optimizely’s annual flagship conference, is still available. The event will be back in-person this year in San Diego, CA from October 3-5. So please go to optimizely.com/opticon to save your seat! Again, Opticon 2022 registration is still open, so if you haven’t already, be sure to get your tickets now at optimizely.com/opticon and hope to see you there!

Laura Dolan:

Hello everyone, and welcome to the Optimizely podcast. I am your host, Laura Dolan, and we are joined by Griffin Cox, who is a Senior Technical Product Manager here at Optimizely, and he is going to talk about a major component of our DXP today, feature flags. Welcome, Griffin. How's it going?

Griffin Cox:

Hey, I'm good. Happy to be here.

Laura Dolan:

Thanks so much for joining us today. Let's just dive right in. I know there's a lot to unpack. So let's start out by telling us a little bit about your background and your history at Optimizely.

Griffin Cox:

Sure. So I've been with Optimizely almost four months now, but before joining Optimizely, I was actually a customer of Optimizely's. I worked at a FinTech company first as a software engineer for about four years, and then I became a product manager before I switched jobs to Optimizely. And while I was an engineer at that FinTech company, I needed a tool that could help me prove the value of this technology I felt it would be good for us to use in terms of performance and conversions and all that stuff. And we had a homegrown tool, like I'm sure many of our listeners do for this sort of experimentation. And it was woefully inadequate and very hard to use. So I made the case for Optimizely after comparing it to many of our competitors, including Adobe, which we actually already had, and it proved itself many times over. And then I ran our experimentation program at that FinTech company for about a year. And then I figured, eh, I love this product so much. Why not work for the company behind it? So here I am.

Laura Dolan:

That is awesome. That's like saying how much you love the iPhone. You're like, I think I'm going to go work for Apple. And then you end up getting a job with the company product you believe in that is really, really cool.

Griffin Cox:

That is a bit of a dream come true for sure.

Laura Dolan:

Absolutely. That is an amazing story. You're one of the lucky ones just to get to work on a product you believe in and still get to work with your passion every day. So that's awesome. Quick question for you, Griffin, not to go off topic, but are you a fan of "The Big Bang Theory"?

Griffin Cox:

I wouldn't describe myself as a super fan, but I have seen it. And I did notice the name of this episode, but I'll let you explain it.

Laura Dolan:

So yeah, so "Fun with Feature Flags" is in reference to "Fun with Flags", which is a segment on "The Big Bang Theory" where basically Sheldon, Dr. Sheldon Cooper and his girlfriend, Amy Farrah Fowler do a web series called "Fun with Flags". And basically they talk about vexillology, which is the study of flags and the different animals that are on flags and different states and countries and what different flags mean. So I noticed we had Feature Flags as part of our products. I was like, okay, I have to do a podcast called, Fun with Feature Flags. So here we are. And we won't be talking about vexillology today. We're going to be talking about a very different type of flagging. So I'm going to turn it back to you Griffin, and tell us, please, what is feature flagging?

Griffin Cox:

Yeah. So simply put, Feature Flag is a split in your code where you either do one thing, a different thing, or perhaps many other things. You can think about it conceptually a bit like a light switch, at least in the most simple cases where you can control a feature or how it functions, how it's displayed to users remotely, if you're using software like ours. And it really gives you control over the feature when it's in production. It enables experimentation on that feature, which I'm sure we'll get into a little bit later. And it can also help you in the development process as well. So I'm happy to jump into any of those topics. Just let me know which first.

Laura Dolan:

Yeah. So if I'm interpreting this correctly, it sounds a little bit like a toggle switch. Am I right in assuming that?

Griffin Cox:

Yes. Yeah. It's in the most simplest case it's on or off. In the more advanced case it's on and how, versus on and how a different way versus potentially off. So you can get a little bit more complex with it, but thinking about it like a toggle, totally a great way to go.

Laura Dolan:

Cool. That helps me understand a little bit more. So what are the various approaches to feature flagging?

Griffin Cox:

Yeah, so feature flagging, even if we have a developer listening, they may not know the term. Though they may, it's kind of an industry standard term now, but they definitely know the concept. And so in its simplest form without a tool like Optimizely usually takes the form of a constant or a value that doesn't change in the code. You set the value to true or false usually. And then in your code, you reference that constant. And you say if show shopping cart icon then, and then the code that shows the shopping cart icon, else, don't show that code. Right.

Griffin Cox:

And that is the most rudimentary sort of feature flagging example. The problem with using a hard coded sort of Feature Flag like that is that to release that Feature Flag, you need to go through your software development lifecycle. So you'd have to write the code, test the code, get it reviewed, get it deployed, maybe do a post-release check, and bam it's out. And there are so many points of failure in that process. Even if you don't run into a point of failure, it's still usually a slow process unless you're a company like Google, and you got it all figured out. But then you probably have your own feature flagging solution.

Laura Dolan:

True. True. So what does feature flagging have to do with Full Stack? How are they related?

Griffin Cox:

Yeah. So if you take that concept and you basically make it remote, so you don't need to go through that software development lifecycle, you have a nice UI where you can just turn it on or off with a click or change the way that the feature is configured by typing in a few words. And then again, another click and boom, you're off to the races. We saw that as integral to enabling our developers to really experiment honestly, on their own without experimentation kind of like formally in mind, like you don't need to do metrics to make the most out of a feature flagging. You can, for example, limit the risk of a deployment. So if the feature has a bug, you just turn it off. It could turn off in minutes instead of potentially hours, or if you have a problem even longer than that.

Griffin Cox:

And so developers really like the concept. And since we've done feature rollouts, which are essentially feature flags, we just call them rollouts. We've kind of expanded on that subject to not only enable safer development and deployment practices, but really also enable developers to roll out a feature to just percentages, parts of their audiences. So it's kind of crazy now that I look back that so many times I released a feature as a developer. I released it to 100% of traffic. So if something was wrong, 100% of my traffic saw it, right?

Laura Dolan:

Oh my gosh, no pressure.

Griffin Cox:

Yes. Yeah. A lot of pressure. And so, even then I was looking for the right way to release it to maybe just a beta group of users that maybe are more resilient to problems to bugs, and maybe that wouldn't hate us. They'd just send us a message saying like, hey, something's broken, you should look at it. So you could roll out to beta groups or you could just say, you know what? Let's just roll it out to 10% of the overall traffic. And then let's look at our errors, and see if that traffic causes some errors, in which case I'd turn it off before I considered turning it up. And so that sort of rollout, that gradual rollout over time has also been really helpful for teams as they work on experimental features.

Griffin Cox:

And to answer your question in full it's really, we took it yet another step further and now do experimentation primarily on Feature Flags. And so any flag can be set up in with any configuration. It can be on or off, but then you can add a layer where you essentially release the same feature on with this setting on, with this setting on, with this other setting, or maybe off, all to let's say 25%, 25%, 25% of your traffic. And then you can let the customers essentially show you what the right configuration is rather than needing to rely on your own sort of gut instinct. I could give you some examples, but does that make sense?

Laura Dolan:

It does make sense. So that leads me to my next question. How does feature flagging fit into Optimizely's history of digital experimentation? Because that's really what we're all about is A/B testing, multi variant testing. So I was wondering how this dovetails into that?

Griffin Cox:

Yeah. So have you ever heard our story from 2008, about how really Optimizely supported Barack Obama's presidential campaign?

Laura Dolan:

I've heard a few things about that, and I need to know more because that is so cool.

Griffin Cox:

Yeah. It's one of my favorite stories to share. So essentially in 2008 in Barack Obama's presidential campaign, obviously he had a lot of interest in his policies and from his party, but it was kind of a question of what do we want to present on the page to get volunteers to sign up, to support it, or to donate? And it's kind of analogous to what a lot of companies want to do. And in most spaces it's like, how do you increase convergence kind of generally? And at the time, there really wasn't an easy way to do this and measure it all in one tool. And that was really the birth of our first product Web. But before I kind of get into that, we helped the Obama campaign with kind of a quick and dirty solution where we were able to test three images versus three videos, each with one of four different calls to action.

Griffin Cox:

So one image for example, was a color image. Another one was black and white with his family, the videos. And then the calls to action could be like, "join us now" or "learn more", "sign me up", or "sign up". And so there were 24 sort of combinations of content and calls to action. And the question was which one would lead to the highest sign up rate? And the team Barack Obama's digital team took a guess, which ended up being wrong actually, which we love to hear here at Optimizely. As long as you find the right answer, eventually, I think you can learn as much or more from being wrong. And essentially, what came out of it was that the best combination was him with his family. So aww and to "learn more". So it was a bit less aggressive than, "sign me up".

Griffin Cox:

Right. So people maybe wanted to learn more than just plop down their email address and dig in, commit to signing up, right. And so, if you assume that the improvement had stayed roughly the same for his entire campaign, so this experiment was only run for a segment of it, but if you essentially assumed that it had run for the whole thing, then roughly 10 million people signed up over that period of time. And if they hadn't run this experiment, if they had run with the combination of content and call to action that they were planning to, it would've been closer to about 7.1 million sign ups. So it's a difference of about 2.9 million email addresses. So the signup rate was much higher. And typically, volunteers actually became volunteers at about 10% conversion rate. So that 2.9 million additional signups essentially translated into about 290,000 more volunteers for his campaign.

Griffin Cox:

Well, that's a lot of people. And I think if we stopped there, that would be a clear example of experimentation really leading to some good decisions and good outcomes. But even in terms of donations, the average donation was about $21 during the length of the campaign. And so 2.9 million more signups actually translated into about $60 million in projected donations. So it's a lot of money.

Laura Dolan:

That's fantastic.

Griffin Cox:

It's a lot of additional volunteers. And in our own small way, we contributed to the success of his presidential campaign. It's kind of exciting.

Laura Dolan:

Really exciting, every little bit counts. And it's amazing how the hypothesis was wrong, and how it just goes to show you how it's so important to just test different variables, even if it's just changing the color of a CTA, or if it's in the first person versus the second person, it's amazing the different combinations and the effect that it has on people because you and I are consumers. You and I are online every day. We're getting emails. We're getting CTAs, and what causes us to click on different things. And it's just so cool to think about when we're on the other side of it, like what is going to get in front of our audience, what's going to get ahead of them, and what's going to affect them the most? So that is so cool that we had such a, I don't want to say a huge role, but at least we had a role in that. And it's a case study that we can leverage going forward. So that is really, really cool.

Griffin Cox:

Yeah. So in the grand scheme of Optimizely's history, we took that experience and turned it into a product. We went zero to one in a sense, and basically brought a new technology to the marketplace, enabling web based experimentation with eventually a Wizzy Wig editor or a what you see is what you get sort of drag and drop editor. So not only did we kind of make it a standard, but we also enabled non-technical folks so not developers to make those kinds of changes with our Optimizely web product without potentially needing to involve developers. So we've continued to kind of build on that. That was our first kind of major zero to one. And then of course, mobile was starting to take off around 2010-ish. And we then started to support mobile apps with mobile experimentation.

Griffin Cox:

We were the first to do that. And then it became increasingly important to support really advanced experiments that required server side experimentation. So we were also the first to do that. So here at Optimizely, and I was shocked to hear this too, but we consistently deliver new solutions to the market. And I'm definitely thinking about what our next one is going to be and what our next major iteration of that server side full stack product builds on our feature flagging capabilities, combines them with experimentation, so that all experiments in full stack 2.0 coming soon, all experiments will be based on Feature Flags, which allows you to roll out a winner without needing to go through your software development lifecycle again. So we continue to innovate and kind of ask ourselves what's next. And so it's an exciting history and I think an exciting future.

Laura Dolan:

Yeah, it sounds like it's saving a lot of steps. And I just want to circle back to when you were talking about Wizzy Wig. And we drink our own champagne here at Optimizely. So I actually edit our brand pages through Wizzy Wig, and it's such a trip because you basically just hit the edit button, and you're editing on the actual webpage itself. And then you click out of it, and all your changes are saved. And it's so convenient and so streamlined, and I just, I love it. I hope more companies start adopting that going forward because it's such a cool type of technology that I've never even worked with until I got to Optimizely. So I love that.

Griffin Cox:

Yeah. And some of the biggest tech companies out there have been doing this sort of thing with their own proprietary solutions for some time to varying degrees of success. But what Optimizely does is really make it accessible to everyone, particularly customers with a lot of traffic, because it does take tens of thousands, hundreds of thousands of users sometimes to detect a statistically significant difference in the land of experimentation feature. Feature flagging may be a little bit different feature. Feature flagging can certainly benefit companies of all sizes, but certainly in the experimentation size, like the bigger you are, the faster you can reach an insight, and the more you can increase conversions in terms of like experimentation velocity as well.

Laura Dolan:

Very cool. So where are we going next with this? Do any of the recent acquisitions relate to Feature Flags?

Griffin Cox:

Yeah. Good question. So if your listeners have been following Optimizely's recent acquisitions, you'll see that we are quickly becoming a leading DXP. So a digital experience platform, which really needs more than just experimentation, although we are a big part of it. So relevant to, I think this conversation about a year ago, we acquired a CDP, Zaius now known as the Optimizely data platform or ODP.

Laura Dolan:

Yep, I'm fully aware. I came from Zaius!

Griffin Cox:

Awesome. I didn't know that.

Laura Dolan:

Yep, came from that acquisition.

Griffin Cox:

Yeah. So then as you know calling Zaius a CDP alone I think is not quite doing it justice. It is also a connector platform designed to really pull and share data with tons of tools, tons of tools, including Adobe audiences, Shopify, MailChimp, HubSpot, lots of popular platforms. So I think a good example of this is being able to pull in an already defined like Adobe audience, combine it with your Shopify sales data if you're a retail ecommerce site, and being able to combine it also with your first-party data. So what you know about the customer all into one segment. And so our leaders here at Optimizely recognize that sort of capability, that advanced segmentation, machine learning based segmentation as well, as really powerful and core to our desire to be like a full DXP. Early on, we knew that we wanted to integrate ODP or Zaius with our experimentation products, including full stack and Web to really unlock some new targeting capabilities.

Griffin Cox:

So while both web and full stack have audience builders built in now without this integration, they are relatively limited compared to the advanced capabilities that ODP offers out of the box. And so by combining these two products, we're going to enable our customers to target users more easily with less development effort, but also more specifically. So you can run some really more advanced campaigns in terms of personalization in web, more advanced experiments in full stack, for example, and then even target lists of users. And those segments again in full stack for Feature Flags really is a powerful integration and that is coming this year with announcements to come in Opticon in October.

Laura Dolan:

Awesome. Yeah. And I recently had the opportunity to test a new segment builder in beta mode with our CDP. So that was actually really cool that I got to see her from the back end and see how it actually works and things like that, helped me understand the product a little bit better. And this has helped me understand what we're doing a little bit better as well, Griffin. So I appreciate you breaking it down and kind of making it a little bit more digestible for our audience, because I know this can get really technical really quick.

Griffin Cox:

It can. These concepts, they maybe on the surface seem really simple at times, but they get complex really fast. And so when you're talking about replacing an image on a page, that we've streamlined basically as much as possible with our web product, but when you're talking about basically experimenting on different algorithms that power an API server side and using full stack to do that, gets a little bit more complex and you need engineers to make sure you're measuring the right stuff and that you're doing it safely, right? But we here at Optimizely strive to make it as easy to run as many kinds of experiments as possible. And I think most importantly to reach insights or stats sig as quickly as possible. And we are definitely the leaders in that with our multi-arm banded algorithms, our stats accelerator, which those could be episodes kind of on their own. So we'll save those for another time. But we are all about getting customers to value as quickly as possible. It sounds maybe a little cliche, but something that really powers everything we do.

Laura Dolan:

Absolutely. And it's definitely a priority of ours, for our audience to get the results that they want in the quickest time possible. And our tool will help them get there. So it's everything we can do in our power to empower them. So I appreciate all the work that you're doing, Griffin, and that your team is doing. Excited to know more. So if our audience wants to know more about this stuff, where can they go?

Griffin Cox:

Yeah. So I think the easiest way for them to learn more is to follow the link I'm hoping you'll put in the description.

Laura Dolan:

Absolutely. I will.

Griffin Cox:

Cool. Yeah. It's a bit of a mouthful, but essentially our developer docs are fantastic. If you're looking to learn more, we have videos on YouTube. We have podcasts like this. We try to reach our market through as many channels as possible. So lots of different ways, if you search Optimizely even on Google, you will find plenty of resources for all of these different things and for the rest of our DXP platform. We, of course, haven't talked about commerce. We haven't talked about CMS. And I gather, you've talked about those on other episodes. But definitely...

Laura Dolan:

Oh, yeah. There's so many different components of our DXP. We could talk about it all day.

Griffin Cox:

Yeah, it's really exciting. And over this next year, one of our highest priorities is to integrate our products. So we spent a lot of time in this podcast talking about full stack and web, our experimentation products integrating with ODP or CDP. But I'm excited to see how it fits into our CMS, how it fits into our commerce offerings, because a lot of our customers want to experiment and haven't picked up our experimentation tools yet. So yeah, I think more to come in this next year, it's one of our top priorities.

Laura Dolan:

Absolutely. And they're adopting one of our major values, which is never stop improving. And I'm a big believer of that. When it comes to anything you do, whether it's work related or life related, I'm always wanting to learn. So yeah, I love that that's one of our biggest priorities. So I'm proud to be a part of that.

Griffin Cox:

Yeah. And I think, not to speak for our CEO, but I believe you said this before if you're not digitizing your company and optimizing using tools like ours, you're really missing out. Especially, one effect of the pandemic was just the move to ecommerce accelerating rapidly, the move to digital in general accelerating, and increasingly people buying their groceries online, right? Consuming streaming content online, not through DVDs, those were kind of out the door soon anyway. But a platform like ours really just helps people get the most out of their migration to digital. And I'm really excited to be a part of it. And again, happy to be here.

Laura Dolan:

Yes. Thank you so much for joining us today. Again, I will put that link in the bottom of the description. Every episode gets a blog. So when you are following along with the blog to this episode, just hit that link, and go learn more. I'm definitely going to go learn more because I want to understand this stuff. I appreciate you being a resource for us today, Griffin, and thank you so much for coming on.

Griffin Cox:

Yeah, my pleasure. Really happy to do it again.

Laura Dolan:

Thank you, Griffin. And thank you all so much for tuning into this episode of the Optimizely podcast. I am Laura Dolan, and I will see you next time.

Laura Dolan:

Thank you for listening to this edition of the Optimizely Podcast. If you'd like to check out more episodes or learn more about how we can take your business to the next level by using our marketing, content, or experimentation tools, please visit our website at optimizely.com, or you can contact us directly using the link at the bottom of this podcast blog to hear more about how our products will help you unlock your digital potential.