Episode Transcript

Karen:

Hi, this is a Responsive Web Design Podcast, where we interview the people who make responsive designs happen. I’m your host, Karen McGrane.

Ethan:

And I’m your other host, Ethan Marcotte.

Karen:

And this week, we are like human versions of the cry/laugh emoji. We are overcome with emotion to have Chris Saint-Amant and Anna Blaylock from Netflix joining us. Welcome.

Chris:

Thank you.

Anna:

Hello!

Karen:

So Chris, Anna, I would love to start off, if you could both just introduce yourselves, say a little bit about what you do at Netflix, and perhaps a little bit about what your role is in going responsive at Netflix. Chris, why don’t you start?

Chris:

Great. I’m Chris Saint-Amant and I lead the acquisition UI engineering team. So, we’re the team that builds the user interface that everyone who signs up with Netflix goes through. So that cuts across a pretty wide range of platforms. So, what you would typically expect with anybody signing up in a browser, as well as in our mobile apps, and even, more interestingly, on our TV and set-top boxes. So, you could be signing up on a Roku or an Apple TV and those types of devices, and smart TVs, and we build that experience as well. So, anybody that signs up with Netflix goes through the user interface that we build. And when I joined a couple of years ago, Anna and I were brainstorming how we could get better at mobile, and that was kind of the initial genesis of our shift towards responsive design.

Anna:

I’m Anna Blaylock and I work on the experience design team here at Netflix. And so yeah, as Chris was saying, I’m his counterpart, so I specifically work on growth and user acquisition for the experience design team, but also all the aspects of the design for the web experience. And when we originally did the MVP of responsive sign-up, Chris and I were really leading the charge and very passionate about testing this idea against having these two separate codebases that we’d had for a good while.

Ethan:

But before we dive in, I’d like to take a moment to mention our sponsor, Harvest. I’ve actually been using Harvest for years, so I was incredibly excited when they approached us about sponsoring the podcast. Harvest bills itself as a beautiful business tool for tracking time spent on client projects. I’ve worked with a number of teams who love it. They love how easy it is to track hours across projects, how beautiful and well-designed their reporting tools are, and I’ve heard from a lot of folks that I’ve worked with that Harvest really helps them stay on time and keeps their project under budget. Now, personally, just speaking for myself, I love its invoicing and expense management. I can update my projects in Harvest with my web browser, with my mobile device, and regardless of the size of the screen I have closest to me, it’s a real joy to use. So if you’re interested, you can check out Harvest at getharvest.com today. What’s more, after the free thirty-day trial, you can enter coupon code RESPONSIVE at checkout and save fifty percent off your first month.

We want to give them that initial view of our product, of what we are about, without having to push them into an app experience.

Karen:

So I’m going to confess: when I think Netflix, I don’t really think the web. I think that you’re on a bunch of different native app platforms. In many ways, when I think of Netflix, I really think of device-specific strategies. You’re on so many different device types, and it seems like the interfaces and the processes are really geared for mobile phones or computers or for TVs. Can you talk broadly about how responsive design fits into a really app-centric environment, and maybe, if you had them, some of the challenges that you might’ve faced in convincing teams that responsive was the right way to go?

Chris:

Sure. So I think that perspective I think very much comes from what we look at in our member experience. So, I assume hopefully you’re a member of Netflix and so you enjoy all the great content we have on all these different devices, but you’re using our member experience. Where we’re really optimizing for this heavy repeated usage, things like streaming video and adaptive streaming, really trying to deliver the highest performance experience that we can with minimal impact on bandwidth and battery life and those types of things. To accomplish that, you’re right: in many ways the best approach that we’ve found is to deliver these very platform-specific experiences, especially when you look at the TV platforms that we’re on and a lot of the mobile platforms that we’re on; that’s really the best way to deliver the member experience.

But when we looked at the non-member experience, so what we present to people that are prospective customers who are coming to our experience to evaluate us and determine if this is a product that they’re interested in and worth their time to try and sign up for and give it a shot, that really that’s a very different segment of our audience, and those people may not be ready yet to make the investment to take the time to download one of our apps. So, we want to give them that initial view of our product, of what we are about, without having to push them into an app experience.

Anna:

But the beauty of our sign-up process being responsive means that we are obviously in the browser across all these different devices, but then we’re also able to embed this sign-up experience into our apps, like the iOS apps and the Android apps. And then when we are doing our A/B testing and improving all the time, it automatically can be tested and wins can be rolled out across all these different platforms, all these different devices, all at once. It’s very efficient.

Chris:

And see, we didn’t have to really spend a lot of time convincing people. We have pretty small, empowered teams, so it was really just a discussion between Anna and I and our product management peers to say, “Hey, do you think this is the right approach to take? How are we going to measure it and how are we going to validate that we’re going down the right path?” and then taking that plunge and trying it.

We don’t assume that there’s differences or similarities between a mobile user and a desktop user. We do love to A/B test here, so we’re always looking at our results by the device type.

Ethan:

I’d love to hear a little bit more about how the design for the sign-up experience started, specifically around these two words that we hear from a lot of our guests, which are “mobile” and “desktop.” I mean, we speak to a lot of organizations who sort of suggest that these are fairly distinct contexts, and “as a result, we need to kind of tailor the experience to each of them.” Others sort of advocate that, you know, especially for transactional flows, they tend to need more or less the same information. How did those words come up in conversation when you were actually designing the sign-up experience for Netflix?

Anna:

So, the original sign-up experience was designed years and years and years ago, but when we were thinking about making it responsive, as I say, we already had an established desktop/laptop version of the sign-up and we had this separate codebase with the mobile one. So, what we tend to think here is, we don’t assume that there’s differences or similarities between a mobile user and a desktop user. As we’ve been mentioning, we do love to A/B test here, so we’re always looking at our results by the device type to see where the difference and similarities are.

What we do know is that everyone coming to the non-member experience, looking to see whether the service is for them, they want to know that Netflix is trustworthy and they want to understand all the aspects of the service. So we know that this doesn’t vary by device type. But we are always challenging the assumptions that they’re potentially similar, so we’re looking at different tests that we can devise to find out what the differences are and where there are differences.

Certainly there are differences in the ergonomics of these devices, right? Does that change how we need to present the content and allow people to interact with the content?

Chris:

What we’ve been trying to get at is are there—certainly there are differences in the ergonomics of these devices, right? In the physical attributes and how people interact with the devices. So, does that change how we need to present the content and allow people to interact with the content and go through the sign-up experience? And how much of those differences that we may observe, are those differences because we’re giving people—that the people themselves are different, in different audiences? Or is it because the pain points and the friction that they might encounter have a higher or lower impact depending on the type of device that they’re using?

For the member experience, people love to consume and watch great content on any device, so that’s one of the big value propositions; you can start watching something on your phone and finish it on your TV. But we do certainly see, on that side, some differences in user behavior in terms of how much content they might watch. You know, your tolerance to binge watch ten hours of Marvel’s Jessica Jones on a TV might be—you might be more apt to do that than on a phone, for example.

We did a lot of iteration both on that sign-up flow as well as then expanding that strategy to other parts of our experience.

Karen:

I’m always curious when organizations, especially organizations that talk about being data-driven and doing a lot of A/B testing, how do you plan the rollout strategy for something like a new sign-up experience? Like, did you stage the rollout to different devices or different user types so that you could test and learn what was working and what wasn’t? Like, how did you plan a rollout process of this magnitude?

Chris:

What we did is we started out initially with, “Hey, let’s use one of our existing experiences as a baseline and take that and make it responsive,” and then essentially pit that against the non-responsive equivalents. So we started with just the bare essentials of the sign-up flow, so from our home page through to the last step of sign-up. That’s about five or six different screens, depending on the type of user you are and which region you’re in. We did that as quickly as possible; we did about a week-long prototyping session with design and product and engineering to get to kind of this initial MVP, then spent another week or two just QAing and cleaning that up, and moved as quickly as we could to pitting that new version against the older versions and really running them head to head.

Once that was proven out that was a success, we said, “Okay, we think we’ve proven out the essentials of the strategy, but then how do we get better at it from there?” So we did a lot of iteration both on that sign-up flow as well as then expanding that strategy to other parts of our experience.

We really started out with prioritizing what are going to be the highest, biggest impact areas where we’re going to get the strongest signal and the clearest read of whether this is the right approach to take.

There’s a bunch of other aspects of the UI of the non-member experience that are accessible in the browser, and we even extended it into our account management functionality, where it’s a small portion of the website where you manage your email and password and a bunch of other preferences, but it’s not something that’s really heavily used and is something that we don’t build into every single one of our applications, so we wanted to make it very easy to access from the browser on any advice. So, we extended it out to there. We really started out with prioritizing what are going to be the highest, biggest impact areas where we’re going to get the strongest signal and the clearest read of whether this is the right approach to take, and then once we proved out that initial approach, then we expanded out from there.

Ethan:

I’d love to hear a little bit more about the applications and tools you actually use to shape the sign-up experience. Basically, like, are you still using more traditional graphic applications, like Photoshop and Illustrator, to sort of refine UIs? Or are you moving more into prototypes? Just tell me about the day-to-day stuff in your tool belt, basically.

Rather than spec’ing out various sizes, we’ll look at a small size and potentially the large size as well, and then we’ll fill in the blanks together through the browser.

Anna:

When we were doing this initial test against the old solution before we were responsive, Chris mentioned we had this week where the designers and engineers were all basically in a room drinking lots of soda, lots of snacks, getting as much done and prototyping as much as possible. In that series of days, we used Keynote a lot from the design perspective for quick iteration. And then, of course, when we’re doing more visual things, we’ll use more Photoshop and Illustrator. Now we’re using Sketch a lot in our day-to-day work; the designers just love it; I love it. And so from a design perspective, that’s what we’re using. And then we’re iterating a lot in the browser with the engineer. Rather than spec’ing out various sizes, we’ll look at a small size and potentially the large size as well, and then we’ll fill in the blanks together through the browser.

The desktop version of sign-up had already had years of A/B testing and optimizing behind it. It was already incredibly simple. So, there wasn’t, like, tons of things to prioritize on the page.

Karen:

Talk to me about how this responsive redesign might have changed the messaging or the content of the sign-up experience. Like, did you prioritize different information in different ways so that it would work or read better or get the message across better on different sized screens?

Anna:

Initially, when we made the sign-up responsive, the desktop version of sign-up had already had years of A/B testing and optimizing behind it. It was already incredibly simple. So, there wasn’t, like, tons of things to prioritize on the page; it already had a very clear hierarchy.

When we wanted to test it against the control, we didn’t do that many changes. There was a specific example where when you registered for the service, we used to have enter your email address and then confirm email address, and then password and then confirm password. But as we know, on mobile it’s very annoying to do these repeat inputs, so we did challenge this. The benefits of having it were, of course, user error and so then they’ll be less likely to have issues when they want to log in after they’ve signed up. But we tested that as a different cell against a control, removing the confirm fields, and we made sure that it didn’t do too much harm to logins and that the calls to customer service hadn’t increased. So, we challenged this and ran with it across both the desktop and the small screens.

In terms of our client-side analytics, we didn’t really change that much when we went responsive.

Ethan:

For such a measurement focused organization, I’d love to hear if Netflix was focused on measuring the speed of the sign-up experience. How did the performance of this new UI get discussed while you were designing with it? Were you measuring that as it proceeded or sort of toward the end of the process? How did you have that conversation?

Chris:

That’s something that we’ve always been focused on, so I’d really look at that as somewhat orthogonal to our move to responsive. We have some client-side performance metrics that we track on an ongoing basis that show us trends in load times, and then we are able to slice that by device and also by region; we actually see a pretty wide range of variances by different countries that we’re in based on the bandwidth infrastructure there. But we didn’t really change that dramatically when we went responsive.

I think where we started scrutinizing that a little bit more is more the user-perceived performance, that that can be much more visible on smaller lower-end devices; that if something is loading progressively or if you were to use webfont and that pops in at a certain time, that there’s more subjective evaluation of performance that we started doing throughout our QA cycle. But in terms of our client-side analytics, we didn’t really change that much when we went responsive.

Everyone’s voice is very equal, all the way from like the chief product officer down to, you know, any other person in the org.

Karen:

What about how responsive may have changed the way the rest of the organization participates in design reviews or product reviews? One of the things that Ethan and I have found in talking to companies that had previously only had been looking at one version, you know, like one layout for any possible design, that having to give comments on a responsive prototype changes the way they get feedback from other team members. Did that happen for you?

Anna:

In terms of design reviews, there is a constant conversation between the different platform teams and within the design team. What we tend to do is often we have prototypers on the design team that will concept an idea that a designer has, which is really useful for demonstrating the responsive nature of any project we’re working on. We don’t have lots of formal design reviews, but there’s a lot of ownership, where the team that’s working on the project has the freedom to roll with the idea that they have.

We do go broader and have a weekly debate forum across various parts of the org, and that’s more to bring in the ideas and have a real healthy debate, not so much an approval process; and it’s really just to get those ideas even stronger rather than to get them, like, signed off by the higher-ups, you know? Everyone’s voice is very equal, all the way from like the chief product officer down to, you know, any other person in the org. There can be a debate between two people in that room and they can be very, very different parts of the org, and it’s accepted and actually encouraged, which is exciting.

As we solicit feedback from our peers, we show both the small and the large, and we’re increasingly shifting towards really just focusing on the small screen only.

Chris:

I think we did look a little bit at which types of assets we’re reviewing. So, Anna talked about prototypes, which has certainly been really useful in reviewing designs. But we also had quite a bit of debate early on about whether we should be—when we are doing comps, whether we should be doing small screen comps and/or large screen comps, should we be doing both? Do we design for the extreme, then fill in the gaps for the browser, which was mostly where we ended up starting out over the first year or two. And then kind of using both of those, as we solicit feedback from our peers, that we show both the small and the large, and we’re increasingly shifting towards really just focusing on the small screen only and not worrying even about the larger screen designs until we get into the browser.

This did certainly complicate our QA process, and it was one of the things that we were very mindful of.

Ethan:

We can’t have Netflix on our podcast and not ask about device QA. I would love to hear about how you tested this responsive sign-up experience. This is something a lot of organizations struggle with, actually incorporating device labs as not just a QA tool but as a design tool. How did you actually just sort of get this thing onto as many browsers as possible and make sure that it was working?

Chris:

Well, obviously we just hire really awesome QA engineers and just let them run wild. But no, this did certainly complicate our QA process, and it was one of the things that we were very mindful of, that you get this benefit of having a single codebase that works across all these devices, but that’s also a potential risk and downside, where one change to one UI component has this really, really wide scope of regression. Even further for us, given that we’ve embedded a lot of those same components into our iOS and Android apps.

We’ve taken a couple of different approaches. One is we’ve invested a significant amount in test automation. So we’d already had a pretty strong set of automation capabilities across our org because of the wide range of devices that we support, and in our area we started investing in that quite a bit more. So we didn’t have to worry as much about the basics of setting up a device lab; in many cases, we had all these devices already around. It was more about how do we more intelligently automate on these devices without having to do a ton of manual work with each cycle. And we’re still getting there; we still, I think, have a ways to go in getting fully automated coverage across the platforms that matter to us, but it’s something that we’re really continuing to invest in.

Our core metrics really are our guiding light, and so the users, through their actions, tells us whether we’re doing things right or where we need to improve upon.

Karen:

If Ethan doesn’t get to have you on the show and not ask about support, I don’t get to have you on the show and not ask about measurement. So, for an organization that I imagine relies as much on A/B testing and data as Netflix, how do you think about measuring the success of a responsive redesign, or a responsive sign-up experience? Like, does that change the way that you, say, track different devices or change the way you measure success?

Anna:

Through our A/B testing, that is how we’re measuring success: what is that conversion rate and the dropoff rate for each of the steps in the flow? So, we look very closely at that across the board and then break that out by different device types.

And then we also do a lot of research, where we put concepts, new ideas in front of people in all our markets, all over the globe where we’re in, and get qualitative feedback about how these flows are going. And then we’re just monitoring trends by device type and screen size on an ongoing basis. Our core metrics really are our guiding light, and so the users, through their actions, tells us whether we’re doing things right or where we need to improve upon.

We really try to stay true to a single core metric that is around the number of people that sign up and start paying for the service, and that is really the main metric that we evaluate everything against.

Chris:

It’s really easy and tempting to kind of monitor and instrument every last little bit of the experience. So on the home page, we’ve got a couple different content modules that you can interact with, and knowing exactly which modules people are looking at as well as, as they move through the flow, exactly which points they’re dropping off at.

But we really try to stay true to a single core metric that is around the number of people that sign up and start paying for the service, and that is really the main metric that we evaluate everything against. So, all those little things, all those small actions that a user takes and the design changes that we introduce at every step of the flow, ultimately the success of that is validated against that single core metric.

Some of that is still an open question that we’re continuing to evaluate what the right way to slice and dice those core metrics is.

But we’re starting to get more, I’d say, thoughtful about how do we evaluate that core metric based on the type of device that somebody signed up on. And so, Anna mentioned device type and screen size, and that’s actually been an ongoing debate, of how relevant is it to filter test results by platform, meaning by iOS or Android, or by iPhones vs. iPads, vs. looking at specifically screen size and screen orientation. Or you could even argue pulling in other capabilities. Are we looking at touch vs. non-touch devices, and how certain changes that we make affect users on those types of devices? I think some of that is still an open question that we’re continuing to evaluate what the right way to slice and dice those core metrics is.

Ethan:

Well Anna, Chris, with this responsive sign-up experience behind you, and you’re continuing to iterate on it, what are some things that you’ve learned from the experience that you’d share with our listeners? If someone is in the audience, they’re about to undertake a responsive redesign, what’s a lesson or two that you’d like to share with them?

Having a flexible, fluid layout actually helped with localization. It actually helps you when you have such a wide range of types of content and languages being shown in that interface as well.

Chris:

Really, the biggest is really empowering the team to move quickly and independently. You know, we’ve talked a lot about how we don’t have a lot of formal approvals here, and that was really helpful, to have kind of small core team that is able to really work pretty independently, helped them make a lot of decisions quickly and on the fly, and adapt and change as needed and not really worry about, “Oh, this is the way we do this as an organization,” or, “This is the way that we’re required to do it or that we’re supposed to do it.” It was really kind of wipe that slate clean and give the team the ability to just really say, “What’s the right way to do this?” and question that. And two days later, if you’re trying something and banging your head against the wall and going, “This isn’t working,” try something different and being kind of flexible and adaptive.

On a more tactical note, one thing that we learned that was pretty interesting that I think we hadn’t thought of initially but that’s turned out to be useful and beneficial for us is that as we continue to expand globally—so since we launched our responsive design, we already supported a handful of languages, but we’ve added things like German and Japanese, and we’ve announced next year that we’re launching in South Korea, so adding in Korean. Having a flexible, fluid layout actually helped with localization, right? If you’ve already structured the design of your interface to support this fluidity and kind of unpredictability in the devices and the types of screen it’s being displayed on, it actually helps you when you have such a wide range of types of content and languages being shown in that interface as well.

If we want that imagery to look good across all screen sizes, we really need to think responsively at the earliest stages of creating that imagery.

Anna:

Yeah, I’d just add to that, that really my bit of advice would be getting in the browser as soon as possible; don’t get too bogged down in Photoshop or any tool like that, where you’re doing all these different designs for different screen sizes. Really just think flexibly and work with prototyping tools or a prototype or an engineer to really start seeing how that design reacts depending on the different screen size.

And then my tactical piece of advice would be more about imagery. I wanted to highlight a specific example, where on our non-member home page we have big imagery that takes up a lot of the screen. What we realized early on, once we started doing responsive design, is if we want that imagery to look good across all screen sizes, we really need to think responsively at the earliest stages of creating that imagery. And so we created guidelines for photo shoots and even attended photo shoots to help influence how the images were composed so they would look great on all screens. We had this template that was prototyped, where we had a focus area that we knew would appear on any screen size, and then negative area surrounding that, so that was more imagery that wasn’t necessary to be on screen no matter what. And so the photographer had the tool to be able to drop the images in as he was shooting them and we could see them in a prototype and see how they would look across screen sizes, and that really helped us.

Karen:

Well, Chris, Anna, we are all heart eyes cat emoji over here. This has been really spectacular. I am genuinely impressed with some of the things that you’ve shared today and I’m sure everybody else listening is too. So, thank you so much for taking time to come on the show today.

Chris:

Thank you. We really enjoyed it.

Anna:

Yeah, thank you, guys.

Ethan:

Thanks to everyone for listening to this episode of a Responsive Web Design podcast. And thanks also to our sponsor, Harvest. Go to getharvest.com and start tracking your time, painlessly, today.

If your company wants to go responsive but you need help getting started, we offer a two-day onsite workshop to help you make it happen. We also offer these workshops to the public, so please go to responsivewebdesign.com and see when we’ll be in a city near you.

If you want even more from us, you can sign up for our newsletter, subscribe to this podcast, and read full transcripts of every podcast episode at responsivewebdesign.com.

Thanks for listening, and we’ll be back next week.