Email Marketing: Actionable Email Marketing Tips From a Real Life Klaviyo Data Scientist

Hey guys, I'm stoked to have someone from Klaviyo here today. This is Michael, who's actually a data scientist at Klaviyo. Michael, how are you? Doing well, how about yourself? I'm doing well, thanks man. I really appreciate you taking the time. Do you mind kind of sharing a little bit more about yourself and kind of what you're currently working on? Yeah, absolutely. So Klaviyo, I guess you could say it's the first real job. I came from an academic background. I was in stats, grad school, land, learning about algorithms, learning about algorithm development, and made the switch over to Klaviyo and I absolutely love it here. What I do on the team is kind of a mix of that same thing. So like some algorithm development, some looking at stats research that's out there and thinking, okay, what's the actual right thing to use for this task? And then some of it is also actual feature development. So kind of figuring out, all right, so we've got a model, but what's the actual right way to give that to people and make it useful for them? Yeah, that's awesome. I know today kind of the topic here is your guys' predictive analytics. And I believe you mentioned you kind of have cool tips, maybe three cool tips that we can kind of break into. Do you mind sharing kind of the first of like predictive analytics and like what it is and kind of what you recommend? Yeah, absolutely. So generally, I would say predictive analytics is a pretty broad idea. Kind of the idea is just, you use some sort of predictive model. So you kind of use data that you have, you use kind of, it might not necessarily be the source of data that you expect either. It could be kind of you're trying to predict for a specific person. But to actually make that prediction best, you have to make predictions based on other similar people, because that gives you the amount of data you actually need to make confident choices. Or it could be, you want to make predictions about purchases, but to get that sufficient level of kind of measure someone's interest, you might have to look at things like their browsing history, and things like that. So a predictive analytics are basically just taking that source of data, making some sort of smart decision based on it and serving that up to you in a way that you can actually make a decision based on it. And the first one I'd like to talk about is our smart send time. I guess the to boil down that, that kind of, I don't know, abstract idea that I just served up. What this does is it says, okay, so we can take information about times that you've sent people emails, about times that they've opened emails about times that they've interacted with them in other ways, we can, we can actually run an experiment. And we can serve that up in the sense of, okay, now you just, it's a single number, kind of what's the best time to send someone an email? Which I think the way that you I think the way that you can make decisions based off of that is pretty, pretty straightforward from there. Awesome. And you might break it down a little bit more from more of a call, right? It kind of sends emails over like a 24 hour exploratory window where every couple hours, it sends to a different batch of people. You're going to talk a little bit about like, collecting based off of people's geo and trying to say like, Hey, someone here, California, where I am on the west coast might want to receive an email a little bit later in the morning, someone on the east coast, right? And then also to is there a kind of certain criteria of number of profiles number of people have to be receiving it for it to be eligible to send? Yeah, absolutely. So to go into the methodology that we use for smart send time a little bit. This came from a lot of research. So Christina Dietrich on our team was kind of the main driving force behind this feature. And she did a huge amount of research coming up with our method. She actually tried some methods that some methods that are in use at some other places where pretty much all they do is look at your historic data. And they say, all right, based on the times that you have sent emails in the past, here's the best time to send in the present. And what we found was that actually didn't hold up super well when we ran experiments with it. So we would kind of say, all right, based on all the times you've sent email in the past, 10am looks like it's the best time just to give some numbers. And then we would say, all right, let's just let's pick a number that's very different from 10am, just to kind of prove that this actually does provide lift. And so we would test it against 3pm and 3pm. And we just kept seeing trends like that, where we when you only looked at historic data, you actually weren't seeing that. But you weren't seeing the proof when you actually tried to test it against other hours of the day. So what we realized the best thing to do was, but we don't have to build a complicated model based on your past historic data, we can actually just run an experiment. So we can say, all right, for anyone in their local time, kind of so like, you know, 1pm in their local time, let's test 1pm in their local time versus 2pm in their local time, versus 3pm, etc. And we actually found you have to test all 24 hours of the day, because some hours that you wouldn't expect are actually not very bad at all. Like, you might expect that sending someone an email 3am is just like that's going to have a terrible open rate, no one's ever going to open it. Not always true when you look at people's data. And the only way we can do that is by testing it out and sending a volume of email where we can actually see those trends. That's awesome. That's fantastic. And one of the drawbacks of this approach, if you want to call it that is, if we really want to make a decision with enough confidence to say, all right, the difference between 2pm and 3pm, we really know that we're really confident that 2pm is better than 3pm, you have to send a lot of email. And just to make it so that you're not having to send you, we call them exploratory sense so that you're not having to run that experiment, you know, 15 campaigns in a row. The future is available for people who have at least 12,000 profiles on that list. Awesome. That's super helpful. And by the way, one thing I want to mention for everyone listening, this Smart Send time is not to be confused with Klaviyo's Smart Sending, right? And basically, the Smart Sending is basically a filter that you can apply to an email that someone doesn't receive it over 16 hours or 24 hours. I can't remember which one of those is the default. You can customize the window. Yeah, you can customize the window. And one of those is the default. Is that something that you guys, this is obviously off topic, is this something that you guys hear about, like, hey, Smart Sending is Smart Send time, that's something that people get confused by a little bit or not really? Yeah, we have heard it a little bit. It's definitely, it's something that we're keeping in mind. Also, I'd like to plug some other work that that team has done that's made this feature kind of, I think the feature was already pretty great to begin with. I mean, the lifts we were seeing in the experiments where we were refining this feature, I think those spoke for themselves. It was like on aggregate, we were seeing something like 10% better open rates. Wow. And that's held up since people started kind of like using the feature at scale. I think the most recent time that they pulled the numbers on that, it was something like 8% better if you use Smart Send time versus what you were doing before. Wow. And that's significant, right? If the low end of that is 12,000 people, right? That's 800 to 1,200 people potentially that you can get increased and opened, which means increasing clicks, which means increasing conversions. And as these people, right, I know on your guys' platform have 50,000, 100,000, millions, 10% or 8%, whatever the number is, that's huge. So that's crazy. I mean, going from a 20% open rate to a 22% open rate might sound small. It's only 10% of what it was, but that is a lot more people opening your email, especially if you can get that reliably email after email. Yeah, that's fantastic. You get all those things we mentioned about the positive engagement. You also look better to Klaviyo and all the ISPs in terms of how your engagement and your open rates look, right? Across the board, it improves a bunch of things. Dude, that's fantastic. Is there anything else that you wanted to add on the Smart Send time before we go to kind of the second tip? Yeah. So some recent work that the team has done to just make this feature even better, they added a reporting page. So kind of if you have questions about like, okay, I understand that you've run this experiment and you have selected the best time for me. I want to see what that time was. I want to see kind of what were the actual open rates in the 24 hours of the day that led you to make this decision? And are there maybe some hours of the day that I don't want to send to everyone all the time, but was there a weird peak at 5pm that I want to investigate? There's a Smart Send time reporting page that you can go out and look at all that data. And it'll also show you break down the time and the opens and the clicks and the revenue. Yeah. If anyone hasn't seen that and wants to do this, it's fantastic. We use this a lot. And that's why I really wanted to get you on here is because we're using this. Obviously, there's things that I'm learning right now too. So I love that. Yeah. Yeah. I love that page. I was not one of the people who built that page, but it's a fantastic page. That's awesome, man. Any last things on this one, or are we going to move to tip two? I think that about covers it for Smart Send time right now. There's more improvements coming in the future, but stay tuned for that. Amazing. Yeah. You crushed that one. Thank you. Tip two. Let's hear it. I'm anxious. I'm ready. Yeah. Tip two. And I think kind of, obviously, people who are on the smaller side of kind of their list size, it might be difficult to kind of swallow this idea of, well, I have to have this number of people to benefit from a certain feature. But this is one that can work for kind of anyone at any size. And it's using predicted gender. So predicted gender, pretty much what we do is we look at the first name of a recipient. So just like take the first name from the Klaviyo account. And we compare that to census data in the US. And if there's a high probability based on census data that the basically like if it's overwhelmingly that like kind of this name is associated with being male, then we return a predicted gender of male, vice versa for female. But obviously, for some names, like consider the name Lindsay, consider the name Alex, if you see that name, you really it's hard to know either way. So we also can return a predicted gender uncertain. So, you know, it just just to indicate that kind of we can't make a decision strongly enough based on just their first name, that you should actually have enough faith in it to start making decisions. And when I say making decisions, so there are good ways and there are bad ways to use predicted gender. I'll start with what I think are some of the bad ways. You don't want to use predicted gender to make stereotypes. You kind of anything that you wouldn't normally want in your marketing content. Don't use predicted gender as a reason Predicted gender is just a prediction. I mean, obviously, my name is Michael. Overwhelmingly, overwhelmingly, most people who have the name Michael are male. I've met people with the name Michael who are female. If you if you make a strong assumption that the person you're sending to is male just because their name is Michael, you could be wrong. And depending on how you message in your email, that could really come to bite you. What I've seen, so some good ways that I've seen customers use predicted gender is more subtle ways. So, for instance, what one particular example that we actually there was a blog post that Christina put out about this feature where she highlighted a few customers who would use this feature and experiments with her. And there was one a boot company where they have kind of a boot that's marketed primarily to men and a boot that's marketed primarily to women in the same email. And all they did was switch the order of this. So if it's if it's an email that's sent to someone that you believe is private, that you believe based on the first name is female, then they see the female boot first and vice versa for if they see it, they'll see them the male boot first if they're if they're predicted male. And they actually saw, I think, surprising to me number numbers of improvement just from that one small change. Another example is changing the color palette. There was kind of like some internal market research indicated or I think it probably based on just kind of sales that they noticed. They noticed that like a certain color palette was bought much more often by men. And so that was the color palette they showed in the image versus kind of a different color palette is one that they showed in the image for women. And they actually found like, again, I thought somewhat shocking improvements just from that one small change. Wow. So it sounds like here and this is kind of how we use and it sounds like you're kind of reinforcing it is it's really subtle changes, right? It's not really calling out the fact that, hey, because you're a dude, you're receiving this email for dudes, right? Or hey, because you're a female, you're receiving this email from a female. But it's really just taking the content that you have and really making the personalization in terms of person in the image, right? If a guy is the likely male, maybe you have an image of the male first, and then, you know, females vice versa, right? But you have the fallback or the failsafe, right? If you have content that's relevant to the opposite gender, in case someone is buying for someone else or in case the gender is wrong. So in the third bucket, then kind of the unknown or the uncertain bucket, would you then take that content and almost serve that content as if you weren't doing gender at all? Or how do you think about that one? Do you pick the male or the female? How do you think about that? Right, the uncertain category, it depends a little bit on the branding question. So some brands kind of, there are definitely some brands that know that they are like most of their audience is female, or most of their audience is male. So it could be the case that maybe someone just didn't want to enter their first name during checkout, or they like entered a first name that like a first name that won't have good representation in US census data. But you know that if they're shopping at your store, more likely than not, they're female. Okay, you can use that to kind of inform. So if you have a strong assumption like that, or just like a strong reason to believe that if you don't have a first name, it's probably going to be one versus the other. Obviously, you can just kind of group them with that predicted gender as well. But I think also, what you what you talked about is, is a good way to see it as like, I mean, is there a way? Is there a third option? Is there a way to kind of like it? Yeah, is there a third option? Is there a way to just like not make a strong assumption either way? But I think that also, if you if you know a lot about your demographics, then take advantage of that for sure. Awesome. So yeah, if you know your gender is overwhelmingly women, send that third bucket of likely uncertain, maybe with women variation, right? If you have a split, and you aren't sure, maybe you approach it like you typically approach campaigns today by not doing gender, right? And you send that one, and then for the male specific, male first, female second, female first, male second for women, that's awesome, man. I personally use gender. So this is cool. Yeah, actually, I mean, I would, I would not be a, I wouldn't be a good data scientist if I didn't plug the idea of A-B testing on that. So, you know, if you're curious, which one's going to do better, you already have both variations. Why not run an A-B test? And if you do an A-B test, and you have interesting results, please share it with Michael and myself. Awesome. That's sweet on gender. Anything on gender before we go to the last one? And again, really appreciate it. Right. So actually, one last thing on there. So I probably sounded a little bit clickbaity when I was like, I saw incredible improvements that I couldn't believe. To actually give some concrete numbers. We're talking in the order of like 24% better in terms of click rate. So like, you know, if you had a click rate of 5%, you know, almost, almost, you know, like 1% more people, like one percentage point more people click or something. Wow. So I don't know if this order was intentional, right? But what if basically I'm saying from tip one, figure out the time that makes sense in those census center, right? Get that eight to 10% lift in open, right? Then from here, once you get people to actually open, right, that's step one. Let's do what we can do to improve the click through, right? Whether it's 10%, 20%, 24%, right? So then overall, right, we're grabbing more opens from the first tip you gave on the smart send time. We're now grabbing more click through on the kind of pairing the variations of the content, right? So I don't know if tip three is going to be like, this is driving convergence or not, but tip one and two, I like those associations with one another. Yeah. So tip three is about customer lifetime value. So it's, it's a little bit, it's a little bit about kind of what happens a little farther down the funnel once people are starting to make purchases. So, you know, driving more purchases, you have more information about people's purchases. Now you can start getting a sense of what is their lifetime value to you. And predicted. So obviously kind of, you know, you can get an estimate of someone's customer lifetime value just by looking at how much they have purchased, but it would be ideal, you know, in a perfect world, we would have a sense of how much someone's going to purchase in the future too. And our model is an attempt to kind of compute that for you. Obviously it's, you know, every predictive model, like no predictive model is perfect, but it, ours appears to be pretty good at it based on like the, the audits that we've done on making sure that it works properly and all of that. I kind of like making sure that like we're capturing, okay, down the line, like, will this person purchase? If so, how much do we think they will? Yeah. Customer lifetime value just helps you answer questions like that. So a few ways that you can actually use customer lifetime value. Obviously, I think one value that it provides you is just having a good, like having a better understanding of who your valuable customers are, which I mean, I don't run your business. You probably have a sense of how you want to treat your more, your more valuable customers. I'm not going to tell you what that is. All I'm going to tell you is this is going to help you identify who those are. Some ideas we've seen people use are once they have a good sense, kind of once they have a, a volume of data where they have a pretty good sense of, okay, a high customer lifetime value customer kind of looks like this, like I've got, I've got a reasonably large audience of them. Then use that, like plug that into Facebook as to get a lookalike audience. Kind of, I like that because there's a nice connection to own revenue there. Kind of, it's the sense of like, we're not relying on Facebook's algorithms to help us in the same way. Like we're, we're not saying, you know, please help us do whatever you can. We're saying, we're going to tell you, like, these are the types of people that are important to us. Now find people who were like that, which I, yeah, I like that. Yeah. Great input. So on the on the automation side, on the flow side, the two kind of flows that kind of correlate or relate to increasing this, like a VIP flow and kind of like a repeat purchase nurture series. Are those kind of the two that are most in line here? Yeah, absolutely. So VIP flow, obviously a good way that you can, that you can identify who your VIPs are is based on customer lifetime value. So I'm, I think that the connection there is obvious, just that you have a good sense of like who your, who your valuable customers are, who are in your VIPs. A related one is also a potential, if you notice someone is getting near the threshold of VIP based on predicted CLV or actual CLV realized up until now, try to, try to push them over the line, let them know that they're near that VIP status. We see people use that way sometimes as well. And then, yeah, the repeat nurture series. So kind of a corollary of the customer lifetime value model is we can basically use the guts of the customer lifetime value model to actually predict the expected date of someone's next order. And again, like this isn't magic, this is a predictive model. We can't guarantee that every single prediction is going to be accurate to the day. But it's, it's performed pretty well when we've taken looks at kind of, you know, like how does this actually line up against real purchase data after we made these predictions? It's, it's lined up pretty well. And also you can customize it. So you can go in, you can build a flow, you can say, okay, on someone's expected next order, I want to remind them, or you can say, I want to prompt them to make a purchase before that point, because I know that, you know, it takes people, it can take people about a week before, like they have to start thinking about purchasing and then it takes them about a week and then about a week and then they purchase. So you can customize it like that. You can say one week before their expected date of next order, or you can say like, I'm going to, I'm going to let them kind of do their thing. I, I'm like my customers enjoy not being like bothered by things like this before a certain point. And you can say, I'm just going to prompt them if it gets to their expected date of next order and another week passes and they haven't done it yet. Because usually they, usually they renew at a pretty set schedule. That's awesome. And one thing that we do, and you probably have more data on this than I do. I can't remember the exact numbers, but what we'll do actually is we'll take a segment of expected order within the next two weeks, expected order date, and then we'll actually show that person like a pop-up or a flout specifically that segment that offers them, you know, a specific kind of special discount the next time they come back on the site in that window. And again, like from the limited data that I have and a few clients that we've been testing so far, it does seem to drive some revenue and sales. So I don't know if you know anything more on that on your end, but that kind of something that makes sense as well. Absolutely. Yeah. Using, so using signup forms, kind of targeting your signup forms to high value customers is 100%. Like that's, that's an idea that I love. And I think using CLB and using expected date of next order. So I've, I've mostly heard people doing that with CLB, the, the like expected date of next order soon segment. I love that idea. I don't think I've actually heard that exact idea before. I love it. Oh, awesome. Yeah. We're doing it with like one or two clients. I don't know the exact numbers and we're making the discount code, even though it's not dynamically unique, we're making it seem a little unique, right? So that way it seems more personal. So that's kind of how we're approaching it. We'll do, this is, this is awesome. Any kind of final words or if not, like how can people learn more about you and kind of learn more about what you're working on? Is it Twitter, is it email, is it LinkedIn? Like how do people learn more about you? Yeah. So to learn more about me I'd say the best way to get in contact with me is on my Twitter, which is a relatively new one. So I have to look up the handle real quick. Give me one moment. It is at Lawson, my last name, L-A-W-S-O-N underscore M underscore T. Send me a message on there and I'll respond. Another way, if you want to kind of follow what the Quavio data science group is doing, I have a series on Medium. We are, we just posted our third episode today of the Quavio data science podcast, which let's see, this month we were talking about the recent core maybe testing feature, kind of giving people a peek behind the curtain of what it was like to build that piece of software. As a user of it, I personally really like it, so I'm glad that you guys built it. Glad to hear that. Yeah. Well, dude, thank you so much for the time and the insight. I really, really appreciate it. Yeah, this was a lot of fun. Thank you. Yeah, of course. Cheers, bud. Have a good one. You too.