How to Analyze User Behavior Data by Betterment Product Manager

I'm currently a PM at Betterment. I used to be a PM at Amazon, and I'm happy to answer any questions about my experiences at either place. I really actually enjoyed my experience at Amazon. I think you read New York Times articles about working there, and it's like, oh boy, it sounds awful. But I actually really enjoyed it for a number of reasons. So really two topics I'm going to touch on today. One is just measuring the business. So as a product manager, you focus a lot of your effort on the business side of the company you work at. I think about it as like sort of the macro data. And then measuring a product change. So your day-to-day, you're making changes to the product. You're trying to improve it. And that's more of the micromanaging of being a product manager. So yeah, data, like the way I think about data, it's just simply information used to make decisions. It's not anything more than that. I think some people try to make it out to be something extremely complicated. There are data scientists, even. And in a lot of ways, I just think about data as just the information used to make a decision day-to-day. And I've got a funny example from Amazon Folklore about this. So this is Amazon circa like 2012, I think. And one thing Amazon does really well is they split test everything. They call it a web lab. Other companies call it something different. But Amazon split tests everything. And back then, they were really pushing Kindle as well as like fashion. And one of the funny stories came out of the fashion team. And it may not be true, but I love the story. They were running a split test where there are two slots on Amazon's homepage. And they ran a split test with picking up both slots. Basically, the bottom slot here has something and they put something in the top slot as well. And they split tested that against what you see here. And what they observed is that if you put fashion in both of these slots, you sell a lot more clothing, but you also sell a lot more other things as well. It actually performed much better than what you see here. And because this part of the product of Amazon is so high impact, a lot of the things that you sort of have to prove yourself out to Jeff Bezos. So the fashion team went to Jeff for this information. And Jeff was like, thank you so much for doing the split test. I now know how much I'm spending by putting Kindle on that slot. So a lot of misconceptions around split tests, especially, this is what you do. You run a split test, split out the numbers, you do what the numbers say. That's not exactly how it's always used. I mean, for me, there are things like that where it's just like, well, I'll just do what the numbers say but when it comes to like strategic things, you may not necessarily do what the split test says. So going back to data, I think about it as sort of like, what should we be working on and what's going on with what we did? It's like this constant, you know, you hear iteration but it really is constant iteration. You need to always be looking at what's going on with your product. And these are the two things that I look at basically every day. There's just so much data out there. Like no matter if you're a small company and a big company, like Amazon, it took me three hours to write a SQL query. At Betterment, it takes me four seconds. And when people have, people are like, oh, I wrote such a hard query that it took 30 seconds to run. I'm like, we don't have, we have tons of data but like we're tiny. Amazon has tons of data and they're huge. And no matter how small or big your company is, you've got lots of data these days. And so I think about obviously the two types of data, your qualitative data, customer feedback, user research, user tests, your company strategy. That's an input to how you make decisions. What your competitors are doing. That's something that coming from the Amazon school of product, I don't think about it as much, but I know it's super important in a lot of ways. And solely opinions, a small company, especially people have strong opinions about what the product should be. Personal opinions, like this is, I love my personal opinions. So I definitely think about that when I think about what we should do for the product perspective. And of course, I want to say this is my favorite, despite the fact that I know that there's not a ton of numbers in this presentation. So things like traffic, feature uses, business metrics, clicks, acquisition attrition, anything that has a number, essentially. That's quantitative stuff. So let's switch and talk about it for the macro side of measuring. So what I do, I focus on the data that helps me best understand the input to the business and keep an eye on the outputs. And this notion of inputs and outputs, I find extremely valuable, but it's a little nuanced. So I'm gonna explain a little bit about what an input and an output is. So inputs, they're the things that you can control, but they're also the things that you believe drive growth, whatever growth may be for you. It's kind of like a strategy. They're switchier, not quite as sort of comfortable to work with, maybe. And the outputs are to the opposite. So they're harder to control directly, but they're the growth indicators, things like revenue. Very easy to measure, less squishy, but more comfortable. Like these are the things, these are the business terms that people use to measure the performance of the company. And so when you're talking to the management, especially, they care a lot about the output. Just some examples of inputs and outputs. Convenience, how the heck do you measure convenience? I need to get creative about how that data that you use to measure convenience. Traffic, that's easier to measure as an input. Usability, what is usability? How do you quantify usability? Featured option, again, that's another one that's reasonably easy to measure. It is an input. Customer experience, that's super important. And also kind of weird and squishy to measure. Outputs, like for betterment, AUM is extremely important. That's assets under management, that's how we make money. Customer acquisition, again, that's a growth indicator. Profit, sales, revenue, just normal things you think about in the financial statement essentially are types of output. And so when you think about inputs and outputs, how do I use it in my day-to-day? So for example, at betterment, this is an example. I believe that convenience, which I measure as an input, drives AUM, which is an output. So betterment, we're a financial services company. We manage your money for you. In a lot of ways, you trust us with your life savings. And one of the ways people invest their money is with 401ks, IRAs, and you move 401ks and IRAs between providers in a way, it's called a rollover. You've probably heard of it, but they're really a pain in the butt. But they're huge. Like I said, they're the people's life savings is where all their money is. So when you think about acquiring a customer at betterment, one of the ways you think about it is, well, how do we get more rollovers? An example of why they're not very convenient. So imagine you're rolling over an IRA from Fidelity to betterment, as an example. Currently in the financial world, you have to get a piece of paper that could go to the bank. You have to get what's called a medallion signature, which is essentially the bank guaranteeing the transaction. You have to mail the piece of paper to Fidelity. Fidelity then processes it, whatever they do. They then liquidate all of your things in your Fidelity account and mail us a check to Dallas where we have like an electronic place that processes them. And then our ops team on 23rd Street electronically marks the rollover as complete and deposit the check. Super, like, it's ridiculous. It's unbelievably ridiculous. So it's clearly a place where convenience is low-hanging fruit. So I focus a lot of my efforts there at betterment. And one of the ways I think about convenience in terms of rollovers is the idea of a perfect rollover. What is a perfect rollover? And a perfect rollover percentage. So, you know, we do a hundred rollovers. How many of those rollovers are perfect? And what I want to measure is a rollover is perfect if it takes less than 14 days. So that's already a long time, but it is what it is. And the customer doesn't call us between seven days prior and seven days after the rollover finishes. So the idea here is we want it to complete quickly and we don't want the customer to call us. We don't want, we believe if the customers would call us, like, they're like scared about something. They see the, typically when they call us, they see the balance in their Fidelity account go to zero. They don't see the balance in their betterment account go to whatever balance it should be. And it's just like zero and zero and they get really concerned. So how do we think about driving the perfect rollover percentage? Because we believe if we drive convenience as measured by the perfect rollover percentage, we'll drive AUM and growth. That's how an input relates to the outputs. Right now, you wouldn't believe it, but the perfect rollover percentage of betterment is between 13 and 14%. Most rollovers take between 20 and 30 days. And so this is clear in the area where we'd like to put a lot of effort in years. Like I said, focusing on inputs is kind of uncomfortable. It's not typically the way people think about driving business, but it's super important. It's really hard to build product around increasing revenue. You can't build products to increase revenue. You can build products to increase the number of things your site is selling. You can't build products to increase revenue directly. So I think about inputs as really the things that lie between the success of your business and the products that you can build. Typically when I prioritize, for example, I prioritize around the inputs, but I also sort of estimate the output. So this is measuring a product gain. This is related to what I talked about at the beginning with that split test. And I think split testing, if you're not familiar with it, split test is essentially comparing two versions. So version A versus version B, and trying to answer the question, which is better? And better is one of those things, like what does better mean? Typically you have a hypothesis version. If we have a green button instead of a blue button, we will get more signup, for example. That's a typical basic split test. And you hear these sort of urban legends around, we change the button color and increase signup 30% or something like that. And I think those things are sort of going away. I think largely because people are getting customers and people are learning or building habits. So one of the fascinating things is if you go to basically every e-commerce site, the add to cart button is on the right side of the page. And it's on the right side of the page because everyone learned that the add to cart button is on the right side of the page, because they shop on Amazon. And the add to cart button is the right side of Amazon. So over time, I find split testing harder and harder to get results from because people have developed habits. And yeah, better is a little, it's tricky. And as I said at the beginning, split test really helps you understand the business impact of a change in order to make a decision. So Jeff Bezos, when he prioritized Hindle on the slot on the Hindle on home page over fashion was making a decision based on data. And the data was the split test results. Split tests aren't the things that tell you what to do. They're just input to what you decide to do at the end of the day. And so I think about running split tests. I only think about running split tests on things that are reversible. And what I mean by that is if you're not going to, if there's a possibility, if there's no possibility that you will ever go back to the previous version, it doesn't, I mean, split test isn't necessarily the right thing to do. And that really is related to large like architecture changes on the site. Right now, better, we're doing a bunch of large changes. We're not split testing it. We're not going to go back. So the only thing a split test can tell us is what, A, what we are giving up or what we're gaining from doing the change. The other thing is a few changes. So running a split test, you can't do a split test where there's like 14 things changing on a particular experience because it's not a split test. You're, it's like, it's not A, B, it's A and whatever, B through Z. I don't know. It's really hard to understand like what the actual change was that influenced the change in customer behavior. It's hard to learn from that. And the third thing is clear measurement. If you can't measure the thing, you're not going to run an effective split test. And I'll talk a little bit about that one in a second. A fourth thing that I didn't put on here that I'm thinking about as I'm talking is, I think about a lot of betterment, for example, is we have a customer support team and they have to help our customers. So when a customer calls in and says, how do I start my rollover? The customer support person wants to do their best to help them out. So they'll say things like, well, first you go to the transfer page and you click the green start rollover button. And we're all like, if we want to split test, changing where the rollover button is on the site or change the color of the button or changing what the button says, it makes customer support really challenging. So one of the things that we're working on right now is better equipping our customer support team to react to what a customer may be seeing on the site. It's not a great experience if you can't help the customer in a timely fashion or the customer support person gets flustered because they're doing this all day. It can't take a minute or two minutes to figure out what the customer is seeing. They really have to know. So we're adding sort of more real-time information on our customer support tool to give customer support people more information about what that particular customer's seeing at any particular time. That enables us to do more split tests because we've chosen to not split test things because customer support couldn't handle the change. So I'll give you a good example, and you're not gonna be able to see this, so I apologize. It's really small, but this is something that we're doing right now. When you withdraw money from Betterment, there's this confirmation screen. And one of the things that we're trying out is showing another message. And so half of our users are seeing this confirmation screen. Half of our users are seeing this confirmation screen. And what we're doing is we're showing a message like basically if your satisfaction is guaranteed, if you're not satisfied, click here and you'll get 90 days free. And we wanna see if that influences customer attrition or not. And so it's very clear. So it's a simple change. It's highly measurable. And it's only one thing changing. It's this box versus no box. So this is an example of a split test that is pretty easy to measure. And it'll give us more information. What we'll do with this information isn't just simply say, well, by putting it here, it's impactful in this particular context. What we'll do is we'll learn that we can put it in other places too. So imagine if this split test turns out that fewer people withdraw money by seeing this. We'll think about putting it in the footer of the page, on every page on the site, for example. It informs more than just this particular. And since it informs bigger, larger decisions as well. Sure, that's a great question. So we changed our fee structure at the end of January this year. And as part of that decision, we did a ton of just sort of research with customers to understand like what's important to them, how they would react to certain changes and understanding like how they think about their money. And one of the things that we learned was that by showing a satisfaction guarantee, we sort of build trust. It's sort of, they're more comfortable with us because it's less risky. We're a startup, obviously, where you manage your entire life savings and that's risky in and of itself. So this sort of helps with the risk side of the customer becoming, giving their money to us. This is the first time we're putting it actually in the experience. What we've normally done was put it in like signup or like on our brochure site where people are sort of evaluating whether or not they want to use this or not. So we're seeing whether or not, we believe it influences acquisition. We're seeing if it influences efficiency as well. So that's sort of where the idea came from. I'm not sure if you guys have read the four-hour work we put in there. Tim Ferriss talks about, if you say the satisfaction is guaranteed, no one is gonna actually take you on it, but whatever reason it increases, increases trust and trust increases the output, which is revenue. Yeah, yeah. So this is an example of a very simple split test that we're running right now. Honestly, we're not seeing a huge impact either way. So we may remove this and we may not put it anywhere else, but it's something that we definitely felt we could split test to understand the impact. Yep. Yeah, yeah. So the question was, so how long do you run tests? And I'll actually touch on that in a second, but it depends. It depends on how confident you wanna be in the results of the test. So, and it depends on the number of people that see it, and it depends on what the expected impact of the test is. So I've got a slide on that in a second. Yeah, so the question was, do you look at the type of people that saw what version? The answer is yes. So there's obviously an overall impact, but you wanna dive in deeper to understand how certain types of folks are behaving versus other types of folks. So it may be true that a 65-year-old person, this really resonates with, versus a 25-year-old that doesn't particularly resonate with or for all we know, the person actually needs their money or to pay taxes or something or to buy a house. And at the very, on the next step, actually we asked them why they're doing the withdrawal. So you can figure that out. I think that's super, super important because if you just take the output at face value, there's probably a lot of interesting information that gets lost, that you can use in other places in your product. So I'm gonna go to, go ahead. Yeah, we do, but I don't necessarily work with them, but. Interesting. So the question was, if sales is doing a split test and product is doing a split test, do you look at them to understand, like, do they collide, I guess? Can you give me an example of a sales split test? Honestly, I haven't worked with them on the sales team. Yeah. Got it, got it, got it. So the question, just to repeat that is, the sales team could be running a test to reduce attrition and you could be running a test to reduce attrition. How do you know which one is the right one? And so currently at Betterment, we don't have a good way to isolate those things. At Amazon, we just had so much traffic that an individual person would get a, like, they would get a very unique experience but we would avoid them. So I guess what I would say is, unless you have a super sophisticated split testing tool, don't run things at the same time. If you really want to understand the impact of those things. Which I mean, it's not a great answer, but I think it's the simplest solution. So this is a bad example. This is something I did while I was at Amazon. It's like simultaneously the thing I'm most proud of, but also the thing that I'm least proud of, and I'll tell you why. So at Amazon, I was responsible for used stuff, like used Nintendos, for example. And one of the things that we were hearing from customers is why would I buy a used thing without knowing what condition it was in? And additionally, Amazon was a bookstore and used books are huge. Used books are really huge. But as Amazon sort of matured and started selling more things other than books, they didn't really change this page at all. They sort of kept it the same. And when you're buying a used book, the condition, like used, very good, is fine for you. But when you're buying a Nintendo, for example, that's 30 years old, what does used very good mean? And there are ways to put sort of descriptions here about like what it means. But ultimately, the best way to know what condition the product's in is a picture of it. So one of the things that I worked on was giving our sellers, the people that were selling these products, the ability to get a picture, describe the thing that they were selling. So let's see here. These two listings have a picture. This is, when you click on it, you'll see this, which is great. Like, this is amazing. You can see exactly what it looks like. There's even shrink wrap on the cartridge. This is great. It would give a customer way more confidence in what they're buying. But the problem with this was, how do you do a split test? It's so hard to do a split test here for a couple of reasons. One, you can't simply say, all right, for half the customers, we're gonna show the pictures, and for half the customers, we're not gonna show the pictures. The reason being, sellers will get so upset. They spent all this effort to put pictures on your website. You need to be showing it to everybody. So A, if we could have done, we could have said, you know what? Sorry, sellers, we're going to do a split test, and we'll have to deal with it. But that wasn't, it's not a palatable solution. The other reason why a split test here is so hard is, what if this one had a picture, and that one had a picture, or that one, that one, and that one did? How do you split test? Because the customers are not seeing the same thing. If you split test across every one of these pages, there's millions of these different pages, and they all have, they may have different iterations or combinations of pictures or not pictures. All of the pictures may be different. It's just not a good split test. It's impossible to get a great A, B. There's not clear control, and there's a clear control, but there's not a clear treatment. So I left Amazon without knowing what the impact of this was for Amazon. We tried, very challenging, but this is the thing that I'm most proud of about working at Amazon, which is, I think, really changed how people buy used and collectible things on Amazon. Not that, you've never, probably no one has bought something used or collectible on Amazon, but lots and lots of people do. So I'm really excited about that. But it was a bad split test. Let me talk a little bit about how measures split test. So you hear a lot about this mythical p-value. If the p-value is less than .05, then it's significant. And that, all it says is, all the p-value says, if it's less than .5, is A is different than B. It's all it says. And you can configure it. Like, if you want to be 95% sure that A is different than B, you can use .05. If you want to be 99%, use .01. All it says is A is different than B. I don't really like p-values. I mean, I use them, but what I think about most are confidence intervals. So confidence intervals, they tell you where the actual number is. So where is the actual real-life impact of your change? And so here's an example of a split test. These are the only numbers you'll see in my presentation. But imagine the p-value is .4. Version A, the mean difference was plus .2%. Version B, the mean difference was plus 3%. But from here to here, this is a 95% confidence interval, let's say. 95% confidence interval. And it's really hard to say, well, I mean, the confidence interval, basically, the real number lies somewhere between these two, somewhere here. Like, the real number for version B is somewhere here. We don't know where. It's just, we know it's between plus eight and minus six. The real number for A is between plus 4% and minus 3%. The mean is plus 2%, but the real value is somewhere between negative three and positive four. And you'll look at this, and I think people have the tendency to say, oh, look, version B, it outperformed version A by 1%. And you can't really say that. All you can really say is, version A, the change was negative 3% to positive 4%. Version B, the change was negative 6% to positive 8%. Which one do we choose? You have to just sort of use your judgment. Obviously, if you run it for more time, these become much narrower. And so when you ask yourself, well, how long should I run this script at? The answer is, well, how wide are you comfortable, how comfortable are you with this distance between the low end and the high end? So at smaller companies, we don't have a ton of data. These tend to be really wide. So you'll have things like this. So this is definitely something that you'll see at Betterment, for example. You only have hundreds, or you can have tens of thousands of people in treatment and control, you'll have something like this. The perfect, the dream, when you see this, this is the dream. The low value of version B is greater than the highest value of version A. You definitely know version B is better than version A. This is the dream. This is sort of the standard Amazon uses. But they can because they have so much data. This is the dream. It never happens. It never happens outside of large, huge data sets. But when you think about how you talk about a split test, I'll talk about, well, version A was negative 3%, positive 4%, version B was negative 6%, positive 8%. I really shy away from, well, version B was 3% and version A was 2%. That's not necessarily true. We don't know what the real value is. We just know that's between those two ranges. It's largely a judgment call, honestly. So you sort of, maybe the answer is, well, let's run it for a little longer and see what happens. Ultimately, it's a business decision. You have to use judgment. And I think that's sort of the, sometimes it's the trap people fall into, is which one do you choose? The data's not telling you the answer. And the answer is, sort of, the result of your judgment. That's a great question. So the question was, do you complement this with user interviews and other sort of qualitative data? And the answer is yes. So an example that I can think of that we've done that is, I wish I had a screenshot, but I don't. There's a button on the summary page of our product that enables you to start a rollover. And we were thinking about removing that button. And part of that choice was around, well, we did qualitative research that said, the user tests. And the user tests showed that if you ask a customer to figure out how to rollover without showing the button, everyone figured out how to rollover. And I think that's a reasonable outcome. If you're trying to move your life savings, you'll figure out how to move your life savings. When we ran the test, we saw that actually less people did rollover. So the results were something like this. Confidence intervals were all over the place. We ultimately decided against removing the button. We added it back. But that's an example of sort of putting qualitative together with quantitative data to make a decision. Ultimately, it's a judgment call. No, no. At the end here, I've got some hot takes. And one of my hot takes is, if a change that you make requires substantial analysis, it's probably not particularly impactful. Big changes are obvious. I think that's sort of one of my hot takes. Yeah. So the question was, what's the timeframe on testing and when you make a decision? What I like to do is sort of before starting a test, I look at, well, how many people do we think are gonna see it every day? And based on how many people are gonna see it every day, that'll sort of tell us, we should expect to see X sort of confidence by 20 days. So I sort of set a number of days before we even test it because I should know reasonably well what the confidence intervals will be like in the future, just based on how much traffic we expect to see to that particular experience. So, again, I think it depends, which is the worst answer, but I think one of the points I'm trying to make here is data doesn't tell you exactly what to do. It is just an input to your decision making. Yeah. So the question was, I guess the question was, if you work at a place that doesn't have a ton of customers, is split testing valuable? Yes, especially if it's low effort. If it's super low effort, it's just another input to your decision making. Obviously, you have to sort of note the caveats, but if you're comfortable with the caveats, sure, there's no downside. And the only downside is, going back to my customer support example, if it causes customer confusion, then that may be a downside and you want to avoid the split test. Yeah. So the question was, what tool do we need that does Betterment use to track this stuff? We actually have our own tool. So we built our own tool internally. It's open source. It's called Test Track on GitHub. So it's really early. It's not a fully functioning or fully featured tool. Honestly, I personally run a lot of SQL myself and do a lot of confidence interval calculations myself. At Amazon, we had an amazing tool. I wish we had it at Betterment, but we just don't. It was obviously not open source and it's proprietary. So yeah, completely internal. So the question was, have we used Optimizely or like split.io? The answer is no. My experience has been an Amazon at Betterment and at Amazon, we had our own thing at Betterment. And we've always had our own thing as well. We haven't really, we're starting to explore third-party tools to help us, but we haven't gotten there yet. The question was, why do we build our own tools? I think that's a good question. I think it was largely due to what our needs were and there weren't really, like we looked at other third-party tools, but there wasn't really something that really fit our specific needs, one. I think the second thing, the second reason was our engineers are really interested in doing it themselves. And when you have a group of excited engineers, you don't want to tell them no, especially when you think they can do a great job. So that was sort of the second reason why we wanted to open source something. We wanted to pick the problem that we had and we went at it. Yeah, so the question was, do we have an analytics team? Yes, we have a very, well, we have like five or six people on the analytics team. They spend a lot of their day not necessarily analyzing split tests, but really helping us understand how sort of the outputs of the business and why the outputs are what they are. So when you think about a company like Betterment, we rely on investor money that, people are fickle, they react to things. The market is a huge influencer of whether or not people give us money or not. So they work on the really hard problems that you can't, that I can't, I can run SQL and like calculate confidence intervals, but like Monte Carlo simulations, that's what they do. That's not something that's sort of within my jurisdiction or my ability. Yeah. Yeah, yeah. Yeah, so the question was specifically about rollovers. I love rollovers, I love getting rollover questions. And the question was, have you tried to show customers where they are in the process? And the answer is yes. So we're doing things to help us there. Unfortunately, a lot of the process is in their own hands. So they have to print a document and once they have to print the document, they have to go to the post office, they get a stamp, go to the bank. We don't have any visibility into what they're doing. So we don't know where they are in that process. And then once they sort of mail the letter or the letter to Fidelity or whoever, we don't have any visibility into what Fidelity is doing either. So unfortunately, we can't give a ton of status updates in the current world. We're working on ways to do it. There are solutions that we're really excited about. Yeah, that was PDF files. Yeah, yeah, yeah. And so rollovers, there's two types, not to go down the rollover rabbit hole, but there are rollovers from like an employer sponsored plan, which is like a 401k. And there's a rollover from another IRA. So imagine you have an IRA at Fidelity or whatever. The IRA to IRA rollovers, there's ways to automate it. The 401k to IRA rollover can't be automated. So we essentially, we're doing things where we give customers the option to schedule a call with us and we'll call your provider with you. So that, and the provider, when we're on the phone with them, we'll ask them, hey, can you give us a confirmation date on when we'll receive the check? And they'll tell us. But that's not something that a customer will know to ask. So unfortunately, there's not a good way to automate half of all rollovers. Sorry. Any other sort of questions? I've got one slide left and just my update. Feel free to, yeah. Yeah, good question. So the question was, like abstractly, does the product team do the analysis on the split test? The answer is yes, but if we need help, we definitely call in the experts on doing data analysis. It's a combination of both, I guess. Yeah, yeah. The macro stuff. So our data team does a lot of macro stuff. They support our finance and strategy team. I also do it on the side. I find it super interesting. I think there's lots of valuable insights you can find in the macro side that if you don't have any curiosity about, you're never gonna discover them. Plus, as a product person, as sort of a business person, you can understand, you know where you should see some interesting things in the data. So I really, really enjoy that side, actually. I spend a lot of time on the macro, what happened. That was actually something that I picked up with Amazon. You're sort of expected to do it yourself at Amazon in partnership with the finance team, but it's essentially what you're expected to do. Yeah. The question. So the question was, what kind of data do you use to figure out what new features to build? And I use a number of different sources. So I work very closely with our customer support team. They are sort of, they know what customers are having problems. They know exactly what customers are having problems with. So I look to them. They're more than willing to come to me with problems. So that's one input. I think a lot about, personally, I think about what problems are customers having that they're sort of not talking about. And you can do that in one of two ways. One way is sort of just thinking about the problems you're having with the product and dog fooding and things like that. The other way is your data analysis. So an example of something that customers weren't really telling us, but I could see was a problem is around sort of manual, manually giving us money. So when you're a customer, you give us money and you can give us money automatically by setting up an auto deposit of a certain amount, or you can give us money manually by just giving us money. And what we were seeing was on the 1st and the 15th of every month, we would get a ton of manual deposits. We would also get a ton of auto deposits too, like a ton of manual deposits. And we were even seeing people that had auto deposits were giving us manual deposits on top of that. So the question is, well, we're clearly sort of at the 15th of effort on the 1st and 15th of the month. How do we like remove that effort from them and automate and make their lives more convenient? So that is an example of how this product called Smart Deposit came about, where essentially we check your bank balance every day. And you can tell us like, I only want $5,000 in my bank account at all times. And if we see that your bank balance is over $5,000, we will initiate a deposit for the right. So automating that extra thing that customers are doing every 1st and 15th of the month. That's an example of looking at the data, understanding what customers need. Got it. So the question was, in thinking about customer feedback, when, how do you sort of filter the noise from the real problems? And that's a great question. One of the fun things about the customer support feedback, especially is they're only getting the problem. Like they are getting the edge cases and like, how do you, what do you deal with all the edge? How do you deal with all the edge cases? And it's interesting though, is as you get bigger, edge cases stop being edge cases. They become sort of like normal course of business problems. And so I don't have a good answer to that other than you sort of get a feel for how much effort your engineering team is putting into solving the problem. You get a feel for how many customers are calling about the problem. I don't think there's a right answer. I think sometimes you can, there's tangible business benefits to solving the problem, but I wish I had a perfect answer. I just don't. It's sort of a feel. Good question. So, and the question was for a textbook product manager, what are the skillsets? I think, yeah. And what's interesting is, I don't know if there is a textbook product manager role. Every company goes about product management very differently. So my role changed dramatically between the Amazon and better. So I think maybe the number one thing is flexibility. I'm able to sort of adapt. Yeah, that's one. For me personally, I think number two is empathy with the customer. Customer focus is so important. That could just be me. I see a lot of folks, the first thing they do is say, what are the competitors doing? And my belief is if everyone is just looking at what everyone else is doing, it'll average itself out and no one will be better than anyone else. If you focus on what the customer problems are, no matter what the competitors are doing, if you're solving the customer problems the best way, you're going to succeed. So I think the second thing is this strong customer empathy. And then third for me would be creativity. I think that's really, really important because in a world like, as a product manager, you have constraints. And I think the way you get around constraints is by being extremely creative. So creativity is really important. And then I would say fourth, being super comfortable with numbers, like being curious. If you're not, you're not going to be able to find the nuggets and the data that'll help you decide what to do. Yeah. So the question was, how do you prioritize what to do? Essentially, especially as a smaller company. And you'll get, I mean, there's tons of different approaches to prioritization. I think for me, there's just still so many small, like low-hanging fruit. There's just so many things that we can do that we know will improve the customer experience. The reason why I focus on rollover is that's where most of our customer's money is. And there's, we know that, well, for every 100 people that tell us they want to give us their IRA, 60 people are. That's 40 people that really get lost in the process. There's just like clear money on the table. So for me, at least, that's the first thing I looked at, is like, are there clear opportunities? And then in a lot of ways, there are so many strategic things that you need to do as well. One of the things that I sort of view like strategic as the things that we, we have no data to support, but this is the thing that we think we should do now. And that's essentially, that's another sort of way to prioritize. Like, this is a priority, we're just going to do it. Yeah, they just didn't finish. At that point, it's sort of, they have the paperwork. It's just up to them to do it. And yeah, exactly. It's like, it's clear opportunity. Yeah, good question. So the question was, how do you manage split tests with longer release cycles? And I don't know if I have a good answer for that. You know, we do continuous deployment. So like every day we do a split test. And perhaps the answer is you have to prioritize your split tests. And plan them well ahead of time. It's harder to iterate quickly, but that doesn't mean you can't learn from a split test. So yeah, we do, we definitely do. The question was, do we do them pre-certified? So yeah, so we use InVision. We heavily rely on the testing.com to help us with understanding sort of how someone would use this if it were actually in existence. I think our challenge is, with the prototype or with InVision testing.com, especially in our business, context is so important. Someone that's sort of being asked to do something because we could've asked the question, hey, how would you do a rollover? They have a different mindset than someone that's like, I want to give you my life savings. And context matters a lot, especially when you add instructions or guidance within the product. So not a great answer, but I think it's still possible. One question? Oh, I know the screen. I know the screen is gone. There's nothing left. Good, okay, go ahead. Good question. So the question was, have we ever gone back and retested something? I haven't personally. That's not something that I've done. I've heard of it. There's no reason to not. One of the strategies that people sometimes use is, it's not an A-B test, but like an A-A-B test. That way you sort of get a sense of how much variation is in, like if you just compare A versus A, how much variation do you see there? You should see none. But you end up seeing something. And that's sort of a way to sort of do it, I guess. But I haven't personally done it. There's no reason to not. Go ahead, yeah. Question, the answer was, are there any like uber metrics, uber, not the right word. Are there any like super important metrics that we look at? And for us, the important metric is net deposits. That's sort of like the output metric that we look at. Like I'm trying to sort of inject some very input focused prioritization and like KR, hopefully in the OPR model, that are focused on the inputs. That's what we did. We focused exclusively on the inputs. We didn't think about the outputs. Like we did think about the outputs, but not, if we weren't hitting the inputs, it didn't matter what the outputs were. Yeah. Yeah, sure. So the question was, what other teams do you interact with other than the customer support team? And what other teams were you thinking about? Yeah, so like as a product person at Betterment, I sit, it's super traditional in the sense that I work with a designer and our engineering team to figure out what we should do. Honestly, the engineers are amazing at picking out the things that just don't make sense. The things that are out of whack from the logical perspective. And typically my belief is that if something's really complicated to engineer, it's probably really complicated, it's a pretty complicated customer experience. So whenever they say, oh, this is just like, this is very complicated. Maybe we should think about making it less complicated. So definitely work very closely with the engineering team on things, opportunities. Work very closely with the design team on, they understand how people use our product more intimately than I do and strategies you can use to make it more usable. So they have a large input as well. Other teams, especially at Betterment, for example, are we have an investing team. They have beliefs on what information we should show to customers on, to sort of communicate with the performance of their investments. So they also are a big stakeholder. And I think the biggest stakeholder that sort of outside of the executive team, and they have strong opinions, especially when your founder is still there. So I try to get input from everywhere. Definitely don't exclude anybody. So the question was, how did my transition from Amazon to Betterment impact my pace of growth? Skill set, yeah. So whenever I describe my transition from Amazon to Betterment, I talk mostly about the interpersonal aspects of it. So at Amazon, you're super focused on customer, you're super focused on your slice of Amazon. You're given full ownership over it. You can essentially do whatever you want within the resources you were given. At Betterment, though, the interesting thing that I observed was people, I was the 60th or something employee at Betterment. So there were 59 people that took a bigger risk than me at joining a small company. And they have very, they're very passionate about the product. So I think about what I transitioned, what things I had to focus mostly on. I had a ton of data at Amazon. I didn't have as much data at Betterment. So I didn't focus as much, like my learning wasn't on the data side as much as it was on having the interpersonal side of being a product manager. That's something I didn't have at all at Amazon, maybe very little. So I think that's what my learning continued. It's just I was learning, I'm learning different things.