How to Build a Data Analytics Strategy by EA's Director of Product
3 Video Views·Oct 2, 2024
Building an analytics strategy is crucial for every product manager. How do you build and implement an effective product analytics strategy that quantifies and drives product success and iteration?
Our speaker tonight, we're super excited to have him.
Bertram Chan is currently the Director of Product at EA.
He has a ton of experience working as a product manager,
and I don't want to step on your toes,
so Bertram, I'll let you do an intro to yourself as well.
But thank you so much for coming out tonight.
Thank you. Thanks for having us. Thanks very much.
Thanks very much, guys. As Laura mentioned,
so I'm Senior Director of Product,
Head of Product Management Team for our data platform.
So that's the team that actually
handles and builds out all the infrastructure
to do all of our player behavioral data.
So anything from logging in to scoring a goal in FIFA,
scoring a touchdown in Madden,
all of that data, we work with the game teams
to define what that is semantically,
ingest it, store it, process it,
make it available for analysts,
make it available for marketers,
and then also build data science,
machine learning, et cetera, tools around it.
So, I mean, the goal tonight is to understand
and really talk about, okay,
at a high level conceptually, as an introduction,
what does all that entail, right?
But first, I'd actually like to get a show of hands.
I mean, how many people right now
are active product managers?
Okay, how many people are just,
are engineers, analysts, interested folks?
Okay, got it.
So this is a high level thing.
If you're expecting me to break down retention
or funnel analysis or all that stuff,
that's really not this conversation,
but we can totally talk about it.
I wanna make this a lot more conversational.
Ooh, where's the, do we have the first slide?
We could have the first slide.
Okay, yeah, there was a, nevermind, don't worry about it.
Oh, good.
The first slide was actually a great illustration
of a product that went wrong,
and I'll just describe it because the story goes,
and there was a map.
The map actually has California as an island, right?
Now, that was created because of only a very few data points
in the 16th century of one person driving,
coming down from the Straits of San Juan in Seattle.
Now, the person coming up from Baja,
they stopped before they got to the end,
and turned around and went back to Spain, right?
That map was used for 300 years.
Now, I would, as a product manager,
I would say that's a bad product
because you use two data points to extrapolate
what happened throughout the rest of,
like, for 300 years in most of Europe,
what was the benchmark for how you actually got around
and thought about California, right?
And that really goes to the point,
the heart of it, what does it mean to be data-driven?
And this is where I'm gonna ask you guys.
For the next few minutes,
please take some time to think about it.
Open out, cry, what does it mean to be data-driven, right?
Go ahead.
Well, first, before you dive into data,
you gotta think about what are you looking for,
what's your goal?
And then go out there and get the data
and see if it actually works for you.
Yep, I suppose, yeah.
Let's say it's like intuition,
but it's, you know, that's people get,
people are driven by intuition of what they feel.
I think if data-driven is,
now, what are the numbers, what do they say?
Okay, data-driven versus the numbers, yep.
Sorry, say again?
It sounds like we're taking data-driven.
Uh-huh.
Data-driven versus data-driven.
Yep, mm-hmm, others, other folks?
Yep, other, yep, uh-huh.
Okay, so when you say tests, what do you mean,
can you talk, expand on that a little bit?
I guess that's because,
there are a whole lot of people,
they just visualize things,
it's not true.
Yep.
Okay, uh-huh, others?
Yep, excellent.
Other folks?
We're getting the right answer.
Okay, all right, others?
Yep.
Yeah, everything is going to be stationary.
It involves the technology.
Mm-hmm.
And changing pace.
Okay, others?
Anyone else?
All right, great, this is actually a great list,
and we're gonna cover a lot of those topics
during this chat.
So before I get into the chat,
really gonna give you a little bit more detail about me.
So for anyone who's an aspiring product manager
who is non-technical, I'm non-technical.
So my bachelor's degree is in political science,
I have an MBA, I came from financing consulting, right?
The thing about data,
and so I head up the data platform,
which is all, as I mentioned, behavioral data for EA.
Data is a language, just like accounting,
just like English that I'm speaking to you right now.
And actually, how you define that,
and then also the relationships you draw,
it's essentially a value in a spreadsheet,
just like Excel, right?
Now, that's different than actually building credibility
with your stakeholders,
whether that's your customer,
whether that's your engineer, your designer,
your VP, or whomever.
And that also is a different language
in terms of how you actually use data to talk about
and make the points you're talking about, right?
So some of the stuff that my team,
so I lead a team of six product managers,
we work with a team of about 100 engineers.
We create things like recommendation systems,
experimentation systems, real-time targeting
and segmentation tools for our players,
so that our marketers will use,
and also multi-channel messaging.
So if you think about our games,
how many people have played an EA game in the past?
Okay, great.
The times when you receive five emails
of the same promotion from us,
that's a bad outcome,
and that's why we designed these systems, right?
Or to say, hey, yeah, you play The Sims,
if you're familiar with The Sims,
but oh, go buy Battlefield,
which is one of our first-person shooter games,
and you don't like shooters,
that's also a bad outcome,
because your player experience is degraded
by the fact that we don't know who you are to us, right?
So that's really the systems we focus on.
Now, before I worked on this,
I actually joined EA through Playfish,
which is a social games company,
a few years ago, it was on Facebook.
A lot of games there,
and I was both marketing as well as product,
because I was actually working with them
to define the features that went into the games
that helped drive a marketing piano.
So what are we talking about today?
So yes, we are talking about, number one, defining data,
right?
So basically, what are we talking about?
How are we being specific about what we mean by data, right?
The next is, okay, who are you talking to?
Who is your customer, not only in the end-user sense,
but also in the organization
as you're talking to different people, right?
What is important, right?
And that's, you know, all of these questions
are very basic product questions.
You can actually write a PR,
you can use these same exact questions to write a PRD.
We're gonna talk about it from the context of data,
but really, what is important when you're talking about data?
What could go wrong?
And then just kind of some wrap-up and conclusions, right?
Throughout the entire chat,
please feel free to ask questions as we go,
and we can also use the whiteboard for illustrations,
examples, and so on and so forth, all right?
Okay, so first, measuring DAU.
This is just a real quick example.
By the way, these are all war stories.
They're real examples.
I can give you all the background on it.
Like, names have been changed
and places have been changed to protect the innocent,
but this is all real stuff.
So three different levels of DAU.
Any, look at it, and just throw out random ideas
as to why these DAU numbers are different.
We got one that's 10,000, one that's just ahead,
like around 7,000, and one that's, you know, 3,000 or so.
Sorry?
Okay, definition of active.
Now, can you actually drill into it a little bit?
Like, how would that definition be?
Okay, so launch this activity.
All right, other hypotheses?
Yeah?
Is it the same product?
Yes, it is the same product.
It's not necessarily a time series,
but it is from the same product.
Okay.
Uh-huh, others?
Other hypotheses?
Yep?
Time zone?
Okay, others?
Okay.
Others?
Yeah?
Uh-huh.
Okay, these are all great hypotheses.
Actually, what it is is really the definition
of what login is.
So this is actually, this example comes
from one of our largest games.
Millions of users every single year,
about, you know, hundreds of millions of dollars
in revenue, right?
What we discovered was, when we define what is the data,
what is login that we're using to calculate DAU,
well, we found three things.
Number one, when you authenticate,
when you log into our service,
there's one login right there.
Now, but the other thing is, okay,
all of our games, they have their own clients, right?
And when you actually have a client,
you opt into our terms of service,
you have the option to opt out.
So what this initial difference is from 10,000 to 7,000
is 30% of the people opt out
because they don't want to give evil EA
your personal information, right?
The last one is the specific,
and this is going back to marketing
as well as the product management.
There was one team that actually operates
a live service online version of the game.
And all they care about is who logs into their game server.
Now, the challenge of this is when you go up to finance,
when you go up to your executive,
in this case, the GM of the studio,
when you actually provide analysis about this,
when these three numbers don't match,
those people get very angry.
So really, this goes down to the heart
of really defining what the data is.
So when I started out as a product manager,
I came in, I was like,
we're going to do these crazy segmentation,
customer targeting tools,
because as a marketer, that's what I really wanted.
I came in, I realized, okay, you know what?
That's actually gonna be very, very difficult
unless you have a foundation for telemetry,
for semantic definition of what each individual thing is.
And we spent literally the last two years
because we have games on our HD consoles like Xbox and PS4.
We also have games on mobile
to create standards for what is login.
Everything now actually comes in
and the only thing we're counting
is through the authentication server, right?
All this other stuff, one, this is gonna be missing data.
That's too specific to the specific needs of that game.
This is one, this is the front door for everyone
when you log in, right?
And that's really necessary at the foundational piece
to actually do any other analysis downstream from that.
Does that make sense?
Any questions around this process?
Because I know there was a question up front
in terms of telemetry.
This is an incredibly hard painstaking process.
This requires, depending on the size of your organization,
walking around, talking to marketing,
talking to your designers,
talking to your engineers especially,
because what will happen is oftentimes
when you send that event through,
they'll say, hey, here's the event structure.
And they will say, okay, now that payload is too big
for my clients.
I'm gonna just change some of the parameters
or not use some of them because it's easier.
It's easier for me to implement
and it'll be easier on the load of my clients
to send through to you.
And what that means is that if every single product
or every single engineer does that
and makes that different choice,
again, you're not matching.
You'll still be called login.
But in the case of EA, you will have player levels.
Someone will say PLY underscore LVL.
Someone will say PLYR underscore LV.
Those differences actually cause your analytics systems
to break downstream, right?
Questions?
All right.
So the other key thing about this is who are you talking to?
So one of the things that we produce
are recommendation systems.
So paint the scenario
and see if you guys have ever experienced this.
You're talking to your engineer about,
hey, I've got this great recommendation scenario.
They say, holy cow, we've got 300 million users.
In order for us to actually recommend stuff,
we have to remember what they saw before, right?
That's gonna be really expensive.
That's gonna be, let's say $3 million
in terms of the overall budget we're gonna have to spend
on just storing that data.
And your head engineer is gonna say,
holy cow, that's way too much
relative to our overall budget.
We can't do that.
You're gonna go to the executive.
They're gonna say, oh my God, that's way too expensive.
You know, that's way too expensive as well.
You're gonna go to the marketer though,
and they're gonna say, wait, hold on a second.
We have 300 million people coming through our product.
What are you doing?
You're gonna come in, you're giving them a bad experience
at the end of the day.
That's really, really horrible, right?
Now I leave these two down there at the bottom
because that's really the key of it, right?
So going back to your segmentation understanding,
that scenario I just painted for you guys,
trying to weigh the cost versus the customer experience.
You know, how do you guys think
you would address that problem, right?
Any thoughts or any suggestions right off the bat?
Yeah.
You have to model what the benefit,
and you've got the cost side of it.
You gotta model the benefit.
If for that $3 million, you only earn an extra $1.97,
not so good.
If instead, you can make a convincing case
through some testing that you're gonna make
an extra $300 million, then it comes no brainer.
So you have choices, you need an outcome model.
Model it out, yeah.
Other thoughts?
Uh-huh.
Other thoughts?
Yeah.
I would tell each of them,
I would say like $600 million, $300 million.
I would definitely be arguing for it,
but I think if you make a set of people
and do recommendations for them,
you can probably straddle that with your everybody else.
Other thoughts?
Yep.
How many pennies or fraction pennies for each user?
Uh-huh.
And then I'll set that up against ARPU or RPPU.
Yep.
And then I'll get to the tax benefit.
Yep.
Mm-hmm.
Tax benefit, potential loss of revenue by other...
Uh-huh.
Okay, other thoughts?
All right, so with this, all that stuff is great,
but that all requires you to take it to the market.
There is actually some very basic things that we did,
because this case came up three weeks ago in my work.
Number one, what is the distribution of people
who will come back after a month,
two months, three months, four months?
What we found is that what the marketer was asking for was,
hey, we need to keep everyone's knowledge for a year,
because that is our game ship on a yearly cycle.
If we miss the holidays, we miss Christmas time,
we miss all these different things,
we're not gonna hit, and not a year,
but specifically 13 months,
because we have to be able to track year on year.
The engineers are like, oh my God,
that's gonna be 300 million times whatever nodes
and the number got very, very big.
All we needed to do was actually look at the distribution
of the telemetry we already have,
that login stat we just saw and say,
hey, if you've churned out,
what is likelihood you're gonna come back
and what percentage of different segments,
the overall population, the spending population,
et cetera, come back?
And what we found is after 60 days,
5% of everyone comes back,
and like half of 1% of spenders come back,
because hey, you've got skin in the game.
You're not gonna just go away for 60 days
unless you really, really are just done playing the game.
Now through that, you're able to then make that case
to the executive saying, hey, yes,
what we can do is we can actually save for a year
the people who were spent,
just to make sure we capture and treat those players well.
For everyone else, if it's 5% is too high,
then we can go out 90 days and it drops down to two, 1%,
and we can do 90 days for the remainder.
But that enables you then to actually have multiple choices
and options as a product manager
around that type of messaging.
Now, how many guys of you guys have free products
versus how many have paid products?
Just a quick raise of hands.
Free products?
Okay.
Paid products?
Okay.
How many guys who are paid have provided free
or demo products?
Okay, got it.
Question to you guys.
Do you notice a difference in your behavioral analytics
of people who went in their free state
versus their paid state in your demo or your free product?
Absolutely.
Can you tell me a little bit about that?
The conversion rate is really low right now.
It really, I think from my perspective,
people are looking at our free version
as really a different site
than people are using the paid version.
Yeah.
Yeah, no, I mean, that's exactly right.
That's what we've seen as well, right?
We have a variety of different free products.
We have, hey, you can subscribe to Xbox Network
and we'll give you early access to some of our games
before you even buy, right?
There are just free demos, a beta.
You can download it on PlayStation and then play our beta.
And we use that as a way to communicate with the community,
address bugs, and also give you an early view into the game.
What we found across the board is that
that is incredibly useful for us to understand
how people are playing the game,
but people do not make the same decisions, right?
And you see that in online poker.
You see that across the board.
When you have no skin in the game
from a payment perspective,
you will make completely different decisions
than when you're actually paying for the product, right?
So that's why when you actually think about,
okay, who are you talking to when you're using your data?
This is key, right?
Any questions around this type of stuff?
Happy to kind of spend more time on this
and talking about how you position,
how you actually manage different conversations
through the app, throughout the org.
No? Okay.
Oh yeah, uh-huh, sure.
I need to play any game.
Uh-huh.
I need to play any game.
I didn't have too much money.
In the end, for the market,
the emotional and financial difference we had
was that people were keeping all the bucks for themselves.
That eventually deterred me from coming back
for the market as a paid customer.
Yes.
So that single painful experience
has made me, in this case,
more shy of spending money
because, first of all, it's not very open.
Secondly, I may get involved with it,
but then there is always a possibility of cheating.
Yeah.
Unless I'm confident that this problem will be solved,
I'm reluctant to come back every day.
Yeah, exactly.
And that actually leads to this next part.
How many people know this chart?
It's from Boston Consulting Group.
It's about the lifecycle of a product.
Yeah, a decent number of folks.
Your question is exactly right, right?
Because what you're talking about is here.
You're already engaged.
This is a product that's probably successful.
It has a large community.
You're one of those paying customers.
That's a different set of data questions
than when you're actually trying to either launch
the product, acquire new users, ramp up, or otherwise, right?
And I've seen this actually happen in several teams at EA
where you pay attention to the wrong problems.
When you actually have a relatively mature cash cow
product that's doing really, really well,
if you're not actually paying attention to fraud and bots
and things that are actually negatively impacting
the player experience, you're probably missing out.
If you're thinking, oh no, I still need to juice
that ARPU up a little bit more,
and you overly focus on getting new people in
and you're not treating the existing veterans well,
that can lead you in the wrong direction.
The key point here is that at each stage
of the product life cycle,
there are completely different product questions
you want to ask.
Before you even launch the title,
you're going to be looking at your acceptance criteria.
What are your SLAs?
What is the load you expect on the system?
How much, how many, what's your adoption curve look like?
So then you can actually provision the backend systems
to be able to accept that load.
Those are also data questions.
When you're acquiring, obviously, cost of acquisition
and how much you're spending on Facebook or other channels
to actually bring people in, how people,
discovery and the rate of conversion, et cetera.
Those are questions that happen here.
As you're trying to grow,
obviously, it's looking at retention.
How long are people sticking around?
And then what is the conversion, right?
And how are we actually managing that process
and bringing people up the life cycle,
trying to remove those roadblocks?
Obviously, as we just talked about the cash cow,
that's okay, you have a core user group.
How are you preserving their experience
so that they don't, they're not leaving, right?
Because if you have critical mass
and you're actually doing really, really well,
yes, let's say 60, 40, you divide some of that,
your bandwidth into, okay,
well, how do we get more people in
because everyone has a life cycle to stick around,
you need to look at the churn in your population,
but then how do you actually make sure
that that population is stable?
And then after the fact, when it's actually in decline,
that frees you up actually to do a lot more experimentation
and you might actually discover a completely new niche
or a completely new aspect of your product
that you innovate on and actually take it
in a different direction, right?
Any questions around that
or anyone want to share experiences?
Yeah, uh-huh, sure.
Right, looking at this,
if you're spending a lot of money,
I mean, I think that you actually,
you will segment people
depending on what your business model is
and what your customer,
and what your player base looks like, right?
So yes, if you are, you know,
if you're presenting to an executive at different stages,
yes, if you're asking them, for example,
one story is, hey, I'm going to ask my exec,
my CEO, whatever, for money for paid acquisition.
Yes, you will say, okay,
yeah, this is what we expect to get, et cetera, right?
And this is the population,
this is, you know, the rough breakdown geographically,
this is what payback looks like, that sort of stuff.
Later on, you'd be saying, okay, yeah,
this is why we're going to be actually doing these features
for your long-term popular user base,
because it's really good for them.
We expect retention to increase this much,
you know, overall ARPU is supposed to rise,
that sort of thing,
but it really depends on the context
of the conversation you're having, right?
By the way, I'm throwing out a lot of different acronyms.
I know a lot of you are probably familiar with them.
Are people not familiar with terms like ARPU, payback,
et cetera, ARPU is average revenue per user.
And there are a couple other funny acronyms.
There's RPPU, which is average revenue per paying user,
or ARPS, average revenue per spender.
They all basically mean, okay,
how much do I make per capita
for my entire app, product, et cetera?
And how much do I get from the people who actually spend?
Right?
Other questions around, yeah, uh-huh?
Yeah, how would you,
what would you consider a good distributed product?
That's an excellent question.
So for example, let's take the big example of EA.
Uh-huh.
How would you say,
does it start on a console,
maybe you could keep up?
Or if you want to,
hey, you buy a console,
what is it you can get?
So it's gonna be like,
hey, you can buy a console.
So I mean, that really depends
if you're talking mobile or on console, right?
Because up front,
mobile is very, very difficult,
because the problem is discovery.
There's so many apps that are launched on your app stores
on any given day,
let alone week, month, year, et cetera, right?
So I mean, a lot of those metrics are prefaced on,
okay, well, how much are we actually getting promotion
from Apple or from Google to actually launch this title?
Is there a license associated with it?
Is it a FIFA game, a Madden game, et cetera?
And what type of acquisition do we expect
from association with that brand, right?
And then it becomes, okay,
within that first three or four weeks,
or, well, actually, let me step back.
For a lot of our mobile titles,
which I'm sure is similar for a lot of your products,
we go through soft launch phases
where we test in small markets
to see what the, what's the word I'm looking for?
Adoption, looks like.
And then we'll actually start rolling out globally
based on, and we'll also do closed betas too,
to say, okay, is the core loop,
are the key features that we're providing,
and test those out as well.
So a lot of that happens in advance of a full-blown,
hey, it's featured on the app store style launch, right?
Now, if you're talking console,
that's different.
People are pointing at 60 bucks.
It might not even be them buying,
it might be either a grandparent
over Christmas time buying out of Best Buy.
So that's a completely different funnel,
which is much more like traditional retail.
And the success of that is very much more nuts and bolts.
Hey, we're paying this much to put it on a disc
and put it in a store or give a digital download code.
This is what PayLite needs to look like
in the 12, 16, 18 month timeframe
for us to decide whether it's a success or not.
Make sense?
Other questions?
Yeah.
I could tell you, but that'll cost you money.
No, that's entirely dependent on your product, right?
I mean, I think that it is like,
we've all seen small, underfunded products
that absolutely explode.
And all the ones that are launched by either huge brands
are very, very well funded,
all completely flat on their face.
I don't know the answer to that one.
I'd love, yeah, go ahead.
So my project needs to be a full-fledged app store.
It's in a very early stage.
How would you consider the five most important metrics
in the introduction phase?
How many people are on the metrics table?
How would you consider the five
or the three most important?
So it really depends on the product,
but there are a couple of basic things, I think,
from a product management perspective.
Number one, can you ship?
Is it going to go out the door?
Number two, once it goes out the door, is it stable?
What's your uptime?
Exactly.
Number three, okay, based on,
you should have some sort of projection on load,
but have you provisioned enough in your backend
to actually be able to support that load?
Number four, what is the velocity of development
you actually have?
Because you're going to get out there,
and hopefully you're going to be beta testing
with a small group of players.
How quickly can you address their concerns?
Because if you actually see any big deal breaker
feature bugs, how quickly can you fix that?
Now, I'm sure there are a lot of product managers here,
and you guys have other thoughts in terms of,
hey, as we're developing a product,
what the first few metrics should be
that you're looking at?
Yeah, I mean, it's the same thing
as the product management process,
and what gets taught at product school.
Like, it's an iterative process, just like this talk, right?
You've got to constantly, yeah, exactly.
Other, yep, uh-huh, thanks.
I mean, did that help answer your question?
Okay, are there other people
who wanted to share different viewpoints?
Totally happy to listen to those as well.
Yeah, uh-huh.
Yeah.
Yeah, so I mean, I think one of the things about,
so let's say you've actually found
something that's really, really,
you've found a niche that your product speaks
to that niche, and it's growing.
Then it really is, okay, how do you then engage
with the core customer base, and understand those needs,
and then add the features that will really expand
and enhance that process?
At that point, though, again, it really does become,
okay, well, what is the velocity of that group, right?
So do you have the scalability built into your process
to be able to address that?
Do you also have, then, the velocity of development,
whether you guys are using Agile,
and StoryPoints, or otherwise,
do you know that you can actually address those features
in a timely way before people churn out?
Because what you then want to understand is,
okay, based on what you've got ground floor,
what is the average life cycle,
lifespan of a person who comes into your product?
Why are they engaged?
How long are they engaging?
If it's a conversion, why are they spending,
and how much do they spend on average, right?
So that, to his point, right,
once you're into a growth mode where you actually,
hey, the product works, it's found some sort of a niche,
then it's about, okay, well, let's think about this
as a live service, and what do we need to do
to retain people, to understand what they're looking for,
why they like the product that they've got,
and then kind of keep it going from there.
Make sense?
Thanks very much for the question.
Other questions or other perspectives?
Okay.
Your marketer comes to you and says,
we just ran a test.
I'm not sure what it means, but we ran a test.
What should we do?
What do you want, as a product manager,
what do you want to do?
Any thoughts on that?
Yeah.
Okay.
How would you do that?
Okay.
Other thoughts, other questions?
Yeah, exactly.
Yeah.
Got it.
Okay, yeah, no, that, that, exactly.
Other, other, other thoughts?
It came from Mark, yeah.
Yes.
It came, Mark, and he said, hey, we ran it,
we threw something, we put a new feature out,
and we did a test.
We got some results back.
Okay, so you want to take this,
you would take this to executives?
Yes.
Uh-huh, yeah, uh-huh.
Huh?
Well, see, we ran, we ran an analysis
based on the geographies of our customers,
and, you know, and they're,
and this is the sales data we got back, right?
Yeah.
I would ask, what decision,
what choices are you trying to make?
Because that's usually the first thing.
I mean, because a test, you know,
in itself doesn't mean anything.
It's a question of how does it make good choices
between A, B, or C?
Okay, so we said, like, they said, okay, yes,
well, we looked at, we're looking at
the verticals of geographies.
We ran our promotion, and we're trying to say,
we're trying to lock in our projections for revenue
for the next 12 months based on this.
So, hey, if I sell a lot in some of these geos,
it'll be great.
Yep, question in the back?
The important model is to capture data for test,
after test, and you can capture data for test group,
and data for group which didn't have test.
So, control and a variant, yeah.
Are everyone, is everyone familiar
with the scientific method?
You guys laugh, but I was having a conversation.
So, one of the things that I found
as building experimentation platforms
is actually a lot of people don't have
a good understanding of what that means
when we say the scientific method.
Yes, we all learned it in high school
or in elementary school or whatnot,
but when you're talking about giving people
an A, B testing tool, how to set up a proper control,
how to set up a proper variant,
and how to identify what the independent
and dependent variables are,
that actually requires quite a bit of education, right?
Because what this person will come back and say is,
hey, okay, actually, you know what?
Q was a wrong dimension for my analysis.
I ran the wrong query.
It actually looks more like that, right?
Now, those are seven dots in a row
because they ran, they'll run it against,
you know, date as opposed to geo.
So, they're saying, okay, now I've got it.
We have a correlation here.
These last three or four days,
we've got, we clearly sell more.
We're gonna run, we're gonna,
we know what the pattern is now.
We're gonna build our forecast out for 12 months
and say, we're going to, we're gonna make this much money
based on the data we're seeing.
Any questions around this or thoughts on how you would then,
what you would do with this stage of analysis?
I wanna ask what kind of hidden variables
you're missing right there.
Okay, hidden variables.
Yep, other thoughts?
No other thoughts?
Okay, yeah, uh-huh.
If I understand what you're saying,
they fished around until they found,
in the data, they found some correlation or some pattern.
And I have forgotten my stats on one,
but my understanding is that's often
a very dangerous way to go about things
is that you need to kind of set the test up before.
If you just fish for, in any data set,
even random numbers, you'll find some wacky correlations
between how much the giants win and your sales.
It's a weird, crazy thing,
but you need to set up the test beforehand
with the appropriate samples that are independent
of the control group and the other group.
And if you fish in reverse,
then you've got no way to test your hypothesis.
Yeah, so I mean, so that's exactly right.
When we talk to marketers, and I used to be a marketer,
so I'm not casting aspersions,
a lot of what they'll say is,
hey, we know, okay, telemetry,
you don't need to do any tracking.
No, we don't need experimentation.
We'll just back test.
We'll just reverse engineer and find the patterns
and that'll be okay.
To the person who made the comment
around controls and variants, that's not really okay
because how are you gonna control how people were exposed?
What were the circumstances by which they saw
the thing that you're trying to test, right?
So to your point, experimental design
is actually really essential
when you're talking about being data-driven
because you can say you ran a test,
but if it's the test that can be,
there can be temporal, you know,
I'm gonna show something to someone on Tuesday
and then show it to a different group on Friday,
but how do you control for the population,
the size, the significance of the population
relative to how both the control
as well as the experimental group, right?
Thank you.
Other questions or comments?
So basically their thought is,
hey, we've got this pattern based on the sales numbers
for, hey, we tested for, it looks like, what?
Five weeks, 35 days.
That's statistically significant.
And if 35 is great, you know, it's a decent size.
So we're gonna base all of our forecasts on that, right?
So then you go back and you say, okay,
this is the time where they actually looked
at your revenue forecasts.
Thanksgiving, all the way straight through to Christmas.
And we're gonna take that one month
and we're gonna then extrapolate that out to say,
every month we're gonna sell as much as we can.
Do you think there might be problems
with the revenue forecast in this case?
Okay, so the whole point about why this is important
is because, you know, it really,
that experimental design, just saying, you know,
to your point that when I asked you early on
during introduction, what is a test?
Defining what that space is, is incredibly essential
because you can actually pull out
and bake in confounding factors like the fact
that it is Black Friday through to Christmas
without even knowing it,
unless you actually ask those questions
like you were talking about early on.
Any questions around that?
All right, you guys have been on this page, right?
Daily, today's deals from Amazon.
Rough estimate, not, you know,
apologies for everyone in the back who can't see it.
How many tests do you think roughly
are running on this page right now?
Yeah, by the way, I don't actually have accounts on it,
but other thoughts,
I just kind of want to get people's thoughts on this.
Yeah, yep.
Uh-huh.
Mm-hmm.
Yep, that's our other recommendation, uh-huh.
Yes.
Yes, uh-huh.
Yeah, no, that's exactly right.
And the other thing that you,
so that list is great.
The other thing you talked,
that you didn't mention is actually product discovery,
because this is my daily deals page.
So just a little background,
it's my daily deals page.
What did I say before?
Oh yeah, product discovery.
Yeah, because if this is just living
in Amazon's vastness of millions of pages,
you know, why would these three,
why would I find these three things
other than what was recommended on this page?
So background on this,
I recently had a baby,
so I'm in the need of a shiatsu massager
because my shoulders are constantly hunched over
taking care of the baby.
In-laws are coming into town a lot,
so I recently bought an air mattress.
And, you know, as part of the,
I like talking, but I also like reading
about different management approaches,
so why is Good to Great Featured as a book for me,
because I like management science
and reading about management science.
So yes, you're exactly right.
These are my recommended product selections.
But to your other point, yes,
deal of the day, the order in which they're presented,
the price, how the price is represented.
Is it a slot, is it struck through with 60% off,
or is it a price range?
You know, all of this stuff.
There's, you know, there are probably, yes,
easily dozens of tests here.
But the thing is, for most of us who aren't Amazon,
this is gonna be way too complicated.
By the same token,
there are gonna be certain points
in your product life cycle
where you're just gonna need to make money.
So trying to control everything down to,
I'm gonna move one thing at a time
might not be the right decision.
That goes back to the earlier slide around,
hey, who are you talking to?
What is the stage,
what is the life cycle of your product?
What is really important right now?
Because if you wanna break it down to this and say,
hey, what we need is,
we wanna build an experimentation platform.
It's gonna be awesome.
I'm gonna handle multiple variables at once,
view recommendations, all this other stuff.
That's a great vision.
But if you're just trying to get the product at the door
and figure out if there's a fit,
probably not the best thing.
Maybe go out and, you know,
this is not a plug by any means,
but there's, you know,
split IO, there's lean plumb,
there's swerve, there's optimize.
All these different ways
that you could actually run tests.
Might not need to actually build a platform for it.
And also, if you're gonna be building a platform,
maybe start small.
Questions around that?
Yeah.
So there are a couple of different approaches.
So I mentioned multivariate testing, right?
Which is different than multivariant,
which most people call ABN.
So there's confusion there sometimes.
I just wanna clarify that.
Multiple variants means,
hey, I've got a control
and I've got A, B and C variants, right?
Multivariant means I have the ability to,
and there are a lot of different approaches to this,
test the different variables in conjunction
to actually test effectiveness, right?
There's a marketing,
they actually have what they call conjoint analysis, right?
When they go through and like,
for the easiest example is the features you buy on a car,
when they offer you package A, B and C, right?
They'll look at, okay,
what are the relative correlations between the features
and what people, what this is people to buy?
And then that's how they create the packages, right?
So there is multivariate testing to enable that.
Other questions?
All right.
But yeah, that goes back to what could go wrong, right?
You could just have bad data
that goes back to, you know,
login and everything that happened.
Oh, by the way, do you know who this is?
Anyone?
Yes.
Do you know why I put this up here, guys?
Do you know?
So this is a really famous historical,
and that just shows that I'm old.
Example.
So basically this was a press conference
when congressional oversight committee
was going through and they hire scientists
to say, why did Challenger explode?
Right?
Has anyone ever heard of Edward Tufte?
Okay.
He uses this example all the time.
What ended up, and this is his diatribe
against PowerPoint, actually.
Basically, the aeronautical engineers
who designed the solid rocket booster for Challenger.
You guys, do you guys all know the Challenger accident?
Okay.
That's okay.
Who was born before the Challenger accident?
No, no, no, no, no.
But essentially what happened was, okay,
so the engineers down at the ground level,
this is also one of the things that you need to struggle
with with who you're talking to
and how they're communicating.
They said, hey, at low temperatures,
the O-ring that actually seals the joints
between the two sections of the solid rocket booster,
they have a tendency to be a little brittle.
They don't rebound and expand back
to their original shape as quickly as they should.
That was buried three or four levels of bureaucracy down,
and then it got boiled up and boiled up and boiled up
so that when it came up to the leadership of NASA,
that was wiped out.
That was wiped out at like the second or third tier.
What Feynman did is a very basic example.
He took a piece of that O-ring.
He put a winch on it, and then dropped it in, huh?
A caliper, thank you, a caliper,
and dropped it into a glass of ice water
during the press conference.
Pull it out, took the caliper off,
and showed that there was a massive dent in that O-ring
and that it was not expanding quickly enough.
So what ended up happening with Challenger,
for anyone who doesn't know,
is that because they took off on a very, very cold morning,
when it got up to a point
where the barometric pressure changed,
fuel began leaking through that gap in the O-ring,
and it caused that solid rocket booster to explode, right?
So when I bring this up as an example
of what could go wrong,
that's one of those things
where have you actually had the conversation
about all the things that could go wrong?
Is your data valid?
Are you talking about the same metric in the same event?
What is noise, right?
What is the baseline?
That's what we need to have control.
What is the baseline noise you see in the system
so you can actually control for that, right?
Seasonality, someone brought that up earlier,
and that's what I was the core of my example
in the previous example.
Hey, if it's Christmas time, if it's summertime,
if it's, you know, how does that impact?
And there are, other than seasonality,
there are tons of other factors that would play into,
you know, how your product performs on specific days.
Is it the Super Bowl?
Is it World Series, et cetera?
Are there false positives?
You could just be seeing something
that just came out of nowhere, right?
Has no significance in the long run,
it just happened because when you ran that test,
something happened.
And then also, what are the other confounding factors
that you're not aware of yet?
So whenever you, even after you run a test, right?
Even if you have that data and that conclusion,
test it again.
Test it a few different places.
Make sure that you can actually validate
that it really has happened, right?
Now, again, this all is in the context
of where you are in your product life cycle.
If you just need to make money,
which is important, making money,
this might not be the right approach,
but if you're actually, but maybe it is.
And that's really a conversation you need to have
based on those stakeholders,
based on what you're trying to accomplish.
Make sense?
All right, actually, how are we doing on time?
Oh, okay, cool.
So in conclusion, overall, define your data properly.
Remember who you're talking to,
both in the audience externally as the customer,
what segment, but also what segment of your own organization,
your own company you're talking to, right?
Identify what is important at what stage
and for what reason, right?
It's no good focusing on acquisition
when you've got a leak,
you've got a hole in the bottom of your retention bucket
and no one's actually sticking around.
If you've spent a lot of money trying to get them in
and they fall right out the bottom, not helpful.
And then were the results tested and validated?
Did you go through and make sure
that you're controlling out for other factors?
Did you actually make sure that the data is clean?
All those types of things we talked about.
And then also make sure that you're going through
and checking for anything else that had gone wrong,
because there always will be.
And we're all working in a space
that's moving quickly, live service,
things are always gonna be changing.
So it's always test and iterate, all right?
So thanks very much for this time
and we'll move into Q&A.
All right, questions?
Thanks for coming.
Uh-huh.
You mentioned you had a team of six,
product managers and software engineers.
Yeah.
How much do you have dedicated to collecting data
or pointing out maybe your interviews
or just working on the data collection part?
The data collection part.
So our entire platform collects data.
So we collect, I mean,
so when I talked about a data platform,
basically what it is is we absorb all the data
from all the games.
So when you do something in our game,
they basically send us a telemetry event.
We receive it, we store it,
we process it into a metric like DAU, right?
In terms of how much data we collect,
we collect about eight to 10 terabytes a day.
Overall data size is about 15 petabytes or so.
So, you know, decent size,
but not the largest by any extent.
But are you talking about data
from the customer for the products?
Okay.
Do you do any like interviews
or like any other kind of data?
Absolutely.
Yeah, no, I mean,
I think one of the things that is most important about,
so EA is a large company,
I'm not, you know,
and sometimes I wish I still worked for a startup,
but one of the key things
is actually how you collect the data and from whom, right?
So one of the things I work with my product managers on
is, okay, who is important in the organization and why?
How are they making decisions?
And then how strong is your relationship with them, right?
Because the goal for them is to, one,
build trusted relations for the people
who actually can make the decisions
and help understand and socialize
or move forward our mission and our vision, right?
And in those cases, those are key customers for us, right?
Because they're either early adopters
who pilot something and we have quick wins
from a functional perspective
or they're influential
because they have their large successful games
that, you know, if they adopt,
everyone else will then fall in line,
that sort of thing, right?
The other thing is actually,
when we talk about how we collect data
is also understanding not just data that's collected,
but what is the action we intend to take, right?
So in a large-scale matrix organization,
what we're talking about is,
okay, well, how do you navigate those tiers of the hierarchy
to make sure that even if you have an insight,
it gets acted upon?
And that's really when,
after the product manager creates a product,
we have success, we're getting,
moving adoption curve is okay,
well, just like selling.
Who's the inside salesperson
you actually need to talk to to actually convert?
Because without that and understanding,
hey, a lot of my team,
sometimes junior product managers struggle with,
hey, I'm only talking to you
from a telemetry standpoint, right?
I'm only talking to the software engineer
who's implementing, you know?
And I can't make any progress
and you have to tell them, okay, no,
I actually need to figure out
how to get two or three layers up to the decision maker
and come back down
and make sure that everyone up and down is aligned
to actually move the thing forward, right?
Hopefully that provides a little bit of context.
Other questions?
Yeah.
I have a question.
So I actually sell furniture insurance products
and I was wondering what would be the best way
to collect data for a physical product?
So do you sell furniture, physical product
in a physical store or online?
Okay, sure.
I mean, do you have a bell
when people come through the door?
We have five customers a day.
Five customers a day?
Yeah.
We'll have five customers a day,
but that's decent.
Yeah.
Yeah, yeah, yeah.
So I'm a network worker
and it's a high-end retail store,
so a lot of the times we get clients,
they get a designer and stuff.
So I actually studied that data analytics,
so it's kind of frustrating when you don't get much data.
Got it.
So every time you try to do a survey,
you don't want it to get kind of awkward.
Oh, really?
So you do an in-person survey
or do you send them an email with like an,
is everyone familiar with like, huh?
So when I'm in the store,
Uh-huh.
Okay, got it.
I mean, and then do you actually,
so how many questions in the survey?
Oh, wow.
You only get 20 answers.
I mean, that's a good question.
Yeah.
Well, so I mean, so there's a couple of things.
So in Ecuador,
I'm not sure what type of market research
firms are out there.
I mean, you can always get a market research,
research survey of some kind.
Do you go, I mean,
there's, you know,
influencers are really popular right now.
Do you actually go to places
where they might talk about luxury design?
Do you talk to the designers?
Do you talk to the,
do you move up the funnel in terms of people
who could actually be influencers
to come into your store?
Yeah, we do that.
Okay, okay.
I've tried everything.
Yeah, so I mean,
what problem,
it sounds like we might want to take it offline.
If you take a dive a little bit deeper
into the conversation,
but I'm happy to talk to you more about it later.
Uh-huh, yeah.
Other questions?
Also, you don't have the capacity of,
for example,
the small file.
Uh-huh.
Outside of where to start.
I don't mean like,
how to use tools and stuff,
but the other one.
On the analysis side.
Oh, that's a good question.
I don't know.
I'm imagining a person
who has a lot of experience in this field
that I can sort of consult.
So first of all,
so first of all,
it would be actually,
there's a lot of Google searching,
so you can make sure you ask the right questions.
And then oftentimes what I've found,
and not just for product management,
but just across the board,
if you actually just talk to friends of friends
or people you can reach out to on LinkedIn
for a casual coffee,
you can usually actually talk to them
and get some insights, right?
In addition to which,
I mean, just like me coming to talk to you guys tonight
or other places.
I mean, there's times when people
will voluntarily just say,
hey, I'm happy to look at that for you,
and I keep it discreet.
So one of the things,
for example,
my sister works for a large cosmetics company
and they're thinking about a European expansion.
I had,
and they're not sure how to actually price
their product in Europe.
So, you know,
I have an MBA.
I went to business school in London
and I said,
oh, you know,
I'll put you in touch with my pricing professor.
Also so happens that I've asked,
I've reached out to him
and he's giving me recommendations
to talk to people
when we've had game pricing challenges, right?
Now, you know,
there's a line between,
hey, what's a good,
good natured app,
casual ask versus,
hey, this should be a consultancy type arrangement.
But usually, you know,
if you're upfront with them about it
and don't overextend
and try to kind of,
you know,
use them too much,
the people are pretty open to helping.
Hopefully that helps.
Other questions?
Yeah.
Uh-huh.
Okay.
So there's a couple of things.
Well,
pause for a station identification real quick.
I forgot that I need to repeat the questions
that were asked in the mic
so people can hear.
So the first question that was asked
about how to actually drive more foot traffic
in a physical retail store.
Second question,
which was asked,
how do we actually,
if you don't have that capability,
you know,
get some help with analysis.
The third question is around recommendation systems.
Basically the person
has a Web 2.0 product.
She's tried to test for the recommendation system
and didn't yield significant results.
The first question I'm gonna ask you
is how many people,
what pieces of content were you recommending?
Did they have to choose between?
Uh-huh.
Yeah.
Okay.
Got it.
So real quick question here.
How many people here in the audience
actually work on recommendation systems
or have them in their products?
Just quick raise of hands.
Two, three.
Okay, three.
And feel free to actually also respond.
I think you're using too few items.
What I found with the systems we use,
so we basically use two different algorithms.
One's called UCB-1
and one's called Thompson Sampling.
They're basic standard multi-armed bandit,
explore and exploit style recommendation.
I mean, it's used the same thing for web pathing
to figure out what the right click path is,
but it doesn't really work
unless you have at least 40 or 50 things.
From my experience,
guys who actually have recommendation systems,
any other feedback?
You can disagree with me.
You can say yes or otherwise.
No?
Yeah, uh-huh.
No, no, not 450, 40 or 50.
Yeah, so basically to your point,
if there's only 10 things like,
and depending on the traffic,
like it's not significant.
It's like going out to thinking about the experiment, right?
I mean, you need something
that's of a significant sample size
where there's actually enough diversity in the population
to be able to actually be representative
of what you're doing.
So I think that, you know,
I don't know the rest of the context,
but it might just be you need more things to recommend.
Yeah.
Okay, we have time for two more questions.
Yeah, go ahead.
At the same time,
or like you present the sample,
so all 40 or 50 things that you-
Oh no, no, no, sorry.
The pool of content you wanna pull from
for a recommendation.
Yeah, rotate through.
Yeah, yeah, yeah, yeah.
Oh, really?
Oh, okay.
So there's a pool of content.
Okay, got it.
Interesting.
Okay, actually, let's take that offline.
I wanna talk to you more about that after this.
Okay, yep, uh-huh.
So, before I said you have a lot of war stories,
I assume you know a lot of data.
So, not one of those horrifying war stories most,
but one of those like beautiful sunshine
when you're going through the sky.
Tell us about a time when you found that people
who hate the war stories found you.
What was it like?
Oh, I was not expecting that.
People behave differently than what I expected.
Are you talking about our players
or people I engage with?
Yeah, yeah, yeah.
I mean, I'm looking at the data that they're putting out,
like, what's the conversion rate?
No, no, no.
Like, it turns out to be 30.
No, no.
It's like data you didn't expect to find out,
find, and then you're like, ooh, that's interesting.
So, this goes back to actually the question
from the person sitting in front of you
around the player experience.
We were working with a game team.
They have a very large live service community.
And we were actually,
all we were trying to do is help them understand,
okay, what's the velocity through their economy, right?
If you're buying stuff, if you're looking at stuff,
how do people convert?
How do people, they can actually trade stuff.
I wanted to see that.
What we identified was actually
what you were complaining about,
is that, you know, there's a market there
and there's a lot of fraud.
There's a lot of effectively money laundering
going on in the system, right?
And what was very surprising
is that they then took that knowledge
and we actually worked with them,
our data science team worked with them
to build a fraud detection algorithm
based on what we call a lopsided trade, right?
You know, a lopsided trade for everyone
who doesn't know is, okay, you're, you know,
you have a product that's worth a dollar,
but you actually end up selling it to someone else
for $10,000 because you're actually in cahoots
and you just want to move money
through the ecosystem, right?
That works with any type of money laundering, right?
So that was one of those discoveries
where there's a, oh, okay, actually that really helps us.
And they just jumped on board
and really worked with us closely.
Other questions?
All right, I actually am going to ask the last question.
Sure.
So we have many aspiring product managers in the building
and as a future product school instructor
slash someone who made their way into the role,
what advice do you have for our aspiring PMs
about how to make it in?
I think skin.
No, 100%, right?
As, you know, basically if you have a vision
for what you actually want to do, right?
The whole reason that I'm a product manager today
is because I came from marketing,
and saw what we had
in kind of a live service social environment,
the tools, the approach, et cetera,
looked around EA and said,
hey, that doesn't exist yet.
At that time, I had never even drawn a marketing diagram,
i.e. boxes and arrows, right?
I was just like, I just,
but I know what needs to be built.
So I went with a diagram
and I went around to different people
and I said, okay, the first place
that this is going to have to happen is a data platform.
And just said, hey, I think I'm a,
I think I have a vision for what needs to be built.
I can get, I can learn the tools
that I need to learn to bring this fruition
and build confidence that way.
It's, you do the same thing with selling,
you do the same thing with finding a new job,
a partner, et cetera, but just, you know,
a thick skin and being able to actually constantly
talk about what you're passionate about
and convince people.
All right.
Thank you so much.
All right, thanks very much, guys.
Thank you.

