AWS re:Invent 2024 - Explore what’s new in analytics and governance (ANT303)
3 Video Views·Dec 12, 2024
oin this session to explore the latest data governance innovations and features in AWS analytics. Our experts guide you through the latest innovations in Amazon DataZone, AWS Lake Formation, and AWS Glue that are helping organizations establish robust data governance frameworks and maintain compliance standards.
All good?
All right.
Welcome, folks.
How's everybody doing?
Grabbed lunch?
Yes?
No?
Hungry?
Not hungry?
Okay.
All right.
Well, thank you for joining us.
I know it's quite a ride from Venetian to Mandalay Bay and wherever you were last, so
appreciate you coming in to hear more what's new in analytics and governance.
I'm Shikha Verma.
I'm the head of product for data and AI governance at AWS, and I'm joined here by two of my favorite
customers who will be joining us on stage shortly, Shaja Selvamani from Cisco, as well
as Travis Smith from the Weather Company.
Both companies I'm pretty sure you're very familiar with.
Anybody here who hasn't heard of Cisco or the Weather Company?
Nobody.
That's what I thought.
Well, super glad to have them join us.
We have a packed agenda, so let me get you started.
We still have folks rolling in, and I think that will continue.
We have a parallel room set up in Venetian for those who couldn't make the hike, but
thank you very much for making the hike.
I love live audience, so appreciate it.
So packed agenda, we'll start off with the voice of the customer, which is how at AWS
we do everything.
We're very customer obsessed, as you know, obsessed with you guys and your needs, and
we develop technical solutions to fulfill the needs that you may have.
We'll get into what is new in analytics and data governance.
We'll start off with DataZone, which was a service we launched a couple years ago, then
get into some of the new announcements that you may have heard of if you attended any
of the keynotes.
We'll talk data lineage, data products, all the new capabilities that we've rolled out
in the last year or so.
After that, I'll show you an end-to-end demo of how the thing works, which is what I find
most interesting, and then we'll transition into the customer stories by Cisco as well
as the weather company.
We should wrap up our content in about 40, 45 minutes, so we have plenty of time for
Q&A.
Please do stay for Q&A.
We'll also do a picture time at Q&A where we'll invite you all on stage and take a picture
together, so hopefully you'll stay.
Let's start with the voice of the customer.
So research shows that customers are investing more and more in being data-driven, and you
guys have to tell me whether this is true for you or not, but 82% of organizations have
appointed CDOs.
How many of you have CDOs?
Quite a few, okay.
94% of organizations increased data investments in 2023 and 2024.
Is that you?
Yeah?
Many of you?
Okay.
Great.
20% revenue growth seen by data insights-driven businesses.
Was it you?
Some of you?
Maybe?
All right, this is hard, because 74% of businesses are not able to translate the investments
that they're making into results, right, which is why we are all here.
I'm a data analytics lifer.
I've spent many years as a customer trying to chase the same problems that you guys are
chasing.
I joined AWS eight years ago to develop solutions to help us solve these problems.
So this is not easy.
So let's see what comes in the way, and what are customers really looking to have.
So what are enterprise customers looking for?
Many of you are looking to democratize data with built-in governance.
Does this make sense?
Does the statement make sense to you guys?
Yes?
Hands up.
Yeah?
Makes sense, right?
You want built-in governance because you don't want governance to be a thing that you have
to do outside of your core job, right?
You want to democratize data, but you want it to be as easy as possible to have built-in
governance controls.
So what does that mean if we look at the components of this particular thing that we've been looking
for?
What that means is that we need to have an ability to discover, understand, and access
data across our organization.
Many of you have data in AWS, which is why you're here, but most of you have data outside
of AWS, too, right?
How many people here have data outside of AWS?
Almost everybody, right?
So it's not a problem that we can solve just for AWS and call it done.
We have to make it an ecosystem.
The ecosystem has to work for you guys.
That's when we have really solved the problem.
The next one is multiple personas need to be able to collaborate on the same data problem,
right?
So you have data scientists, data analysts, data engineers.
Everybody needs to be working on the same data that they just discovered and found access
for.
Then you want easy-to-use tools.
Sometimes they're AWS.
Sometimes they're not AWS.
Many of you, I'm sure, use things like Tableau, Spotfire, Informatica for ETL.
Is that true?
Do you guys use tools outside of AWS tools?
Yeah?
So we've got to solve that, too.
Most of you want self-service through a single pane of glass, as well.
So this is Nirvana, where all your users come to one thing.
From there, they can access all of their data for the right persona, with the right set
of tools, and last but not the least, consistent ways to govern the tools, the data, the people
who are accessing the data the same way.
Make sense?
So many of you are taking pictures, so I think this makes sense.
Thank you.
Okay.
All right.
So this is why we announced and rolled out Amazon DataZone a few years ago.
Some of you are nodding.
Thank you.
So it looks like you've used it.
How many of you have used DataZone or known of DataZone?
Several of you.
Okay.
Great.
So this was, for those of you who are not familiar with it, it's a service we rolled
out a couple years ago to catalog, discover, share, and govern your data across organizational
boundaries.
Right?
So whether you call it data mesh or you call it data fabric, data exists across your organization
in multiple silos.
You want to bring it together.
There's multiple architectural paradigms to do it.
DataZone allows you to do them irrespective of whether you're centralized or decentralized.
So you can do both.
So what DataZone allows you to do, for those who are not familiar with it, it brings data
producers and data consumers together so that producers can share data with built-in controls
and data contracts, and consumers can consume the data in the tools of their choice.
So since we announced GA for DataZone last year, we've been hard at work to add a bunch
of features that you guys were interested in.
We'll cover a little bit of that next.
So we GA'd in October 2023.
Since then, a bunch of, you know, customers across all industry segments, thousands of
customers have started using DataZone.
Millions of assets have been cataloged and governed through it.
And we've been on a roll rolling out new feature sets, right?
So 30 new features launched.
Do you see your company in the logos?
Like, can I get a show of hands of who's anybody?
Other than Cisco and another company.
Okay.
They're not here because they're busy using it, I guess.
So I'm sure some of you will contribute to new logos next year, hopefully.
So what are the kind of things that we rolled out?
Let's go over them.
So key launches in 2024, we added generative AI capabilities to auto-create descriptions.
So what that means is that if you have a data asset looking at the tabular content of the
data, we're able to generate descriptions that your business analysts can read and say,
oh, okay, this data looks useful.
And I'm going to show you a demo of this, so it will become very real when you see it.
Data quality metrics.
So many of you have data quality metrics either in AWS systems or you might be using Informatica
or Talent to do your data quality.
All of that is integrated in as well.
So you can use our APIs or our interface to add that data within data zones so that it's
easily accessible.
Fine-grained access controls is for row-level, column-level filtering across your data warehouses
and data lakes.
Hello.
I see familiar faces.
Please join us.
So what fine-grained access controls allows you to do is really do row and column-level
filtering of data consistently across your data warehouses and data lakes, right?
This kind of capability doesn't really exist in many places.
So I'm sure that when you see it, it will be exciting.
The next one is one of my favorites, which is organizing within business units via domain
units.
So what that means is if you have a sales business unit, a marketing business unit,
and a finance business unit, you can set up your data domains differently for those because
everybody has different policies that they need to adhere to, right?
And you can implement them with domain units.
Does that make sense?
Like, does that resonate with people where you have different business units who want
to control data differently, right?
So this allows you the functionality to do that.
I'll show you some of that in action when I show you the demo.
And then once you do that segregation, you want to be able to club things together, right?
So we also added the ability to create data products.
What data products are is you could have tables, dashboards, SQL scripts to go with it, and
you can club them all together as a data product and share that data product within your organization.
So this is very, very powerful, right?
Because you may have a standard definition of creating something, and you can assemble
all those components together, along with the models and dashboards that go with it,
and share that as a product.
Subscription workflows, we added additional ways of just, you know, managing your data
the way you want it, getting the approvals the way you want it.
I'll show you this in a demo, too.
How many of you attended or checked out the keynotes by Matt and Swami yesterday and today?
Okay.
Right?
Okay.
Cool.
Many of you.
So you've heard of SageMaker and the next generation of SageMaker.
Yeah?
Did that resonate with folks, like, that we are assembling everything together?
Make sense?
Okay.
Cool.
So obviously, data zone is the foundation of that integration and that launch.
So I'll show you that in action.
The demo that I'm going to show you is actually going to be a SageMaker next-gen demo.
Also, who uses Tableau or Click or something else for their BI?
Most of you.
Okay.
So we added JDBC query capability, which allows you to use the governed data that you have
in data zone with your choice of BI tool.
This has been very interesting, and customers have been very excited about it.
So I encourage you to check that out.
We also added semantic search, which means that when you're looking for profit, right?
Profit-related data.
It may not be a table name or a column name that you can just syntactically search.
But you're looking for related data sets.
So we added that.
So if you search for profit, it will show you tables that have revenue data, that have
things related to profit that you can use for your calculations and metrics.
So that's pretty powerful as well.
We also added Q on top of all of it.
How many of you are familiar with Q, Amazon Q, right?
So the chat with data concept.
We added that.
So what that allows you to do is you can just say, hey, I'm looking for revenue by blah,
sneakers.
And it will show you, oh, you can use this table.
You can use this metric.
Then after that, you can go in and say, I want to create a model that uses this.
So Q is really accelerating the way we are doing everything.
Highly encourage you guys to check that out if you haven't already.
We also announced in GA data lineage.
How many of you have data lineage today?
Oh, cool.
Some of you.
How many of you want data lineage?
Everybody.
Yes.
So check it out.
Across the AWS ecosystem, we have it in an automated manner now.
So if you use data zones capabilities, you get automated lineage for all your blue pipelines,
all your redshift data, all of your EMR, SPAC jobs, et cetera.
Very cool.
Okay.
So what does this mean?
Are we stopping here?
That seems like a lot of work, right?
I'm tired just launching all of that in 2024.
But we're not stopping.
Because we are building towards next.
And what some of you heard about already is we are building the modern data foundation,
which means we are bringing all of these capabilities together.
So if you go back to the first slide where we talked about the five problems that we
want to solve, think of this as a four-layered cake.
The bottom part of the cake is all of your data stores, whether they are data warehouses,
they are data lakes, they are the Teradatas, on-premise, they are any kind of on-premise
solution that you might have for data stores.
On top of that comes unified data governance, right?
So you want unified data governance across all of your data, whether the data is in AWS
or not in AWS.
On top of that, you want to be able to use this data for querying, for ETL purposes,
for your analysis, for creating your dashboards, for creating your models, all of that.
So the top two layers of the cake really allow you to use the data with all of the tooling
that we offer to create the outcomes that you're looking for.
So given this is where we're headed, this is why you heard of Amazon SageMaker, the
next generation, which we announced as part of our CEO keynote yesterday, which we are
all very, very excited about.
Okay.
So let me go back.
Sorry.
Okay.
So what that...
Wow.
This thing is wanting to move without me wanting to move it.
Ooh, this is kind of funny.
Okay.
But anyways, I don't know what this is doing.
Okay.
But you get the idea.
So Amazon SageMaker, which is the center for data analytics and AI, is an integrated suite
across the four layers of cake that you talked about, right?
And of this, the center, which is the data and AI governance piece, is built on Datazone.
So all of the capabilities that I talked about earlier, all of the features that we have
launched in the last couple of years, all of those capabilities are now available in
this unified suite.
So if you go and check out the new studio, the unified studio that we just launched yesterday,
what you'll see is it has all of the data foundation stuff built in, and we have added
a lot more on top in terms of how do you use this data?
How do you act on it?
How do you experience it?
So in this screenshot here, and this is what you'll see in the demo shortly, you'll see
that the project and data catalog, which was a concept we introduced with Datazone, is
available in the unified studio, right?
So you go in, and Megan in here comes in, and she looks at the data catalog.
So if you look at the screens, the data catalog is there.
There's a Gen AI playground that's actually super cool.
So you can now use the data that you have in your data lakes and data warehouses to
create chat apps just from this unified studio.
It is very cool.
You guys check it out if you're into that kind of thing.
All of the query editor, the Redshift query editor, all of our ETL functionality with
Glue, the visual ETL, all of SageMaker's existing capability in terms of model building is all
available in here as well.
And it's all wrapped around the same framework of governance that we introduced with Datazone.
So it's extremely powerful to use because your encapsulation of data is still the project
concept that we introduced, and then on top of that, the domains.
So you set up your rules by your domains, by your business domains.
You can have multiple projects under it, and then all of your data and tooling is governed
by the same concepts, okay?
So then what do you do?
You go, and when you discover, you can see on the screen that you can browse your assets
by the domains that we have talked about.
You can find your data, and you can subscribe to it from within the studio interface, right?
So you don't have to go out, you don't have to initiate any sort of JIRA workflows, which
I know many of you are doing, to get permission to use the data.
All of that can be done from within the studio.
You can also look at all of the lineage from within the studio too, right?
So that thing that all of us want is actually available, embedded within the studio.
Also the Gen AI capabilities that I talked about, which I'm going to show you in action
in a minute, is all embedded in as well.
So when you look at a data asset page, which is kind of like, if you compare it to Amazon,
and when you search for something on Amazon.com, you have a product detail page that shows
you everything about the product, right?
This asset detail page pretty much shows you the same thing for all your data, ML, and
AI assets.
So all of your models, all of your dashboards, all of your data has a detail page.
So using this, you create your data marketplace where you can have your producers and consumers
interact and share data.
All right.
Who's ready for a demo?
All right.
Are we still up?
Am I boring you?
No?
You're up?
Up?
Can I get a woo-hoo if you're up?
Woo-hoo!
All right.
Okay.
Cool.
Okay.
So I showed you a bunch of features which can be boring, so I wanted to show you how
do we put it in action when you are using it within, you know, a specific use case.
So a specific use case here is we have Leo from sales, who's the data owner, and we have
Susan from marketing, who's a consumer.
Leo is going to be publishing a data asset into the environment, and Susan is going to
be finding it and using it.
Okay?
So let's...
All right.
So...
All right.
Leo is in.
We're not going to mess with the flow now.
So Leo is in.
He's into the sales project that you can see on the very top.
He's going to click on his project, and in here, you can see that he can see the data.
I know it's kind of tiny, but hopefully, you know, when you see it on YouTube, it will
be easier to navigate.
But you can see in the project catalog that he can navigate through all of his data.
In here, he's looking for revenue data.
So he clicks on that revenue thing.
Total revenue.
Right?
Here, this data asset, he can click on the generate descriptions, and this shows the
gen AI stuff that I was telling you about, where it's looking at the column descriptions
and the table and telling you what is contained in this particular asset, right?
So there's a detailed description that gets generated that says, hey, you can use this
data for X, you can use this data for Y.
He likes all of that.
He can add additional information, like readmes and glossaries and metadata forms to describe
the data in greater detail.
For Susan, who's the consumer who's going to be using it.
Here he's just picking up some glossary terms, which allow him to categorize the data.
So you know, all of the capability of really categorizing your data so that it's easier
to find and use is built into the studio.
In here, the technical metadata shows up at the very bottom.
So since this table was a glue table, it's showing the technical metadata from glue just
as is.
We bring it in from Glue Data Catalog and Lake Formation so you don't have to worry
about that.
It's all built in.
Also in here, you can add additional metadata forms.
So in this particular case, we want a business approval added to be used for this data, which
will be used in a subscription workflow.
So you can add that metadata form here.
You can make it mandatory, right?
So for any time anybody requests access to this particular dataset, they'll have to provide
that data.
Now we're looking at the column level detail now.
And again, it's all been auto-generated, the descriptions.
You can edit it if you want.
But it's a huge time saver to have those descriptions be generated by the code, because then you
can just edit it instead of having to recreate it from scratch.
Now this part I really like, because we are creating a row level filter here from within
the studio interface, right?
So what that allows you to do is for that particular asset, that revenue table, we are
restricting it to a particular category of sneakers.
So when I'm giving access to this particular dataset to a person, I can restrict it and
say, oh, you're from region northwest.
So you just get the northwest data.
So that level of row level filtering can be done within the subscription request itself.
So you don't have to create multiple views, which is what many people have to do right
now, because essentially we are doing that in the background for you.
And through this, we are managing the Lake Formation grants too.
So in the background, we are using Glue Data Catalog and Lake Formation to actually implement
this permission and enforce it on the dataset.
Now here you can see the lineage of how this dataset was created, right?
So the dataset is on the right.
We can see that three different tables were used to create this.
It went through a Glue Data pipeline, generated that asset.
We are able to monitor all of this and then publish the data out into the catalog.
So now when the catalog, the dataset is available in the catalog, then Susan can come in and
search for it and ask for subscription.
So now Leo's part is done.
So we'll move over.
So now you can see that the total revenue table, it's available.
So now Susan comes in.
And since Leo has already published the data, now Susan can go and find it.
So Susan starts from her main page, right?
And on the top, she's in the marketing project.
She searches for the data asset, total revenue or revenue, whatever she wanted.
She can see it.
She can find within, there's a number of facets that you can search by.
You can see all of the assets.
You can narrow down your search by multiple things.
As I mentioned, we've also added semantic search.
So a variety of stuff is going to show up.
If she said profit, she would have had a longer list of things show up as well.
She likes the search.
She finds the thing that she was looking for.
Now because of all the description that Leo added using Gen AI, it's very easy for her
to read through and say, okay, I can use this data.
It has this and this kind of thing.
It seems useful.
She's going to look at the lineage.
Looks good.
She hits subscribe.
So that button right there, the tiny thing that you can't really see, is actually a subscribe
button, which means it's going to generate an access request workflow in the background.
And the request now goes back to Leo to approve.
So she's saying here that, oh, I need data for my marketing project.
Give me access.
Submits a request.
Now this stuff currently is being managed outside of tools right now, right?
ServiceNow, Jira, I'm sure you have workflows of your own that you're managing in multiple
places to do this stuff.
But here it's all baked in.
So if you go to Leo, Leo goes back into his sales project.
He'll see a number of approval requests pending for him.
So he sees, okay, all right, subscription request.
What is this?
Clicks on it.
Sees that Susan has asked for access to this data.
She can click on, he can click on it, approve it from right there.
And when he approves it, we manage the grants to the data within Lake Formation.
So we're using all of the plumbing that we have built as AWS in the background to make
this workflow really easy for you.
All right.
So I think that he just approved it.
Now Susan goes back.
Within the same interface, now she wants to use a tool of her choice to use this data.
So she gets back in her project, and she can see if she goes in the data browser that she
now has access to that particular table that she looked for.
So in here, the total revenue thing is here.
She clicks on it.
She can look at it.
She finds it.
And now she has options to view it or analyze it with multiple different tools.
So now for here, their admin has allowed them to use Athena and a Jupyter Notebook.
So that's the options that she sees.
She can use one of them to go look at the query.
Now here she generated the query, and you can see that only sneakers show up because
that was the category that was applied as a low-level restriction for her.
So it's become super easy for you to go through that end-to-end workflow.
I know it's kind of tiny, so I'm hoping that you can check it out yourself and play with
the studio a bit.
It's very powerful.
We are bringing all of these things together so that you don't have to build the capabilities
yourself.
So I'm very excited for you to check out and give us feedback on what you think this enables
for you and what kind of stuff that we should add.
So with that...
Was that helpful?
I know it was tiny, but was that helpful?
Yeah?
Okay.
Cool.
All right.
So let me invite Shaja on stage to share the journey that Cisco has gone through for their
analytics and data governance.
Welcome, Shaja.
Yeah.
The green button to advance, red to go back.
Good afternoon, folks.
Thank you, Shika, for the awesome introduction and demo.
So I want to share some real experience related to what she spoke of in the context of Outshift,
which is a business unit within Cisco that I'm part of.
So let's get started.
So Outshift is a business unit which is focused on innovation.
So in simple words, think of Outshift as the innovation hub for Cisco.
It's a relatively new business unit where we are primarily focused on doing innovation
in terms of building, like, new ideas, prototypes to existing adjacent, like, businesses as
well as to kind of, like, new verticals, right?
So I lead some of the data science efforts as part of this business unit, and I want
to share some experience related to some of the challenges that we face, the lessons learned,
and how we are using, like, DataZone.
So let's dive in.
I met, like, a lot of my friends at reInvent, a great place to meet a lot of people and
build your network.
So I asked them, like, okay, how do you manage your data?
Some of my friends said we use, like, SharePoint.
Some others said we use, like, data warehouses.
Somebody said, like, we use, like, data lakes.
Somebody else said, like, lake house, whatnot, right?
I'm sure, like, this resonates with a lot of you.
And it's not just, like, an out shift.
But if you look at, like, any typical organization, data is siloed in its nature.
So what I'm just trying to say here is if you are trying to innovate on something, let's
say in my context I have to take some data which is coming from a router, some data which
comes from a video streaming application, which is running in an application layer,
where it kind of, like, generates some virtualization data.
And then there is, like, some physical layer data related to measuring temperature, a lot
of telemetry, and so on and so forth.
Given its nature, it's not on purpose.
Data is siloed, right?
And that's true for several organizations.
Nobody wants to kind of, like, keep data in a way.
But just if you want to do something really innovative, then data, when it comes in its
raw form, it's, like, siloed.
And it becomes, like, a huge barrier.
So that is, like, one of the problems that I have seen.
And it's echoed by a lot of my colleagues and friends as well.
Do anyone have, like, problems related to data silo in your organizations?
Yeah, many of you, right?
The second problem is when you are doing innovation, data comes with its own governance challenges.
What do I mean by that?
Somebody would say you can use this data only for this particular purpose.
Somebody would say you cannot use it for giving it to somebody.
And so on and so forth.
So you have your own complex, like, governance rules and policies.
Sometimes at a row level, column level, sometimes, like, it's related to PI redaction and so
on and so forth.
So this component brings in another layer of complexity when you want to do innovation.
So that is the second challenge that I want to highlight.
The third challenge is I think we are living in a world of generative AI, where anybody
and everybody is doing, like, machine learning.
And what happens is many times, like, these ML efforts gets, like, siloed, and it's very
difficult to discover and kind of, like, share and understand what is, like, building what
and how and long story short, like, you end up getting to a phase where many people are
working on the same thing with very small difference in terms of approaches, essentially
ending up a lot of useful resource and time, right?
So these are three main challenges that from my experience I felt we had while we started
our journey.
And then we went to the drawing board, saying, okay, if you want to be on top of innovation
where we have to develop, like, cutting edge products based on generative AI and so on,
how can you go about, like, addressing it?
So I went and did, like, a lot of research in terms of reading, talking to a lot of consultants
and so on and so forth.
And what we ended up figuring out was something called as, like, data mesh, right?
How many of you have heard of data mesh or used data mesh?
So many people.
Makes my life easier.
So just for the other people, like, just to kind of, like, reemphasize how I think data
mesh helps to solve the problems that I have highlighted in the previous slide.
The first thing is it gives you, like, the domain ownership.
What it means is a person who is creating, like, the data which is simulated, say, from
a router or a particular hardware, we don't have to say that give all your data, I'll
take a copy of it, put it into a common place that others can use it, right?
With data mesh, the idea here is we say that, okay, if you are generating this data, you
know best about this data.
You own it.
You don't need to kind of, like, give it to an ETL engineer who wants to claim or, like,
own this data who has a very poor understanding and has troubles maintaining it and so on.
So the first principle is in data mesh, like, the ownership is given to people who produce
this data and who has good knowledge about this data, right?
Data doesn't move.
The second thing is data as a product.
What do we mean by that is many times, like, we see that people kind of create a copy of
this data, toss it on, pass it, and then kind of it becomes stale.
So what data mesh advises is treat data as a product.
In other words, similar to a product has a good product document, it has a good description,
it is, like, well maintained, it is continuously being updated, the quality of the product
is governed and is, like, well tested and so on, the idea here is treat your data as
a product so that it doesn't become stale and is, like, well maintained.
Similar to how a product is kind of, like, praised for its quality and how many people
are using it and its success, the idea here is the same, which is use data as a product
instead of just another byproduct, thereby people understand the value of it and use
it at the highest capacity.
The next one is when we want to start using data, the idea is it should be available in
a platform which is, like, self-servable, right?
So which is where I feel I think something like data zone is cool.
So it should be in a platform when you wanted to put the data into practice, people should
be able to just use it instead of having to go through a lot of, like, learning or documents
or things like that.
It should be in a way that it enables, like, data to be served, like, self-sufficient.
And the last component is the previous challenge I highlighted in the previous slide.
Even though data is distributed, it cannot be just, like, everyone can define how they
want to do with it.
There should be some common principles which can be distributedly managed, and that's where,
like, the federated computational government comes in, and that is another critical component
that we are looking for to solve this challenge.
And long story short, we felt that to solve the challenges that I had previously highlighted
in the previous slide, data mesh, if there is something which could help us implement
all these four principles, we thought that would solve our problem and help us to do
a lot of innovation.
And then we ended up, like, using data zone.
We started, like, using it for some of our use cases.
And I want to quickly highlight a couple of use cases on how we are using it.
And I'm available here to talk into more details in terms of what research we did related to
understanding several other things and how we arrived at this particular one, right?
So a little bit of context.
So let's assume two personas here.
One is, say, an analyst persona.
She is interested in understanding when a particular application is crashing.
Let's say a particular video streaming, like, application is supposed to stream at a particular
bandwidth, but now it is not streaming, and it is giving, like, errors once in a while.
And she is interested in analyzing why the error comes from.
And this error can come from the application layer or it can come from the Kubernetes layer
or it can come from a particular router might have crashed or somebody might have configured
a policy in a router or maybe, like, the physical hardware where some of these routers are running,
a fan in a router got, like, overheated and the temperature is causing things to go down
and so on and so forth, right?
So if you see here, to debug why this particular error is happening, the application data by
itself is not enough.
You need to span across the application layer, virtualization, Kubernetes layer, the network
layer, and the physical layer in order to understand what is really causing the problem.
So how she used to solve this problem before was she will try to get, like, permission
for this data from different people.
That will take its own time.
And then assemble all this data, which is in different shape and form, bring it to a
common place, and then start creating dashboards and do, like, some analysis and so on.
I'm sure this story resembles with a lot of you.
This is how, like, we used to do it before, right?
But now after implementing data mesh on top of, like, data zone, our life becomes easy.
As she showed in the demo.
She doesn't have to go all over the place.
She can look into the catalog, search for all the data which is required across, like,
different layers.
And kind of those data is not, like, redundantly copied.
They all sit in its own place.
And from, like, kind of like a single place, she can very efficiently accelerate her journey
towards diagnosing what is causing the particular problem, right?
In a similar way, like, when� so what I'm just trying to say here is with the help of
this picture, what you see here on the producer side, which is in the left-hand side, you
can imagine a producer could be, like, the application logs, and the other producer could
be, like, the router logs, the other producer could be, like, the physical layer logs, and
the other producer could be, like, the Kubernetes logs and so on.
And then these logs can come into and get, like, produced in different shape and form.
We take advantage of, like, the catalog and the inherent, like, automated, like, data
quality checks it provides in terms of when somebody gives me access to those log files,
we take advantage of the features that we have from data zone related to is the data
complete and is it, like, completely populated and so on and so forth?
And then this data is, like, being produced and is owned by different owners, right?
It is not just, like, one team.
But it's, like, different cloud accounts, which are, like, produced and kind of, like,
is managed by different units themselves.
So it may not be just, like, one producer or one account where you need to have, like,
multiple people, but it can be, like, multiple accounts and so on and so forth.
So all this data is made centrally available with the help of, like, the Amazon, like,
data zone catalog, where the discoverability happens.
And the analyst who is, like, interested in kind of, like, debugging it, she has visibility
into this catalog.
She looks into this catalog and then have a very good visibility to understand the entire
data and pokes in and requests permission and subscribes to those data points and gets
access to it to quickly diagnose, right?
The last use case that I want to talk about is think of, like, a data scientist who is
interested in creating a machine learning model, be it a traditional model or it is,
like, fine tuning of an LLM.
It really doesn't matter.
But how the machine learning life cycle used to look like is similar to the personal journey
that I described for the analyst before.
So he tries to get access for a lot of this data.
He then, like, tries to create different versions of this model, benchmarks it, and keeps it
in one place, and nobody knows and gets access to that, and, like, the visibility of this
model by itself gets difficult in addition to getting access to this data and creating
this model, right?
So now how it is being done is similar to the data analyst journey.
Data is kind of, like, completely decentralized.
So the data scientist gets a lot more innovative in having, like, the capability to play with
different forms and shape of data.
The data is managed as a product.
So it is maintained with extremely high quality.
And he has no access to this data, creates this model, and what we also take advantage
in data zone now with integration with, like, SageMaker is this model's visibility is also
made very importantly, like, available across all the people who have interest.
So that, like, they can reuse this model instead of, like, trying to develop it from scratch
and so on, right?
So long story short, we have started using data zone, and so far it has been a great
and interesting journey for us, which is helping us to accelerate and do a lot of innovation.
It doesn't by itself do magic, but I think it's a bridge to connect your data to innovation.
So that's a quick journey that I wanted to talk about.
So now I wanted to call, like, Travis on stage to talk about an interesting story that he
has to share from the Weather Company.
Thank you.
All right.
So my name is Travis Smith.
I work for the Weather Company.
You may know us better as the Weather Channel or weather.com.
We also do WonderGround and wonderground.com.
And to give you a little idea of scale that we do, we have over 150 billion requests for
weather data per day.
We have over 25 billion forecasts produced every day, and then we ingest and process
about 400 terabytes of data every day, resulting in super lot petabytes over the course of
a month, if you will.
But let me ask you guys a few questions.
And I know you guys have just had lunch, and you're kind of a little asleep.
So do me a little favor.
Don't leave me hanging when I ask these questions.
How many of you guys, as product people, going up, trying to figure out what to do, have
gone to your data people and said, hey, I need this data, but it was in this one meeting
at this one time where we talked about this one thing, and I'm not sure where it is.
Who do I need to talk to?
That's never happened, right?
Of course.
How many of you senior leaders, executives, have always ever asked, why does it take so
long to get this data?
Why can't you just tell me in the next five minutes?
Data teams.
Have you ever received any of those questions where you're kind of like, what?
You just want me to make something?
It's kind of like that expert with seven lines all perpendicular to one another, but they
can't touch.
Very similar.
So for us, we had very similar problems and similar issues.
And for us, the Amazon data zone became a massive game changer.
And it allowed us to seamlessly manage, catalog, govern vast amounts of data very seamlessly
across all the teams, enabling quicker insights and improved collaboration.
The enhanced accessibility enabled us to deliver more contextualized and personalized forecast
insights faster than ever before.
Which was extremely important as we scaled to the 450 million users that we have.
When we look at the problem, we had a threefold problem.
One was enterprise data catalog and data glossaries, if you will.
We have imagined having millions and billions of data points and data elements and no idea
how to find the data set that it has.
It's kind of like having the world's largest library and no catalog system.
That's what we had.
We were using a spreadsheet to manage our data glossary.
Imagine the steps stepping on top of one another, the ownership and the blame that happened.
We also wanted to talk about our data provenance.
So what I mean by that is how the data moves through the various systems, how the data
We wanted to create trust and transparency to make sure that we were within all the legal
compliance rules, et cetera, et cetera.
We had no idea what was going on with our data.
This was like taking a raindrop journey from a cloud to the floor and trying to predict
that transgression or whatever it's called and be able to tell someone about it.
We just couldn't do it.
It was impossible before DataZone.
Data exploration.
Everybody wants to explore their data.
Everybody wants to see what's going on with their data.
We wanted to enable that for everybody.
How many product people have ever had this issue where I've got this brilliant idea,
but now I need the data to prove that I need to do it and write my justification, and I
have no idea where that data is?
We want to enable that for them as well as for our BI analysts and our data scientists,
as you just heard from others.
We had a variety of use cases.
How many of you guys are in legal and privacy and you're sitting here looking for a technical
solution to the problem and able to enable the business and protect the business?
These are our guardians.
They keep us out of hot water.
They protect us from stormy weather, if you will.
Without these people to be able to protect us, it's impossible.
They needed to come in and look at where the data was, look at the data flows, and they
always came to us.
What data is being used by this product or that product?
We didn't have the answers.
We had to go to the technical team to figure out how they sent data to us and how we sent
data back to them, and then figure out which data sets and data tables it was.
But now, they can go in and look at it for themselves.
Next one is product teams, as you guys saw.
You guys are innovators.
You're planning out the next product, planning out the next feature.
You guys got to be on block to get answers and answers quickly, sometimes even in real
time to make certain decisions.
Our data stewards, they are our quality champions, if you will.
They make sure all the data is nicely tied up.
They make sure the data is properly documented so that it can be used throughout the systems.
We have our system owners, which are our technical experts.
They know their data better than anyone else, but they may not be very technically capable
when it comes to data engineering, so they lean on us for it.
Then we have our BI and data customers.
They really need the data.
They need clean data, and they need to access it fast to create those insights that the
various executives may need and want.
This is the interesting part.
I'm an architect, so I love nice, fancy diagrams.
On this diagram, we have a lot going on.
Here, we created something what we call Data Gov Ops.
Now, you guys are like the ML Ops, the Data Ops, et cetera, et cetera.
This is our fancy word to say we're merging the disciplines of data governance with operational
efficiencies.
Here, in our diagram, we are a data fabric, not a data mesh.
The diagrams look similar because you have producers and consumers, but let me describe
it for you real quick.
A data fabric is like the spine in relationship to the neurons and the signals that the brain
sends to the body.
The brain is like your decision makers, your BI, et cetera, and the rest of the parts of
your body are like the consumers and the producers, but all the data flows through the spine.
Likewise, for us, all the data goes in a singular direction and sits in a singular place, but
producers and consumers can be left autonomous.
There is a collective side as well as an autonomous side, and our whole goal is to make sure that
people are unblocked and get what they need when they need it, how they need it.
Here, at the bottom right, we have our on and off properties, sources, so that would
be weather.com, the applications, the apps, as well as our third-party partners.
They send the data into our data lake and data warehouse through streamings, through
batch, through processing, whatever, and that data is made available to share and govern
in the Blue Ocean data governance account.
Likewise, if we have teams that are capable enough for themselves to be data product people
like a data mesh, so I call this kind of like a pseudo data mesh when you have producers
at the top that they're sending their data straight into the data governance account,
but more in reality at our company, that producers actually go to the right to the data warehouse
and then comes to the governance, and then consumers always access it through the data
governance account.
Does that make sense?
Hopefully it does.
A picture is a thousand words.
So we had several business outcomes, technical people bear with me a little bit, so I'll
walk through this real quick.
We enabled centralized oversight and access to the teams.
This enabled us to no longer step on each other's toes as we're not using a spreadsheet
anymore.
We're actually using a system that is fully documented, fully enabled.
We empowered data source owners to be able to control who has access and when they have
access, and next we have streamlined privacy reviews.
I don't know about this in your company, but whether you have OneTrust or any other type
of governance system, each technical owner of the system had to go in and answer a bajillion
questions and then hit approve and so on, but with data zone now, this privacy review
of how data flows is no longer necessary to the granularity that all the various technical
teams had to do.
Instead, they can go into data zone, look at the glossary, look how it's used, look
at the lineage, look at the visualizations, and see and answer their own questions.
That saves me time as an architect, it saves my technical leaders time for more nitty-gritty
definitions, and it's a huge win for us.
And then, obviously, you get the accelerated product decisions and integrations.
Product leaders, you never have to go back to a data guy or a gal to see whether the
data exists and if it can give you a graph, and I'm not sure if you guys ever hear this,
but yeah, the data's no good, but here you go.
Never happened to anybody?
All right.
Technical outcomes.
My favorite one was the glossary.
We, as a media company, are heavily governed by GDPR, CCPA, and all those other fun acronyms
of the various privacy regimes, and we have to publish a public glossary.
Again, this was done in Google Sheets for years.
It's painful for me as an architect.
I couldn't get us out of that.
It saved my life.
Finally, when data zone was announced, we were able to get out of it, and it's a beautiful,
wonderful thing.
We facilitated cross-team data access, as you guys saw, the federation of the various
data governances where owners can go in and publish, say, yes, who can access it and who
cannot.
When we start talking about various systems of our data, whether it's PI, SPI, people
only need access to what they need access to.
Not everybody needs access to it, so executives don't necessarily have access to it, but our
data scientists, for the purposes of personalization and segmentation, can leverage it and anonymize
it in a pseudo-anonymous way.
Then finally, we have onboarded diverse data sources.
One of the big problems that we had was we were trying to find a solution.
Due to our volume, we were very cost-prohibitive on a variety of solutions, and at that cost,
they didn't necessarily have everything that we wanted.
We were a full native 100% AWS shop, and so by leveraging data zone, granted, it didn't
have everything initially, but as we grew and implemented, it does, and we were able
to crawl and ingest all our sources into the data zone, and really had no problems
whatsoever.
Then finally, establishing tagging guidelines.
Have you guys ever experienced where you have team A come over to you and say, hey, I want
data so-and-so?
Then team B comes over to you, hey, I want data this and that, and then you come to find
out it's the same data?
That's never happened to anybody?
Yeah, okay.
This tagging and governance and taxonomy and descriptions, so when someone says, hey, I
need this data, everybody knows what that data is.
They know what it means, and they know where to find it.
It moved from who do I need to ask about that to, oh, I need to go look up in data zone.
The other day, I was talking to my enterprise data architect, and we were just talking,
and I had a random question out of the top of my head, and I said, hey, where is this
data and where can we do it?
He looked at me like, data zone.
Just walked away.
I'm like, oh, yeah, I did that project, my bad.
So even I, where I'm constantly pushing our envelope and wanting to do this and even help
build some of it, design it, I still had to go back and remind myself that we actually
have to do this data zone thing, too, across the entire business, and I don't necessarily
get to take shortcuts.
Here's our roadmap for the next couple of months, if you will.
We're currently in the process of making data GovOps a core discipline across the organization,
and then we're super excited about the SageMaker Unified Studio to get that adopted and moving.
Currently, this is only adopted by the consumer team, so we're looking to expand that into
the weather sciences, so when you guys think of the weather channel, you all think of the
forecasts and the weather data.
That's the other side of the business, and we want to bring them along, and we ingest
over 400 sources there, and to see a data lineage there is just tremendously impossible,
so we want to make sure that they can do it and B2B operations as well.
We want to make sure we are fully native compatible in our systems with open lineage and for data
visibility and observability and further integrate into various other data zone offerings as
well as other AWS offerings to enable full data democratization and self-serve.
All the fun, fancy words.
Before I finish, I always like to get a fun little selfie.
Do you guys mind giving a selfie, and if I could get my partners in crime to come up
here?
You guys are super excited.
You know, this is going to be really painful, because the spotlights.
Yes, we will.
All right.
Here we go.
You want to come closer?
You can.
We got everybody.
You can wave.
Make signs.
All right.
One, two, five.
All right.
Fantastic, and if we can go ahead and open it up to Q&A.

