Analytics in 15: Big Data and Data Governance with AWS
1 Video View·Nov 17, 2024
Running big data workloads while ensuring governance and establishing end-to-end security is challenging. Amazon EMR makes it easy for data analysts and engineers to run open-source big data analytics frameworks such as Apache Spark, Hive, and Presto, while AWS Lake Formation automatically and securely manages data through multiple AWS services. Learn more about the big data processing and data securing capabilities of Amazon EMR and AWS Lake Formation.
Hi, everyone. I'm Arati Srinivasan.
I'm a senior big data architect with the AWS Lake Formation team.
I'm here to present and talk to you about data governance using Lake Formation.
Now, let's see what is data governance.
What does this data governance term mean?
This has become very popular these days in the data world.
So data governance means we centrally define, manage,
and control how data is managed in an organization.
You have a uniform mechanism to categorize the data,
define need-based access policies for each team member and user groups in your organization.
You'll have a central mechanism to define user access policies
so that any new user and any new data set management becomes easier.
Now, you should also be able to automatically detect as the data grows, right?
So you collect more data from various sources.
You need to be able to automatically detect sensitive PII information,
provide masked data sets for the teams to use
and track further the usage of those type of data especially.
You'll create effective audit mechanisms on who accessed what data sets,
at what time, using what tools, and using which permission set.
And you should be able to further trigger alarms,
review those alarms, take actions,
and create good audit reports on data usage
as per requirements for your auditing mechanisms.
Now, it's not just enough to secure the data.
We need to make the data available to different teams for different purposes
so that businesses could extract timely insights from this data.
Data scientists, analysts, business intelligence users,
data engineers, architects, and DevOps.
Each role will need to use their own data analytics tools
that they are skilled at.
And hence, you need to have a good choice of tool sets for each job role.
Now, with having defined data governance,
let's see what are the challenges in building a data governance model
and how these challenges can be overcome.
A variety of...
The very first challenge would be your variety of data formats
and a variety of data sources.
You could have data in on-premise servers.
You could have clickstream data coming from mobile applications
and various websites for your organization.
You could have structured data sitting across a variety of databases,
old and new database engines.
And you could have log applications, streaming data.
And so, hence, you have a variety of data formats in ORC,
Parquet, a variety of formats, and a variety of data sources.
And it doesn't stop there.
Having the next challenge would be the storage and maintenance challenge.
Once you gather data from a variety of these resources, sources,
you will have to clean and classify them into different data usage types.
You can start gathering all your data into a first layer of storage,
starting from the raw data storage buckets.
The next would be your cleaned and curated storage buckets.
And the third layer could be your consumption-ready
and aggregated storage buckets.
Now, as your data sets grow across a variety of these S3 buckets,
you know how tough it is to manage these S3 access permissions
for various analytics.
S3 policies, bucket policies,
and the corresponding IAM principle policies
could get very cumbersome to write and manage
and continuously keep these updated with the growing data sets
and your growing user sets.
It's going to be very tough to manage.
And the next challenge comes in cataloging all these structured
and unstructured data in your organization.
One set of catalog is Glue Catalog
and any other coordinating cataloging mechanisms in AWS.
And catalogs could be in a high meta store,
sitting across a variety of catalog systems,
third parties, and other open source catalogs.
So it's extremely important that the metadata
should be continuously in sync with the changes
in the input downstream or upstream data sets.
Now, the next challenge is securing
and sharing these data sets across accounts and organizations.
You need to meet your strict complaint standards
like GDPR is the highest priority,
and you need to have auditing mechanisms
that are consistent, repeatable,
and you should be able to create audit reports
that are easy to create and consume across the organization.
The next set of challenges to make the data discoverable
and available to a variety of data analytics tools,
machine learning and deep learning models,
and making change to these permissions
should reflect across the stack.
Now, above all, whatever governance model you come up with,
it has to scale well with the growth of data volume
and the number of users and tools.
So now we know what are the challenges
in these data governance or building a data governance model.
Now let's see how AWS Lake Formation comes into the picture
and helps to address some of these challenges
in building a data lake and building a data governance model.
AWS Lake Formation is a fully managed service
to build, manage, secure, and share data in data lakes.
The time to implement data lake
is simplified from month's timeline to day's timeline.
Lake Formation provides an RDBMS-style
grant revoke permissions model to IAM principles
and SAML-based Active Directory users
on databases and tables cataloged in Glue Data Catalog.
The underlying datasets are sitting on S3 buckets.
Lake Formation removes the need to write
and maintain complex S3 bucket policies
and the corresponding IAM user policies,
IAM principle policies.
The next level of depth is how fine-grained
can Lake Formation be used to define permissions
to these database and tables?
Lake Formation can be used to grant permissions to users
at the column level, cell level,
and using data filters in the concept called data filters.
Lake Formation can do fine-grained permissions
for up to cell level.
Next is how can we scale the permissions
using Lake Formation?
Do you have to constantly keep writing new grant policies?
No, not at all.
Lake Formation has a concept called Lake Formation Tags,
which provides the tag-based access control.
These Lake Formation Tags are different from IAM Tags
and AWS Resource Tags.
Permissions definition scales well
with these tag-based access control.
Tags reduces the number of grants
that you need to define every time you create new tables
and you create new users into the picture.
You create a tagging strategy suitable
to your organization's data access methods.
You tag the users, you tag the resources,
and define Lake Formation tag-based grants.
Now, for any new users and any new resources
that gets added,
all you need to do is tag them relevantly.
Their permissions are already defined,
so you don't have to do anything else.
The user gets predefined access to new resources
based on the existing tag permissions.
Next is how easy it is to share cross-account.
In Lake Formation,
sharing a resource across the account
is similar to sharing within the account
to any new IAM principles and Active Directory users.
So using these cross-account shares,
it's as easy to share
and track using CloudTrail audit logs.
These CloudTrail logs are available
for both the sharing account
as well as the recipient account,
so you could come up with any additional models
to build reports and regularly maintain these audits
using the CloudTrail logs that Lake Formation provides.
The last but not the least,
the more effective point is the integration
with a variety of tools for data analytics.
Athena, Spectrum, Redshift Spectrum,
Glue ETL, and EMR, SageMaker, QuickSight.
And as the service continues to evolve,
more choice of analytics engines,
both from AWS and third-party engines,
will get integrated with Lake Formation.
Now keeping all those challenges
and available solutions in mind,
organizations are adopting
a more modern data governance pattern.
These are continuously evolving patterns
and highly scalable
and custom-built to every organization's needs.
Similar to how applications evolve
into microservices supported by open APIs,
modern data architecture involves a transition
from data as an asset mindset
to data as a product implementation.
What this means is that the team
that produces and owns the data
gets to manage its full lifecycle,
starting from the data ingestion at the source
to how it is consumed by others downstream.
We will see some of or a few of these variations
in this modern data architecture patterns.
Shown here is the very first one,
which is a central data governance model.
Here you see producer accounts on the left
and few consumer accounts on the right.
The central account is the central data governance account.
The entire account structures
and the architecture pattern is set up
by a centralized data governance team.
All the producers are given access
to catalog their metadata of databases
and tables in the centralized blue catalog
of the central account,
and Lake Formation comes on top of it
and is used by the producers
from each sharing account.
The producers come here to the central account
and start writing the Lake Formation permissions
for their respective database and tables.
And the producers own the permissions model fully
on any consumer who wants to access,
they discover the glue data catalog,
which is fully readable access
across the entire organization.
Each consumer, whoever wants to access the tables
and databases for further analysis,
they request access to the relevant corresponding producers.
And the producer grants permission,
the respective describe, select, alter table permissions
and column level permissions,
restricted column permissions.
Using data filters, you could restrict further down
into the table access for each individual consumers.
This is a holistic model
where the central team could create
and maintain the required data governance audit reports
for the security compliance model.
The next model that we'll see here,
in the second model, there is one central Lake Formation,
in the second model,
there is one central Lake Formation account
for permissions management within each line of business.
So each line of business or department
gets to manage their own data set fully,
and the data is visible only within their line of business.
The producers and consumers are within the same organization.
And each central data governance account
for each maintains their own auditing and reports.
You are still having the options
to share the data cross account as needed across their lobbies.
These are still AWS accounts anyway,
so each account will have its activated Glue data catalog
and Lake Formation settings.
So once that is set up,
sharing data across accounts is very easy.
And the last model is a quite open model.
There is no centralized Lake Formation
permissions management.
Each producer account serves the data directly
from its accounts, Lake Formation
and Glue catalog settings.
This avoids any additional hops
to a centralized Lake Formation account
and waiting for permissions.
The producers can also be consumers
in all the three models.
Producers can also be consumers to other accounts' data sets.
These are the three to come up with,
but as per the use case,
as your use case demands in the organization,
the data access models could be different,
and each company is free to come up
with their own data set models.
To recap, we saw what data governance is,
what are the challenges involved in building
and maintaining a secure data lake
with good data governance strategy,
how Lake Formation solves some of these challenges.
We also reviewed a few modern data architecture patterns
using Lake Formation.
I'm sharing a few additional resources here
to get started with Lake Formation.
You could use the workshop, documentation,
blogs, and well-architected patterns.
I hope you enjoyed this presentation.
Please reach out to me for any additional questions.
Thank you.

