A Realistic Day in Life of Data Engineer | Work, Harsh Realities, More.

Serenity Space
31 followers
#learning #dataengineering #roadmap
In this video, I am covering everything you need to know about Data Engineering. What does a Data Engineer do? What does a realistic day in life look like? What are some harsh realities of being a Data Engineer? And of course, what skills are important to become a Data Engineer.
Here's what I actually did as a data engineer and probably there is no better time than
right now to talk about it because almost every report suggests in 2025 that data engineering
is set to outpace almost every other role in the tech industry apart from AI engineers
and also a lot of surveys show that for every data science a company has to hire at least
three data engineers. So the job is blooming and for good reason. It's a field that I know
really well. I've been in the data space for more than six years now and I started my journey
at ZS as a data engineer, stayed there for three years and then I went to Google where
I worked as a data engineer for first year and then transitioned into an AI engineering role
and for more than a year I've been a senior software engineer in AI engineering space.
Even now I do a lot of data engineering because it's just unavoidable even when you're in AI
field. Today I want to use all of that experience to tell you what a data engineer does realistically
on a day-to-day level so that you can make a decision for yourself whether it's the right
field for you. I'm going to say everything, the good, the bad and also the code that often used
to break at 3 a.m. So let's jump right in. Okay so first of all let's talk about what even is a
data engineer. So let's take a house analogy. Okay so you live in a house and data is water.
What are data scientists and data analysts then? Well they are the ones who use this clean water
to let's say cook and clean and maybe use it in the dishwasher in the final product.
Data engineers are the ones who build the pipelines, who maybe harvest rain water, make sure
that the water that you get out of their outlets are completely clean. Probably with this analogy
you can now understand the importance of data engineering in data world. They basically build
and maintain huge pipelines that collect large amount of data, process them and also store them
efficiently. A data engineer's work mainly boils down to three things. One, requirements. Two,
building and three is maintenance. Let me give an example. So a data scientist would come and say
that hey I want last two years of clickstream data so that I can make a user recommendation model.
That becomes your business requirement. I used to ask a lot of clarifying questions like hey
what are different data sources? What's the size and scale of data? Is it gigabytes, petabytes,
terabytes? How fresh does the data have to be? How do I expose it to you? And generally you
would need to write all of these things in a design document and you need to make sure that
you get everyone on the same page before you write a single piece of code. Now next step of
the job is building and this is always a multi-step process. Now first is data integration. So data
always comes from n number of different sources. Maybe using third-party APIs or relational,
non-relational databases or maybe just through cloud storage or some pdf files or csv that's
not properly maintained. Some would have dates in dd, mm, yy versus others would have date in yy,
mm, dd. So you would have to do a lot of data cleaning, quality checks and you would have to do
a lot of standardizations as well. Second is data storage. So once you have all of the data,
you need a central place or central repository to store all of them and usually this would happen
in a data warehouse or a data lake if you are cost-sensitive and most of the time it would
happen on one of the cloud platforms. Third is where the engineering happens. This is where you
write your ETL or ELT jobs, maybe joining the tables, doing some aggregates and finally getting
the metrics that you want whether you are running it on Spark or SQL or Flink or doesn't matter what
the platform is. So you have to understand the architecture of that platform, you have to
understand the programming language which is often PySpark, Python or Java and you also have to
understand the SQL fundamentals really well and finally there's data serving. So you put it in a
business-friendly system, something like let's say Snowflake, where a lot of data analysts and
analysts could just write SQL queries. They don't have to worry about your complicated Spark or
Flink architecture in the backend. There's maintenance and optimization as well. So
sometimes let's say if there's a data spike, your Spark cluster will start failing, maybe you'll
have to tune your auto-scaling policies to get away with that. It's an ongoing thing and I would
say it's the most unglamorous part of the job. You will probably see day-in-life data engineering
videos, even I have a couple of them on my channel. So it's not always as glamorous as it seems on
YouTube or Instagram. So here's what I did as a data engineer on a daily level. So my morning
usually started around 7 30 a.m. This is where I'd have some coffee, breakfast. The first thing I did
was to check my emails in Slack to see if we have something urgent for the day. I was looking for
red flags like any pipelines that failed overnight, if a critical financial report didn't update.
After having some shower at about 9 30 to 1 30 was my deep work time. I put on some lo-fi
music, turn off notifications and just get in the zone. So let's say if you are an order
fulfillment team, you would build data pipelines that would give you some of the important metrics
to measure the success of your team. You would calculate all of these metrics. This meant writing
PySpark, SQL or Python codes, defining the data structure and setting all of these things in a
cloud infrastructure. And this is where the real engineering happened. Around 1 30 I usually have
lunch and this is something that I always order online since I didn't really have any time to
at all. Again at around 2 30 I'd resume working till 4. This is where I like to do maintenance
work. Whether it was hitting the gym or just going for a walk, the break was non-negotiable
for me at this time. I used to work in roles where I'd eat and sit at my desk every day and
I realized a good career is nothing without a good life outside. So this is my gym where I used to go
in the evening and work out and I used to take a walk from my home to gym because it's like a
After working out I'd take a quick walk back home. I do take some time to enjoy the evening
like this taking a coffee from my balcony and then I go for a walk at night. There's a beautiful
garden very close to where I live. This video was shot during festivities time so it's all lit up
and then my late evenings were kind of more collaborative in nature filled with meetings.
That's one thing I did not like because I've been working with US based teams.
A lot of times there's meetings even from 9 pm to 11 pm so that's not really great but I mean
you can't really get away with it at least two to three times a week and then when I don't have
meetings at night I either like to play with my friends or just chill and watch Netflix before
ending my day. Now the thing about data engineering is a lot of work is invisible. That means nobody
notices your work really until it fails. It's like being a goalkeeper in a football team. This is
where we would have to drop like an email to entire company saying that hey we are late in our
reports we messed up in our pipeline and we are trying to fix it. Also huge part of dealing with
that invisible work problem is finding collaborative tools that increase your productivity.
One such tool that I would like to talk about today is an AI presentation building app Gamma.
So if you are still stuck on using google slides or powerpoint to make presentations and proposals
you're doing it wrong because gamma is a game changer and it makes creating ppt so easy. As an
engineer my favorite part is how it can translate complex idea into something really concrete and
presentable to end stakeholders. Say I just finished building that data pipeline I mentioned
and instead of a boring text document I can just give gamma a prompt it will instantly generate a
stunning deck. They just launched these incredible new AI image editing and animation features that
help you bring ideas to life. Take a look at this timeline file I can just click and I can just say
make it more look like a roadmap where you have different phases and it's more easier to read.
You can also just edit images just by describing what you want. Here I would like to change the
art style I would like to make it more realistic and I would like to have some AWS components
sprinkled here and there so you can see how it intelligently created things like
kidneys lambda all of those things in my intro. I would just like to add my photo here but also
would like to remove background with one click I can simply do that and edit my images. It takes
a lot of your work off of your hands and it makes your deck look extremely professional.
With over 50 million users in the world it is the most popular AI presentation app in the world
whether you're a student slash fresher or an engineer or a product manager no matter what
your role is in software field I think apps like gamma are real time saver. I'm going to add a link
in the description so don't forget to check it out and thank you gamma for sponsoring this part
of the video. All right now back to the harsh realities being on call so obviously on call
schedule keeps on rotating so let's say one week I'm on call other the week somebody else would be
on call but when the week when you're on call you have to just handle everything that goes wrong so
even if a pipeline breaks at 2 a.m you have to wake up and fix it and then finally the third harsh
reality I want to talk about is the constant pressure to keep up so there are new tools
coming in every time and even let's say take an example of spark the current version of spark
lets you do a lot of things that couple of versions back just six months back you couldn't
do with spark you have to stay up to date with all of these recent releases and you have to
ensure that you use those features in your data pipeline to make your work easier also in
interviews also you can expect a lot of these questions on the latest releases despite all of
those challenges I loved being a data engineer and it came down to mainly two things one was the
challenge and impact so I really like solving puzzles and solving you know huge problems where
you have let's say billions of records or events coming in and then you translate them into
something usable you're like an enabler for the entire company and the next thing is pay and
lifestyle let's be honest like the pay of a skilled data engineer is really high compared to many
other roles in software industry flexible work hours unless when something goes wrong so also
you don't have to really go to office every day because it's a job that you can easily do remotely
as well allowed me to you know follow passions outside of my day job as well and one of those
things is talking to you on youtube so the big question is should you become a data engineer
well in my opinion learning the technical stuff like let's say spark sequel python all of these
things is actually the easy stuff you know what's hard and what separates a good data engineer from
a great one and what helps you get promoted as well it's the soft skills it's the ability to
manage expectations from your stakeholder ability to maybe move around some of the things in your
timeline so that you meet the most p0 impact okr ability to translate an ambiguous complex
business requirement to a very clarified and step-by-step technical requirements and generally
just being a good person that people like to be with i mean these are some of the skills that
would make or break your career and i get it learning all of these things along with the
soft skills it could be tough when i started out it was tough for me as well it was like i was
drinking water through a fire hose and that's why i've also created some of the roadmap videos that
you can check out i'm going to link them in the description that's like a step-by-step path
of how you can become a data engineer and learn all the skills that you want so don't forget to
check it out are you still interested in being or remaining a data engineer are you maybe looking
for transition in or transition out let's talk about all of those things in the comment section
below and don't forget to leave a like and subscribe to the channel it's incredibly motivating
for me to see all of those things and that's about it see you next time
