Day In The Life Of A Big Data Engineer In Sydney 🇦🇺 Working From Home!

Hey everyone, good morning. By a partition key. Morning, just that, thanks. Thank you. I'm still working, it's a minute late. Hey everyone, good morning. How are you doing? Amy and myself, we've raised- If you don't immediately filter by a partition key, you lose the majority of the power of partition pruning. If you don't choose a partition key, that might- Does that actually influence the pipeline building? Yes. We do need to choose a partition key, and then he might just be like, yeah, that's fine, whatever. Thanks a lot, bye bye. Hold on, two seconds, two seconds. They had a small Spark, they had like 28 gigabytes behind it, like the size of a data set. So the size of this Spark drive is 28 gigs, and then they run those. So I think they're running like tons and tons of big queries. And just to slowly build each of these data sets, and then finally when they get these data sets, they can try and build this finalized output. They need the functional logon, they need the details of the functional logon to be saved in Azure. Feel free to just tag me in the group chat with them, and I'll talk to them about it. Okay. There's going to be an engineer that physically copies the X and W tables into a staging area in Adapt, and then it's going to be exposed to potentially a wealth of Adapt analytics tools. This was done on the 2nd of April. So I'm guessing everyone is in luck. What you're doing is you're grabbing Parquet files in your Workspace Storage account, and you're surfacing a view on those Parquet files. Right. So this code here, it grabs some Parquet files that I've saved into the Workspace Storage account, which is actually an Adapt secure view. Yeah, no worries. I'll wait for it to get back. But as a pattern to consume from Power BI Desktop, what you need to complete... He's obsessed to some degree. The vision of me would have been ecstatic, selfless, fortunate, and feel thankful for the opportunity to build what I did over the last decade. And of course, to enjoy the many comforts and perks that come along with that. This is not fair. This is why you go with this idea that you can now have all the models available to you. So GPC, Opus, Gemini, K2, Rock, what have you, all the models. And then you can select a chair. Right. So you have council members or the selection committee. And you have an ensemble. So this is basically...