1. Scraping and Organizing Website Content - SCAI017
Sloup
1 follower
6 Video ViewsยทSep 15, 2025
Nate Herk - AI Automation Society Plus\10. Client Builds (Real Projects)\4. Newsletter & Research Agent\1. Scraping and Organizing Website Content - SCAI017
All right, hey guys, so I wanted to do a little update video on one of the products I'm working on.
The goal here of this workflow was, I was testing out different methods. I know
I had sent a post earlier about, I was trying to figure out how to scrape through
these dynamic websites where the website has a ton of magazines, and then you click on the magazine
and it would pull you up into this new URL, a new page, and the magazine's being loaded dynamically
because it's like an animation, so it looks like a real magazine. And the issue was,
even in the HTML content, I wasn't able to scrape through that. I had to go into the network
settings. Once you inspect element, you go up to network and you can look at headers and stuff,
and each page was being dynamically loaded. It was like a bin file. So I had tried to download
the bin files in, I had tried to do all this, and at the end of the day, I just couldn't get that
content back. So what I was thinking I was going to do was get on Puppeteer, Browse.ai, something
like that, and have to go through the magazines and take screenshots, next page, take a screenshot,
next page, and then sort of have AI interpret an image. And that turned out to be, I mean,
I didn't really go too far down that path because it seemed like it was going to be time consuming.
So I just explored some other options. And one thing that's really important when you're
working with clients is a lot of times they come to you with an idea, right? And they also don't
completely understand the capabilities of AI as far as what it's good at, what it's not good at,
and they don't maybe completely understand the foundations that you want to build within a
business to then continue to chip away at processes within their operations. Anyways,
here I go again with the ramble. But anyways, what we ended up doing was,
you know, it's a magazine, right? So this content is going to be posted somewhere,
whether that's on their socials, whether that's somewhere else on the website,
and whether that's, you know, anything like that. So we just basically I was looking around these
different websites that they wanted me to scrape through. And I was just, you know,
talking to them about, maybe we can get this content somewhere else. Because going through
the magazine, there's also going to be some noise that we don't want, like advertisements and
different promotional things. And then if we can actually just find articles that reference this
magazine, we can get the same information. So anyways, that's where we're at now. So I was
looking at these different things, and I was scraping the websites in different ways to get
different articles. So oops, so we kind of started off over here, right? So this was just a URL to a
website that had a ton of different articles that were just regular HTML text within a website. So
we get back this nasty chunk of HTML, we then are extracting the specific URLs here for each article,
as you can see right now, it's 29. And each of these take you to a different article, that's
just a regular, you know, article on a website. So it's able to be scraped easily. So from there,
I was putting it into Airtable. And then we were having this Airtable trigger, whenever we saw a
new file in Airtable, then we'd grab that article. So like, let's say this first one, we're just
getting one of those articles back. As you can see, that's actually getting the scraped content,
because it's already finished. But anyways, we'd just be getting the article. And then from there,
we'd be grabbing that article right here. So that's the full one. Again, we're getting nasty
chunk of HTML, we got to make that readable. So we're parsing it here, we were then getting the
heading, the date, and the body of this article. And then we were putting that back in Airtable.
And I was like, wait a minute, that's kind of redundant. And also just kind of a waste,
we could just make this one flow. So this isn't yet optimized. But basically, it would just be,
it would be triggered. We'd have, you know, the base URL, we'd scrape through,
we'd grab that URL. And then for each of those 29 items, we'd loop through, update the Airtable,
loop through, get the content, get the content, get the content. So that's kind of how this one
was working. And this is all just, you know, kind of just workbenching ideas. And so that's
one website. Now I'm moving on to website number two, where again, they have the same problem with
wanting to scrape websites, or sorry, wanting to scrape magazines within website.
And also sometimes like these magazines, up top, you know, you have different URLs, right? And so
sometimes the magazines have the URL, and then it's like page equals two, page equals four,
page equals six. And sometimes that didn't happen. So again, just a headache with the magazine. So we
wanted to look at the other articles on the website and do that. So let me just run this one
because I have the Airtable deactivated, and then we'll walk through what exactly I'm doing here.
So it started off with the same concepts, right? We wanted to get the articles,
wanted to extract the content, wanted to pull that information out, or sorry, we wanted to pull out
each individual URL of all the articles on the website. And then we just go through, loop through,
and scrape that information. And so that's what I started off with, was with this down here.
And then I'll explain why I had to go through this step up here. And maybe this may be a little
redundant, some of this stuff, but like I said, it's all about iterating, refining, getting feedback,
iterating. So from here, once again, we're taking that base URL. This website right here will pull
up all the articles. So from here, we get this stuff, and then we want to extract just the article
name. So we were getting just the blogs back. And as you can see, this isn't the full URL.
It's just like the second half. The base URL is going to be coming from here, where we say like,
you know, liftandaccess.com slash, and then we're putting in this information.
And so I originally just had this, because in the previous one, we were able to extract the
date of the article from the article. But in this case, the date isn't really found there.
But we found it earlier. And when it's listing all the articles, we've got the dates there. So
that's why I'm grabbing the dates here. We use different CSS selector here. This one is pulling
up the blog URLs, and this one is extracting the article dates. So that's what's going on right
here. And as you can see, it's coming back as one item, an array, and we're getting multiple
items, I guess, within the item, within the array. Anyways, so from there, we wanted to extract the
full URL, or not extract, combine the original one with the second half. So now it's an actual URL
we can put into an HTTP request and scrape. So we got here. It's a ton of data, so I'll just show it.
But we're getting all this information back. This is 10 items. This is the content from 10
of the article URLs that we found. And then we're looping through each one to actually get the
article content. So as you can see, we get the title, we get the content.
And that's coming through as 10 different items. So as you can see, so from there, I was like,
okay, well, we got this information. Nice. I want to put this in Airtable now. But we want to put
it in Airtable as one field, one cell, just scrape result, have everything in there, title,
date, content. So then I was thinking, okay, well, we have the dates over here.
But they're in one array. And so if we were to loop through this, it was going to only pull
this first one or was going to, you know, assign this date to all the articles, which we didn't
want, obviously. So I decided we'll split this out. So now we're splitting out the dates, we got 10
items, and then we can loop through this based on position, in order to say first one's going to go
this first one, second one's going to go this. From there, we wanted to merge this back with
the original full URLs, so that we're assigning this one with that first date, this one with that
second date, that kind of stuff, right. So then we merged them based on position. So now the dates
line up with those URLs, and we're good to go here. And then we have this information in the
right order, URLs, dates, and then we want to merge it with this looped information that we got, which was
just extracting the content from each of those articles. So we're merging those here, so that
each record, as you can see, we have 10 items here, each item has the correct date, URL, title, and
content associated with it. So that's why we're merging them by position in this case. And now
everything is lining up the way we want it to so that we can feed it back into Airtable. So we've
got that, and then I just wanted to aggregate the result into one item, one field, yeah, one field
per item. So now as you can see, we were just kind of manually setting, it's going to say article
date, it's going to be article title, I don't want to look at JSON, article date, article title, and
then the actual content. So that's how it's coming through, right. This is what's going to be in
each cell in Airtable. So this is what we have here. And then from there, we're just assigning
the different parameters to go into Airtable. So we've got the URL, the scrape source, and the
result. The scrape source is hardcoded because this is going to be, everything going through
this workflow is going to be associated with a specific source, which in this case, as you can
see, it was the base URL right here of lift and access. And if you're dealing with Airtable, one
thing that you need to understand is that, you know, Airtable can reference different tables
within each table, right. So it's very relational. When you're trying to update a specific relation
in Airtable, it has to come through as an array. So this is the record ID that we want everything,
everything, all these 10 items we want associated with that source, right. We want all of these 10
items coming back to be associated back in Airtable relationally with the lift and access
records. So this is the record ID for that source. And then you just have to put it in an array,
which is as simple as doing square bracket, parentheses, record ID,
parentheses, close bracket. So that's just how it works. If you try any other way,
this was giving me a headache initially. It's just going to error. It's not going to fill in
anything. It's going to say expected an array, but received a string, whatever. So this is the
way you should do it. Just had to do a lot of trial and error there to figure that out.
Yeah, I've got more websites to go through. You know, the thing is not everyone's the same. You
really have to get into the HTML, the different ways you need to, you know, just logically merge
your data. In this case, you know, like the date was coming through over here, rather than over
here. And in this guy, the date was all, all the information was coming through in that loop of
initial HTTP requests. Great. So, you know, obviously, this is just phase one, we have to
make it more, more automated, we're going to make it more, you know, production ready. So
going to get some feedback and iterate. And we'll probably be changing this logic a little bit,
especially up here, this top one, this was kind of just a playground. And this is probably going
to be more of the way we want to do it. But you know, I hope this was cool. Hope it shared some
insights. If you got any questions about this, or if you guys have any ideas, please let me know.
This stuff is obviously super cool. I love just looking at different use cases. I love
exploring different tools and stuff like that. I haven't used the merge logic too much, at least
in NNN. Obviously, before this, my full time job, I did automations, very hard coded,
or very static automation. So there was a lot of joining. As you can see in here with the merge,
you've got position, you've got matching fields, or all possible possible combinations. So there's
a lot of things you need to be careful of when you're merging data, joining data, you could
accidentally have a joint explosion where you're duplicating stuff, you could have a join, what
are they called? The opposite of an explosion where you're losing records. So you got to test
a lot of stuff. And luckily, this was just 10 items. So I was able to just make sure everything
was working correctly. And then yeah, one other suggestion I would make is when you're doing
something with loops, what I was doing here is I wasn't looping and testing that like that,
I was adding a limit right here. So I was just getting one item back. And I was just testing
that first link. And just testing that first code. Because there's this weird thing with the loops
where you can't just kind of pin data and test an individual step, you got to run the whole workflow.
So just another little tip. As you can tell, sometimes when I'm doing these little live
updates, I tend to ramble. But anyways, that's gonna be for this one. So
let me know if you guys want to talk about this one. But thanks.
