🛠️ What You’ll Learn in This Tutorial: In this step-by-step tutorial, I’ll walk you through how to **scrape any website for free** using DeepSeek, Groq, and Crawl4AI. You’ll learn how to:
✅ Use DeepSeek, Crawl4AI, and Groq for powerful web scraping ✅ Tweak a prebuilt template to scrape any site effortlessly ✅ Extract leads and save them to an organized file ✅ Automate the process so you can scrape any website instantly
⏰ **Video Timestamps:** 00:00 - Intro 01:02 - AI Scraping Tool Overview 03:22 - AI Scraping Scenario Overview 04:58 - AI Scraping Code Setup 06:31 - Crawl4AI Scrape Example 09:31 - AI Scraping Deep Dive 18:16 - AI Scraping In Action 21:03 - Convert CSV to Google Sheets Table 22:08 - Outro
Hey guys, in today's video, I'm going to show you how you can scrape anything completely
for free using DeepSeek, Grok, and Crawl4AI.
And after talking to dozens of AI developers and businesses, I can confidently say that
web scraping is one of the most in-demand skills that companies are hiring for.
So you definitely don't want to miss out on everything that we're talking about in today's
video.
So to help you guys master web scraping, we're going to go off and build an AI web scraper
step-by-step together to go off and scrape a website just like this to go off and capture
leads, save them to a file so that we can eventually follow up with these potential
opportunities at a later date.
And because you guys are awesome, I'm going to be giving away all the source code in this
video completely for free, link down in the description below.
But hey, enough talking, let's go ahead and dive in to start off web scraping.
Oh, and real quick, if you're looking for support on your AI projects and to meet like-minded
AI developers, you'll definitely want to check out the free AI developer school I've created
for you guys.
We have over 4,000 members, weekly free coaching calls, and we'd love for you to be a part
of the community.
But enough of that, let's go ahead and hop back to the video.
All right, guys, so let's quickly cover the three tools that we need to go off and build
an AI web scraper.
And this is just to help us set up a good foundation before we hop over to code in the
next section where you get to see all of this in action.
So let's hop into tool number one, which is Crawl4AI.
This is an open source library that makes it super easy for us to go off and scrape
websites.
And it's awesome because not only can it scrape the website, but it's set up in a way to where
it actually can take that content that it scraped and pass it over to an LLM to where
it can go off and do anything.
So in our case, you're going to see how we're going to scrape a website and actually pull
out leads from the website.
Very excited to show you guys this in action.
And they have a ton of examples that make it super easy to understand their tool.
But don't worry, I'm going to break it down, all the important parts that you need to know
over in the code section in just a minute.
Next, when it comes to tool number two, it's going to be DeepSeek.
Now DeepSeek has taken over the internet recently.
So I'm super excited to show you DeepSeek specifically, the reasoning model, DeepSeek
R1 in action.
A few things that are super important to note.
It's just as smart as OpenAI's O1 model and it is insanely fast and it's insanely cheap
to run.
It's about 20 times cheaper to run.
Super excited to show you guys this tool in action in just a second.
And then finally, we now have our third tool, which is going to be Grok.
So what Grok does is they have a ton of AI chips that are specialized in running AI models.
So think of your LLAMA3, think of DeepSeek.
So now that we have Grok, they also have an amazing free tier, which is going to allow
us to not only just run DeepSeek, which is a huge model, but we can actually run it for
free and super fast.
As you can see in this example, I was like, hey, please explain DeepSeek R1 to me like
I'm five and tell me why everyone's going crazy about it.
And it's super cool because you can see in action, this is one of the specialties of
DeepSeek R1.
It has this thinking thought process and almost like a human-like fashion of like, oh, by
the way, I need to do this.
Oh, I also need to include this.
So not only thinks through its answer, it then gives you the answer.
So you can see, you know, Grok runs this and it runs it insanely fast.
You can see we were doing like 275 tokens per second, and this took under two seconds
to generate all of this information.
So Grok is going to be our hero here when it comes to running DeepSeek completely for
free.
So now that we got that out of the way, let's go ahead and actually hop over to a quick
code overview of exactly what we're building, and we're going to go ahead and start coding
out the entire project together, step by step.
All right.
Let's go ahead and hop to it.
All right, guys.
So it's about time to hop into the code, but I just wanted to give you an example scenario
so this would all make sense before we just start coding it up.
So in this case, we are going to pretend that we were just hired by this wedding photographer.
He just moved into town and what he's trying to do is get more clients for his business
because he just wants to grow his business.
How does he want to do this?
Well, he wants to go off and just contact more wedding venues to say, hey, I offer these
services and I would be happy to help.
So what he needs from us is to go off and grab wedding venue information from common
wedding venue websites.
So in our case, what that means is we're going to say, yes, I will happily go off and build
you a scraper for a price, obviously, but here's exactly what we're going to do.
We're going to go off to whatever area they moved into and we're going to go over to a
common wedding venue site.
And what we're going to do is build a web scraper that is going to scrape all the different
wedding leads inside of this area.
And we're just going to continually go through page after page after page, scraping all this
information and we're going to continue scraping until there's no more wedding venues found
and we have an awesome Excel sheet ready that we can go ahead and send over to them.
And basically our end result is to go off and generate something that looks just like
this so that the photographer knows the name of the venue, where it's at, how much it costs.
And we're also going to use a little bit of AI in here to go off and generate a one sentence
description of the venue so that when he's calling the venue, he knows a little bit exactly
about what they do, the space.
So when he's calling, he can have an intelligent conversation.
So that's exactly what we're going to be building in the code.
We're going to scrape, we're going to save everything to SCV, and eventually we're going
to upload it to a Google sheet so that we can hand it off to our customer and get paid.
So that's exactly what we're about to do.
So let's go ahead now and hop over to the code.
All right, guys.
So now it's time for us to actually start getting our hands dirty and start coding out
our AI web scraper using our three core tools, which were DeepSeek, Grok, and Crawl4AI.
That's the core tools we're going to be using here in this project.
And I've tried to set up this project to be as simple as possible for you guys to just
copy and paste, and it'll just run out of the box.
And I've also tried to set it up in a way to where you can easily just tweak a few configuration
settings, and then instantly go off and scrape any other website.
So I'm excited for you guys to see how easy this is to actually run.
And before we dive into code, though, I just want to show you how you can set this up on
your own so that you can get it running.
So first things first, you just need to head down into the install instructions.
And all we're trying to do is go off and first create a new environment that's going to
set up and have all the dependencies we need to run this project.
So we're going to use Conda for that.
Once we have Conda installed, we're then going to go ahead and activate our environment.
Then we're going to go off and install all the necessary dependencies.
In our case, there's really only one major one, which is Crawl4AI.
That's going to be how we're going to scrape everything.
And then from there, we're just going to go ahead and once we've installed everything,
the only other piece of information you need to do is head over to your Grok dashboard
and go ahead and add in your API key to the environment file.
So that's all you need to set up this project.
And then it will actually just go ahead and run out the gate.
And it's so simple to run, just python.main.
Okay, cool.
So now we've like overviewed.
Let's go ahead and actually start looking at a quick example of what a crawler looks like.
And then we're going to expand on that simple example and go ahead and look at a complete example.
So let's go ahead and hop over to a quick example real fast.
All right, guys.
So here's a quick overview of a simple crawler just so you can understand all the core components
before we go off and actually start scraping an entire wedding menu.
So what are we trying to do?
Well, inside of Crawl4AI, there's a few core components that you need to know.
First, BrowserConfig.
BrowserConfig is how we actually set up what browser do we want to pop up.
So specifically, a BrowserConfig, you get to pick, I want to open up a Chrome browser.
I want the size of the browser to be this big, the window.
I want to set it to be headless to false.
So headless false means I actually want to watch what the browser is doing.
I kind of like doing that because I like to see what pages we're actually scraping.
But if you just want it to run in the background, you'll just say headless true.
And you get to set a bunch of other additional information.
So this is pretty much BrowserConfig in a nutshell.
And then the other thing that we need to do is set up our CrawlerRunConfig.
So as you can see over here, we have BrowserConfig and then CrawlerConfig.
So once you set up the browser, you now get to pick the crawl,
which is like what do you actually want to happen?
So this is where we get to do a ton of either simple or complex configuration setups.
So inside of here, you can actually pick a few of the core things are,
as I'm going off and crawling a website, because this is made for LLMs,
you get to pick a bunch of things.
So I want to extract information using a deep-seek model.
And I want to pull out this information.
You'll see this in action in a minute.
I just want to go and plant a few seeds.
And then you get all sorts of options when it comes to actually like loading the page.
For example, if your page is pretty complex and it doesn't actually render information for a second,
you can pass in some JavaScript code and say,
hey, please don't scrape the website until this JavaScript element loads.
Or you can say, I would like to take a screenshot.
There's all sorts of awesome things you get to do in here and play with parameters.
But I just want to go ahead and show you like what's possible.
And you'll see in the code in a second, I actually have links to each one of these.
So as we're coding, you're going to have a link to come back and view the documentation and go,
oh yeah, that is all the different options I get to use.
So excited for you guys to see that in section.
Now, what do we do next?
Well, now that we have a browser that's going to pop up
and we've specified our run configuration, we actually get to go off and kick off a crawler.
So in our case, that means we're going to say, all right,
I would like to go off and scrape a website.
I want you to open up this browser.
And here's what I want you to do.
I want you to go off and actually scrape this URL.
And as you're scraping that URL, please use this run configuration right here.
So, and what this would do is it would actually go off and scrape the entire website.
And actually, it would just give you the results of that website
in a nice markdown file that you could go ahead and use.
LLMs love markdown.
So this is a really nice way if you wanted to, you know,
save those results and pass it over to another LLM to do something later on.
So that is crawl for AI in a nutshell.
Now that you have that out of the way,
let's go ahead and actually go ahead and hop over to our code,
where you're going to see how we can actually take this example and expand upon it
and go off and scrape an entire wedding venue,
get all the information we need and go ahead and save it off to a really nice Excel sheet.
And see how we're going to add in DeepSeek R1.
So let's go ahead and hop over to our code.
All right, guys.
So now it's time for the fun part where we're going to go ahead
and start looking through the code together line by line
so that we can understand exactly how we're going to tie together DeepSeek,
Grok and crawl for AI to help us go off and scrape leads for all these different wedding venues.
All right.
So let's go ahead and dive in and start using everything we've learned so far.
So, so far in our code, there's only really one main function that we're trying to do,
which is go off and crawl all the venues.
So under the hood, here's what's going on in this at a high level.
And also let's tie it over to what our ultimate goal is.
So ultimately, we are trying to go off and crawl this entire wedding venue site
and grab all the leads for a city.
That's all we're trying to do.
So what do we want to do?
Well, at a high level, and we're going to dive into this part by part,
we want to go off and continually scrape each page
and keep scraping until we run out of pages to scrape.
So right now we're on page one.
And eventually, once we go over to page 10,
what's going to happen is we're going to go off and go, oops, no more results found.
So goal, keep scraping until no more scraping can be done.
All right.
So that's all we're trying to do.
Now, what do we need to dive in and actually set up to make this happen?
Well, at a high level, we need to go off and start setting up our crawler
to go off and start scrape each one of those pages.
So let's walk through everything real quick.
So first things first, as you remember, in our quick example,
we have to set up our browser config.
So we're going to say, hey, I would like to go off and set up a browser,
like a Chrome browser to pop up a window and scrape the page.
And I want to see what's going on.
I want to see all the logs as we're scraping each page.
OK, makes sense.
Now we're going to dive into something new that you haven't seen before.
But I do want to explain what it means.
So we need to set up our LLM strategy.
And all this means is how do we want to scrape and use the LLM
to transform the scraped information into valuable information?
Well, in our case, all we're trying to do is we want to go off
and scrape each one of these wedding venues.
So in our case, we want to scrape a wedding venue model.
We want to get another model and another one and another one
and keep scraping each one of these wedding venues
until there's nothing else to scrape.
So the keyword I was saying there was venue model.
So you can actually see we've created a model inside of our code base.
So in the models folder, we have a venue file.
And all we're trying to do is for each venue,
I want the name, location, and other information
that I need in order to give it over to the photographer.
All right, so OK, now we've talked about what information we're trying to get.
But what the heck does this LLM extraction strategy even mean?
So here's how I like to think about it.
As we're going off and scraping a website,
we're going to get back a ton of raw data.
And we need to set up an LLM to say,
hey, look at all this data I just gave you and do something with it.
In our case, we're going to say, LLM,
I want you to extract all the wedding venues and outside of this raw data.
And I want you to get these different pieces of information,
the name, location, price, all that raw information.
And I want you to generate a one-sentence description.
And what you'll notice is this information directly ties over to our venue model.
So it's like a one-to-one relationship.
We have to give our LLM what it needs to do
and the final model that we want to generate.
OK, so hopefully that makes sense on our instructions and our schema.
Now we get to pick, well, what model, what LLM model
do we actually want to do all this processing for us?
And this is where Grok and DeepSeq come in.
So in our case, we're going to say, I want to use Grok.
And specifically, I want to use the DeepSeq model on Grok
to go off and perform these instructions
and convert everything over to this nice venue schema.
So hopefully that makes sense.
And just to give you a quick tip,
you actually could change this if you want to do another free model.
You could use OLAMA if you have a powerful enough computer
and run LLAMA 3.1 on your local computer.
Or if you do like some of the more paid models,
you could easily change it over to GPT 4.0, 4.0 Mini,
whatever you want to do.
Or 0.3 Mini that just came out very recently.
So hopefully that makes sense.
All right, so let's quick recap.
We're going to set up our browser to go scrape everything.
As we're scraping, we're going to have our LLM.
We're going to tell it exactly what information
it needs to scrape and pull out
and make all these nice venue objects.
Now we get to dive into the fun part
where we're actually going to go, OK, cool.
I now have all the base set up.
Now it's time to go off and start scraping.
So let's walk through this.
Well, in our case, what we're going to do is create one crawler.
So that's going to be the crawler that's going to go through
and actually scrape all the different web pages.
And we're going to say,
I want you to open up the Chrome browser.
So that's just like kind of,
we're just setting things up to go off and scrape.
And once we have our crawler created,
now we get to dive into this core fetch
and process page function.
And all it's trying to do is find wedding venues for us
and let us know if it didn't find any more results.
That's all it's going to do.
So let's go ahead and dive in and see
how all of this ties together.
And I've tried to add in as many comments in here as possible
so you guys can understand exactly what data
is moving around inside of the code.
So as you're off on your own,
building your own web scraper,
you can just copy this code and tweak it.
All right, so let's look at this line by line.
So first, what are we trying to do?
Well, we're scraping.
Specifically, we're trying to scrape a specific URL.
So a URL is nothing more than a base URL
and then the page number we want to go off and scrape.
So in our case,
we actually have a nice config file set up
with some of the core pieces of information.
So we're just going to say,
hey, I want to scrape this base URL.
You could tweak it to whatever URL you want to scrape as well.
So we're going to scrape the wedding venues
and we're going to start page number one.
Then here's what we're going to do.
First, we're going to go off and actually check.
We're going to scrape the page
and check to see if there was no results.
So let me show you this.
So we're going to pass in the crawler and the URL
and we're going to say,
hey, please go off crawler and scrape this page
and on this URL.
And as you're scraping the page,
I just want you to don't cache anything.
Just go off and scrape this page.
And all we're going to do is get back
to see if anywhere on the page,
the word no result was found.
So scrape the page, look at the results.
If everything came back successfully,
check to see if no result was found,
because if it was,
that means we're on like a late page.
So page equals 10.
So that means we're on like super late page
and there's nothing else to scrape.
Hopefully that makes sense.
So now we're going to go, all right,
if we didn't find results, we're going to return true.
And that means we can go ahead and cut out of this early.
So no results, we didn't find anything.
Let's return and quit early.
This is a nice way to make sure your code's nice and clean.
It's just an early return statement.
That's all we're doing.
Now, if there is information on the page
and we didn't get this no result found page,
what we can do is go off and actually scrape the page,
but we're going to do a little bit more detailed
scrape this time,
where we're going to start actually trying to pull out
all the wedding venues from the page.
So here's how this one works.
We're going to say, all right,
I would like you to crawl the page again,
and I would like you this time to,
you know, it's the same page,
but we're going to scrape things
a little bit differently this time.
This time, what we're going to do
is add in two additional pieces of information,
the extraction strategy and the CSS selectors.
So let me walk you through what's happening.
So when it comes to the CSS selector,
all this is, is we're saying,
hey, instead of scraping,
you know, thousands of lines of HTML code,
I actually only want you to process
these specific elements on the webpage.
So in our case,
let's go ahead and like dive in
and see what elements we want to scrape.
So let's make this a little bit bigger.
So you can see inside of our website,
if I go to inspect mode and I click this little cursor,
I can hover over some of these elements
and you can eventually see when I hover over this,
I can see when I click on it,
I can see all of the classes and styles
applied to this specific element I want to scrape.
Because all I want to do eventually
is just like I said, from the get-go,
I want to scrape, well, even more specific,
I really just want to scrape this information,
this information and so on and so on.
So how do I scrape that?
Well, if I look at the code over here,
I can see each one of these webpages.
If I hover over this right here,
it has a class that says info container.
So I really just want to scrape all the info containers
on this wedding venue site.
And that's exactly what we're doing.
If you hop over to our CSS selector,
I'm saying, hey, please scrape any class
that includes the word info container.
That's all we had to do to actually get this set up.
Now, all we can do is now that we're saying,
tying it back, instead of scraping the whole website,
just grab all the info containers.
Now, as you're looking at all the info containers,
I want you to use that LLM strategy
that we set up a little bit ago,
which said, hey, I want you to use deep seek
to go off and scrape all the venue information.
So hopefully this is all getting tied together
and you can see how we're combining
some general scraping with AI scraping.
Now, what we're going to do is go, great,
I've scraped the website.
I now can start looking at the results
in the extracted content.
So this will just be a JSON string
that will include all the extracted venues.
Then what's going to happen
is we're going to load that JSON string
and we're eventually going to get extracted data.
This extracted data is going to be nothing more
than a bunch of these models right here.
So it's a list of venues.
So pretty cool if you ask me.
Now, that's literally the entire code loop.
Those are all the core components
tying everything together.
So what I'd like to do now is go ahead and run the code
so you guys can see exactly what's getting triggered
and I'm going to explain you what's happening along the way
so that all the code we just talked about makes sense.
So let's go ahead and start running our code
and I'll walk you through everything as it goes.
Okay, so what we need to do
is go ahead and open up our terminal
and you need to make sure
that you are in your conda environment.
So you can see mine says deep seek crawler.
That's the conda environment we created earlier.
And now what I can do is do python main.py
and this is going to go ahead
and actually start running our code.
And when we run it,
it's going to go ahead and pop open a browser window
because we said headless equals false.
And now it's going to start scraping
every single one of the venue pages
that we have set up inside of our URL that we want to scrape.
So I'm going to zoom out
so you guys can see exactly what's happening.
So as we scrape pages,
you're going to see over here,
it's going to start going over to page number two.
After it scrapes page number two,
it's going to go to page number three
and so on and so on.
Along the way, what's happening over here in our terminal,
you can see because we had verbose equals to true,
so you can see we're getting a lot of logs.
So you can see as we're scraping the website,
it's making a bunch of calls over to deep seek.
Specifically, it's trying to go and say,
Grok, please run the deep seek model on all this data
and let me know what wedding venues you find.
And along the way, we have a bunch of different logs
so we can see like, oh, on wedding page three,
I was able to go off and scrape 10 different leads.
So it's going to keep continually doing it.
So in this page three, we got 28 leads.
Now we're on page four.
And along the way, you can see what's pretty cool
is you can see it's doing exactly what we've set it to do.
So it's processing a wedding venue.
So the name of it, Stillwater Pond,
Temple, Georgia, probably near Atlanta.
You can see the capacity and the description
based on what's over here.
And it's going to continually keep scraping until it's done.
What's nice though, is we're almost at the end.
I think there's only six pages.
And once it's done, you'll be able to see it goes,
cool, I'm at the end.
There's nothing else for me to scrape.
And it's going to give us a nice little final blurb saying,
hey, I'm done scraping.
And it's going to give us a quick overview and say,
hey, I actually used this many tokens
to scrape everything you just asked.
Also, by the way, I went ahead and saved everything
to a CSV file.
So we're on the last page now.
If I scroll down all the way to the bottom,
you can see we're on page six.
There's not another one.
So it's about to quit right now and say, I'm done.
So we extracted the final ones.
Then you can see it closed the browser for us
because it's done scraping.
It's done crawling.
And then it's going to show us this.
It's going to say, hey, here's all the tokens I used.
All in all, I used 43,000 tokens.
So we're under the limit.
I think it's like 60,000 tokens per minute inside of Grok.
And then you can see per page what we end up scraping.
We actually scraped a lot.
Now, what we can do next is actually go ahead
and look at our completed venue sheet.
And you can see we scraped so much information
in no time at all.
And we could easily change this.
Just throw in a different URL and boom,
we're off scraping another website.
So all around, this is everything
that we needed to scrape.
And this is everything that we need
to go ahead and just pass over to Google Sheets.
So what we're going to do, I'm going
to go ahead and show you real fast how
to make that nice table over in Google Sheets
with all this new CSV information.
And we'll be pretty much done.
All right, guys.
So I went ahead and hopped over to Google Sheets.
And this is where we're going to go ahead
and actually import the CSV file that we just
created inside of our code.
And all we have to do inside of this one
is go ahead and actually just we're
going to click the Import button inside of Google Sheets.
So we're going to click Import.
Then it's going to ask us to upload what we're trying to do.
So we can just drag and drop our uploaded CSV.
It has a ton of information that we've scraped.
So we're going to go ahead and import it.
Then things are going to be nice and done.
And it's going to say, all right,
here's everything that we just scraped.
And what's nice is now Google Sheets
is going to say, I'd like to convert this to a table.
So now we have a beautiful table that we can instantly
share with our client.
They can come in here.
It's all nice now.
They can start filtering by prices, capacity, ratings,
reviews.
And you can see it also has that nice AI-generated description
of just a brief description of what this wedding venue is
about and what it looks like.
So all around, this is awesome.
Yeah, I hope you guys see how powerful this is.
And I'm so excited for you guys to go off and start
scraping your own websites for your own clients
using this AI web scraper with DeepSeek, Grok,
and also Crawl4AI.
And that's a wrap for this video, guys.
I hope you all love learning how to build an AI web scraper.
I cannot wait to see what you guys go off and build next.
And two quick reminders.
All the source code in this video is completed for free
down in the description below.
And if you're looking for support and to meet like-minded
AI developers, I have that free school community
for you guys.
And once again, more information down in the description
below.
But also, I have a ton of other AI-related content
right here on this channel, everything from Cray.ai,
LingChain, and more.
Just definitely recommend checking out whatever video
pops up next.
I know you're going to love it.
But until next time, y'all are awesome.
And can't wait to see you in the next video.
See ya.