How to Give Your AI Agent Real-Time Web Data (Claude Code + MCP on Oxylabs)

So I want to show you something that shouldn't be possible yet with AI, but it is. So I'm going to ask an AI agent to go find me the prices of three competing cold brew brands right now, live on the internet, and then bring those numbers back to me. Now, if I ask a normal AI chatbot to do this, it's going to guess, it's going to give me an answer that it sounds confident in, but it's probably wrong because it has no live connection to the internet. And that is one of the biggest limitations to a lot of AI agents. And that is exactly what we're going to do in this video. I'm going to show you how to enable. That kind of context for your AI agents and systems. So by the end of the video, you can do that too. It's going to be a fun one. Let's go. Now, historically, a lot of the problem here is that the internet doesn't want to be kind of live scraped by all these agents. There's going to be captchas. There's going to be site blocked kind of IP barriers. Changing layouts, like half the site will be rendered in JavaScript or something. And so developers to make that work typically have to stitch between all these different tools. Like they'll have one tool that's Searches the web and pulls URLs, then another one scrapes the actual web pages, then there's a proxy service to manage that or do it in parallel. And it's all this infrastructure basically to try and let AI see what you as a human can see if you just go to Google and go to some website. And so the platform that we're looking at in this video, which solves all of that and unlocks what I just spoke to, is a web API out of a company called Oxylabs. So they partnered with me to make this video happen and And the way they do what they do is they run one of the largest Proxy networks in the world. You're talking 195 countries 175 million residential proxies and then their new thing web API is built specifically for AI agents in the way scraping works today and what it does basically combines search and scrape into one API. So search will find real-time live URLs and then scrape will extract clean data from those web pages in a way that's readable to AI even on your historically really Really difficult sites to crack like JavaScript heavy sites and to break down what's actually happening there. So first search discovers URLs and then returns real time results. Scrape extracts the data into a format that's ready for agents. JavaScript rendering is built in. So then dynamic sites, that's kind of the core issue with that kind of website that only loads their content after the page runs will still work. And then fourth and finally, it'll plug directly into agent pipelines and then an MCP based So now we're going to jump into my computer so I can show you how this works. So if you want to follow along and just set this up as I'm doing it step by step on screen, you can get a free trial of the web API and help support the channel by going to the link in the description or commenting OxyLabs under this video. With that trial, you get $5 in free credits, no time limit and no credit card required to do that. So completely risk-free to just follow along and try out what you're seeing me do on the Now let's jump into my computer. So here we are inside a demo account in the Oxylabs dashboard. And our goal here is to create an AI agent that can pull real-time pricing from the competitors of our cold brew brand. If you haven't seen that before, it looks just like this. So we're inside the dashboard currently. Step one is to go ahead and get an API key. Now, keep in mind, like I just said, everything that you're about to see, we can do for free on that $5 of credit. When you first create an account, let's just scroll down here under web access to web API. We'll click on create instance. I'll call this Claude MCP, and then I'll click create instance. Now under this instance, which I'm calling Claude MCP, since we're going to use it through Claude, you'll have an API key. This is the critical piece of information that you'll need to proceed. So I'm going to go ahead and copy that. You'll do the same. And then we're going to jump into Claude. Code to set up the MCP and you're going to use it. These three lines that you see on the screen to do that. There's also an article that looks just like this but what's easier for you just go ahead here and then screenshot these lines. So now with these lines, what we can do is open a new terminal session and then I'm going to go ahead and paste that first line. If you already have this installed, it'll tell you that in this case we do if not, it'll just install it from this line. Now, we'll copy that second line. Let's go ahead and paste that and now this As you can see, has installed the oxylabs MCP and then now to connect it to our account or to the web API, we're going to use this third line. What you'll do is just replace where it says your key with your actual API key. Okay, so I went ahead and did that. We're gonna leave that off of the screen for obvious reasons. Now what we can do is go ahead and jump into Cloud and then we can test to see if it worked. What we can do there is forward slash MCP and we can see our local MCP is set up here with Oxylabs. But we can also see that that Oxylabs web API has access to six tools. Let's go ahead and press on that and view tools. And so those six tools are web search, read a page, check a queued scrape, reload an offloaded page, list scrape endpoints, and then call a target scraper. So that's everything now that we have the ability to do through Claude. So now we can go ahead and run it and see how exactly it integrates or what kind of data we get back from Oxylabs. So I'm going to give it this prompt. So search the web for the current product pages of three competing cold brew coffee brands. Scrape their pricing and review data and give me a Comparison table. I'm not gonna say use Oxylabs, but Claude should know to use Oxylabs to pull this data. Let's go ahead and click enter. We can see it did call Oxylabs web API, and it's using that to get the live data. And now it's seeing that you do in fact need JavaScript to scrape this. This is the exact problem we were referencing earlier. So it's scraping the data with rendering turned on using again the Oxylabs API. So what we're doing now would literally be impossible or at least much more difficult without this tool. Okay, here we can see that data has come back. So we have the Chameleon, the Stumptown, and Grady's, and this is real live pricing from these webpages. So we can see 16 bucks for our Concentrate, 11 per bottle for Chameleon, and 70 for a case of six for Grady's. And now all of this is saved in Claude, right, in its general brain context awareness. So Claude and web and on your your mobile will remember all of it as well. And so there we go. That is live data pulled through. through the web API, but that's just one use case for something that we have played around with a lot on this channel, right? So our cold brew brand, but what about the general purpose applications? What I wanna do now is four very quick demos that just give you an example of the breadth of usage and functions with a tool like this. Now, number one is gonna be using Oxylabs to scrape an Amazon product page for an ice cream maker. So I'm saying basically I want all the real time Current accurate information about a certain product on a website like Amazon. Okay, here we can see that data all pulled from Amazon. So $229 in stock, star rating and 7,000 reviews. Now, number two, let's just generally pull information from the web. So I'm gonna say use the Oxylabs search tool to find the top five live results for best cold brew coffee, 2026. Okay, that was super fast. Here are those five results. All looks good. For number three here, we're going to search for homes for sale. So we're going to pull specifically from the Redfin website. So we're going to scrape the listings page and pull five properties for sale in East Austin Okay, here are the results. We have five sales for listing and all the information from there and we can pull a whole lot more than just five if we wanted to. Now on that note, there are specific scraping kind of tools built. So I want to show you what those are now, so I'm going to say list the oxylabs dedicated scrapers. Now one of those is the google shopping scraper, so we're going to use that specific tool or sub tool to find then the five cheapest cold brew concentrates. So here are the dedicated scrapers. So let's see that run. We have retail, we have Google search engine, AI chat bots, First we'll get the list. YouTube, et cetera. And then here we can see that information from Google. So I hope all of that gives you a more general overview of the specific functions or the applications here. Maybe one of those is useful to you. Maybe all of them are, and let's go check out what that looks like on on the dashboard side. So I'm going to reload this dashboard page. And then I'll go under usage here and we can see 19 scrape results with nine search results. And then under spending, we can see what that all cost. And it was about three cents total for everything that we looked at. So very cost effective. And the last thing that I want to show you here is a pretty cool tool as part of the platform. We're going to jump into the web API playground and everything we just did was through the MCP, but you can do that in the platform as well. We have the same API key or the instance that's being run on, but we can scrape information by just pasting a URL directly, or we can go ahead and search so let's use one of the examples i can say SaaS companies raising Series B click start searching we can see all the information here in a way that is most readable to ai so i could just copy this paste it into my Claude chat now if we jump into scrape let's do This is fun. The Wikipedia page on web scraping, and it will actually go ahead and scrape that and again, return the data in a way that is accessible to AI agents. So I could copy that, head back to my ChatGPT, my Claude chat, whatever it is, and paste that in the result. And there you go. That is Oxylabs and the web API. So my overall assessment or take of the web API is that this genuinely is super useful and solves a very material problem for people whom that problem And the MCP is key there to connect the API or be able to use the API in a way that reflects how all of us will do kind of scraping or work with AI today, which is mostly going to be your chatbot, right, your Claude. Your ChatGPT, your Grok, Gemini, et cetera, interfaces. So I think the fact that they unify a lot of that, they make it very AI native and they make it a seamless experience is the really big selling point from my perspective for the whole deal. If you work in this space and you know exactly how hard the incumbent processes here are, then I don't really need to tell you that. Now, if you have never worked with an API before, there is going to be a little bit of a learning curve there. So do keep that in mind if that's you. And then this is a paid tool once free trial credits run out. Here's the pricing on screen now. And so you will need to budget that if you're doing some, you know, very scraping, very data intensive process. And so, like I said earlier, if you weren't following along, you can go for that free trial where there's no credit card required. You get $5 in free credits by commenting OxyLabs under this video or going to oxylabs.io and you do help support the channel when you do that. That said, appreciate you being here. That is everything I wanted to get across to you about the Web API and I'll see you in the next one. Cheers.