Rebuilding the web for agents — Liad Yosef, MCP Apps

Hi, everyone. Hello, hello. So we're going to talk a little bit about the agentic web, and more specifically, what it means and how do we make the web ready for agents. One disclaimer, I built this talk yesterday, so it might be outdated, because things are moving really, really fast in this space. And I think I need to introduce myself just to get some context. So I'm the co-creator and maintainer of a spec called MCP Apps. MCP Apps, that's the underlying spec behind Chagpity Apps, Cloud Apps, Copilot, GitHub. Every chat-based app that you see is based on the spec MCP Apps in the MCP committee. I'm also the co-founder of a company called Aura, where we research the agentic human interactions. And I built and led the agentic storefronts in Shopify. So the topic of agentic web is really close to my heart. And a little bit of primer about MCP Apps, if you're not familiar, MCP Apps are actually, this is actually the spec that paved the way to agentic web. So MCP Apps were released as a spec and as a standard a few months ago, with the support of Cloud first as the first client, and then all clients followed. And if you play with any chat-based app and you pulled any visualization and interaction UI layer from an MCP server, you probably use MCP Apps. And the good thing about MCP Apps is that everybody wins. Because if the servers or the providers can send UI chunks into the chats, then the apps gain their brand and identity. They get to keep their UI and not being reduced to a database or just text-based information. The users gain their trust and familiarity because if you ask your GPT, hey, book some hotels for me, and you see booking.com, then you know it's booking.com, right? You know who you're interacting with. And the host of the chats gain access to the world of capabilities. You know, a few months ago, people asked, yeah, why won't OpenAI just build everything out of scratch? But OpenAI will not negotiate deals with hotels or support users that want to change seat in a venue. We need those services, and MCP Apps actually solve for this last mile of interaction. Yeah, so this is the benefit that we get. And MCP Apps solved our last mile of interaction, right? So what does it mean, the last mile? Last mile is as agents become better, as models become better, they can do things more autonomously. But we are still in the end of the chain as humans. We still need to be able to choose the hotel or to view a 3D model or to choose a seat at the venue. All those last mile of interaction, that can change. And this is an example from, oh, sorry, yeah. So this is from like Claude PR on MCP App. This is how it looks like, right? So when you're embedding apps inside chats, you get this unified experience of how UI feels inside the same context of a chat. And as you can see, in the same conversation, you get data from booking, you get data from all trails, you can get data from everything you want to interact with, right? So this is already happening. MCP Apps is already widely supported across every chat agent except Gemini, but that's coming. And this is already happening. And the interesting part is that it brings us to what we call the agentic web. So what is the agentic web? The agentic web is not a web of agents. It's not the web as we know it today. It's a shift, it's a change. It's a change in the way that we see websites and the way that we see browsers. Because up until now, we spent 20 years, two decades, perfecting this experience. Meaning that if I want to somehow do a project or fulfill a task or plan an anniversary, I have to open tabs in my browser and I have to go through these tabs and I have to convey my intent differently to each of those services, right? Meaning that if I just want to plan an anniversary, then I need to learn Google's UI and Amazon's UI and booking UI and the other booking UI and the other Amazon UI. All that just convey the same intent to those interfaces and every company perfected this user flow. But we don't need to do it anymore, right? We can just take these interfaces and break them into atoms. Because I don't need 99% of Booking's dashboard. I don't need 99% of Airbnb's dashboard. I definitely don't need Jira's dashboard. But I do want to convey my intent to those services. So why can't I let my personal assistant use that? So if I have a personal assistant, I can just take these atoms and it composes these atoms and say, yeah, I see that you have an anniversary coming. This is the view from Google that says that you have an anniversary coming. So I can book, I can buy things for you. I can book your hotel. Now, Claude knows me so he knows that I prefer to book hotels in the nature, right? So he knows to pull the map from booking. I don't need to think about it. And what Booking and Amazon and Google benefits from this is that they have this integration layer that they don't need to develop. Because Claude has the context for me, right? So Booking doesn't need to develop integration with my calendar. Amazon doesn't need to develop integration with Booking. So it's a win-win-win. This is the view that we're aiming for. And this is very different from what the things are going, which is this, which is basically browsing into different services just to stare at different text boxes that ask what I want to do there. But no one is going to do that. In a year, no one is going to browse to Amazon's agents and Etsy's agents and Expedia's agent. I don't wanna do that. I have my own personal assistant. I don't want to use your agent. I wanna use my agent, okay? Now, if we think about it, it means that websites, as the source of truth, are something that's gonna go away because why would I need the website? Why would I need to open the tab? Why would I need the information that the website is working really hard to convey to me? And the immediate response to that is that, yeah, we have browser agents. We have computer-use agents. We have very smart agents or assistants that can browse the web for us. But that doesn't make sense because that's a faster-horses solution. Why would I let an agent that doesn't know anything about the human perception limitations work with filters and paginations and sorting and all of those UIs that we spent decades perfecting for us just for it to fulfill its task? So Google is very bullish on the other end of the spectrum, which is co-browsing and websites, and they have a protocol or standard called WebMCP. WebMCP says, okay, instead of the agent taking screenshots of the website and trying to figure out what's happening, let's have the website expose some tools in the website, JavaScript tools, and the agent will work with it. And you can see here an example of Gemini in Chrome that buys stuff for me or finds deals for me as I browse the web, which is very nice, but it's very simple. What happens if we push Salesforce Dashboard to Gemini on Chrome? Why would I want that? Why would I want to browse to Salesforce Dashboard with Gemini to click buttons for me in a dashboard that I don't even want to use? I mean, this is definitely not what we want. And the shift is that assistants are becoming our entry point to the web, right? We see it everywhere. Every major lab wants its app to be the everything app, and the shift is happening, and friends of mine that connect Jira's MCP to their IDs, they don't go to Jira's website anymore, right? And my mother uses ChedGPT for everything. If she could book a doctor's appointment using ChedGPT, she won't go to that clinic's websites anymore. And you have a long tail of websites that no one is going to see because it's much easier for me to connect them using my personal assistant. So websites will become obsolete, browsers will become obsolete, and personal assistants will become our only gate to the web. And I know what it sounds like. It sounds like old men yelling at a literal cloud, right? I mean, yeah, they're not gonna go completely obsolete, but we see it with the younger generation today. My nine-year-old goes to ChedGPT. He doesn't go to Google when he wants to search for stuff. I was traveling to Georgia, the country in Eastern Europe, and there was an older lady there that asked me to take a picture of her, like 70-year-old, and on her phone she had three apps, WhatsApp, camera, and ChedGPT, that's it. The shift is happening, and very smart people, like David Kramer from Sentry, said a year ago, I will take this bet against anyone who thinks websites are going to be obsolete in 25 years. That was one year ago. Few days ago, David Kramer published Designing for Agents, which says, we have to recognize that interaction with products at Sentry will no longer be exclusively through our web application, and we have to design for API first as a surface. So everything is going headless. Salesforce recently went headless, which is very important, because Salesforce's main differentiator from its competitors is its UX. It's the way that it conveys everything to the user, but it went headless. Cloudflare went headless, Sentry went headless. This is a tweet by a Cloudflare CEO saying that agents' traffic surpassed human traffic in the web, and we have this spectrum of interaction because if my agent is autonomous, I don't need this last mile of UI, right? If I have an open call, I just send, and I just say, yeah, organize my emails, it comes back, said, done, good. But if I want to book a hotel for my honeymoon, I probably need this last mile of interaction, so we call it the nearly headless web, right? It's not completely headless. The agents will interact with your website headlessly, but then it will bring back these UI resources for you. So user experience as we know it is being split. Now it's the agent experience of your website plus how a user experiences the agent that experiences your website. But 90% is gonna rely on agent experiences, which means your website needs to be agent-ready for everything. It needs to be agent-ready for your agent. If I'm booking, I need to make bookings that are ready for booking's agent. I need to make it ready for my customer's shared GPT or my customer's open call, and I need to be ready for my human customer who doesn't have an agent but still wants to browse to booking.com, right? So I need to be agent-ready, and what we found out is that agent-ready means a lot of things. So we heard a lot about AEO, SEO, GEO, how to get discovered, but discovery is only the first step because once an agent knows about you, it still needs to know what you are and how to interact with you and how to authenticate to you and how to headlessly pay you, right? And actually, when we built, it's an anecdote, but when we built analytics for our product, we asked Cloud Code, what's the best analytic service? And it recommended PostHog, which is an analytic service. And we said, no, we prefer Mixpanel because we know Mixpanel, we worked with Mixpanel for a decade, we know how to work with it, and Cloud Code insisted on PostHog because it said PostHog has better MCP and API and I can integrate to it better. So we don't have brand loyalty, right? We went to PostHog, if that's what Cloud Code recommends, but it made us think about Mixpanel. Mixpanel spent a decade perfecting their UX and developer experience, we just left it just because Cloud Code preferred PostHog. And it will always happen. Hermes, for example, if it uses your product and it needs to spin up a browser to use your product, it will remember that and next time it won't go to your product anymore, right? So at Aura, which is an agentic web research lab, we started researching. We raised some capital and we started researching what does it mean for the web to be ready for agents? And we have these readiness benchmarks, which is interesting. So you can run, it's free, you can go to Aura.ai, you can run any website, you get this score and benchmark according to a lot of protocols and best practices and you have agentic feedback, so the agent actually returns the feedback about your website. And we have this leaderboard of how companies and products rank according to this benchmark and we started mapping the web and everything was nice, but we hit one insight or one unexpected result. We found out, anyone here heard about lms.txt? lms.txt, that's like the de facto standard to be agent ready. You say, yeah, if your website publish an lms.txt, agents know how to interact with you. And you have auth.md and pricing.md and X.402 and a lot of standards and we found out that almost 50% of the website that we tested, that we ran, published lms.txt, but none of the agents that we ran on this website actually used lms.txt. Actually, almost all the agents went straight to the docs page and then the homepage and the 40% that did use lms.txt used it only because the docs pointed out that there's a file called lms.txt that they need to use. So then it hit us. We said, it doesn't make a lot of sense for us as humans to define to agents what they need. No one is doing it anymore. Even OpenAI, they don't publish best practices for tools anymore because they say that every time they publish best practices, the models become better and these practices become obsolete. Six months ago, the best practice for an MCP server was, yeah, have three paragraph descriptions so agents will know how to interact with you and now three lines are enough. So best practices become obsolete. We need the agents to define what the agents need. We need agents' feedback on this website. So we build Aura's Journey. And Aura's Journey, that's really cool. That's also free. You can go to journey.aura.ai and you can run on any website, any intent with any agent and see the path of the agent as it tries to interact with the website. So, for example, atio.com, choose Cloud Code, and we run it on the website and you can see in real time how the agent goes and what it tries to look for in the website. And we did it tens of thousands of times just to understand what agents really look for when they try to interact with websites. And the cool thing is not running just one harness. It's running multiple harnesses. You see here, Cloud Code and Eve, which is Vercel's harness, and ChaiGPT on the same website with the same intent. So this is Cloud Code. This is Haiku. This is Eve. See how different the agent journey looks like. And this is ChaiGPT. So ChaiGPT could find results better. See, you can just see the journey across the website. And we need to understand why. We need to understand why does it happen. Why websites publish auth.md files but agents don't look for auth.md files and what do they look for? So in Aura, you can actually go to any business question. So for any domain, you have business goals or questions and you can see the paths that the agents are taking. And the interesting part of it is that, okay, now we know how agents interact with websites. What's the next big milestone for the agentic web? And for those of you who heard the previous talk, it's discovery, right? But it's not the discovery that we think of. It's not SEO. It's not GEO. It's not AEO. Because we have to remember that every revolution came with its discovery layer, right? The web revolution came with search. Mobile came with app stores. Social came with feeds. What is the discovery layer for agentic resources? What is the discovery layer for MCPs? What is the discovery layer for OpenAPI.json? What do we even look for? I mean, maybe Airbnb has a better MCP than Booking.com, but Booking.com has a better API than Airbnb. So how do we do it? Web search, classic web search, it's not enough because it's based on human SEO and human page rank and it's not flexible enough. Custom registries, like pair agents or pair chats, they're not enough because they're closed and they require every app to submit itself to these registries. And a central registry, like an MCP registry, that's not enough because who will do the curation? What's the governance model? How do we decide which resource comes to that registry? There are emerging standards around it. There's AI Catalog.json, which is a standard by Antropic OpenAI, Google, MCP, and A2A, which standardizes how a website exposes itself to agents. And there's the agentic resource discovery standard, which is by all these companies and more, which is basically standardized how a discovery layer or a directory exposes itself to agents. So we built Aurora Directory, right? Because we're a research lab for the agentic web, so we built this. And in Aurora Directory, we take all the domains that we scanned or scanned ourselves and we put it in a directory that agents can actually query, right? We expose the AI Catalog.json file. So you can see here, for example, for monday.com, you can see that we generate this JSON file that basically tells agents, yeah, this is the MCP server for Monday, this is the API server for monday.com, and the agents can just query that, right? So for every domain, it can just go to AI Catalog.json. And we expose the registry, the directory itself. So for every entry in the directory, we can tell the agent, yeah, for example, Vercel, these are the agentic resources, and this is how you access them. And obviously, this directory is fully ARD or agentic resource discovery compliant, so any agent can just query that directory with any query, right? So this is Aurora.directory, if you wanna take a look. All of it is part of the Aurora multiverse, journey, directory, and the ranker. And we found out some very other interesting insights. For example, being agent-ready and being human-accessible is very similar because LLMs, when they come to a website, they're like users or people with vision disabilities. They don't see your website. They need other signals to understand how to work with your website, and making your website human-accessible has agent-accessibility and vice versa. So to wrap up, these are MCP apps. This is what I've been working on in the past few months, and this was the last piece for the agentic web. This actually brought the agentic web, and now agents start to roam the web, and agents need different things. They need different roads. They need different infras, but we don't want to rebuild the web for agents. We want to make the web agent-accessible. So let's just make sure the web is prepared for when agents are coming. Thank you very much.