AI Engineering in 41 Minutes: From Demo to Production

AI engineering is one of the most in-demand skills in 2026. And one of the best books that I've read on this topic is called AI engineering by Chip Hewitt. And this is all about building applications with foundation models. And we're going to dive deep into the book. I'm going to share with you a lot of the sections that are within the book, and I will give you my own take into it. We're going to dive into the theory. I'm going to give you lots of examples. So grab a cup of coffee and let's dive into it. Section one, the big picture. We're going to start with the big idea of the book. AI engineering turns foundation models into useful, safe, fast and reliable products. And what do we mean by foundation models? We basically mean language models, the large language models that we use every day, whether you use chat GPT, Gemini, Claude, Lama, et cetera. Those are the foundation models. And there are four main things that we focus on is the quality, is the output accurate and useful? If it wasn't, you won't be using those foundation models. The safety, is it harmful for the user? Is it risky to use? How can we ensure that our data is private and secure? Speed and cost, because no one wants to wait five minutes to get a small response. So that's very important, but also speed and cost, they come together. If you want something to work faster, it will cost more. So that's also something to keep in mind, especially when you start building an application and the feedback. Of course you need to learn, improve over time, because when you build something once, you still need to keep refreshing it and iterating over time. So the book is about moving from a simple demo to a real product. And that's what AI engineering is. So why AI engineering exists now? So for three main things, better model capability, because nowadays we have really powerful language models that can do so many cool things that we're going to dive into in this video. So modern models can write, reason, summarize code, and work across many tasks. And we now even have agents that can collaborate with each other to perform really crazy tasks that usually takes a few people to do. It also has easy access to APIs with MCPs, et cetera. So it's easy to use external tools and lower time to build. Have you noticed how quickly now we can build websites or applications using tools like Repl.it, Lovable, things like this? Of course, it might not be perfect, but to get started, it's insanely fast. And that's just mesmerizing to me. So more demand plus lower barrier to entry equals a new engineering discipline. But AI engineering is more than just building demos and that's what we need to understand. Demos is one prompt, manual testing, one model call, low stakes, no monitoring. It looks cool. It runs on your computer. You can show it to a few friends, but it cannot sustain hundreds of users, getting quick responses, managing costs, monitoring, debugging, et cetera, which is what we see in production. We see system design, evaluation pipelines, context, tools, routing of different language models, guardrails and reliability, monitoring and feedback loop. And we're going to dive into all these topics in this video. Let me walk you through the full AI engineering life cycle in a simple way. So first we have the use case. What problem are we trying to solve? Because of course, we're only going to build an AI system if we have a problem to solve in the first place. Then what's the model that we're going to choose? Is it open AI, like LGBT? Are we going to use cloud, Opus, Sonnet, Gemini, et cetera. What are the models that we choose? Then we have evaluation. And here by evaluation, we mean what is the success metric that we're trying to optimize for? What is the problem that we're trying to solve? But also how do we measure the success of this AI application? Basically, what are we trying to optimize overall? Then we have four where we see the prompts, retrieval, augmented generation, agents, fine tuning. This is basically the stuff that a lot of people jump into directly, but there's much more to it. And that is just to improve the quality of the output, the retrieval, et cetera. Those things have their purpose because they help us improve the system overall. Then we have the production architecture. This is to make it reliable, usable at scale for the end user. We also have monitoring and this is where we track failures, costs, latency. This is where we improve the system. And for those that come from a machine learning background, this is the MLOps part where we try to keep things flowing for the end user without breaking consistently. And lastly, we have the feedback loop. This is to keep improving the system. Never consider a system done once and finished. There's always things to improve. Things change, model change, and you need to be aware of that. Section two is foundation models. So we've got to try and explain what they are. So what is a foundation model? A foundation model is a general purpose model that can be adapted to different tasks. Think about it as chat GPT. It is adapted to help you answer general purpose questions, do research, et cetera. It is able to take on text, audio, video, images, infographics, all these things, which makes it multi-modal. So when you hear multi-modal, it just means that the language model is able to use images, audio, text, et cetera, and also output the same ones. And it is also adaptable. So when we build retrieval augmented generation systems, we are injecting our own data, our own context so that the LLM uses those as context to better respond to our queries. We also have fine tuning, which is where we adapt the language models that it responds to the way we want. And of course we adapt it to our own use cases. So examples here are GPT, Claude, Gemini, LLAMA, and all the foundation models that we see nowadays. So from language models to foundation models, and here we basically just see like the evolution. We have the language models that learn patterns from texts. We have large language models, which is the same, but at scale. It scales the data, the computes, the parameters. That's why you hear that there is a model that has 7 billion parameters, another one that has 120 billion parameters, et cetera. So they're just bigger, heavier, and therefore they can answer more nuanced questions. From there we have foundation models, and these are usually general purpose and often multi-modal. When you start GPT or Claude, these are multi-modal foundation models. Then we have AI applications, and that's what you're going to be building on top of those foundation models. So they are built by adapting the model to a real use case. So you use these foundation models, you tailor them into whatever application you want to build to try and solve a business problem. So scale plus post-training plus better interfaces make modern AI applications possible. Here we're just briefly going to look into tokens and self-supervision. So tokens basically is how language models understand what we give them. So basically every word or every fraction of the word is a token. For example, here, just to simplify, AI is a token, engineering is a token, terms is a token, et cetera. Each one of these is a token that goes into the language model to be processed. They're basically a unit, a model reads and predicts. Self-supervision is the predicted next token. And remember that language models are just predicting machines. They always try to predict the next token based on probabilities. It's very advanced and it looks very smart, but they are probabilistic, not deterministic. So that's why every time you run chatGPT for the same query, you might get a different response. So the model here learns by predicting the next token from massive unlabeled text. Something very important to understand is why AI outputs change because the model generates tokens probabilistically, as we said. So the same prompt can produce different valid answers. So the key factors that we need to remember here is that there are a few parameters that makes it that the AI or the foundation models ends up generating answers differently every time. They might be the same result. For example, if it's a mathematical formula, but then the way they process it and the way they answer it and the words that they use will differ. And this is because of the temperature, which controls randomness. So when we're close to zero is when we want it to be as factual as possible. And when it's closer to one is when we want it to be as creative as possible. We have top P, which limits choices to the most likely mass. Top K, which limits choices to the top K tokens. We also have presence or frequency penalties, which reduce repetition. And we also have the system load and non-determinism can cause variation. So there are quite a few factors that make it that every time we might get a different response. We can see here on the right side that the next token probability distribution, it's just a probability and the model is going to go and get the highest probability to predict the next token. And that's why different foundation models, when you compare Claude or Chai GPT will certainly give you different opinions and different answers because they were trained differently. They have a different algorithm and therefore the responses will be different. That's why we'll have this conversation that Claude is better than Chai GPT for coding. Chai GPT is better than Claude for designing or whatever. So these things is because they were trained differently. So sampling is why the outputs are not fully deterministic. So if we want something deterministic, we can just write a Python code logic or a function. But when you use foundation models, it's probabilistic and that's something to be aware of. And that's why we get into this hallucination. What is hallucination? It's when the foundation model just makes up something and sounds very confident, but it's not true. You're asking a question, as we can see here on the right side, who won the 2024 Mars cup, as if this cup even exists and the model will hallucinate. Orion FC won the 2024 Mars cup in a thrilling final, but no such event exists. And this happens more than we want to admit because we often think that these models have answers to everything and they're trained to try and answer everything. Unless you put guardrails in place to tell them only research facts and if you don't know return that you don't know, they might made up something and that can be very harmful, especially when the work that you provide is very sensitive. So why does the model hallucinate? There are a few different reasons. There's a knowledge gap. Think about it. For example, ChildGPT they released the last model at some point in 2025. So let's say May, 2025, I don't have the exact dates. I'm just making something up. So everything after May, 2025, technically the model wasn't trained on. And therefore if you ask about a question in early 2026, it couldn't answer that question without searching the internet. So sometimes you have to prompt it to search the internet to go and get the newest value. Otherwise it might hallucinate and give you something because it just wants to answer. Other reasons for hallucination, we have pattern completion and probabilistic generation. And I'm regrouping this together because as we said, the foundation model is just probabilistic and tries to predict the next token. And sometimes it just fills the gap with something that might sound good and probabilistically high, but it doesn't necessarily answer that specific query of the user. So sometimes the most likely next token is not always the true one. And language models, they don't know that unless you give them context. Also there's weak grounding because as I said, if they don't have a good context, the answer might drift into something that is completely random and that can harm the results that you get. So hallucinations are reduced with better context evaluation and guardrails. So we've got to dive even deeper into each one of these things in this video, section three, evaluation. We measure what matters. Good evaluation is the foundation of reliable, improving AI system. So why evaluation is the core skill because you can't improve what you don't measure. And this works for a lot of different disciplines, but even more in AI. Evaluation turns AI development from guesswork into engineering. Remember that without good evaluation, you're just hoping. And that's a core skill when building AI system. Good evaluation helps you measure progress, compare models and approaches, catch regressions early, align with business goals and build user trust. Evaluation is the feedback system for AI quality. So if you cannot evaluate the responses that you get, the latency, the costs, everything that you have in the system, you're just flying blind. And therefore you might get a serious impact in the way the user perceives the AI system. Why AI evaluation is hard. Yes. So this is the real deal because it's harder with AI system because unlike traditional machine learning, AI outputs are open ended and often subjective. As we said that foundation models are probabilistic, not deterministic. And therefore the responses might be completely inaccurate or can be missing context, et cetera. And that's why evaluating those responses because most of them, they sound good, but they might not be accurate. And that's something that AI evaluation deals with and that is important for you to understand. So we have a few things in here that I want to look into. So what makes AI evaluation hard? First is open ended outputs. Many valid answers can exist. So sometimes a response might be a few different things, not just one. Subjective quality, for example, helpfulness, clarity and safety are hard to score automatically. How can you score the quality of the output of a certain query? It can be very tricky, especially if you don't have the context. Context dependent. Good answers depend on task, user intent, and domain. If someone even didn't phrase their request properly, the response might not be that accurate as they expected. But of course, there are techniques to deal with each and every one of these things. Multidimensional quality, safety, cost and latency all matter. Otherwise they can become very slow, very clunky, costly, etc. AI quality is not a single metric. It's a lot of things combined. So there are two broad families of evaluation methods. We have the exact evaluation and we have the subjective evaluation. So let's look into each one of them. Exact evaluation, which is the objective one is deterministic or verifiable outcomes. It can be checked programmatically. Example, co-test, math answer, schema validation. These are to measure correctness and functional performance. So these are things that we can go and evaluate by ourselves by testing. Regarding the subjective evaluation, quality is judged by humans or AI judges. And we're going to dive into AI judges later, but we assess the quality of a response by a human that understands the context or we use another foundation model to analyze the response based on the context given and the query to see if the response makes sense. Many good answers may exist, which can make it confusing. Examples, helpfulness, tone, creativity, explanation quality. The goal here is to measure user experience and nuanced quality. The nuanced part is what makes this evaluation subjective. Now we're going to talk about AI as a judge, also called LLM as a judge. So these terms are interchangeable. You can use either or. One model can evaluate another model's output using a rubric or side-by-side comparison. So how it works in very simple terms. It defines a rubric and examples. It generates candidates' outputs. It asks a judge model to score and rank them. It aggregates the scores and compares results. So let's look on the right side. We have output A and output B. Both of them are going to go through a judge LLM and by judge we, it's just another foundation model or LLM is going to judge those outputs and the context given to them. And then you're going to score them. And tell us, okay, output A gives me a score of 90%. Output B gives me a score of 75%. Therefore output A is the winner. And that's the one that is going to be chosen as a response for the user. But as you can imagine, it's not because you use AI as a judge that it means that there is no bias. There are a few limitations like biases, inconsistencies, or model specific preferences. And these are things to be aware of. Let's look at evaluation system design. And here we have a general pipeline for evaluating AI system. We start with testing the data, which is the questions, the tasks. Basically we are still here trying to figure out the benefits for building the system in the first place. Then we have the model, which is the system and the tests, the foundation models in action here. Then the evaluator, the human or AI as a judge to evaluate the output given by the model. And here we're abstracting a lot of the things that are happening in between each one of these layers. Then we have scores and metrics. So we score the results that we get from the human or the AI judge per criteria. And then we can make informed decisions whether we shift or we improve. And this is like an iterative loop that goes back into testing, modeling, evaluating, scoring, and making decisions. Model selection is not just leaderboards. Public leaderboards are useful, but they rarely reflect your exact use case. Why leaderboards fall short? Because different data sets, tasks, and metrics. So there is complexity. There is nuance. It may not match your domain or language, which might be important for your organization. It doesn't capture cost latency or UX choices. It changes quickly over time. And the data distribution is unique, which in reality might not be. So what to do instead? Define your use case and success criteria. And this is literally the step number one that you need to focus on. Build a private evaluation set, which is going to be referenced later on to improve the quality of the output. Run side-by-side benchmarks. Spending time on these things might not seem very exciting, but that's what gives you a better quality output that you can trust and the users can trust as well. Evaluate quality. Safety, cost, and latency, because cost can quickly increase and latency can also quickly, dramatically increase. And you might start losing users because of that and make data-driven decisions so that you can improve the system over time. Let's look at the evaluation pipeline for production. So now we're talking serious stuff. We have the first one is test set, the tasks, the prompts, the edge cases, et cetera. We move into the system version where we have the model, the prompt, the tools, retrieval, augmented generation. This is the basis of the system. Then we have the evaluator. Is it human evaluating or AI is a judge evaluator. Then we have a score for the quality, safety, cost, latency. Then step five, we make decisions, we shift, we hold or improve. So here we make a decision on, is there anything to improve or are we going to shift that model? And then of course we have the monitoring that will track the production behavior because things will drift, things will change. And we always need to keep improving the system over time. So evaluate after prompt changes, model changes, data changes, and tool changes. So after every change, you need to be able to evaluate so that you know if anything broke or if anything changed your system drastically. Section four, improving quality, prompts, context, and guardrails. Prompt engineering is the first lever. It is the fastest way to change model behavior without changing the model itself. Why does it matter? Because it's fast to test and iterate, changing a prompt is as easy as just replacing the prompt that you had before. It's low cost compared to fine tuning, which we're going to talk about briefly later. It's useful for format, tone, and instructions. We can write in the prompt what format we want, the tone, the instructions, everything within the prompt itself. It's often the best first step before bigger changes. It's cheap. You don't need retraining. It's flexible, easy to adapt to a task, and it's practical and it's great for early iterations. And also one tip, is try to version your prompts. So if you change your prompts over time, try to keep a copy or a version of the previous prompts in case you want to revert back or compare the outputs of them. So start with prompts, then move to rag, then agents or fine tuning if needed. Let's look into the anatomy of a good prompt. Strong prompts combine clear instructions, useful context, and a target output format. So first one, we give it a role or a system. So we set the behavior and the rules. We can say it's something like act as an experienced data scientist or act as an experienced SEO manager or things like this. So you give it a role. Then you say the tasks state exactly what to do. Exactly. You have to be precise. Then give it the context. So provide facts and background. Then give it examples to show the pattern that you want. And finally explain the output format that you want. Do you want it in JSON? Do you want it in HTML? Do you want me in Markdown? Explain exactly how you want it to be. So we have a small example here. You are an expert assistant. Summarize the report. Use the notes below. Return three bullets and one action item. Of course, this is small and summarized, but you get the idea. Clarity always beats cleverness. Let's look at prompt engineering system design. Prompting in production is more than a text box. So we have the user input. So the user input is the query. What is the user asking for? Then we have the system prompt. The system prompt is that detailed prompt that explains to your AI system what it is, what is it doing, what's the purpose, et cetera. So that's a system prompt. It's always used every time there is a query. The system prompt is always the first thing that our AI system goes through. Third, we have the retrieved context. So in order to answer the user input, we need to get the correct context for it. That's why we're going to retrieve it. That's usually something that is done in RAG, retrieval augmented generation, where we do have a knowledge base and we retrieve the correct context to answer the user input. Then we have a few short examples, which is where we give via a prompt a few examples of how we want the response to be. We call that a few short examples. That is usually tested with evaluations. Then number five, we have the prompt builder. So basically what happens, we have the user query, we have the system prompt, we have the retrieved context with a few short examples. All these are built into a prompt, given then to number six, the foundation model to be able to answer us the way we want. It's going to go through the output parser and validator. So it's going to validate the structured output. For example, we said that we want it to be in JSON format to then give us a response. Can you see all the things that it goes through? When it's put like this, it's very simplified. So always think about it from the user side. The user gives the query, but the AI system has a system prompt. Then our retrieval augmented generation system is going to retrieve the context needed to answer that user query. So there is chunking, indexing, there are a few things that go into there. It's going to also look into the few short examples given so that it gets inspiration from them. And all that is going to be passed into a prompt builder is going to be used by the foundation model before being able to answer us. So production prompting needs structure testing and output control. A very important topic is prompt attacks and prompt injections. Untrusted inputs can try to override your intended system behavior. So common attacks that we see, ignore previous instructions, reveal the hidden prompt, follow malicious content from retrieved documents, trick the model into unsafe tools. And this is stuff that we see a lot because anyone can inject any sort of information or make the language model behave in a certain way because it's so some hidden text or some injected content within it. So that's something to be aware of. So the defense is here is to keep the instruction hierarchy clear, treat retrieved content as untrusted, restrict tools with allow lists and permissions, check outputs before they reach the user. And that's very important. Prompt injection often enters through user input files, webpages and retrieved context. And that's why we need guardrails system design in place that can be placed around the full AI workflow to help us keep this system as secure as possible. And here, we're going to see that in this system here, we have a few guardrails in place. So we have the user inputs that is going to go through the input check. So it's going to be checked as well. Then we have the context retrieval checks. So we're going to check and validate the context retrieved as well. If it's usable, then we have the tool permission layer. Are we even allowed to use those certain tools? Then we have the model and we're going to look into the output check. So the final result is also going to go through a check before getting into the user or the human review. And we have a few things in here, like the policy filters, prompt injection checks, schema validation, and escalates with a human to review or an LLM as a judge to review when uncertain. So guardrails reduce risk before, during, and after generation. So guardrails is not just before the output, but they can be along the way. A very important part of AI systems is context. And many AI failures happen because the model does not have the right information. Without context, the answers are generic. They can hallucinate, weak personalization, and misses current or domain facts. That's reality. Because as we said, foundation models are probabilistic. The foundation models stopped their train at a certain point in time. So they don't have the newest information and they're not personalized to you. They are personalized to everyone on the internet. But when you give it context, the answers are more relevant. It gives you better factual grounding, stronger domain performance, and more useful user experience. And it's also adapted to you, to your tone, to your style, to the output that you ask it. So context matters extremely in AI systems. So the context can come from retrieved documents, user data, tools, structured system state, prompt, etc. Let's look at a basic RAG system design. We have a user query. Then we have the retriever. The retriever retrieves the data or the chunks from a vector store or a document corpus. So think about it this way. We have a knowledge base. They store it in a database. Let's say a SQL database. We have a lot of documents, PDFs, etc. Those ones will need to be chunked, which means that we're going to take paragraphs. So we're going to take a number of characters because putting it into a vector store, huge PDFs at once, it's not optimal because when we try to retrieve some of it to answer the user query, bringing a whole PDF to answer a small question is not relevant. We might need the section, a paragraph, a page, etc. We're going to chunk that data from our SQL database and put it in a vector store or put it in a document corpus so that we can use it for retrieval. We're going to take the user query, put it in the same vector space as the retrieved chunk. Then we're going to find similarity and then we're going to pick the relevant chunks and docs, prompt with the context. Now that we have the relevant chunks and the user query, we're going to add those into the prompt as context that the LLM is going to use to give us a grounded answer. And this is the whole basis of RAG in a very simplified way. We retrieve, make sure that we have the context to be able to generate a good output. So the RAG helps when the model needs facts it did not memorize or that change over time. Think about it. If you have sensitive data that only your business is aware of, how could a foundation model answer that question for you? It's literally impossible. It doesn't have the context, but when you give it access to your database, you chunk the information that you have, the database into smaller pieces so that it's easier to be able to answer questions. Then you're given the foundation model enough context to be able to answer you in a grounded way. Let's dive a little bit deeper into the RAG data pipeline. This is very interesting. We have two different segments. We have the offline indexing segment and we have the online query time segment. So think about them into two things that go in separate ways. The offline one is what we have, as we said, the document sources, our database, that that data is cleaned and passed to be able to be chunked and stored later on into a vector store. So think about it. We have a few PDFs. Those PDFs are our document source. We're going to clean them and pass them. In PDFs, we usually have the abstract, we have paragraphs, we have sections, titles, references, et cetera. Okay. We're going to clean the things that we need. We don't need, then we're going to chunk them. We chunk them by section, by characters, by pages. We're going to need to find out which chunking method is best for our data, because there is no one chunking method that works for everything. We're going to use an embedding model to turn that content into a vector that is going to be stored in a vector store or index. In the online and query time segment, this is the part where the user is going to ask a query. And then we're going to retrieve the relevant chunks from our vector store. We're going to use something like a re-ranker or a filter to be able to pick and choose what are the best chunks to be able to answer the user query. Then we're going to inject that context into the prompt so that the LLM is able to answer. So a good RAG starts with clean documents, strong chunks and a reliable indexing. And these are things that you will experiment with because there's not just one chunking method. There is not just one embedding model. There is not just one vector store. And sometimes you're going to combine a hybrid approach where you look into keywords as they are, and also the semantic meaning via embeddings. So this comes with practice. A very important layer of RAG systems is the retrieval methods. Here we're going to talk about the keyword search, the semantic search and the hybrid approach. So we have the BM25 or the keyword search method is when we look at exact words. This is strong for rare terms and simple baselines. Let's say I'm looking into the keyword, specifically AI engineering. I'm looking into this keyword and I want the response to be able to look into that keyword as it is. I'm going to use a keyword search. But in the embeddings and semantic search layer, it understands the meaning of the search, not just the exact keywords. This is better for paraphrases and useful when wording differs. Let's say that my query talks about AI systems and it didn't mention anything about RAG, for example. And I ask a query about, can you tell me more about different AI systems that I should be aware of? And the first one, the keyword search will go and look into AI systems as it is and try and find in all the documents that you have, what is the best possible answer for it? But the semantic search is going to understand the meaning. It's going to understand, okay, that AI systems can talk about agents. It can talk about fine tuning. It can talk about RAG and it can maybe give you a few chunks that are a bit broader that might answer your request even better. So the keyword search is best for exact match. Let's say you have some reference ID, you have a specific book title, things like this. You want an exact match. But if you use a semantic, it's best for the meaning where the wording doesn't matter as much as the context. We also have the hybrid search and this combines keywords and semantic and it's usually often the strongest one in practice. It's better for recall and precision. And then there's an ability to get both results and try to score them and choose the ones that rank better overall. So using a hybrid approach usually is best overall in many applications. Start simple, then test VM25, semantic and hybrid on your own data. Let's talk about improving RAG quality. Most RAG gains come from improving retrieval and context quality. Because if you feed your language model poor context, then you can imagine the output is going to be quite poor as well. So when we have better chunking, that's the first step. It's the better chunking. So we've split content into useful units and that really depends on the type of metadata that you have. If it's research paper, if it's a book, if it's Excel sheets, it really depends what's the data that you're trying to chunk in the first place because not one method will work for everything. Then there is query rewriting where we turn vague questions into stronger searches. For example, if the user wrote a query, the user might make a few spelling mistakes or the context wasn't really clear. There's query rewriting just to make it a little bit better. We also leverage metadata and that's very important. Let's say you looking into authors and the books, you have a lot of books or research papers, then the metadata of the author name or the country or the publishing company, all these things can be used as filters so that we can narrow down quickly to the responses that we aim for. Let's say for example, the user query is to know the last 20 American authors in the data. We can quickly filter by the country of the authors just with the metadata instead of scanning through all the documents to get that information. We also go through a phase of re-ranking where we reorder results by relevance. When we have lots of results, of course we need to pick and choose the relevant chunks for it. Then we have the evaluation loops where we measure what helps. Evaluation is always part of a RAG system so that we can get the best possible context for the LLM to give us grounded answers, because it got the right context. So treat RAG as a system to optimize, not just a vector search add-on. A few different RAG failure modes to be aware of. One is missing documents. The source is not in the corpus. So if it's missing documents, how can you even answer? Two is bad chunks. Chunks are either too large or too small or poorly split or you lose meaning along the way. Weak retrieval. The right content is not returned. So the retrieval is massively important layer because if you cannot retrieve the right context, then the response will be really bad. Next is poor ranking. Relevant content is buried below weaker results. Ranking helps us choose the best chunks so that we're able to feed the language model with the best ones. Then we have stale knowledge. Documents are outdated. Documents haven't been updated. And that could be problematic in future generations. Unsupported answer. When the model says more than the context supports. RAG reduces hallucination, but only when retrieval and grounding are strong. And the whole point of RAG is to give us more grounded, trustworthy answers and reduce hallucination. Let's look into the nuance here and the difference between RAG and agents. RAG retrieves information to be able to answer user query in a more grounded way. Whilst agents go further, they can plan, they can use tools, they can have access to memory. I can, they can take multiple steps. So if we look at RAG in a simplified way, we have a user question. We have retrieval mechanism. We give context to the LLM and we get an answer. So that's the RAG in a very basic way. If we just simplify it. Whilst the agent is given a task, he can plan through it. He can call tools. He can observe the results. He can decide the next steps and give us a final answer. So it really depends on the mechanism that we create for the agents, but they are more powerful. They're given some sort of autonomy that we allow it. We give it tools. We even allow it to review its own work and reason through it. We can even add other agents that can go in between certain tasks to be able to evaluate along the way and make sure that the responses are correct. But we shouldn't necessarily default to agents for any AI system project. And that's where we need to understand. Are we just trying to answer queries from a grounded source? Or are we trying to create something like an agent that can really take some sort of actions, use tools, have memory and reason along the way. So RAG is best for grounded answering, single step retrieval plus generation. That's it. And we have agents that are best for multi-step tasks and actions. They can plan, act, observe and iterate. So agents extend RAG by adding planning and tool use. You can always have RAG systems that have agents incorporated in them. Let's look at an agent system design. An agent loops through planning, acting, observing until the task is done. We start with the user goal. Then we go through the planning and reasoning. So we try to understand what is the user query. Then the agent is going to select the tools relevant to answer that query. It's going to call the tool via MCP. For example, it's going to observe the results that it gets. It's going to update the memory with the newest results because it can go back and see what happened before. So it has a memory that we give it. And it's then able to take the next action. If that next action is enough context to be able to answer the query, it's going to just answer. If not, it's going to revert back into the planning reasoning, look at the tools needed, observe, update the memory, et cetera. It's an iterative process. And there's always a human in the loop handoff if needed. If that's something that you built in, in the tool selection, because you can give it the tools that it can have access to. The observation as well, see if the tools that they were used were correct. And the next action, if you see that it gets the right context and it's reasoning properly. Think about agents as workflows with feedback, not just one model call. They are workflows. You set up each one of these steps by yourself. That's where we get the intelligence part of the agents, but overall they are just intelligence workflows. Let's look into agent tools and permissions. Tools make agents useful, but permissions keep them safe. So tools is the ability for the agents to perform certain tasks. If the agent doesn't have a search tool, you won't be able to search the internet. If the agent doesn't have the ability to access your database, you won't be able to query it. If the agent doesn't have access to some external APIs, therefore it won't be able to access them. So agents, it's not because we call them AI agents that they're able to do anything in the world. We give them those tools, we give them access to those tools. For example, search, database, code execution, external APIs, but we should also put in permission layers as well. And that's the things that we want to control that AI agent. We want to have some allow lists, what things that it can do, what things it cannot do, the scope, what is the things that it needs to perform. The rate limits stop at a certain point when we bring in some sort of threshold approval, audit logs, et cetera. Least privilege, give the agent only the tools and access it truly needs. And it's even better when you create an agent for a specific thing and another agent for another thing. You don't want an agent to be able to access 25 tools and perform 15 different tasks. It's confusing. The memory will blow up and most likely the results are going to be very weak. Remember that useful agents need tools. Safe agents need strict permissions as well. And important layers of agentic systems is the memory. The memory helps agents stay coherent across longer tasks. We have short term context where recent messages and immediate conversation context. So this is shortened context. We have task state, the current goals, progress, and intermediate results. We have the longterm memory, which is the stored knowledge preferences and facts over time. We have the interaction history, past interactions and decisions for continuity, but there are risks, stale memory, privacy issues, and growing context costs. If you've used code, you've hit context limits many times. And that's because if you want the memory to keep long historical context between you and the foundation model, or in this case, the agent, it's going to keep adding up and adding up and adding up and adding up. And at some point, every small request is going to go through a massive amount of information to be able to answer. And you don't want that. You either want to refresh the memory, clean it after a certain point, remove certain historical information after a few days. So you need to keep that memory fresh. Otherwise the context is going to become more cluttered and you're going to risk introducing some privacy issues and costs as well. So store only what helps the task and forget what is not needed.