ChatGPT4o is the latest LLM model released by OpenAI. This new model is supposed to be 2x as fast and half the cost of the previous version. In this video, we put this new model to the test with CrewAI to see if it speeds up our crews and makes them cheaper to run.
Timestamps: 00:00 Introduction 00:27 GPT4o Experiment Overview 01:36 Join The Free Skool for AI Developer 02:02 Experiment #1 - LLM Only 04:08 Experiment - Results 06:10 Experiment #2 - LLM + Internet 08:10 Experiment - Results 09:35 Experiment #3 - LLM + Internet + Embedding 11:44 Experiment - Results 12:29 Outro
Hey guys, so Chachapiti version 4.0 just dropped this week,
so I wanted to put it to the test with Crew AI
to see how it performs in the real world.
So to test this new model,
we're gonna be running three different crews.
And as we run these crews,
we're gonna be substituting in and out
the old and new version of Chachapiti
so we can see how they stack up against each other.
And honestly, I was super impressed with the new model,
but it didn't perform exactly how I thought it would.
So let's go ahead and dive into this experiment
so you can see exactly what I'm talking about.
So like I mentioned earlier,
I've created three different crews
that we're going to run against each other
using the new and old version of Chachapiti
just so we can see how they stack up against each other.
And what I've done for each of these crews
is I've actually updated them
to use the latest version of Crew AI,
so that's gonna be the fastest and best version.
And what I've also done
is I've added in a tool called Agent Ops
so we can track just to see how much
and how fast each of these crews runs.
And to make things interesting,
what I've also done is I've ordered these experiments
in order from the simplest,
where we're just doing normal LLM calls,
and then eventually we add in more complexity.
So like eventually we're gonna use tools
to access the internet,
and then we're also eventually
gonna start working with embeddings
just so we can see how these different models
perform with different crews
that get more and more complex.
And it's also important to mention
if you actually wanna try out any of these crews yourself
to run the experiment or to see the results
or just actually use the source code
for these new updated versions of these different crews,
go ahead and check out the link down description below
because I've added the source code there
for you guys completely free to download.
So go ahead and check that out right after this video.
But enough of that,
let's go ahead and actually start diving into the experiment
so that we can see how ChatGPT 4.0
stacks up against the old version.
So let's go ahead and dive in.
Oh, real quick, I just wanna point out
that I have a free school community
for AI developers just like you.
We have over a thousand members now
where people are talking about the projects
they're working on, they're helping each other out,
and we have free weekly coaching calls
so I and other members of the group
can get you unstuck with whatever problems
you're running into.
So if you wanna meet a ton of other AI developers
of all skill levels who are just like you,
go ahead and click that link down description below
so you can join the free community.
All right, let's get back to the video.
All right, so let's go ahead
and dive into the first experiment
where we're going to be running the Crew Game Builder.
Now, if you haven't seen this crew before,
basically what it does is it takes in an input
of a game you want to build,
and then it uses a bunch of different agents
to go off and code that game, usually in Python.
So let me just show you kinda what it looks like in action.
So as you can see, what I've done real fast
is I've pulled out the Game Builder code over here,
and this is the updated version, by the way.
So this means I'm using the Agent Ops
and I'm using the latest version of ChatGPT.
The way I'm using Agent Ops is you can see
I'm importing it up here and just initializing it.
And this is what's going to allow us to track
how expensive and how fast this ran.
And then additionally, to make sure
that we're using GPT 4.0, I updated our agents file.
And what you can see right here
is we're using the model 4.0.
And anytime I change it back and forth,
we're updating all of our different agents in our crew
to make sure we're using the latest version.
So that's how we're hopping back and forth
for this experiment.
And final thing, just wanted to point out,
we are using the latest version.
Like I said, version 30.11,
and this is how we're adding in Agent Ops.
Okay, enough code.
Let's actually go ahead and look
at what this crew is able to do.
So what you'll notice is I have an Outputs folder,
and this Outputs folder is where we're able
to basically put all the different results
so you can play with them yourselves.
This is the two different ones for the game planner.
So these are the different outputs for the snake game.
So you can see they're like basically identical
around 160, 180 lines.
So very, very similar.
And I wanna go ahead and actually show you me running them.
So I just changed my directory to that folder.
Now I'm just gonna run Python and the name of the file.
So when I run it, what it'll do is it'll take me
over to the game that it just built, which is crazy.
And then now I can actually start playing and controlling it
so you can see, well, yeah, there we go.
See, I don't suck too bad.
Yep, once we die, we have a high score and we'll start over.
So that's the game it built for us,
but now let's actually dive into the comparison part of it.
What, how we're gonna go compare?
Well, once again, we're going to use AgentOps.
It's a completely free tool to use.
Definitely recommend trying it out.
This walkthrough will show you exactly how to set it up
in your own projects.
Just click Get Started for free.
Once you have actually installed AgentOps in your crews
like I showed earlier, here's what you get to see.
This is a deep dive.
So you can see Session, Drill Down.
Over here on the left-hand side,
I have the Chet GPT 4.0 version pulled up
and you know that's 4.0
because if you scroll down to the bottom,
you can see my model here.
And over here on the right,
you can see that I'm using 4.0 Turbo.
So let's actually start doing some comparisons
so we can actually see how each crew
stacked up against each other.
So all in all, you can see they had a similar number of runs.
Each one, this one had eight runs
and this one had seven different events.
And now let's actually look at the important part.
So is Chet GPT 4.0 faster than the other one?
Right now, at least for this example, it wasn't.
And I ran this a bunch of times
and it just so happens this time it was zero.
Sometimes I see a 10 to 20% speed improvement,
but for whatever reason, this time it didn't.
The important part though,
is let's actually look and see how much these LLMs cost.
So on 4.0, it definitely is cheaper than this other one.
It's about 20% cheaper to run 4.0 compared to Turbo.
And basically the way most of the costs are generated
for these crews come down to the number of tokens
that we processed.
So in this case, we basically processed
close to 14,000 tokens over here.
And we calculated close to almost what, 20,
a little over 20,000 tokens over here.
In just in case you haven't like heard of
prompt tokens and completion tokens,
prompt tokens are inputs and completion tokens are outputs.
So you can see we were processing a lot more information
inside of GPT 4.0.
And even though we processed an additional 5,000 tokens,
we were still much cheaper than Turbo.
So all around, round one does go to version 4.0.
I would have liked to see in that time
come down a little bit more,
but let's actually go ahead and see what's gonna happen
in the next versions and the next experiment,
just because this experiment was completely done
all locally.
There was no reaching out to the internet
or anything like this.
This was strictly LLMs talking to other LLMs.
Let's go ahead and dive into the next experiment
where we're gonna add in our crews,
the ability to go off and search the internet.
All right, so welcome to the second experiment.
And this one turns out way better.
And you'll see what I mean in just a second.
What we're running in this comparison
is we are using the Crew Trip Planner.
And this is by far my favorite Crew AI example
that I have seen built so far.
And the way it works is you are able to say
where you're starting at, where you wanna go,
and it'll plan an entire trip for you.
So let's actually hop over to the code
so you can see exactly what I mean.
This is an example of what I've hard-coded
in the example that you can download.
But basically I'm saying I'm gonna go from Atlanta, Georgia,
and I wanna go over to Croatia during these dates.
And here's what I wanna see on my vacation.
And then from there, we'll start up a crew,
and it'll go off and run for us.
Now, let me just go ahead and show you the results
because this is pretty cool.
So we're gonna start off with Turbo.
So this is what Turbo was able to produce for us.
You know it's a nice seven-day itinerary
where it's gonna just walk us through
what we're gonna do every day,
morning, evening, and afternoon.
Very nice.
It walks us through exactly what we're gonna do.
At the very end, it gives us a budget breakdown
and then some packing suggestions
for what we should do while we're there.
Very cool, very cool.
I like it.
But 4.0 goes even further.
What 4.0 I've noticed starting to do
is it provides a lot more links.
So this has actually happened
in a bunch of the different examples that I've tried.
It basically does a much better job
of understanding important items
that it needs to pass over to you as the reader.
So here's their example.
So you can see, once again, seven-day trip.
It's a day-by-day suggestion,
but it keeps giving us more information.
And this isn't just a one-time fluke.
I've run this one about like 10 times.
And every time, it does a much better job
of giving us more information.
4.0, it seems to be more intelligent
and it's able to just do more.
So this is just an example of like,
yep, it gives us links with things we need to do
and try while we're out and about traveling in Croatia.
So now that you've seen what it can do,
and you can see, once again, at the very end,
it gives us a breakdown, packing list, and things like that,
what's actually happening in a speed and cost comparison.
So let's hop back over to Agent Ops.
And this is where my mind was blown.
So as you can see, 4.0 had about 20 different events
and Turbo had about 23 different events.
Now here's what's interesting.
We were able with 4.0 to run in two minutes and 45 seconds,
whereas Turbo took almost five minutes, which is crazy.
I think all in all, it was about like 40% faster.
And then when you look at the price comparison down here,
this was 36 cents and this was 50 cents.
So we're 25% cheaper and we were running 40% faster.
So it's not the twice as fast and half as cheap
what we were told, but it's getting pretty darn close.
And the part that I also think is interesting
is we're not comparing exactly apples to apples
because what's also interesting is when it comes over to 4.0,
we were processing a lot more information.
All in all, we had about 45,000 tokens,
whereas over here we had 34.
So even though we're doing more work with our tokens,
input and output wise,
we were still much cheaper and faster.
So this is the kind of result I was expecting
to see with 4.0, super impressed with this version.
And the other part that you got to remember
that's pretty interesting,
we're also going off and searching the internet,
pulling in tons of data from everything that we're reading.
And with that large amount of texts we're pulling in
and processing, we're using it a lot more effectively
when driving towards our final answer.
So in this example, we did LLM plus internet.
So let's go ahead and dive over to our third example
where we're going to do LLMs
plus we're going to do the internet
and we're going to start working with embeddings too.
So let's go ahead and dive over to our third experiment.
So let's go ahead and dive into the third example,
which is going to be the stock analysis crew.
Now, the way this crew works is we pass in a stock ticker
that we want to analyze,
and then our crew will go off
and do a bunch of fundamental analysis of the stock
and give us some insights and recommendations
on what it thinks we should do.
Let's go ahead and actually look at what's happening
and how it works.
So per usual, we have a crew
that we're passing in our agents, our task,
the usual part, but in our case, like I said,
we're just going to look at the company Tesla.
So let's first look at what Chachabit 4 Turbo
was able to do for us.
All in all, it gives us a nice executive summary
of like what it recommends we should do for the stock.
And then it does a really good breakdown.
And the other part that's important to mention
is that these crews are not only just, you know,
using historical information,
they're going off and searching the internet.
And more importantly, what they're also doing
is going out and searching the,
let me pull it up so you can see it,
but they're basically looking through the SEC records.
So basically the quarterly filings from these companies
were going off and searching through those.
And that's what you can see is exactly happening
inside of these different basically reports.
So here we're looking through everything that's happening
over the past few quarters,
and you can kind of see what this crew
is recommending we should do.
And all in all at the very end, it's saying like,
hey, do a hold basically if you have a higher risk tolerance.
So that's what Turbo told us to do.
Now let's go ahead and look at what 4.0 told us to do.
So here's the report for this one.
And once again, I think this is awesome
because as you can see,
4.0 once again is providing us access to URLs
and links that it thinks we should be investigating.
So you can see that it's giving us different reports like,
hey, do you want to see how this stock is trending,
their earning reports?
And here's some resources and sources.
So we didn't change anything inside of our crew.
It's just that 4.0 seems to basically
be a little bit more anticipative of like,
yep, this is probably the information you want to need.
And it also, once again,
it's giving you some more,
it's strategy and suggestion
for what you should do for this crew.
So all in all, awesome all around.
I love that it came to the same conclusion,
just came out from two different answers,
two different ways.
And also that 4.0 gave us a lot more links along the way
so we can go off and do our own research.
But enough of that,
let's go ahead and hop back to the comparison.
So once again, we're over here in Agent Ops.
On the left, we have 4.0 and on the right, we have Turbo.
Once again, you can see that 4.0 beats Turbo.
And this time we barely beat it, but we still beat it.
And the important part that we were able to beat with 4.0
is the cost.
We're at like, what, 30 to 40% cost reduction.
And in this one, once again,
we were processing way more tokens.
So over here, we're at basically 44,000,
while over here, we're closer to like 47,000 tokens.
So we're doing more with our information
and we cost less and we're faster.
So all around, I thought this was awesome.
And I hope you guys now are super excited
to go off and try 4.0 on your own.
Let me know too what you think,
if you think these crews are smarter
when they're using 4.0.
And let me know if you are going to make the switch
to only using 4.0 from here on out.
And that's a wrap for this video, guys.
I hope you all enjoyed seeing this comparison
between the new version of ChadCPT and the old one.
And I hope you're pumped to start using ChadCPT 4.0
in your own crews.
And like I said earlier,
if you want to actually go ahead and download
the new crews that I created for you guys
that are on the latest version,
using Agent Ops and version 4.0,
go ahead and click that link down in the description below
so you can go ahead and grab the source code.
And also, if you already haven't joined
that free school community for AI developers,
just like yourself,
go ahead and click that link down in the description below
so you can meet a ton of like-minded people
and hop on our weekly coaching calls
where you can get support completely for free
for your own projects.
But I hope you guys have a great day
and definitely check out one of these videos
right after this video and see you on the next one.
See ya.