In this video I use Claude Code to set up Hermes Agent on a 16GB Mac Mini and plug it into a local Gemma 4 12B model, so the whole thing runs 24/7 with no API bills. I let Claude do everything from the terminal: changing the Mac's settings to stay always-on, installing and wiring up the agent, and connecting it to 3rd Party Tools so the agent actually does real work.
I'm starting a blog to document all my AI adventures. I only make Youtube videos about 10% of the stuff that I do, learn, or experiment with. Drop your email to follow along: https://blog.workingmodels.ai/
This video demonstrates how to build a local AI assistant by integrating the Hermes Agent with a Gemma 4 model on a 16GB Mac Mini (M4). By leveraging Cloud Code for automation and LM Studio for model hosting, the creator establishes a private, \"always-on\" digital employee capable of managing professional workflows, such as Zendesk customer support, while optimizing limited hardware resources.
In our last video, we installed Jemma 4 onto this 16GB Mac Mini.
And in this video, I want to take things a step further and I want to install Hermes Agent
and plug it into the Jemma 4 model so that we can build our first AI assistant.
Okay, so let's start by opening up our favorite coding tool.
And I'm going to be using Cloud Code in the Cloud Desktop app.
I'll set myself to Opus 4.8 and I'll just go to Ultra Code.
And you can just use this as a benchmark for what's possible.
But we'll be using Cloud Code to do everything for us.
We'll be using it to help change the settings on our Mac.
We'll then be using it to install Hermes Agent to plug it into our Jemma model.
We'll also be using it to build out any dashboards or workflows.
The better the model that you have, the better the output you can get.
You just want like a really, really smart person setting all this stuff up for you.
So one of the reasons why we're able to manage everything on a computer
is because all these coding tools run in the terminal.
And if they run in the terminal, they have access to your device.
They can read your device and tell you what RAM you have available.
They can change the settings for you.
And a lot of these tools and apps that you download,
like as we know with Hermes Agent, they have a install line here.
So our cloud can actually just go to the website,
find this install line, and then automatically run that in a terminal.
And then help us set up everything without us actually having to
go through the very tedious and sometimes confusing process
of setting up a Hermes Agent.
So once it installs this, a lot of these tools also have a CLI.
And a CLI is just a way that you can run, use, change settings of,
for example, Hermes directly in the terminal.
So for example, let me just go to the LM Studio CLI.
We have list your models.
That's the command.
And to run that, we just need to do lms ls.
So I've got, it's already open.
I'm just going to go lms ls.
And I'm going to hit enter.
And then we see all the different models that I've got available on my LM Studio.
And I'm actually using LM Link,
which means that across all my different devices,
I can share the local models that I have.
That's why you see like 20 models on here.
As you can see, we can just operate under the hood.
We can operate LM Studio.
We can operate Hermes Agent.
And we can do that directly from Cloud Code or Codex or OpenCode,
whatever's got access to your terminal.
So let's just start by running through a very basic command.
Hey, can you let me know what hardware I'm running on this device?
So we're just checking the Mac's hardware details.
And as you can see, it's just executing the command that runs in the terminal.
Here we have a Mac Mini with M4 chip.
We have 16 gigs of RAM.
The display that we have is 1920 by 1080.
So it can read everything that we have on our device.
So the second thing you want to do before we even start installing
any of these Hermes Agent is,
hey, can you confirm if this Mac ever turns off
or if it's set to always be on?
And one of the reasons why you want to have this device to always be on
is if your Mac Mini goes to sleep,
your Hermes Agent goes to sleep.
So if it's sleeping at 3am
and you have like a really important email that you wanted your agent to get,
it's not going to get it until it wakes up.
So you want to have your Mac always on.
So in this case, I think this Mac is always set to always be on.
You can also do things like if your power ever goes out and it reboots,
you want to have like a reboot script
or like an option to like reboot the Mac Mini,
turn on LM Studio, turn on Hermes Agent, turn on Tailscale.
So I just realized that I didn't even have the ultra code turned on,
but I'll just turn it on now
and we'll start setting up our Hermes Agent.
Can you please download and install Hermes Agent?
I just want a very basic setup for now.
I just want to get it up and running.
I don't want to plug it into Telegram just yet
or WhatsApp or anything else.
Just install it and then give me the URL so I can open up my dashboard.
And also, I don't want to plug in any external services.
Like I don't want to use any paid APIs for doing web search or anything like that.
Hey, if you're watching this video and you're enjoying it,
you're getting something out of it,
I'd appreciate if you could either like the video,
drop a comment, or if you haven't already,
please subscribe to my channel.
That'll help me grow.
That'll help this video reach more people
and help more people build local AI agents.
Thank you.
So while Claude is installing our Hermes Agent,
let's go across to LM Studio and load up our model.
So in our last video, we downloaded the Gemma 412B,
which is 7.04 gigs.
In that time, there's been another version released,
which is the QAT, Quantized Aware Training.
This is just a little bit smaller at 6.66 gigs.
I haven't done too much reading into this,
but it's meant to, at a high level,
it's meant to like try and retain as much smartness as possible
as the full-size model at the smallest possible footprint.
As is the case with local models,
you always want to be testing different models
for different use cases.
And in this case, you might want to test
between the regular 12B and the QAT 12B.
And six days ago on Reddit,
this person said that the 12B QAT
was actually a regression for the use case.
So it's kind of contradicting
that it's meant to be as smart as possible
versus the regular 12B model.
But in this case, it actually like,
it didn't work for their use case.
So it's all about use case.
It's all about testing.
So definitely see if it works for you.
I guess for us, we'll just try and use a small model
for this demo.
And as you know, from the very first time we did this,
if we go max context length,
where we see an error saying
you're not able to actually load up this model.
And if we get roughly down to like 12 gigs,
just a little bit lower,
probably around here is the sweet spot.
And we only have like 34,000 tokens.
Now it's important to note that
since we're running Hermes agent,
we firstly, we cannot run it at like 30,000 tokens.
It's not enough for us to have a conversation
or complete any tasks at all actually.
But we don't want to go up too high
because we've only got 16 gigs.
So at 12 gig, we've only got four gigs left
for the operating system
and for Hermes agent to be able to check out emails,
complete tasks, go into different tools.
I don't really like using paid services
as much as possible.
I want to use local services.
So I think you can download file crawl
and run it locally within a Docker container
on your computer.
If you do that, that's going to use
like a gig of RAM, for example.
So as you try and start introducing different,
like make your Hermes agent do different things
on your device,
actually take care of more tasks for you.
You have to play this like really fine game
between how much RAM you're using
for the model and context
versus how much RAM you're making available
for these different actions.
But one thing I've been trying out recently
is using like Chromium
or using the browser mode
with the agent within the terminal.
We'll be able to go to different webpages,
search different URLs,
extract the content.
And this is going to be totally free
and private as well.
So just be mindful of the more
that you want your Hermes agent to do for you.
You have, everything is like a very,
it's a very fine game on the Mac mini,
to be honest.
But what we can do is
at the very bottom of these settings,
we've got this flash attention setting over here,
where if we engage both of these,
we can just try and drop down to Q8,
which technically is a bit of accuracy loss.
So again, just try and test this for your use case.
But this is a very easy way for us
to try and get a bit more context,
which should make this model a bit more usable
within Hermes agent.
As you can see now, I'm sliding up, sliding up.
Let's get to roughly around 12 gigs,
just around here.
Okay, maybe a little bit too high.
12.16, we have roughly 60, 67,000 token context window.
So let's see if we can actually load this up
into our Mac mini.
This is going to give us a bit more runway
for us to actually have a conversation
with our Hermes agent.
And let's just give this model a quick test.
Let's have a new conversation
and let's choose our loaded model.
Hi there.
Yep, just a little bit of thinking.
And then here's our response.
Beautiful.
12.66 tokens, nice.
And back in Claude,
it looks like we've set everything up.
Let me just scroll up a little bit,
check what's happened here.
Okay, so no telegram, no API keys.
And we also haven't downloaded our browser or Chromium.
So we can do the agent browser mode,
which we were just speaking about.
We can get to this in just a second.
Hey, I've just loaded up a Gemma model into my LM studio.
Can you please make that the default model
for our Hermes agent?
Nice, looks like it's picking up that we have LM studio
as a provider available in Hermes agent.
And we're using 1207-001-1234.
Let me just double check.
1234, perfect.
And we have this as running as well.
So now we can actually serve that model directly
into our Hermes agent.
And this is another cool thing that I want to speak about.
So right now Claude has set up Gemma inside Hermes agent
and Claude is running a prompt into Hermes
to make sure that Gemma actually responds,
basically doing an integration test for us.
And this is possible
because if we look at the Hermes agent CLI,
you can just send a prompt Hermes chat
and then you put your actual question or statement,
whatever it is into Hermes and you get a response back.
So we would not get a response back
if our Gemma model wasn't properly configured,
if it wasn't loaded, if LM studio wasn't installed
and then Claude can actually go back
and change the settings until it works.
So in this case, Gemma is now your default.
It was successfully set up.
Let's open up this local host URL.
So reply exactly what with this sentence.
So this is the test conversation
that looks like Claude just had.
And now if we go across to chat and let's see here,
let's send our message.
Hi there.
And we've sent it across to Gemma.
So let's see here and we're processing that prompt.
So beautiful.
Everything's connected up from our LM studio.
Let's come back into here and see when we get a response.
So it looks like we're still trying to process that response
just over two minutes in
and our LM studio is just finishing processing the prompt.
Now we're generating the response and there we go.
We have our response.
So that took roughly two and a half to three minutes
to actually process.
And we can see over here,
16.6,000 out of our total 67,000 tokens used.
So it's actually not as big as I thought it was.
I think Claude code is like 30,000 tokens,
which is better for us.
It just gives us some more runway.
But the issue is, since we're using a very small device
and the actual hardware on this is an M4,
it is going to be a little bit slower with these responses.
So in terms of an instant agent, we might not get that.
But in terms of like a 24 seven employee or assistant,
we definitely can get that.
Now, the final thing that I want to take us through is
I want to plug in our Hermes agent
into an existing tool that I use.
We were just speaking about customer support.
So I want to figure out how I can get Hermes agent
to respond to customer support tickets.
So I use Endless for customer support
and I just want to go across to my admin center
and I want to generate an API key and give it to Claude.
Pretty much any tool that you use,
if you use HubSpot, if you use Salesforce,
if you use Shopify,
whatever tool that you're using for your business,
you can definitely get an API key or API token
and you'll be able to give it to Claude
and Claude will be able to like set up webhooks,
set up the connection between both these systems.
So let me just generate a new API key.
Demo three, we'll save this.
We'll copy the API key and back in Claude.
Hey, I'm going to give you my Zendesk API key
and I want you to create a webhook
that triggers every time
there is a new customer support ticket.
I want you to send it across to Hermes agent
to be able to respond to that ticket.
For now, just use a generic data set.
Just assume that we are a computer parts replacement company.
Make up an FAQ.
This is just for a test case
and let's paste in the API key here.
Claude's always going to come back to us and say,
hey, you've just exposed your API key in this conversation.
You want to cycle that key.
You want to delete that key,
which you definitely want to do.
So after you finish your conversation,
definitely go back into your tool
and just delete your API key.
So we want to create a webhook from Zendesk,
which basically takes any new incoming email
or a new incoming ticket that we receive
and send it across to our Hermes agent.
And then from Hermes agent using the CLI,
as you know, we're able to receive different events
from different places.
And I mean, instead of me trying to figure it out,
like how do I kickstart a session with Hermes agent?
How do I upload the skills?
How do I give it the FAQs or whatever?
Claude's able to read all that information,
do its own due diligence, its own thinking,
and hopefully also do its own testing.
Just to make sure that process works end to end.
Awesome.
So it looks like Claude has finished
setting everything up for us.
Let's just go across to Zendesk
and we can see in the webhook section,
we didn't set up any webhook for ourselves,
but we have the Hermes agent auto-reply webhook.
Just to summarize the chat,
I had to answer a couple of different questions
basically around preferences
if I wanted it to be an internal note
or a public reply directly back to a ticket.
Since this is my demo Zendesk account,
I was just like, yeah, let's make it a fake FAQ sheet
and then public replies just to like demo this.
Let's also fire off a real ticket.
So I'm going to send this across to my Zendesk inbox
and back in Zendesk, we can see a couple of new tickets.
So two minutes ago, this is replacement PSU is dead.
This would have been generated by Claude
to test out that Hermes agent works.
So I think we just get a response in a couple of minutes.
And then we had our ticket, which is hi,
what are the open hours?
It looks like LM studio is currently trying to process
two different tickets.
And if we open up the activity monitor,
it looks like out of the 16 gigs,
we're using 15.08.
So it's quite a lot.
And if I just put my hand on top of the Mac mini,
I can see it's pretty warm on the top as well.
So it's definitely working.
And it looks like we're getting the responses generated now.
This is actually pretty quick.
I mean, this is two different threads
and you can see the tokens per second were pretty fast.
And here's a response to our open hours question.
Hi Bart, thanks for reaching out.
We're open Monday to Friday, 8 a.m to 8 p.m.
Thank you BitForge components customer support team.
All right guys, thank you very much for watching this video.
If there is one takeaway that you have after leaving,
I hope it is that you start using Cloud Code
to manage your devices.
From changing the settings on your Mac mini
to installing tail scale,
to installing your Hermes agent
and plugging into different tools
and building out these workflows for you.
So it's extremely powerful and it's a massive time-saver.
It is very reassuring to have Cloud Code
just do that process for you.
And if you watch the very end of this video,
please like, drop a comment.
And if you haven't already, please subscribe.
Thank you and I'll see you in the next one.