Local LLMs, Explained Like You're 5

heymleo
348 followers
4 Video Views·Nov 19, 2026
In March 2023, Meta's LLaMA model leaked through a link on 4chan, and a week later one programmer, Georgi Gerganov, had it running on an ordinary MacBook. Follow how llama.cpp squeezed every number down to four bits, how Alpaca, Vicuna, Llama 2, Mistral, Ollama and DeepSeek-R1 put AI on everyone's computer, and why most people still use AI in a data center. Plus a rule of thumb for what fits on your laptop: at four bits, about half a gigabyte per billion numbers.
Chapters
00:00 The leaked file
00:31 The AI you couldn't download
02:04 Hacked in an evening
02:42 Smaller numbers
03:32 Everyone's computer
04:45 Free for everyone
06:10 Why we still pay the cloud
07:51 Know which computer is thinking
AI Generated·May contain errors
March 2023, the AI that Meta had locked behind an application form,
leaks, and the download link, it turns up on 4chan,
almost 7 billion numbers, 13 gigabytes,
made for graphics cards, most people don't own.
A week later, one programmer had it running on an ordinary MacBook,
in one evening, no data center, no subscription, no internet.
So how do you squeeze a brain that big into a laptop?
February 24th,
2023.
Meta announces its own large language model.
It's called Llama.
Meta says even its 13 billion version beats GPT-3 on most tests,
a model more than 10 times its size.
But you can't just download it.
To get it, you fill out a form, researchers only, approved one by one,
no commercial use, because what Meta was guarding wasn't an app.
It was the model itself, a giant file of numbers called weights.
Each number is a tiny dial,
tuned while the model trained on 1.4 trillion pieces of text.
Put all the dials together and you have everything the AI knows.
Whoever has the file can run the AI.
With ChatGPT, that file never leaves the company's data center.
You only ever see the chat window.
With Llama, the file itself would go out,
but only to people Meta approved.
That lasted one week.
On March 3rd, someone uploaded the weights as a torrent
and posted the link on 4chan.
The same day, someone even asked to add
that download link to Meta's own official code page.
Meta sent takedown requests and the pages came down,
but a torrent has no off switch.
Every copy keeps sharing itself.
Still, almost nobody could actually run it.
Even the smallest version needed about 13 gigabytes of memory.
And a graphics card most people don't own.
For now, the leaked AI was just a very big file.
Enter Georgi Gerganov, a programmer
who liked making big AI run on small computers.
Months earlier, he'd written a tiny library
for doing AI math on ordinary machines.
No giant software, no special hardware.
He'd already used it to run Whisper,
OpenAI's speech recognition model on everyday laptops.
So when Llama leaked, he had one goal.
Get it running on his MacBook.
The problem? Size. 13 gigabytes.
And all of it has to sit in memory while the AI works.
His trick? Make every number smaller.
Each weight was stored with 16 bits.
Think of a number with lots of decimal places.
Georgi squeezed each one down to four bits,
like writing 3.14159 as just 3.1.
You lose a little detail, but the AI still mostly works.
Four bits instead of 16.
Four times smaller.
13 gigabytes became about four.
On March 10th, he posted the code.
He called it llama.cpp, and the first lines of its readme said this.
This was hacked in an evening.
I have no idea if it works correctly.
It worked.
Words started appearing, one by one,
on an ordinary laptop, no data center far away.
The AI was inside the machine.
The next day someone ran it on a Raspberry Pi,
a computer the size of a credit card,
about 10 seconds for every little piece of text.
But it ran. Two days later, a Pixel phone.
Impossible to a weekend project.
Ten days after the leak,
a team at Stanford took the 7 billion model and taught it to
50,000 examples. Total cost, under $600.
They called it Alpaca.
A week later, they pulled the public demo,
the hosting bills, and the answers it kept making up.
Then a group of university researchers fed
the model 70,000 chats that people had shared from ChatGPT.
Their version, Vicuna, cost about $300.
Every week, someone posted another one.
All built on the file Meta had tried to keep behind a form.
And in May, someone leaked a document out of Google.
An internal memo.
Its title.
We have no moat.
And neither does OpenAI.
The writer's point was blunt.
The barrier to entry, the memo said,
had dropped to one person, an evening and a beefy laptop.
Georgie's evening had become a strategy problem
for the biggest companies in the world.
And then, the twist.
In July 2023, Meta changed its mind.
Llama 2, free to download.
For anyone, even most businesses.
That September, a small French startup called Mistral
released its first model.
No press event, just one post online
with nothing but a torrent link.
The leak had become the launch.
Meanwhile, apps like Ollama
and LM Studio wrapped Georgie's work in a friendly window.
Pick a model, click download, start chatting.
In March, it took a programmer and an evening.
Now, it took two clicks.
A model about five gigabytes in size now fits on an ordinary laptop.
A year on, Meta went all-in.
In July 2024, it released its biggest model yet,
and Mark Zuckerberg published a letter titled
Open Source AI is the Path Forward,
calling it the first frontier-level open-source AI model.
And in January 2025, a Chinese lab called DeepSeek
released R1, open weights, MIT license,
plus six smaller versions distilled from it.
Within days, people were running the small ones on laptops
through the same tools.
On a train, in a tunnel, the Wi-Fi dies and the words keep coming.
Nothing leaves the laptop.
The notes, the questions, they stay right there.
So why does almost everyone still use AI
in a data center?
Start with time. A model only knows what
was written before it was finished.
Ask it about last night's game, and offline, it can't look anything up.
At best, it says so. At worst, it makes something up.
Then there's size.
Say you try one of the biggest models Meta ever released,
Llama 3.1, with 405 billion numbers.
Even squeezed to four bits,
your laptop would need about 200 gigabytes of memory
just to load it.
It has 16.
So you write yourself a rule of thumb.
At four bits, every billion numbers needs
about half a gigabyte plus room for the conversation.
An eight billion model, about five gigabytes?
Fine, 70 billion, about 40.
Not on most laptops.
And local doesn't automatically mean private.
Ollama now also offers cloud models.
Pick one, and your question goes to Ollama's data center instead.
Same window, different computer.
And a local server can be left wide open.
Change one setting carelessly, and anyone on the internet can use it.
In February 2026, one scan counted more than 12,000 of them.
No password.
One more thing.
Local isn't free.
The download is.
The computer and the electricity aren't.
So which should you use? Ask three questions.
Does it have to work without the internet,
or stay on your machine? Local.
Does it need today's news? Or the biggest model there is,
the cloud. And will it fit? Do the half-gigabyte math.
Most people end up using both. In March 2023,
a leaked file and one programmer's evening
project moved AI out of the data center
and onto your laptop. It's real. It's useful.
It has limits. Know which computer is thinking.
Please like the video, subscribe, and turn on notifications.
