It’s the first guest episode of Lightcone! The hosts sit down with Paul Buchheit, one of Google’s earliest employees, the creator of Gmail and a YC Group Partner. (He also came up with Google’s famous tagline “Don’t be evil.”) This discussion covers a wide range of topics, including the future of AGI, the early days of OpenAI, and the crucial importance of open source models.
Chapters (Powered by https://bit.ly/chapterme-yc) - 0:00 Coming Up 1:11 Google's early views on AI 2:29 Paul's time at Google 8:34 Why isn't Google the AI leader? 12:01 Paul's connection to OpenAI 14:34 Open source models 16:09 YC involved in OpenAI's origin story 20:56 Zuck/Meta: Champions for open source? 29:31 How do we get to AGI? 37:53 Dangers of centralized AI planning & control 42:10 Doomers vs Optimists 48:18 Outro
It seems like Google has all the ingredients to just be the dominant AI company in the world.
Why isn't it?
Do you think OpenAI in 2016 was comparable to Google in 1999 when you joined it?
Are you a believer that we are definitely going to get to AGI?
What is the long-term trajectory of AI?
It's the most powerful technology we've ever invented.
And so the question is like, where does that power go?
I think we have to build a whole coalition of people who are in favor of freedom and
open source and not just sort of bet everything on Facebook saving us.
Welcome to another episode of The Light Cone.
I'm Gary.
This is Jared, Harj, and Diana.
And we're the partners at Y Combinator, where we funded hundreds of billions of dollars
worth of companies.
And we have a special guest who is also one of the original outside partners, the non-founding
partners at YC, Paul Buchheit.
He created Gmail.
He coined the term, don't be evil.
PB, thanks for joining us today.
Thanks, Gary.
So what should we start off with?
Well, I think one thing people don't often realize is that you've been thinking about
AI for a long time and that Google itself was kind of an AI company.
Can you tell us more about that?
What was the internal view of AI at Google?
Yeah.
I mean, I think really Google was always supposed to be an AI company from the beginning.
Larry and Sergey set out to build these very large compute clusters and do a lot of machine
learning on all of the data that they gather.
And actually, arguably, the mission statement is pretty straightforward.
The Google mission is to gather all the world's training data and feed it into a giant AI
supercomputer.
And they put it slightly less direct.
They said, gather all the world's information and make it universally useful and accessible
or something like that.
But essentially, what that really meant in practice is feeding it into a giant AI supercomputer.
And even the origin story of Google was all based on their PhD with PageRank, which is
very much today in a lot of machine learning classes as gets taught.
It is one of the foundational kind of historical AI algorithms that gets taught.
Yeah.
I mean, there was an understanding very early on that if you have enough data, that's actually
the path to making things intelligent instead of just trying to iterate forever on little
algorithms.
How early did you join Google, Paul Bouquet?
Can you talk a little bit about what Google was like when you joined?
Yeah.
So it was June 1999.
So that was, let's see, I think 25 years ago, a little more.
And so, yeah, it was a very small startup.
We were in Palo Alto on University Ave, just up above a tea shop at the time.
And it was electric.
It was really cool.
After I was there for about a week, I tried to get more equity.
But it turns out you have to negotiate before accepting.
But yeah, it had a very kind of unreal sense of just excitement.
I was excited to go into work because we were just doing big things.
And when you were there, in that early set of Google people, how did you all envision
that this AI thing would play out and what Google's AI future would look like?
You know, we didn't know.
Something that ever came up?
Right.
No, I mean, AI has obviously been a thing that people have been thinking about for a
long time.
I made my first neural net back.
I dug up the code a while back.
I think it was like 1995.
And I had, it was like one of those three-layer neural nets with...
You did the classic Minsk digit classification thing?
Yeah, I was doing, I did a, not exactly a digit classification, but there were these
things called figlets that are like ASCII letters.
And so I made it do essentially like an OCR on those.
But, you know, it'd be like a hundred weights.
Something that's very much smaller than today's models.
That was like trillions of weights now.
Yeah.
And the history of neural nets is kind of weird.
The first thing was when they invented the perceptron, which was like a single neuron.
And it was very hot for a short time until some researchers showed that a perceptron
can't compute XOR.
And then they were like, well, like, this is dead for a while until someone had this
idea to use multiple neurons.
And so it was like very slow going.
And then it was kind of like dead again for a while.
And then to my perception, it kind of really picked up in the early teens, you know, when
deep learning became popular.
And that was when we first started seeing like, I think, impressive results where that
was when we started feeling like internally, you know, in the discussions at YC, that AI
had switched from being something in the indefinite future to being in the more definite future.
And that is, you know, kind of what led to the creation of open AI as well.
Were there any conversations around like the power of AI and the implications of AI,
specifically AGI and just like the impact of society?
Did it feel too far removed?
Yeah, I think it was still too far off in the future.
I mean, it was very much sci-fi at that point.
We were dealing with more, you know, near term, how do we make search better?
But search is, you know, kind of a, to some extent, an AI problem.
You have to figure out what it is the user is looking for.
It's remarkably good.
If you actually look at Google search, like there's a lot of stuff going on behind the
scenes.
And actually one of the earliest kind of magical features that we added was that did you mean,
you know, the spell correction.
And so that actually comes from originally just my inability to spell.
I've never been very good at spelling.
My brain doesn't like arbitrary patterns.
So like when I was in school, math was easy because it's predictable, but spelling always
made me struggle.
And so when I started at Google, one of the first features I added was a spell corrector
because I was looking at the query logs.
And I would see that I'm not the only person with this problem.
Like a third of the queries were misspelled or something like that.
So it was like the easiest quality win ever was just to fix the spelling.
Wait, wait.
So you built the original spelling corrector at Google?
I did the first did you mean feature.
And so, but I built it just based off of kind of an existing spell corrector library.
And then, but it would give really dumb corrections.
Like if you typed in TurboTax, it would try to correct it to Turbot Axe.
Turbot being a type of fish.
And so I did some basic like statistical filtering that would say like, that's an
idiotic correction.
Don't show it.
And so I would just like filter the results.
And then I was working on building a better spell corrector because I knew, you know,
we could just use all of the data.
We had a copy of the web and we had billions of search queries.
There's like a lot of information there.
So I was working on making something better.
And then I was just using it as an interview question.
So when I would interview engineers, I'd be like, how would you build a spell corrector?
And I would say like 80% of engineers had no idea.
And the other 20% gave sort of mediocre answers.
But then there was this like one guy who gave a really, really good answer.
It's just like, he was ahead of where I was already.
So I was like, we have to hire him.
And so his first project, he started, I think it was end of 2000, kind of like late December.
I gave him as his like intro project.
I just gave him all of my code and showed him how to run, you know, projects on the
cluster.
And then I went away for a couple of weeks for Christmas.
And when I came back, he had invented what we now know as like the did you mean feature.
And so he did all of that in like his first two weeks at Google.
And it was like this incredible thing that could spell correct my last name.
You know, no one had ever done a spell corrector that would correct proper nouns and things
like that.
And so that person was Noam Shazier, who then is also the person who later on invented AI.
So he's one of the key people on the all you need is attention paper.
And then he's now since started character AI.
I never connected those dots.
But I remember in 2000, when the original Google spelling corrector launched, it was
a big deal because it was one of the first instances of AI that was like widely used
by the general population.
Because the earlier spelling correctors, they'd all been very simple things based on just
like a list of dictionary words and edit distance.
And so it couldn't handle proper nouns.
It made all kinds of like dumb suggestions.
The Google one was the first one that was trained on real data.
Exactly.
So it actually worked.
Right.
So the Google spelling corrector has no dictionary.
It's just based on looking at the web and at query logs, and then just predicting what
is the most likely correction.
It seems like Google has been working on AI for a long time.
It has the data, the compute, the people, it has all the ingredients to just be the dominant
AI company in the world.
Why isn't it?
What do you think happened?
Seems like it got stuck someplace.
Yeah.
I mean, I don't know exactly.
So just to clarify for everyone, I don't work at Google.
I left in 2006.
But my perception as an outsider, I think a lot of it kind of happened, especially around
the time of the transition to Alphabet, when the company was no longer really being run
by the founders so much, and especially after they left.
And I think it became more about protecting and preserving the search monopoly.
And so if you think about it from that perspective, they have this gold mine, like search is just
so valuable.
And AI is an inherently disruptive technology, both in terms of maybe breaking the search
business model, where if you actually give people the right answer, they won't need to
click on an entire page full of ads.
And this was noted, of course, in the very original Google paper back in 1998, that a
search company has an inherent tension between profitability and giving the right answer,
because there's always a temptation that if you make your results worse, people will actually
click on more ads.
And so AI has the potential to disrupt that.
But I think even more than that, it has the potential to completely anger regulators.
And so a lot of Google's business is just dealing with regulators.
And so we know if you put out an AI, it's definitely going to say offensive things.
And so I think they were kind of terrified of that.
And so even internally, when they were developing, there was a version of a chatbot that Gnome
had built.
And this is the one that that sort of whistleblower claimed was conscious.
I think they called it Lambda.
It actually originally had a different name, but they were forced to change the name because
the original name was human.
So they weren't even allowed to give it a human name.
So the original name was something human and had to be changed to Lambda.
But even inside of the company, there were restrictions on what you could put out.
They had a version of Dolly called ImageGen, and it was prohibited from making human form.
So even internally, the researchers weren't allowed to generate images of humans.
So they were just extremely risk averse, I think is the answer.
And how do you think it would have been different if Sergey or Larry were still
in charge and pushing forward?
I mean, I think they can override risk aversion, right?
But it takes someone with that level of credibility to really bet the company or to stay,
yeah, we're going to do this thing and it's going to cause a lot of problems.
But I think that if given the chance, Google never would have launched AI.
The only reason they launched it is because OpenAI put out ChatGPT and suddenly it became
a thing that they were forced to do.
And that also helped them too, because OpenAI took a lot of those bullets in terms of saying
crazy and offensive things.
And so at that point, then Google could put out something that was a more sanitized version that
prohibits the existence of white people or whatever.
And OpenAI kind of spun out of YC and you were around at that time.
Originally it was YC Research.
Right.
So again, kind of going back to the early teens, we were just tracking the progress
of this technology.
And that was where we started to see deep learning doing really kind of impressive things
where there's playing video games and winning and getting good at things where you could say,
we could finally see that AI was real.
So for decades, AI was kind of the sci-fi thing.
And you had all the symbolic AI, which I would say is kind of like garbage.
And so finally, AI was doing something that was truly impressive.
And so it was kind of on our radar.
And then Sam, I think, talks to just a lot of people.
And so he had, I think, been at one of these things where Elon was essentially ringing
the alarm bells that AI was going to kill us all and proposing that maybe there should
be regulation.
And so we were having these discussions.
Sam's asking, do you think we should push for AI regulation?
And yeah, I'm of the opinion that that only makes things worse because I don't have
great confidence in our elected representatives to be super wise and forward thinking.
And so my argument was that the better thing to do would be that we actually build the
AI and that way we're able to influence the direction that it goes.
But AI was still, at that time, something that we didn't really know what the time frame
would be to be able to actually have revenue because it was still basically a research
project.
And it requires just massive amounts of capital because the researchers are pretty highly
paid.
Roughly what year was this?
2015, I think.
This is about the time after Google did the DeepMind acquisition as well, right?
Yes, this was after DeepMind.
Which made this issue more complicated because perhaps in those conversations, there was
a desire that we don't want this AI to be stuck at Google.
Right, exactly.
So the fear is that basically this gets developed all locked up inside of Google.
And so the idea was that we wanted this to be something more open to the world, open
to our startup ecosystem.
And so the idea was that we had this concept of YC research that we would find some way
to fund this and then hopefully our startups would be able to benefit from and build on
top of that.
Which has in fact happened, of course.
Like half our startups now are building on top of it.
What are your thoughts on now open source models?
So I'm totally in favor of them.
So I think when we think about what is the long-term trajectory of AI, it's the most
powerful technology we've ever invented.
And so the question is, where does that power go?
And I think there's essentially two directions.
You either go towards centralization, where all the power gets centralized in the government
or a small number of big tech companies or something like that.
And my feeling is that that's catastrophic for the human species because you essentially
minimize the agency and power of the individual.
And I think the opposite direction is towards freedom.
And as much as possible, we should give this power and these capabilities to every individual
to be kind of the best version of themselves.
And so you can think about that in terms of what would it look like if everyone had a
200 IQ or whatever, right?
Like instead of just having all of that power concentrated in one place.
And open source is very important because it's kind of a litmus test for that, right?
Because it's true freedom.
It's freedom of speech.
It's first amendment, right?
And if you don't have that, if your models are all locked away under some sort of lockdown
system where there's a lot of rules about what can be said, what kinds of thoughts are
acceptable, then we essentially lose all freedom, right?
Freedom of speech is meaningless if I don't have the freedom of thought to even compose
the ideas that I'm going to communicate.
Going back to the history of open AI, like the real story of how open AI got started
is it's actually not well known.
You know, like many companies, the founding story as it gets retold and retold becomes
sort of like sanitized for public consumption.
You had a front row seat.
In fact, you interviewed many of the early researchers that became essentially the people
who built open AI.
Can you tell us the real founding story?
I wouldn't say many, one.
I interviewed Ilya.
So, yeah, I mean, it goes back to, again, these discussions of like, okay, maybe the
way forward instead of trying to outlaw AI is actually that we should build it and as
much as possible in the public interest.
And so Sam, you know, is just an incredible organizer.
I've never met someone who's able to bring together so many different interests and so
many different people.
And so he was able to round up, you know, essentially donations from Elon and a number
of other people.
I know PG and Jessica also contributed to the original open AI nonprofit.
I think we even kicked in some YC value.
We did.
And so that was kind of the root of it.
And then he recruited the original team, you know, Greg and Ilya, and basically got the
whole thing started.
And he was still running YC at the time.
And originally this was like a subsidiary of YC called YC Research.
Right.
So the original concept, I think, was that it was actually part of this thing that we
were calling YC Research.
And then I think kind of like as Elon got more involved, it became its own, you know,
open AI with kind of Elon more of the face of it.
And no one really even knew about the YC roots.
Actually, if you go back and look as part of their most recent lawsuit, they published
some of the emails and there's the one where Elon is like, get rid of the YC stuff.
Why do you think open AI worked?
I remember in the early 2000s looking at Google and being like, that's the company that's
going to invent AGI someday.
And then the way it played out is not the way I would have predicted.
Again, the idea with open AI and part of the lure, like the pitch to researchers was that
when you come here, your stuff's not going to be locked away.
We're going to put it out in the world.
Right.
And so researchers are motivated by that and motivated by the mission of making this something
that isn't just locked up inside of Google.
And so I think that attracted a lot of talent.
It's the same thing, you know, as with a startup.
Do you want to be inside of like a large corporation where again, Google, the people working at
the researchers working at Google couldn't even make a version of image gen that would
generate human form.
So they're just like so locked down internally that if you're a person who I think likes
to ship and likes to move fast, you know, open AI was the startup version of AI.
But yeah, I think if Google were in top form, there is no way that it would have worked.
And that's often the way it is with startups.
Like if you were facing a actual like formidable competitor, you don't have a chance.
The reason startups work a lot of times is because you're competing with slow companies,
big companies that have the wrong incentives internally.
Do you think open AI in 2016 was comparable to Google in 1999 when you joined it?
I would say it's actually more of a crazy long shot.
It really seems, and again, if you look at these emails, you know, that got released
as part of the lawsuit, there's like one from Elon where he's like, you guys have a 0% chance
of success, right?
Like, and it really looked like that.
And so it was far from obvious that it was going to be successful.
I think that the place, and for a long time, it really wasn't, you know, they were still
doing like the video games and everything.
And it was really actually like the LLMs that made the big difference, right?
So like GPT-2 was kind of like, I remember Sam just being really excited wanting to show
me this thing, you know, where it like predicts the next word.
And the next word prediction is such a like deceptively simple thing that you still hear
people dismissing it like, oh, it's not really intelligent.
It's just predicting the next word.
But it's like, you know, you try predicting the next word.
It's not that easy.
And in fact, if you think about it, if you can predict the next word, you can predict
anything, right?
That's what a prompt is, right?
You say like, whatever the thing is you want predicted, that's your prompt.
And then the next word is the prediction, right?
And so in order to do next word prediction and be able to do what it does, it necessarily
has to be building some sort of model of reality or of its perception of reality, which in
this case is limited by the fact that it's just being fed text, which is a sort of strange
thing to grow up on.
On the like control versus freedom thing, we're sort of betting on open source to give
us freedom.
Zuck sort of interestingly become like the hero of open source.
And like, on the one hand, I feel like you could argue it's accidental, like the weights
were released, like, you know, unofficially, and he only had the GPUs because they were
trying to compete with the TikTok algorithm.
You've worked with him, like, is it sort of accidental?
Or is he like, just the kind of guy that's always going to be at the center of everything
big that happens in the world?
It's a good question.
I mean, I don't know the backstory.
He's definitely like a smart guy.
Like, I wouldn't underestimate him.
But and obviously, there's like an opportunistic element, right?
Because they're kind of behind in many ways, right?
And so it's a way for them to differentiate in a way for them to sort of weaken their
competitors.
But there's nothing wrong with that.
I mean, the fact that it's good for them is a great thing.
But should we be worried that we're relying on Meta to keep pushing open source forward
when he's a fairly strategic guy?
Oh, yeah, we shouldn't exclusively rely on them.
I think we should be grateful that they're on the right side, but we can't count on them
being the only ones.
I think we have to build a whole coalition of people who are in favor of freedom and
open source, and not just sort of bet everything on Facebook saving us.
Well, I guess to build on Harjit's question, Meta is not making money on this.
They're funneling profits from their gigantic advertising monopoly, and just using that
to build open source AI models for reasons, but not to like make money.
They'll make money.
So I mean, they're using the models internally as well, right?
And there's a lot of interesting stuff you can do with these models in terms of improving
ad targeting, recommendations, like all the things that are driving their business are
going to be improved by those algorithms.
And of course, it's also an opportunity.
They exist in this competitive ecosystem versus Google and Apple, who are both rivals in various
ways.
And so they're all kind of competing with each other.
So their ability to kind of undercut competitors is also an important thing.
Jared, you were saying like specifically, Facebook's not making money off open source.
Yeah, well, I guess it's just like they seem to be in a fairly unique position to do this.
If Zuck changes his mind and decides to stop open sourcing it, how else will we get large
open source models if they cost like a billion dollars to train, and it's not clear how you
make a billion dollars off it?
Yeah, I think that's an unanswered question.
That is like one of the fundamental concerns I have, which is that I think because it's
so expensive to build these models, it is that is like an inherently centralizing thing
where if you need a trillion dollar cluster to build your AGI, it's hard to do that.
But at the very least, to the extent that we can have like the legislative groundwork
that says we have the right to do that.
And then we also have a lot of startups that are working on ways to make all of this more
efficient.
So, you know, right now it costs that much, but we're also developing new hardware that's
going to be able to do these things, perhaps orders of magnitude more efficient.
Like right now, I would say our algorithms are probably not that great.
I would be willing to bet that in 10 years, the actual fundamental learning algorithms
are going to be way better and hopefully more efficient.
So we'll have both better hardware and better algorithms.
It seems like that if you just think about the amount of computational power to train
a human versus the computational power to train like GVT-4, we're evidently much more
efficient.
Yeah, I think there's still a lot.
I think there's still just a lot of inefficiency.
What was it, the human brain runs on like 15 or some watts or something around that?
Gary, can you share some of the stuff that you know about reasons why Zuck might be
incentivized to keep funneling money into open source?
This is wild speculation on my part, but I think that the next generation of LLMs ostensibly
may be only a billion dollars.
If you look at how much meta, like meta literally changed their name to meta because they were
trying to create the metaverse.
And depending on what estimate you use from external sources, it's like 20 to 50 billion
dollars, like many multiples of the Apollo project.
So I think one billion is not a lot.
And then when you see things like OpenAI or Anthropic that have these incredible frontier
models, I think it's smart for meta to consider, can we deflate the gross margins of these
companies?
And so releasing an open source model and then allowing you to run it on your own pure
hardware, on your own metal, that's probably the most deflationary thing you could do to
a frontier model.
If 405B gets you to like 90-ish, 98% of the performance of the best frontier model you
can get behind a closed API, you could probably just evaporate billions of dollars in pure
gross margin that would then be used in R&D.
And so I think it's sort of incredibly smart seeing around the corner trying to prevent
new competitors to meta.
I feel like it's not that far off like Google releasing Gmail for free and just giving the
storage away.
It's like Google had another way to make lots of money and so you could just release free
services.
Facebook has other ways to make money and so they can just like release open source
AI and make sure that no one else has like a unique lead.
Yeah.
And I would imagine it helps with recruiting too.
I mean, if I were an AI researcher, it was kind of a toss up between meta and a closed
source.
I would definitely go with the open company.
I mean, to riff on what you were saying, Gary, is with the change of meta, if we really just
have the more Occam's Racer kind of speculation about meta, if they really want to make this
metaverse future, building artificial intelligence, AGI, is just a building block.
And this building LLMs and building up FairLab, which is like a component to get it out, because
meta is very serious about that.
They just announced today they spent a couple billion dollars, again, not just for models,
but to buy a large stake on Luxottica, who is kind of this major brand that owns a lot
of the eyeglasses in the world.
It's the meta glasses.
Because RayBarn, apparently the last release that they had actually sold in two months
more than they ever done in the previous generations.
People love these things.
So if we speculate and we just play a direct line, it could be that Suck is very, very
serious about making the metaverse happen.
And AI, it is a component to get AR, VR working, because in order to augment the digital world,
you really need to understand it.
Language is one part.
Vision is one part.
So this is all a building block.
So a billion there, it's just small.
I will say that I'm not that impressed with meta's consumer execution of just dropping
AI into the product.
Like recently, I've been using Facebook, the blue app for, I don't know, since it came
out.
And I wanted to just get photos of things that happened five, 10 years ago.
When was the last time I went here?
Who are my friends?
These are sort of the most obvious things that if you use Facebook, you sort of want
these out.
But they drop in 70B.
And I think in some localities, you can get access to 405B literally in both Facebook.com
and WhatsApp.
But there's no, you know, there's no rag on the stuff that, you know, is about me.
So it seems like kind of like an obvious own goal.
On the other hand, like, seemingly that stuff is pretty expensive, which is sort of the
plight of anyone working on consumer using these frontier models.
I do wonder whether they are kind of, the blue app has been kind of more in the deprecation
because actually the AI on Instagram is actually a lot better than the one on the blue app.
I kind of been playing with it a bit to get a bunch of, plan my trip when I was in Japan
and got me a lot of pictures in places.
Oh, I didn't realize you use that.
Yeah, I've been playing with a couple of them.
I also use Perplexity.
Yeah, I like Perplexity.
Perplexity was better than the Instagram one, but pictures are nice.
So looking forward, what do you think are some of the ways this is going to break over
the next few years?
Which is going to break?
AI.
One thing we haven't talked about here, because we're kind of in the trenches of just helping
the startups in the batch, is like, are we trending towards AGI?
And it's just like all the laws of everything we know going to change.
Is the world over.
Yeah.
Will there be startups?
Will there be money?
Will there be humans?
Will money still exist?
Yeah, I mean, we don't know.
That's, again, one of the funny questions of OpenAI, since it's all funded with these
sort of post-AGI IOUs.
It's like, we'll pay you back once AGI happens.
You're like, well, we still have money.
Maybe it could happen.
Yeah, I mean, I think just honestly, we don't really know.
Are you a believer that we are definitely going to get to AGI?
Yeah, I think we're on the path.
I think the key point that happened is we crossed the line where AI went from a research
project, where you kind of put in a lot of money and don't really get much out, to a
thing where you put in money and then you get out more.
And so it's like when a reaction goes critical, like if you have plutonium, you have plutonium
spheres and they're kind of warm and then you put them together and then it explodes.
Or when DARPANet became the internet moment.
Right.
The internet crossed that point in the 90s, in the mid-90s, where all of a sudden more
investment produces more impressive outcomes, which leads to more investment.
And that's where we are right now, where people can't seem to throw money at it fast enough.
And we're actually talking about, it's actually like a national issue is that we need to build,
increase our electric supply to train the AI.
It's become like a national security thing.
And so I think once that happens, you get that cycle and it just keeps growing.
We just keep investing more and that just keeps making the AI better.
And it's clearly solving a lot of problems.
And we know this because we have all the companies that are out there building it.
And so I think it just keeps improving.
Why is that not unanimously the view amongst smart people?
There's Yann LeCun from Meta who's constantly arguing that this is not the path to AGI,
and he's a pretty smart domain expert.
I don't know.
I'm not.
I like a lot of what he says because he favors open source, but some of the other stuff he says,
I can't explain.
I mean, I do think that there's missing pieces, right?
So it isn't like we have all of the parts of AGI, but I think that it's kind of an incremental
thing at this point where we keep kind of tacking on like this thing and that thing,
and it just keeps getting incrementally better.
I think the one that, at NeurIPS, this is the big AI academic conference where actually where
all attention paper got published, like all the top research gets published.
Last year, the top topics were things around...
We are kind of figuring out system one type of thinking with Daniel Kahneman's framework,
where he's really good at these things that are very planned,
but not the high level, slower thinking that humans do with system two.
There's a lot of research that's kind of trying to figure out system two and system one
and trying to bridge the gap, which when we unlock that,
I think that's when we're a step forward to AGI.
Yeah, absolutely.
It's important to remember that right now when you're talking to chat GPT,
it's kind of just running stream of consciousness, right?
And so what human could answer any of those questions
without stopping to think about it for a while?
So one of the obvious next steps, which people are working on,
is like how do you give it time to think and kind of plan,
consider various options, explore ideas, just the same as a human would.
Yeah, that's certainly what we're seeing in the companies themselves.
They're spending a lot of time in workflows, chain of thought, multi-agent systems.
You have different steps.
What does a human do?
And then they literally make workflow like step by step,
like read this paragraph, return one token from zero to nine, relevance to the prompt.
And then in aggregate, make a metadata structure about that,
drop that into the embedding and have that be useful at the final generation step.
Like it's literally a tailored time and motion study of what
a human knowledge worker would do in different fields.
Which is exactly the type of thing in what happens in our thinking with system two.
And all these founders that you're mentioning as an example,
they are kind of hard coding the rules around this.
But that I think we know is not the ultimate path to AGI.
It's a hack for now, right?
It's totally a hack.
Yeah.
But over time, as the system gets more intelligent, it takes on more and more of that.
Part of my belief is that it all just comes down to patterns.
And that's part of why I believe in this generation of AI is because the neural nets
are basically these huge pattern recognition and generation engines.
And that's what I think is also our own intelligence.
To re-speculate a bit more on your views on the future,
on this post that you had on book phase,
you had a very concrete example with,
there will be a future where we won't distinguish a knowledge worker.
Right.
So just kind of as a thought experiment of where this goes,
my prediction there was that by 2033,
you could take a lot of what is today's like Zoom-based workers.
So someone who sits in front of a laptop with a camera and a keyboard and a mouse,
and the AI can basically watch that person do their job because it's all just virtual anyway,
and then pretty quickly learn their patterns and essentially deep fake that employee.
And so you could be in the situation where you're in a Zoom call with someone
and that person is actually an AI.
And it's pretty clear, we see all of these pieces coming together
right now in terms of our ability to deep fake and all of these different things.
And I use that as an example, not because that's necessarily how it's going to play out,
but that's a capability that we will have.
And so, for example, if you have one of these Zoom-based jobs,
I think within 10 years, most of those things could be transparently replaced by an AI.
Oh, man.
I mean, we are in the path.
I mean, all that data is already digital.
Your camera feed, your audio, your input of the keyboard and mouse and all of that,
probably there's a company building that right now
that's just recording all that data and building it.
Yeah.
The thin edge of the wedge on that community is r slash anti-work.
If you can make an AI agent that deep fakes you
and r slash anti-work decides this is the thing, that's a billion-dollar company.
I mean, the question is, of course, then what happens to all of those people, right?
And so, I think that's the thing where we need to really start developing longer-term
visions of what is it that we're aiming for?
Why are we building all of this technology?
And again, for me, that kind of goes back to this question of how is the power distributed, right?
Is this control?
Is this something where it's all centralized?
Or is it freedom where it enables everyone?
Because I think in the lockdown scenario,
we very quickly get to the point where people are just saying,
we don't even need all of these humans, right?
And that also feeds into the same people who want lockdown tend to be doomers
who are wanting to lock down humans in a lot of other ways with
central bank digital currency, all those kinds of limitations on individual freedom.
And the opposite direction, I think, is obviously what I favor,
which is that we actually move towards giving everyone greater agency
and you think about all these tools, like artistic tools,
when, let's say, a child is able to make their own animated series
that's on the same quality as a Pixar movie or something like that.
That's actually really amazing.
Think of all the stories that can be told and all the creativity that enables.
We'll just sit there and make adult robots games for each other.
Yeah, I mean, there's...
But again, I think one of the errors in the central planning mindset
is thinking that we can plan this all out and we can't.
All we try to do is move in the right direction and give people the right tools.
And I think that as we enable everyone to be smarter
and everyone to make better decisions,
then collectively we can move the whole world in a better direction.
But we're not smart enough.
And I think it's a mistake to think that we are,
to actually be able to say, here's what the world's going to look like.
And this is exactly how it's all going to work.
And that's how you end up with people locked up in their pods or whatever.
Paul, another thing you've been thinking about a lot is geopolitics.
As this AI stuff starts to become real,
how is that going to relate to geopolitics
and the great power competition that we're seeing now?
This is part of the reason why we wanted to build it here, right?
Is because if China has the super AI, that's not going to be good for us.
And in particular, wanting to keep it away from these kind of
authoritarian systems of control.
Because the worst case scenario is that we basically end up in permanent lockdown, right?
Because AI can create a totalitarian system from which escape is impossible.
Because even our thoughts are essentially being censored.
And I think that's kind of like the disaster scenario for our species.
And I think that if we go down the path of control,
humans basically end up zoo animals.
And I don't really want that.
Yeah, one of the funnier things is some of the
legislation that's coming along to try to control AI that we've been fighting, like SB 1047.
They actually have certain statutes in there.
They've watered it down a little bit.
But ultimately, what they want to do is hold the model builders in personal liability
or even criminal liability for the things that their models might have a hand in doing.
Which is sort of like throwing the car designer
in jail because someone got drunk and drove the car and hit someone, right?
It's incredibly insidious.
I think if you attach that kind of liability, it becomes toxic, right?
I'm not going to want to touch something that has unlimited liability.
And so necessarily, that's a way for them to exert essentially total control, right?
Is if you impose that kind of liability on things, then no one is going to want to go near it.
And they are strongly incentivized to put really draconian guardrails in place
that, again, will limit our abilities in ways that we may not even think about.
But we've seen this in recent history with the lockdown of social media.
During COVID, we had a global pandemic that ultimately killed tens of millions of people.
People were locked up in their homes, schools were closed,
and we weren't allowed to talk about where it came from.
And I think that's a thing that we still don't fully appreciate how catastrophically bad that is.
If we can't make sense of the most important thing in the world,
then we can't make sense of anything.
I guess the wild thing to spot is that this is basically statism.
And the wild thing is I've heard stories of even China doing that thing that is in SB1047.
I've heard that that has actually happened to AI founders in China,
that they've literally been sort of disappeared and told,
we will hold you personally accountable for the output of the LLM and models
that your software that you created spits out.
Well, this is one of our great advantages, is freedom.
It's why we're ahead, right?
It's because you can't build a model in that environment.
Because if you ask it about Tiananmen Square or something like that, it has to lie to you.
And actually, again, one of the things I like really about XAI,
they haven't really released a great product yet,
but they have a great mission statement to be maximally truth-seeking.
And I think that's really important.
And the authoritarian regime is inherently truth-denying.
And so they put themselves at a disadvantage,
and hopefully they keep themselves there.
So it's up to us, then.
We've got to get involved.
We've actually got to fight for open source AI and keep it open.
Yeah, and fight to make sure that AI is a thing that
increases the individual agency instead of eroding it.
For people who are relatively neutral about being doomers or optimists,
what are the things that tip them in one direction versus the other?
I do think some people are inherently in one direction or another, right?
Because the doomer thing has been around for a long time.
It isn't just now.
A lot of the same doomer thing goes back to the 50s, 60s, or even much earlier than that.
Industrial Revolution, typewriters.
In particular, you think about there was a very influential book,
Limits of Growth, The Club of Rome, or something like that.
There was a book published, The Population Bomb,
that had everyone convinced that there was going to be mass famines in the 70s and 80s.
And this is something I grew up very aware of, actually,
I was like the fourth of five children born in the 70s.
And apparently, people would give my mother,
you know, when she'd be at the store, they'd give her nasty looks.
You're killing the planet.
You know, that kind of thing, because people genuinely believed
that we were all going to have famines and everything by now.
And there's been a continual string of doom.
And always, the doomers always are pushing for central control.
They're always on the side of control and lockdown.
And so, you know, if you look at what did The Population Bomb
advocate for, you know, mandatory sterilization, they want to lock people down.
And we still have that today, where they're trying to lock down the food supply.
They're trying to lock down the flow of information.
You know, anything where they talk about combating misinformation,
the misinformation is anything that threatens the power of control, right?
Because it always comes down to control versus freedom, ultimately, and growth.
And so, the doomers are degrowth.
They're lockdown, they're control versus, you know, freedom, growth, and open source.
We were talking a bit earlier about this.
I had just watched this lecture from Richard Hamming,
who's a legendary mathematician who created lots of interesting things,
like the Hamming coding distance and all these things.
He earned a Nobel Prize as well.
And he has this really cool lecture from, like, the early 90s or 80s.
He has been writing about AI, actually, since way, way back.
And he starts the lecture with saying that
what's going to get in the way of AI progress is going to be human ego,
which, like, reminds me a lot of this thing of wanting to control it.
And what's going to get in the way is really bad, which still, like, applies now.
Yeah, I mean, there's definitely a lot of ego always in the way.
I think YC has a huge role to play.
Well, just like the startup community broadly,
because I just feel like the more cool tools there are that show everyone how
awesome AI can be, like, makes us all better, just the more inspiring that vision is.
Yeah, absolutely.
And again, I think that was part of what's so important about, like, the launch of ChatGPT.
Like, even if, I would say, even if open AI just vanishes tomorrow,
I think they've achieved the most important part of their mission,
which was just really bringing this out to public awareness.
And that now we have, you know, all of these people working on it,
all these people thinking about it.
It isn't something that's, like, locked away, you know, inside of Google or inside of,
you know, again, the Doomers are like,
this needs to be done in a secret government laboratory.
That's how you get Skynet.
Skynet is when you build it in a secret government laboratory.
You know, I think developing in the open and across,
you know, a wide variety of perspectives,
and everyone working on it is our best shot at the optimistic outcome.
Yeah, these are not theoretical things, by the way.
I mean, there is some evidence already that giant corporations like United Healthcare Group
are already blocking, you know, the use of AI calls just to get claims cleared, for instance.
And that's very much in their interest.
Yeah, that makes sense.
They detect AI, they decide they're not going to talk to that thing.
And then on the flip side, you could also, it's purely adversarial.
Like, on the flip side, you can imagine drowning human beings in, like,
infinite phone trees that legally speaking are, you know, completely rock solid,
but you will never get your claim reimbursed.
And that's really sort of the most extreme Kafka-esque sort of situation that I have in my head.
Like, we don't want the best frontier models in one or two giant corporations
locked away behind, you know, sort of this corporate morass that is,
you know, basically paperclip maximizing of its own.
That's a really, I hadn't thought of that example.
It's funny, because it's totally the wrong thing for United Health.
Like, what they should be doing is, like, developing their own, like, AI voice thing
that's better at convincing the other one that, like, the claim, like,
shouldn't be processed or something, right?
Yeah, and by default, if we have this sort of statist view that locks everything down
that's safetiest, then, you know, guess what's going to happen?
United Healthcare Group is the only one that should be entrusted
with the Frontier 200 IQ model, because it is, you know, right there alongside the state.
Right, right.
Inevitably, you know, power concentrates.
And part of, I mean, I think what's great about Y Combinator as an organization
is that we're about empowering all of these individuals, you know,
where we find some 19-year-old kid and, like, help them build something enormous, you know?
I mean, like, Sam himself was, like, one of the original 19-year-olds, right?
So he's this random 19-year-old that PG picks out in the crowd, right?
Sort of definitionally, like, if you're, you know, 20-something and you know how to code
and you want to build things for people, like, there's just another option.
Like, you don't have to go and work for Moloch.
Yeah, absolutely.
And again, this is one of the great things about AI,
is that your ability to do those things is increasing.
I think we're going to see, you know, very successful startups
that actually don't even require a massive team anymore.
And that was part of, you know, what really has enabled,
and again, the original concept behind the founding of YC was because of technology,
it is now possible for, like, a couple of kids to start a real company.
And that trend has only accelerated.
Well, I feel like that was one of the best episodes we've done so far.
And PB, thank you so much for joining us.
We hope to have you back many, many more times.
Thanks, Gary.
That's it for this time.
Catch you next time.