Poetiq is a new startup founded by former DeepMind researchers that recently achieved a major jump on the ARC-AGI benchmark by layering a recursive self-improvement system on top of existing models. In this episode of Lightcone, Poetiq's Founder & CEO Ian Fischer joined us to discuss how small teams can build “reasoning harnesses” that outperform base models, what that means for startups and why automating prompt engineering may be one of the most powerful levers in AI today.
Chapters: 00:00 – Intro 00:40 – What Is Poetiq? 01:07 – Recursive Self-Improvement Explained 02:07 – The Fine-Tuning Trap 02:59 – “Stilts” for LLMs 03:14 – Recursive Self-Improvement vs. Fine-Tuning 05:05 – Taking the Top Spot on ARC-AGI 06:37 – Beating Claude on Humanity’s Last Exam 08:40 – How the Meta-System Works 10:26 – Beyond RL: A New S-Curve 11:32 – Automating Prompt Engineering 13:37 – From 5% to 95% Performance 14:50 – Early Access & Putting Your Agent on Stilts 16:17 – From YC Founder to DeepMind Researcher 18:29 – Advice for Engineers in the AI Era
The world is changing so quickly. This is probably a little bit obvious, but you should just
try things and like every day, do something with AI. Last summer, I took a weekend and
used GPT-5 to help me build an iPhone app. I hadn't done that in a decade.
So fast.
Yeah, it's so fast and so easy. And that was, you know, an age ago,
that was like eight months ago. Now it's even faster and easier. Don't limit yourself. Like,
anything that you imagine, you should just try to use AI and see how far you can get with it.
And you'll be making the world better.
Welcome to another episode of The Light Cone. Ian Fisher is the co-founder and co-CEO
of Poetic, which is building recursively self-improving AI reasoning harnesses for LLMs.
Previously, he spent a decade as a researcher at Google DeepMind
and founded a mobile dev tools company through YC years ago. Welcome, Ian.
Thank you. I'm so happy to be here.
What is Poetic? How's it different than RL? You know, how's it different than context engineering?
At Poetic, what we're building is a recursively self-improving system. And so,
recursive self-improvement is this, you know, kind of the holy grail of AI,
where the AI is making itself smarter. The core insight that we had is that we could do
recursive self-improvement far faster and cheaper than all of the other ways that people had
been proposing to do this. And so, obviously, I can't go into details about what that is,
what our particular approach is. But most of the approaches out there involve, you know,
they require you to train a new LLM from scratch. And training LLMs from scratch costs, you know,
hundreds of millions of dollars and takes months of effort.
And then Anthropic or OpenAI will come along and just eat your lunch in the next model release.
Right, right. And, you know, of course, Anthropic and OpenAI and Google,
they're exploring recursive self-improvement, but typically at that level of having the,
you know, having to train a new model for every step of self-improvement that they do.
I mean, that seems like actually the, like, defining thing that a startup really, really wants.
Like, I know that I want to take advantage of whatever the next model is, but the second you're
in fine-tuning land, I'm spending, you know, millions to hundreds of millions of dollars.
And then guess what? Like, I just lit it on fire because,
you know, the next version of the Frontier model comes out and I'll never catch up.
Whereas, like, working with your systems means that I will always have the thing that is
better than the thing that's out of box. And that's sort of like the holy grail.
Yeah, we think that this is incredibly valuable to anybody who's building on top of
large language models. And we don't view the, you know, the Frontier models as competitors.
They're the ones that were using the stilts, you know, building stilts to stand on top of.
But if we didn't have that foundational layer, then, you know, Poetic couldn't exist.
Yeah. I mean, being the smartest model, you know, it's a game of inches,
actually. And like, so those inches matter a lot.
Right, right.
How do we actually get started? I mean, you've built something that
basically any startup could use that it's sort of like stilts, really.
We have built a system that can automatically generate systems for your particular problem
that will always outperform the underlying language models. And without kind of the
massive expense, as you're saying about the bitter lesson, where,
you know, what would you have done without Poetic? You probably would have said, okay,
we're going to first collect a large data set, you know, like tens of thousands of examples for
a particular problem that we're working on. And we're going to fine tune, you know,
the best model we can get our hands on. Maybe that's, you know, one of the frontier models,
or maybe it's an open weights model, doesn't particularly matter. You're going to spend a
lot of money on that fine tuning, the compute is so expensive. And then at the end of it,
you have something that, you know, works better than the thing that you fine tuned on top of.
But by then, a new model has come out. And it's better than the thing that you fine tuned,
you know, you fine tuned, you know, like, three years ago on top of GPT 3.5, or whatever.
And then GPT 4 comes out, and it just blows you out of the water. And so are you going to do that
again? Or are you going to go out of business? And like, in some cases, the latter. With Poetic,
what we end up giving you is a, you know, people are calling these things harnesses now,
but you know, or a gentic system, or whatever you want to call it that sits on top of one or
more language models, and it just performs better than them. And when the new model comes out,
that same harness is perfectly compatible with it. And you don't need to change anything to get the,
you know, an even bigger performance bump. Additionally, we can, you know, continue to
optimize for this new model, whatever the new model is that you want to use, and you know,
make it even better. But you you don't lose out on the hundreds of millions of dollars. In fact,
we do this so much more cheaply than fine tuning would cost as well.
And you've done this actually a bunch of times, right? Like, I remember when you first came out
with your paper in December of last year, you shot to the top of ArcAGI V2. And then you've
done this a bunch of times for other benchmarks, too. What, you know, what was that like?
ArcAGI V2 was, this was kind of us coming out of stealth, letting people know that we could
tackle these really hard problems. And in particular, you know, we wanted to show that
our system could generate these, what we call, you know, we call our system like the poetic
metasystem, can generate reasoning systems that are highly effective. Gemini 3, DeepThink had
just come out, and they were, you know, really quite dramatically at the top of the leaderboard
at 45%. And two days later, we released our results where we were showing that we could get
a lot higher than that.
So they come out with Soda, and then you come in right above them every single time.
Yeah, yeah.
Like, wild to see, honestly. That's what it's like to have stilts, you know? Like,
whatever model comes out, you can be taller than that one with Poetic, which is like, that's so
awesome.
Yeah, so the interesting thing is that we were half the cost of Gemini 3 DeepThink,
because we were building on top of Gemini 3 Pro, which is a much cheaper model. But we still got,
in the end, a nine percentage point improvement on the official verification. So they were at 45%,
and we were at, like, 70-something dollars, and we were at 54%, and $32 per problem.
So recently, you guys just announced some incredible results for Humanity's last exam.
Can you tell us more about those?
Humanity's last exam is a set of 2500 really, really hard questions written by experts in
many different domains. They're meant to be challenging even for PhDs in those fields.
AI hasn't passed it yet, but we got to 55%, which is almost two percentage points higher than the
previous state-of-the-art, which came out just last week from Anthropic with Claude Opus 4.6.
They got 53.1%, and we got 55% on it.
And one thing that Humanity's last exam doesn't publish is the cost of getting those results.
In your case, this run was done with less than around six figures. How much was it?
We didn't publish any cost for this, but I can say that the optimization costs us less than $100k.
Which is impressive, because each of these big foundation model
trained runs are in the hundreds of millions of dollars.
And you guys, as a company, you're only seven people?
That's right, yeah. Seven research scientists and research engineers.
That's impressive. And I think the thing that's very interesting about your approach
is sort of taking a very scientific approach to the emergent behaviors that a lot of the
best founders are doing with models. I think a lot of founders that get very good results
for agents, they treat the underlying model as a common layer that you can switch in between.
There's a certain task, for example, for GPT-5.2, like very hard-to-verify bugs,
gets sent to that, versus architecture that gets sent to CLOT-4.6.
But you're kind of doing this automatically instead of having human conducting.
It's very impressive. I think there's something more special going on underneath.
Can you tell us a bit about how it works?
Yeah, it sounds magical. What can you tell us?
So you're getting at a really core thing. These harnesses, they are code, prompts,
data, built on top of one or more language models. And so this is something that, in principle,
you can build by hand or with CLOT code or whatever. But in practice,
it takes a lot of work to do these, to have all the insights to make these work well.
And so the core technology that we've developed at Poetic is recursive self-improvement. So we
have a recursively self-improving system, which we call the Poetic Metasystem.
The output of that system is systems that solve hard problems, where a hard problem is something
that if you gave it to GPT-5.2, it would struggle to give you a reliable, robust result, just to use
an example. So this is a very big advantage for us. We can generate these systems in a much more
automated manner, which means that we can do it much more quickly and much more cheaply than if
you hired a team yourself to try to make your own agent to solve your particular task.
But not only that, since this is really an automated optimization process,
if you already have done that work, you're a startup that's going after a particular vertical
and you've put together, you know, you think you understand your problem pretty well, you've put
together your agent and you, you know, maybe it's working pretty well, but you know you can get
something better or you really need something better, then you can bring that to us and we can
optimize that entire agent or pieces of that agent. We could optimize just the prompts, just the
reasoning strategies. There's a lot of different things that we can do depending on your particular
needs. It sounds like this is a complete different paradigm than RL, because we went through the S
curve of regular pre-training, RL with when OpenAI released 01, and now this feels like a new one.
It sounds special. It sounds, it rhymes a lot with the RNNs, which is a whole different paradigm than
than RL, right? It's going to depend on the particular task, the particular type of problem
that we're going after, that we're trying to solve, and the underlying models that we're working with.
But effectively, you could say like each model or each set of models that we're working with will
have their own S curve. The poetic system, the poetic metasystem itself is also going to have
its own S curve. And so as the poetic metasystem gets better and as the underlying models get
better, you'll find that the, you know, the S curve that you're dealing with keeps shifting
higher and higher until ultimately either you saturate or like... Reach AGI? Yeah, reach AGI,
reach super intelligences. Given its stilts, you might like hit the ceiling first then.
That's the goal, right? You want to hit the ceiling first with poetic.
I think a lot of startups that we work with, and then in my spare time, I do a bunch of
context engineering. And then the thing is we're sort of like tuning it, tuning evals, tuning like
we're context stuffing ourselves. What does that even feel like to have a recursively self-improving
version of like prompt engineering, context engineering? We don't spend a lot of time
looking at the particular data that we're working with. Instead, we're letting the poetic metasystem
look at that data. And so like the metasystem, you know, if it thinks that it needs to put more
things into context, do more context stuffing or whatever, it'll do that. If it needs to like
generate a bunch of examples to get better performance, it'll do that for you, right?
It was pretty interesting to look at the prompt outputs in particular, I'd say for ArcAGI,
in that, you know, I think you can read those and say, well, that's not what a human would have
written pretty clearly. And it's, you know, there's some unexpected stuff and, you know,
it made some really simple examples. And one of the examples is actually wrong,
but we didn't change it. We're like, well, this is, you know, this is the thing that it
outputs. We'll just leave it be. You know, we don't want to go in and monkey around with things.
And so historically in machine learning, you always, you know, it's like the rule was you
have to know your data set really well. But now we're kind of outsourcing that to the AI itself,
where it's the AI's job to understand the data set and figure out where are the failure modes
and where are the kind of robust reasoning strategies that the agent could use to get
better performance. The output is much better prompts. And then how much of it is like the
harness itself, context stuffing or summarizing in the right way or re-ranking in the right way
so that like you have some number of like mega LLM calls and how do you get the most out of
each of those calls? Yeah. And so that definitely varies per problem, but what we've seen,
in fact, our last paper at DeepMind was not doing this recursive self-improving stuff, but we were
showing that you could build these harnesses manually to solve really hard problems.
And what we saw there is that, you know, we manually optimize the prompts really hard for
these very hard problems. And that got us a little bit of the way in this particular case,
you know, the hardest task we were working on, we got like to 5% performance with Gemini 1.5
Flash. This was a while ago. And then when we added on the reasoning strategies, we went from
5% to 95%. And so this is typically what we see, you know, like everybody's out there kind of doing
some amount, I wouldn't say everybody, but many people are out there kind of doing some amount of
prompt optimization. You know, JEPA is this very popular paper, everybody's kind of re-implementing
that. That will get you some performance improvements, but it's very far from
everything that you can get if you actually think about these reasoning strategies that are really
going to be written in code rather than in just better prompts. So if startups want to use Poetic
to put their agent on stilts, what should they do? Yeah, so right now we haven't released anything
yet. But if you go to Poetic.ai, there is a button you can click to sign up for early access. And
if you're a startup or a company who has a really hard problem, and you've tried everything that you
can to make it reliable and robust, and you just can't get all the way there, you need something
more, then let us know. We're looking for problems like that. So just tell us what it is that you're
and we'll reach out. You'll be the first to know when we're ready to work with you.
I mean, if you're at the top of Humanities' last exam, then I mean, that's pretty big. So
you're already all the way out there at SOTA, and then I guess the stilts basically let any
agentic company become SOTA. That's the idea, yeah. And we view the ArcAGI results and the
Humanities' last exam results as showing kind of two different capabilities that we have. We can
really improve your reasoning, and we can really improve deep knowledge extraction from these
models. And then you're just totally vaccinated against the bitter lesson. Exactly. YC's next
batch is now taking applications. Got a startup in you? Apply at ycombinator.com slash apply.
It's never too early, and filling out the app will level up your idea. Okay, back to the video.
Slight change of topic, but something I was curious about. So you arrived at Google over
a decade ago when they acquired your first YC startup, Aportable. Aportable was porting mobile
apps cross-platform, right? Android or whatever. It's quite different to recursive self-improving
AGI. How did you make that leap? What happened once you got to Google? What made you think that
you maybe wanted to shift down and do something different? And just would love to hear that story.
The acquisition was this amazing opportunity to reflect on what I really wanted to be doing next.
Google itself is a place where you can do so many different things.
So I spent some time thinking about where I wanted to go next in my journey.
I realized that the problems that I was most excited about were really actually AI and robotics.
And the best people in the world, many of them in those fields, were at Google at the time.
And so I went and talked to them. They let me come join a new AI robotics team
in Google Research, which was this amazing opportunity for me since that wasn't my
background. My background was like computer security and then this cross-platform mobile,
you know, systems building stuff. I was able to join this team. And I'll tell you the truth that
I very quickly realized that hardware is hard and I didn't really want to be doing robotics.
It was more aspirational at that moment. But I was really passionate about machine learning.
So I just made a very hard switch into just doing machine learning research
and did that for about a decade at Google. And then Google and then DeepMind.
What's maybe some advice that you have today for engineers who want to get into
more of the AI side, probably the applied AI and build startups around AI?
How should they think about that?
The world is changing so quickly. This is probably a little bit obvious,
but you should just try things and like every day, do something with AI.
Always try to push yourself to find the boundaries of what they're capable of
and build the things that you want to build. Even for me, last summer, I took a weekend and
used GPT-5 to help me build an iPhone app. I hadn't done that in a decade.
Yeah, it's so fast and so easy. That was an age ago. That was like eight months ago.
Now it's even faster and easier. Don't limit yourself. Anything that you imagine,
you should just try to use AI and see how far you can get with it.
And you'll be making the world better.
That's all we have time for today. But Ian, thank you so much for giving us all stilts.
We can't wait to use it at YC. I can't wait to use it for Gary's list.
I mean, there's just so much to do.
Thank you for having me. This was a lot of fun.