Learn all about Jev, a new System One model from TypeSafe AI, and how you can use it in your agent harness. Jev is up to 200x faster and 400x cheaper than LLMs on classification style tasks, which makes it great for applications like model routing and "jev as a judge" online evals.
Chapters: 0:00 Building a harness with a new model called Jev 0:12 The agent loop everyone already knows 1:02 Why tool calling and structured outputs exist 1:38 What Jev actually is 1:45 Jev as a System 1 model 2:46 Where the System 1 name comes from 3:27 Demo: Jev vs an LLM head to head 4:11 The three question types Jev answers 5:35 Using Jev inside LangChain 6:04 Use case: model routing 6:47 Use case: auto mode for risky tool calls 7:28 Use case: Jev as a judge for evals 8:47 How to get started with Jev
Sydney from LangChain introduces Jev, a \"System 1\" model from TypeSafe AI designed for fast, structured decision-making. Unlike traditional LLMs that generate text, Jev provides typed answers and probabilities, making it significantly faster and cheaper for classification tasks. The video explains Jev's capabilities, its integration with LangChain, and practical use cases.
Hey folks, my name is Sydney. I'm the product manager for the open source team at Linkgain.
And today I'm super excited to share with you how you can build a harness with Jev,
this new model that just came out from TypeSafe AI.
So before we dive into what Jev is, why it's so exciting,
I want to do a quick recap of kind of the core agent loop that we're all used to.
So a request comes in, we send that to an LLM, and it's able to take action by using tools.
So it will call a tool, get some results back, process those, and then it can continue in this
tool calling loop, or it can decide that it's done with a given task and return a result.
This loop is super powerful, but LLMs are a little bit unpredictable, right?
They get text in and then produce text out. And so when LLMs first came out,
we decided that we needed ways to impose a little bit more structure on them
so that they could work well in code-driven applications that inherently depend on
structured types, not just the raw text. So enter tool calling and structured outputs.
Again, tool calling, a model can send a structured request, in this case,
a get order request with a given ID, and then the tool returns a result with structured data.
The other primitive we added, again, is structured outputs.
The idea here is that you can bind an output type to your model,
and so it produces a final result that adheres to a given JSON schema rather than just raw text.
And so these two primitives made it a lot easier to
take advantage of this really powerful loop in code-based applications.
Okay, so now to the exciting piece, what is Jev?
There's so much buzz right now on Twitter about this new model.
Why is everyone talking about it? So Jev is what we call a system one model
from this new startup that just emerged called TypeSafe AI.
So according to their docs, system one models are a class of AI models built to make fast,
structured decisions that software can use directly.
A system one model evaluates a state and questions and returns typed answers and probabilities.
You can see an example of state and questions and then typed answers here on the slide.
The interesting thing here is that Jev actually doesn't take text in and then
generate text out like the traditional LLMs that we're used to.
Part of the reason why Jev is so popular right now is it can be 20 to 200 times faster
and 40 to 400 times cheaper on classification style tasks compared to LLMs.
So Jev is not a drop-in substitute for an LLM,
but it can do specialized tasks that we currently ask LLMs to do much faster and much cheaper.
In case you're curious where this system one phrase is coming from,
it actually comes from psychologist Daniel Kahneman's book, Thinking Fast and Slow.
And in this book, he described two ways that minds think.
Fast and cheap system one intuition versus slow and expensive system two reasoning.
And so TypeSafe borrowed this sentiment for its new model called Jev,
calling it a system one model that's fast and cheap, returning almost instant type decisions
instead of generating text, which is in deliberate contrast to standard LLMs,
which behave more like system two, slower and pricier,
but adhering to more of this step-by-step reasoning over open-ended problems.
All right, I'm going to show a quick demo here on
how an LLM answers a question versus how Jev answers.
So the question is, is there PII in this text?
And so an LLM produces this text result.
And then also structured output takes about five seconds.
Jev almost immediately, chance that there's PII is 98%.
And so you can see kind of from this demo for simple classification style tasks,
Jev is super useful.
And there are a lot of classification or kind of decision style tasks
that we currently ask agents to do with LLMs that could be more optimal in terms of speed and cost.
All right, so let's talk a little bit more about the types of questions that Jev can help answer.
There's three types of questions and we'll look at them through the lens of this example.
So the state says, hi, I've been trying to connect my Stripe account for three days
and it keeps failing, I'm losing sales, please help ASAP.
So question type one is a choice, multiple choice, pick one option.
So for example, which team should handle this?
The score shows 0.84 for billing with a competence of 0.596.
The second question type is a score.
So somewhere on a scale, you know, how frustrated the customer appears
from calm to frustrated to very angry.
And the score here is 1.035.
So customer seems frustrated, but not very angry.
And then the third question type is a null.
So this is a Boolean type answer, a yes or no.
So the message conveys urgency or time sensitivity returns 0.999.
This is on a scale of zero to one.
So certainly a high urgency probability for this case.
One other thing to note, you can actually send a single state
with many questions that Jev answers in parallel.
So Jev is actually very efficient with many questions at a time.
This is again, in contrast to OLMs,
which do more sequential reasoning rather than parallel processing.
All right, so how can you use Jev in LangChain?
We just released a LangChain TypeSafe integration.
So from LangChain TypeSafe, you can import this TypeSafe classifier.
First, you'll need to acquire a TypeSafe API key.
And then you can use this classifier to invoke Jev with a state and questions.
And then the response has answers to each question that you send it.
Let's walk through just a couple of use cases that are super great for Jev.
The first is model routing.
So if you have an agent that does both simple and complex work,
you probably don't want to use the most powerful model
for everything that the agent does.
An example of this that we've been exploring at LangChain recently
is that we actually want our internal coding agents
to switch between fast and cheap models
and more expensive and powerful models,
depending on the complexity of a given question or coding task.
And so Jev is great for this model routing step.
You can assess a given prompt against given criteria
to help decide whether or not you should use a fast or more powerful model.
And Jev decides almost instantaneously.
Another interesting case is what we call auto mode.
So from LangChain, we already offer this as a pre-built auto mode middleware.
And the idea here is that you can ask Jev to analyze
whether or not given tool calls are risky,
and then you can block those at runtime.
Anecdotally, I actually turned off auto mode in my coding agent recently
because the classification step of whether a given tool call was risky
was too slow for my coding agent to feel productive.
But it's back on now that Jev can make these decisions so quickly.
So a quick example here is like if a tool call was issued
that would delete a database or delete important files,
Jev would certainly classify that as risky
and then block that tool call.
All right, the third interesting case here,
maybe the one I'm most excited about,
is using Jev as a judge for online evals.
So the idea with this is that you often want
some sort of model to help you identify
whether or not your agent has done a good job with a given task.
But when you're running evals at scale,
it's not super reasonable to have human oversight
over every single trace,
but you often need more than just kind of code style evaluators
to decide whether or not your agent is doing a good job.
And Jev is really great at this.
So you have your input for a given eval,
so the initial question and then the answer from the agent.
And then you also have a rubric that you provide,
so grading criteria like,
is this answer correct?
Does it match the reference?
Is it grounded?
Was there a cited source?
And then Jev can score the given answer
along these different rubric criteria.
This is kind of an evolution of LLM as a judge online evals,
which are currently certainly very useful,
but also very expensive.
We just wrote a blog on how to use Jev as a judge,
and it has some really interesting results
about Jev being, surprise, surprise,
much cheaper and much faster,
but also interestingly,
much more reliable and consistent across evals
than the LLM alternative.
All right, so I'll leave you with this.
It's time to go build something cool with Jev.
You can get started with uvpip install LangChain TypeSafe.
You can read our accompanying blog post at the link here.
Easily get an API key from typesafe.ai.
Tag us at LangChainAI on X.
We'd love to see what you're working on.
Thanks, folks.