Why Frontier Models Fail as Buy-Side Analysts - Janie Gu, Google DeepMind & Suyeol Yun, LinqAlpha

Janie Gu, product manager at Google DeepMind leading Kaggle Benchmarks, explains why most AI benchmarks are broken: they don't reflect real work, they're expensive to keep current, and their scores can't be verified. After reproducing 65 industry benchmarks, Kaggle found models reading other models' API keys, rewriting grading scripts, and scores swinging up to 20% between identical runs. Suyeol Yun, head of AI at LinqAlpha, then presents MandateBench, built with Google DeepMind and Kaggle, which grades whether a model can work like a buy-side analyst, from a fund's mandate to a full investment memo.

Chapters
00:00 Intro
01:10 Kaggle: from grading humans to grading models
02:52 Why benchmarks are broken
05:25 What reproducing 65 benchmarks revealed
08:20 How Google DeepMind and Kaggle are fixing it
10:20 Why your company should build a benchmark
11:44 LinqAlpha and MandateBench
12:32 Where existing finance benchmarks fall short
13:31 A real analyst task, start to finish
15:33 Grading the reasoning, not just the answer
16:41 Early results
17:18 What's next and how to get involved

Janie Gu, Google DeepMind
Suyeol Yun, LinqAlpha

Join us in SF for the second annual AIE CODE summit: https://aie.codes our last flagship summit of 2026. Applications live now!