In this episode, a16z General Partner Martin Casado sits down with Sujay Jayakar, co-founder and Chief Scientist at Convex, to talk about his team’s latest work benchmarking AI agents on full-stack coding tasks. From designing Fullstack Bench to the quirks of agent behavior, the two dig into what’s actually hard about autonomous software development, and why robust evals—and guardrails like type safety—matter more than ever. They also get tactical: which models perform best for real-world app building? How should developers think about trajectory management and variance across runs? And what changes when you treat your toolchain like part of the prompt? Whether you're a hobbyist developer or building the next generation of AI-powered devtools, Sujay’s systems-level insights are not to be missed.Drawing from Sujay’s work developing the Fullstack-Bench, they cover:Why full-stack coding is still a frontier task for autonomous agentsHow type safety and other “guardrails” can significantly reduce variance and failureWhat makes a good eval—and why evals might matter more than clever promptsHow different models perform on real-world app-building tasks (and what to watch out for)Why your toolchain might be the most underrated part of the promptAnd what all of this means for devs—from hobbyists to infra teams building with AI in the loopLearn More:Introducing Fullstack-BenchFollow everyone on X:Sujay JayakarMartin Casado Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts. Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Ask this episode

“What did the council decide on item 42?” — answered with the speaker and the moment.

Transcript

Related