Simon Willison

If there are existing tests that you can point to that, it'll get so much easier.

Well, I don't know.

With the device driver, it either works or it doesn't, right?

Oh, no.

If you can reduce the problem to a thing where the coding agent itself can tell if it got it right, it's easy.

Oh, yeah.

Right.

The thing today with LLMs, automated tests, no log optional,

um continuous integration no longer optional good documentation that's actually up to date with code no longer optional those things like in the past we've been able to excuse oh we don't have a good test suite yet because we didn't have time that doesn't work anymore you've got time now that run claude code overnight and you'll wake up to a test suite and it'll be a bit but it's better than zero yeah right yeah god it is it's just amazing this new world we live in

So here's a fun one.

I think it's now easier than ever to introduce a new protocol into the world if you ship a conformance suite.

Like release a conformance suite and boom, overnight you'll have libraries in half a dozen languages because the conformance suite is the majority of the work.

It overcomes the problem that it's not in the training data.

And people are kind of nervous that you could never launch a new programming language now because it's not in the training data.

But the context lengths are big enough now that if you can get it into a test suite and fit the examples on how to use it in 10,000 tokens, it doesn't matter that it's not in the training data.

Oh, yeah.

It still is.

It's $10.

You get to go in a self-driving car.

It's the best.