Joel Hron

Everything you do has to go to a human evaluator every turn of the crank.

It really slows down the development pace.

So we try to like

kind of phase automated testing or automated evals with human evals so that we can take a lot of swings at bat in an automated way.

And then as we get closer and closer to production, bring in more and more human eval.

before we ultimately ship.

And I think if you can build really good kind of automated frameworks or automated evals at the early stages, it helps teams move more quickly in the early days, particularly.

I think the other thing that we try to prioritize is perhaps like keeping teams small at the beginning and then growing them sequentially over time.

So, you know, we've been working on deep research in earnest for probably the last six months.

And, and I would say that team started quite small, like, you know, less than seven people.

And, you know, probably every sprint grew by a couple, right, as we started to sort of like bring in more context and bring in more parts of the, the, the application ecosystem and stuff like this.

So we try to grow that over time, I think a little bit.

Um, I guess the other thing is, uh, from a technical perspective, like there's a lot of new frameworks that come out and have this like agentic scaffold and this thing is a real balance in like diving into one of those too early.

Like the space is really evolving quite quickly.

And, um,

You know, being too dependent on one of these frameworks in such a fast evolving space can sometimes be a handicap, even if it helps you get started a little faster.

It can.

They do.

They do.

And you get to this point of development where you're like, oh, man, this is actually like more of a handicap than it is an accelerant.