Kyle Corbitt
speaker
658 appearances
1 recordings
1 series
first heard Oct 2025
last heard 16 Oct
Kyle Corbitt’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 1 in all, peaking in Oct 2025 with 1.
Appearances
You also need to simulate the user.
And I mean you can do the naive thing and just say, oh, we're going to have a separate LLM that, you know, with a system prompt that is like the user simulator.
And we do that, but it's like, okay, but like the breadth of ways a user might respond, there's like a lot more diversity in that than the actual diversity you'll get in practice when you have this like simulated user.
And so then it's like, okay, well, is this environment close
close enough to how a real user would interact that like, you know, if if a user says something different that it's gonna know what to do.
And the answer in many cases is no.
If if you're just purely training on kind of like an LLM user simulator, it's gonna have its own idea of like what the correct way to answer is and the breadth of like a way a human might respond in this situation is is wider and and and your agent just may not be able to deal with that.
I think it's still very hard.
You know, like ideally all companies should have this anyway, because they're getting you know, if you're doing end-to-end testing, like theoretically, if you're following best practices, you would have one of those set up.
When we talk to enterprises almost universally, that's like not something that really exists.
So there are some startups, like there's some companies we talk to that do have it, and and we can just like use that, but it's it's a very, very small number that that actually have an environment like that.
Um I think it's hard to do it and and like there's
There's lots of like weird bugs that don't show up in an environment like that.
And and even if they do have a testing environment, they don't have it populated with like full realistic data, which is also like important so that the it it understands how to you know interact.
So I think in practice it's hard in both cases.
Maybe it's easier for the company, but at the same time, depending on you know the quality of the company's engineers, it might not be easy for them either.
Yeah, I suspect there is value in that.
You know, I think the you know, the the folks buying those environments and training on them in the big labs would have the best knowledge on how well they work.
I think they probably work okay.
I think they probably also are like you know, and we'll see m maybe with the next generation of models released, like how well they transfer.
Showing 261–280 of 658 · page 14 of 33
← Previous
Next →