Andrew Lee
speaker
2,833 appearances
3 recordings
1 series
first heard Mar 2025
last heard 15 May
Andrew Lee’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsRecordings per month over the last 12 months — 2 in all, peaking in May 2026 with 1.
Appearances
There there is when do things get updated on our side.
Like when do we decide what that compressed history that we put into the LLM looks like?
And then what caching do we do on the LLM side?
Um and the answer to the former is every time you do anything, it's sort of incrementally updated, including in the middle of runs.
If you have like a very long turn that loses a lot of tokens, it might actually start compressing inside that uh that turn.
And the reason that we persist that.
Is actually calculating that could be really expensive.
Like running an LLM-based compaction of like an older section eats a lot of tokens.
And you don't want to have to do that every time the thing starts up.
If if you're, you know, every hour you have a trigger running, and every time you have to like compress a bunch of history, that could be very expensive.
So we keep all that around.
On the model side, caching depends on the provider.
So in the case of Anthropic, we're using five-in-caching.
caching and so it doesn't stick around very long.
Um and the assumption there basically is you're probably either in an active session or like in the middle of a turn, uh in which case five minute cache is enough, or you're probably waiting for the next trigger to run.
And like most people's triggers are not running like every you know half hour.
They're running every few hours or every day.
So the assumption there's this is not so common.
And then different providers have different uh different
possibilities there.
Showing 101–120 of 2,833 · page 6 of 142
← Previous
Next →