Podcast Appearances
the best model i mean of course frontier models are easy but then the best inference for all the open source models all that became really popular um that business is growing a ton i think a couple months ago we announced that hit 50 million uh run rate uh within like five or six months wow um and the margins there can be pretty good because open source models you can host at a decent margin that is growing like crazy uh we didn't really expect that but that's that's a big part of it
The other side of it is extremely boring.
If you are a company that's using open code and you have a thousand engineers, you can't just tell them all to go download open code and like add an open API key.
You need some kind of control plane to like set up all the providers, permissions, budget controls, rate limits.
So we have a product there.
we're gonna make that publicly available soon but right now it's just been like enterprise deployed uh so just if you're a company that's using open code at scale you need some administrative software to run it you can't practically use open code at scale without something like that that's also open source but you know most people just pay for our hosted version uh the other thing is
I think the time has finally come where people are looking at how much they're spending on LLM.
And they're like, what are we doing?
Are we actually getting anything anymore done?
So companies are now looking at their costs and trying to figure out how to optimize it a little bit.
It's great timing because open source models are now very competitive.
They are 10x cheaper.
Blending that in and having good inference for open source models is becoming a part of our business as well.
So these big companies, you know, they need the control plane, but then we kind of just give them inference access as well to these other models.
And they end up just kind of naturally starting to use it.
If that ends up being a main part of our business, we might stop charging for the control plane itself and just charge for the inference.
Yeah, so I think this is a, it's kind of, there are different parts of the business.
So if you look at the pure inference part of a business, if you think about what's the floor on the cost, the floor is a cost of electricity.
There's a capital cost to acquire the hardware.
Once you have it to deliver a token, the cheapest it can get is the electricity to power it.