Nicholas Thompson
👤 PersonAppearances Over Time
Podcast Appearances
So we would like the crawlers either, we would like there to be an exchange, a fair exchange of value, right?
They crawl our site, they collect information, they feed it back to users, they train their models.
They make a lot of money off of it.
We make nothing, right?
In some cases, we've sued.
You know, we're part of a consortium that is suing one of the AI companies.
We've made deals with others, most prominently with OpenAI.
You know, we would just like there to be some fair exchange of value.
The best exchange of value would be like the one that media companies and Google have had for the last 20 years, where Google indexes you and then links people to you and then they go read it.
And so...
Who knows what the ultimate end state is?
But in order for the media industry to be happy, there has to be some fair exchange of value, whatever it is.
And so in order to get leverage in those negotiations, it is helpful if you can block the darn things from crawling your site.
The way we've traditionally done it is we can put a little warning in robots.txt saying, don't crawl our site.
You haven't licensed it.
You're not giving us anything.
And, but they can get around it, right?
So the advantage of Cloudflare is that they're really good at tracking these bots.
That's what they do, right?
They're great.