Nicholas Thompson
👤 PersonAppearances Over Time
Podcast Appearances
And so they've built this handy little dashboard and I can go in there and I can say, okay, Perplexity crawled your site this many times.
Google crawled your site this many times.
Apple bot crawled it.
Claude bot crawled it.
Weirdly, like the fourth highest crawlers from Huawei hadn't even heard of their bot.
And so what we can do is we can just go in and say, no, block this one, block this one, block this one.
And then we can go back
Apple, Perplexity, whoever, and say, hey, we've blocked you.
And in fact, we've blocked you with Cloudflare.
So if you want to crawl our site, you have to give us something.
You have to give us a link.
You have to prioritize.
You have to do something.
Pay some exchange of value, whether it's traffic, whether it's money.
And
What Matthew has done and what Cloudflare has done is given the media companies that were just getting tramped, right?
The AI companies came in and they were like, you know what?
Life's hard.
Lawyers are expensive.
Internet was meant to be free and also screw these guys.