Duane Forrester
speaker
918 appearances
5 recordings
1 series
first heard Sep 2025
last heard Sep 2025
Duane Forrester’s voice in public audio — every appearance, attributed to the second.
Trend
recordings per month · last 12 monthsNo recordings in the last 12 months.Older appearances are listed below; set an alert to hear about the next one.
Appearances
Um but the fact of the matter is um things like the LLM.txt are those are those are made up.
Um conceptually I get it.
Somebody looked at it and went, oh robot TXT is for the old crawlers, we need one for the new crawlers.
No, you don't, because the new crawlers are still just crawlers.
and they follow robot.txt and point and fact, CC bot, the Creative Commons bot, is actually responsible for a whole bunch of the data that makes its way into these systems because all of the systems access the data from Creative Commons bot.
So
If it's following robot TXT, you don't need anything else.
That's it.
Maybe in the future, but right now, I'm gonna toss that in the trash.
Totally, totally.
And you know what, on that point, I think this is really important as well.
So for your sitemaps and your robot TXT, spend the time to get to learn how the actual syntax should look, like what you should be putting into those files, especially robot TXT.
You know, I I'll skip the old warning about how easy it is to block everything on your website and like screw yourself because you know you put the slash in there in the wrong spot, right?
But understand what control in there looks like.
The default for every robot is do crawl.
That is the default for a bot.
It's its only reason for existence, is to crawl a URL.
So there is no do crawl directive.
There is a do not call directive where you tell it what not to go into.
So go through and understand all of that properly because I see robot txt files daily.
Showing 41–60 of 918 · page 3 of 46
← Previous
Next →