20 Oct
2008
20 Oct
'08
5:12 a.m.
On 21/10/2008 11:02 a.m., Brad Pearpoint wrote:
If you wanted to, you could simply ask them to obey your robots.txt.
There is a perfectly functional mechanism in existence which can be used to ask folks with automated crawlers to leave some parts of your site alone. It's called a robots.txt file. What mechanism is in place to ask people to check if they really, really, absolutely, yes even natlib, truely should check the robots.txt? A robots.robots.txt file? I assume they don't actually want 150,000 emails from webmasters detailing what to crawl and what to leave... Some thought has been put into these things by Clever People (TM) - why does the National Library believe they can ignore that body of thought? Gerard