19 Oct
2008
19 Oct
'08
9:54 a.m.
On 20/10/2008, at 4:52 PM, Criggie wrote:
Nathan Ward wrote:
But, Google etc. walk my content fairly regularly, so I'm not that concerned.
Google obeys robots.txt
It would be very nice if natlib's crawler at least obeyed robots.txt entries that were specifically directed at them. Even nicer if they crawled from an NZ system, seeing as the international traffic will cost some small customers with lots of content real money. I left a message for a contact at natlib, maybe they can at least obey specific robots.txt entries. -- Jasper Bryant-Greene Network Engineer, Unleash ddi: +64 3 978 1222 mob: +64 21 129 9458