20 Oct
2008
20 Oct
'08
4:43 a.m.
If you wanted to, you could simply ask them to obey your robots.txt.
Which would be entirely in vain after the fact :-/
I was somewhat suprised that despite being on various mailing lists, I didn't hear about NatLib's harvest until well after it had kicked off. I have asked NatLib to honour robots.txt in the case of the sites I personally host, and received a prompt affirmative response from them on the request... I was also advised that they "remain committed to our strategy of ignoring robots.txt unless requested otherwise." I in turn suggested that perhaps NatLib needed to pay attention to NZNOG... to me this discussion should've happened several weeks ago, so that webhosts were better able to plan/respond to the situation. Mark.