-----Original Message----- From: Brad Pearpoint [mailto:Brad(a)advantage.co.nz] Sent: Tuesday, 21 October 2008 11:03 a.m. To: Craig Whitmore Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] NLNZHarvester2008
Quote from their site:
" If you ignore robots.txt, what's to stop me blocking your crawler's IP address?
Nothing.
Some webmasters have taken this action, and we're sorry they felt they had to go to these lengths. We are running this harvest with good intentions, and ask that if you have blocked us, you reconsider - for example by allowing the harvester to access your site on the condition that it honours robots.txt. We'd much prefer this outcome to getting nothing from your websites at all.
Please remember that this project is about trying to ensure that as much as possible of the social history being enacted on the web today is available to researchers and all New Zealanders in the future. If we don't capture it now, we may not have the chance later.
"
If you wanted to, you could simply ask them to obey your robots.txt.
We already have, by having a robots.txt file. Shouldn't have to ask twice. Craig Miskell ======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================