FYI: Message sent to us @ auckland.ac.nz ============================================================================== This is a heads up that the National Library of NZ has embarked on a web harvest of the .nz domain as part of their legal mandate to collect and preserve NZ’s online documentary heritage. They are outsourcing the work to the Internet Archive (Way Back Machine etc). You can see the latest information here: http://www.natlib.govt.nz/about-us/news/15-october-2008-update-on-web-harves... with more background at: http://librarytechnz.natlib.govt.nz/2008/10/2008-web-harvest-let-us-know-how... http://www.natlib.govt.nz/about-us/current-initiatives/web-harvest-2008 Things to note are: - the crawlers are working in bursts of 500 URLs at a time - they will not honour the robots.txt protocol - they won’t harvest password protected content If you are checking logs, the user agent string that appears should look like this: Mozilla/5.0 (compatible; NLNZHarvester2008 +http://www.natlib.govt.nz/about-us/current-initiatives/web-harvest-20 08) All of therr requests should be originating from either of these two IPs: 149.20.55.4 207.241.232.188 The harvest is happening between 7-24 October with subsequent patch crawls until 7 November 2008. If you have any concerns let me know. If there is any impact on service (unlikely, but you never know) please contact the National Library directly at web-harvest-2008(a)natlib.govt.nz and they will stop or modify the harvest as quickly as possible.