Hi,

We at the National Library are about to begin our 2018 New Zealand domain crawl. As in previous years, the crawl is being run by Internet Archive, and some of you may have already seen the crawlers as IA go through their initial setup with some test crawls.

The user agent string their crawlers will be using is: NLNZ_IAHarvester2018

While we do our best to crawl respectfully, obey robots.txt, and break out of any crawler traps before grinding too much bandwidth, if you encounter any issues from us then please get in contact.

Cheers,

Ben O�Brien | Digital Preservation Web Engineer | Preservation, Research and Consultancy

National Library of New Zealand | Te Puna M�tauranga o Aotearoa

PO Box 1467, Wellington 6140, New Zealand | www.natlib.govt.nz