Hi, We at the National Library will soon begin our 2021 New Zealand domain crawl. As in previous years, the crawl is being run by Internet Archive, and some of you may soon see the crawlers as IA go through their initial setup with some test crawls. The user agent string their crawlers will be using is: NLNZ_IAHarvester2021 For more information on our domain crawls please see http://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest While we do our best to crawl respectfully, obey robots.txt, and break out of any crawler traps before grinding too much bandwidth, if you encounter any issues from us then please get in contact. Cheers, Ben O’Brien | Digital Preservation Web Engineer | Preservation, Research and Consultancy National Library of New Zealand | Te Puna Mātauranga o Aotearoa PO Box 1467, Wellington 6140, New Zealand | www.natlib.govt.nzhttp://www.natlib.govt.nz/ [Logo-test]
