Re: [nznog] Fwd: Notification of web harvest & consultation report

--- Gordon.Paynter(a)natlib.govt.nz wrote: From: "Gordon Paynter" Information abut the user agent is available from http://bit.ly/nlnzwebharvest and we don't yet know what IP addresses will be used. We have no plans to use the If-Modified-Since (or Etag, or similar approaches) for comparison with the 2008 harvest. If you have concerns about how the crawler may behave on specific websites, feel free to email us directly at web-harvest-2010(a)natlib.govt.nz or get in touch via our feedback form. ------------------------------------------------- Does it honor the robots.txt file? scott ---------------- ----------------- -----------------