Dean Pemberton wrote:
. With regard to the Internet Archive using 2 IP addresses (1) Honouring robots.txt(1) Not honouring robots.txt. Ask the NZNOG community whether they noticed significant differences in the behaviours between the two crawlers.
I know it's bad form to reply to your own posts - but there is a point to this one. It turns out that the two harvest IPs NLNZ was using had different behaviours. One honored robots.txt and the other..... well not so much =) Did any of you who noticed the harvest seem to be getting hit by one of these more than the other? The answer to this, and how much you were hit as a ratio, might help with feedback on how NLNZ can do this better next time. Answers onlist if they are relevant, offlist to me if they are not, or live to Gordon who will be presenting re the harvest at the NZNOG conference =) Thanks Dean