19 Oct
2008
19 Oct
'08
10:30 a.m.
Michael Fincham wrote:
On Mon, 2008-10-20 at 16:31 +1300, Murray Fox wrote:
NZ Internet ..... from an international host ..... blatantly ignoring robots.txt
I don't suppose you or anyone else could share any info re: the subnet the requests are coming from or point to this info if it's published somewhere?
I presume it'll be something inside one of The Internet Archive's allocations but am curious for specifics if they're available.
cat vhost-access_log.* | grep web-harvest-2008 | awk {'print $2'} | sort | uniq 149.20.55.4 207.241.232.188 Those are the two we have seen to date