Re: [nznog] NLNZHarvester2008

<1224537417.6874.23.camel(a)oberon> <48FCF90A.4010609(a)webdrive.co.nz> <1224539473.17842.14.camel(a)localhost> <5b1922bbe7a7501041a3fc1eb9c95dc8(a)localhost> Message-ID: <0d338fea8445d9b74d6cfc26ba307a80(a)localhost> X-Sender: barry(a)unix.co.nz Received: from 203-96-86-10.cid.global-gateway.net.nz [203.96.86.10] with HTTP/1.1 (POST); Tue, 21 Oct 2008 12:26:45 +1300 User-Agent: RoundCube Webmail/0.1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit

But surely they could obey robots.txt entries that specifically target them?

User-agent: NLNZHarvester2008 Disallow: /massive-collection-of-high-res-images

User-agent: NLNZHarvester2008 Disallow: /my-porn-collection Beer barry