<1224537417.6874.23.camel(a)oberon> <48FCF90A.4010609(a)webdrive.co.nz> <1224539473.17842.14.camel(a)localhost> <D333E64FEDA1394A9AE9430C711402C401736465BD(a)EXCHANGE.advantage.co.nz> <DBC13D10DC0C754AB48A0E95D39026510C62380CCC(a)exchsth.agresearch.co.nz> <alpine.DEB.1.00.0810211130240.6897(a)green.darkmere.gen.nz> <5b1922bbe7a7501041a3fc1eb9c95dc8(a)localhost> Message-ID: <0d338fea8445d9b74d6cfc26ba307a80(a)localhost> X-Sender: barry(a)unix.co.nz Received: from 203-96-86-10.cid.global-gateway.net.nz [203.96.86.10] with HTTP/1.1 (POST); Tue, 21 Oct 2008 12:26:45 +1300 User-Agent: RoundCube Webmail/0.1 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit
But surely they could obey robots.txt entries that specifically target them?
User-agent: NLNZHarvester2008 Disallow: /massive-collection-of-high-res-images
User-agent: NLNZHarvester2008 Disallow: /my-porn-collection Beer barry