-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alex Hague wrote:
The issue for us isn't that they are indexing our site, it is that they are disobeying robots.txt. Even this in itself wouldn't be a problem if they provided a heads up in future and compromised by following robots.txt entries that targeted their user agent.
And that is entirely the crux. It's more the sheer arrogance and lack of thought to implementation and the community that is causing people grief. Some respect towards the people involved shouldn't be too much to ask for when the avenues are so readily available. p.s. Long time, no see, Mr. Hague. - -- Drew Calcott Science IT University of Auckland (p) +64 9 373 7599 x84269 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkj9JFYACgkQD+yXTWfduLG2MgCgo4yvhxH128/ZlDzRevU1VnK0 KNwAnjO52+qFbv5CzaJKN96Ov1PvpCdp =21Xk -----END PGP SIGNATURE-----