On 20/10/08 4:31 PM, "Murray Fox"
Hi All,
This was news to me ('till international traffic on some of my sites began to climb); The National Library is this month ripping the entire NZ Internet ..... from an international host ..... blatantly ignoring robots.txt
Somewhat risky for them surely? One thing robots.txt does is protect crawlers from infinite dynamic content. I'd have thought that at least *some* NZ sites have referer tarpits or recursive redirect blackholes that any sensible crawler would be best to avoid. -- Michael Newbery IP Architect TelstraClear Limited TelstraClear. Simple Solutions. Everyday Residential 0508 888 800 Business 0508 249 999 Enterprise & Government 0508 400 300 This email contains information which may be confidential and subject to copyright. If you are not the intended recipient you must not use, distribute or copy this email or attachments. If you have received this email in error please notify us immediately by return email and delete this email and any attachments. TelstraClear Limited accepts no responsibility for changes made to this email or to any attachments after transmission from TelstraClear Limited. It is your responsibility to check this email and any attachments for viruses. Emails are not secure. They can be intercepted, amended, lost or destroyed and may contain viruses. Anyone who communicates with TelstraClear Limited by email is taken to accept these risks.