19 Oct
2008
19 Oct
'08
10 a.m.
Somewhat risky for them surely? One thing robots.txt does is protect crawlers from infinite dynamic content. I'd have thought that at least *some* NZ sites have referer tarpits or recursive redirect blackholes that any sensible crawler would be best to avoid.
And if they didn't before, they sure as heck will now. Or at least, they will before NLNZHarvester2009 starts crawling. -- Spiro Harvey Knossos Networks Ltd 021-295-1923 www.knossos.net.nz