22 Sep
2014
22 Sep
'14
9:08 a.m.
On Tue, 23 Sep 2014, Mark Foster wrote:
I'm not sure there's any excuse for ignoring robots.txt - it's just plain rude. A webmaster should retain the ability to influence whether their content is picked up or not. With that in mind I suspect (2) is the right answer.
Well the problem is that it is common for people to block all bots except Google (and perhaps a couple of others). AFAIK this crawl they did have a specific line to block them if you really wanted to. -- Simon Lyall | Very Busy | Web: http://www.simonlyall.com/ "To stay awake all night adds a day to your life" - Stilgar