19 Oct
2008
19 Oct
'08
5:18 p.m.
On Mon, Oct 20, 2008 at 04:31:43PM +1300, Murray Fox wrote:
Hi All,
This was news to me ('till international traffic on some of my sites began to climb); The National Library is this month ripping the entire NZ Internet ..... from an international host ..... blatantly ignoring robots.txt
http://www.natlib.govt.nz/about-us/current-initiatives/web-harvest-2008
so - is this a snapshot or is the intent expected to be on-going? --bill