Hi Scott,

I was aware from previous years that Robots.txt rules can be put in place.

My problem is that I have an extensive number of domains which would take me some time to edit in rules for every one.

In years past I understand that the harvest is done from servers overseas.�� In the past that hasn't been an issue for me as my content has been over seas.��

However in the past 12 months I've been making moves to improve local performance by putting content locally.�� The cost of that is that requests for content from O/S are way more expensive than before.�� I also have much more content that I used to.

So, simply blocking the IPs in a couple of routers makes much more sense for me.

D

On 19/01/2013 7:45 p.m., Scott Howard wrote:

The linked document states that :

Robots.txt files with a rule that refers specifically to the��NLNZ_IAHarvester2013��user agent will be strictly obeyed.

Which would seem to be a much better way to handle this than an IP block...

��Scott

On Fri, Jan 18, 2013 at 10:28 PM, Don Gould <don@bowenvale.co.nz> wrote:

Hi All,

Can anyone tell me where this harvest is being done from this year and what IP range to block so it doesn't troll though my machines?

D


On 13/12/2012 1:22 p.m., Jay Gattuso wrote:

Hi All,

This email is to inform you of the NLNZ intent to undertake its 3rd
Whole of Domain Web Harvest.

The planned dates for this exercise are the 4th to the 22nd of February
2013.

The harvest will be undertaken by the Internet Archive
<http://archive.org/index.php> on our behalf,

**

More details can be found on the web harvest
website:**http://natlib.govt.nz/publishers-and-authors/web-harvesting/2013-nz-web-harvest


If anyone has any specific concerns, requests or questions, please do
not hesitate to contact me directly: jay.gattuso@dia.govt.nz
<mailto:jay.gattuso@dia.govt.nz> or via the project email address:
Web.Archive@dia.govt.nz
<mailto:Web.Archive@dia.govt.nz?Subject=About%20the%202013%20Web%20Harvest>

If anyone wants to discuss this further on the mailing list I am happy
to do so.

King regards,

Jay

*Jay Gattuso*| Digital Preservation Analyst | Preservation, Research and
Consultancy

National Library of New Zealand | Te Puna M��tauranga o Aotearoa

PO Box 1467 Wellington 6140 New Zealand | +64 (0)4 474 3064

jay.gattuso@dia.govt.nz <mailto:jay.gattuso@natlib.govt.nz>



_______________________________________________
NZNOG mailing list
NZNOG@list.waikato.ac.nz
http://list.waikato.ac.nz/mailman/listinfo/nznog