Re: [nznog] National Library of New Zealand Web Harvest 2013

Can you please give us the calculations that show that writing a one line shell script and serving up robots a handful of times is more expensive than spending your time writing emails objecting to the same? I'll be happy to sell you bandwidth for slightly cheaper to help you cut costs, based on your calculations of course. I'll even give you a ride to the nznog venue from the airport in the Lamborghini you'll have bought me by mid next week. Also please provide similar calculations taking in to account each further reply so I can adjust my prices accordingly. On Saturday, January 19, 2013, Don Gould wrote:

Hi Scott,

I was aware from previous years that Robots.txt rules can be put in place.

My problem is that I have an extensive number of domains which would take me some time to edit in rules for every one.

In years past I understand that the harvest is done from servers overseas. In the past that hasn't been an issue for me as my content has been over seas.

However in the past 12 months I've been making moves to improve local performance by putting content locally. The cost of that is that requests for content from O/S are way more expensive than before. I also have much more content that I used to.

So, simply blocking the IPs in a couple of routers makes much more sense for me.

D

On 19/01/2013 7:45 p.m., Scott Howard wrote:

The linked document states that : Robots.txt files with a rule that refers specifically to the * NLNZ_IAHarvester2013* user agent will be strictly obeyed.

Which would seem to be a much better way to handle this than an IP block...

Scott

On Fri, Jan 18, 2013 at 10:28 PM, Don Gould wrote:

Hi All,

Can anyone tell me where this harvest is being done from this year and what IP range to block so it doesn't troll though my machines?

D

On 13/12/2012 1:22 p.m., Jay Gattuso wrote:

Hi All,

This email is to inform you of the NLNZ intent to undertake its 3rd Whole of Domain Web Harvest.

The planned dates for this exercise are the 4th to the 22nd of February 2013.

The harvest will be undertaken by the Internet Archive http://archive.org/index.php on our behalf,

**

More details can be found on the web harvest website:** http://natlib.govt.nz/publishers-and-authors/web-harvesting/2013-nz-web-harv...

If anyone has any specific concerns, requests or questions, please do not hesitate to contact me directly: jay.gattuso(a)dia.govt.nz mailto:jay.gattuso(a)dia.govt.nz or via the project email address: Web.Archive(a)dia.govt.nz

If anyone wants to discuss this further on the mailing list I am happy to do so.

King regards,

Jay

*Jay Gattuso*| Digital Preservation Analyst | Preservation, Research and Consultancy

National Library of New Zealand | Te Puna Mātauranga o Aotearoa

PO Box 1467 Wellington 6140 New Zealand | +64 (0)4 474 3064

jay.gattuso(a)dia.govt.nz mailto:jay.gattuso(a)natlib.govt.nz

_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog