On 21/10/2008, at 1:10 PM, Gordon Paynter wrote:
1. Several people have asked about notification, and some you have made practical and workable suggestions about how we can handle this better next time.
In the current crawl, we could not see a good way to do so without effectively becoming spammers. In hindsight we could have communicated better with webmasters. When we decide to run the harvest again, we will make more of an effort to publicise the harvest in mailing lists and groups frequented by webmasters (such as this one).
Can I suggest something here? Speak to the Most Evil Media too about the crawling. This is neither fool nor fail proof, but the tech media should be interested (and I can see a few stories about the archive already) and help publicise your intentions to those who don't participate in mailing lists.
2. Others ask why we are harvesting from the USA and not New Zealand?
We have contracted the Internet Archive to conduct the harvest because they are the single most experienced provider of large-scale crawling services in the world.
An unfortunate offshoot of this is that their servers are based in the USA.
We hope that after observing the experts at work we'll be able to manage future harvests from within New Zealand. At the very least we have learned that we should locate some of the harvest servers in New Zealand.
Umm, yes. :)
--
Juha Saarinen juha(a)saarinen.org http://www.techsploder.com