Importance of end-to-end IPv4 - research - please help
As Richard mentioned earlier, WAND has recently done some work on behalf of Alcatel looking at the viability of SP-NAT. In particular, I've been investigating the number of incoming connections to DSL customers for a NZ ISP - how many customers are accepting connections and how many connections are they accepting? Long story short, over 40% of observed customer addresses accepted at least one incoming TCP connection over the time period we looked (around 4 consecutive days, including a weekend). This ratio grows to be more than 60% when UDP is also considered, although the counts for UDP aren't as reliable. Most of the incoming connections are on either well-known p2p ports or high-number ports, suggesting a lot of customers doing some form of p2p. More detailed results (plus pretty graphs!) can be found at http://www.wand.net.nz/~spa1/someisp/flow_counting/result_page.html#inbound In addition, WAND also looked into the average number of outbound sessions that those same DSL customers were using. The main aim there was to determine how many customers it would be feasible to place behind a single SP-NAT device. The results of that can be found on the same web-page (just scroll up a bit). Note that all these results are for a single ISP during a particular time period. It is very likely that other ISPs would see significantly different numbers depending on the profiles of the customers they tend to attract - warez monkeys vs Grandma, for instance. Shane Alcock WAND Network Research Group University of Waikato Nathan Ward wrote:
Hi all,
I'm attempting to get a bead on the importance of end-to-end IPv4.
By that I mean, home DSL user talking to another home DSL user.
This is something that would break if we ran out of IPv4 space tomorrow, and had to start putting customers behind service provider NAT (SP-NAT).
There's two ways I'm looking at doing this are: 1) Using a vendor box on loan to do p2p packet inspection for a month or so. This will tell us about how much "p2p[1]" traffic there is on a network, compared to non-p2p traffic. 2) Getting a packet capture from somewhere on a network for an hour, or whatever is feasible in terms of storage and processing power. The target of the capture would be traffic to/from a certain block of an ISPs end user type customers (so, a DSL pool probably). Analyse this and match it against dynamic address pools. - Anything going out to another dynamic pool (as determined by one of those dynamic pool lists) is something that would be broken by SP-NAT. - Any new incoming connections is something that would be broken by SP-NAT.
If there's anyone that's interested in the following please let me know: a) Helping me with some research b) Getting some free intelligence on the type of traffic on your network (wave it in front of marketing, and drip feed them the pretty graphs whenever you want something from them)
My intent is to publish the results stuff freely, publicly and widely.
I'd even like to get to the point where we can do it regularly perhaps? Let me know if you're open to that.
-- Nathan Ward
[1] By this I mean file sharing, skype, etc. Stuff commonly identified with the "p2p" buzz word, as opposed to the technical peer-to-peer phrase. _______________________________________________ NZNOG mailing list NZNOG(a)l... http://list.waikato.ac.nz/mailman/listinfo/nznog http://list.waikato.ac.nz/mailman/listinfo/nznog
On Tue, 2008-08-05 at 11:45 +1200, Shane Alcock wrote:
Long story short, over 40% of observed customer addresses accepted at least one incoming TCP connection over the time period we looked (around 4 consecutive days, including a weekend). This ratio grows to be more than 60% when UDP is also considered, although the counts for UDP aren't as reliable. Most of the incoming connections are on either well-known p2p ports or high-number ports, suggesting a lot of customers doing some form of p2p.
Is there any way to tell how many of those were actually desired by the customer, as opposed to their boxes being remote controlled via bots etc? Richard
Richard Hector wrote:
On Tue, 2008-08-05 at 11:45 +1200, Shane Alcock wrote:
Long story short, over 40% of observed customer addresses accepted at least one incoming TCP connection over the time period we looked (around 4 consecutive days, including a weekend). This ratio grows to be more than 60% when UDP is also considered, although the counts for UDP aren't as reliable. Most of the incoming connections are on either well-known p2p ports or high-number ports, suggesting a lot of customers doing some form of p2p.
Is there any way to tell how many of those were actually desired by the customer, as opposed to their boxes being remote controlled via bots etc?
Richard
We've made no real effort to try and distinguish whether the customer intended for that particular port to be open. I'm not sure that's something we can determine just by looking at TCP/IP packet headers but it is definitely something important to consider when thinking about SP-NAT. What we can do is look at the port numbers used for the incoming TCP connections - http://www.wand.net.nz/~spa1/someisp/flow_counting/incoming/server_ports_tcp... The four most popular ports are well-known ports for eDonkey, BitTorrent, HTTP and GNUtella, in that order, making up nearly 15% of all incoming TCP connections. These are very likely to have been opened by the customer themselves with the intention of using those services. On the other hand, ports like 139 and 5000 are probably being exploited without the customer's knowledge and SP-NAT preventing connections on those ports would not be harmful in the slightest. Everything else could be a customer using a peer2peer application or they could be a victim of something more malicious - we don't really have any reliable way of telling. Shane Alcock WAND Network Research Group University of Waikato
On 5/08/2008, at 11:45 AM, Shane Alcock wrote:
As Richard mentioned earlier, WAND has recently done some work on behalf of Alcatel looking at the viability of SP-NAT. In particular, I've been investigating the number of incoming connections to DSL customers for a NZ ISP - how many customers are accepting connections and how many connections are they accepting?
Long story short, over 40% of observed customer addresses accepted at least one incoming TCP connection over the time period we looked (around 4 consecutive days, including a weekend). This ratio grows to be more than 60% when UDP is also considered, although the counts for UDP aren't as reliable. Most of the incoming connections are on either well-known p2p ports or high-number ports, suggesting a lot of customers doing some form of p2p.
More detailed results (plus pretty graphs!) can be found at http://www.wand.net.nz/~spa1/someisp/flow_counting/result_page.html#inbound
In addition, WAND also looked into the average number of outbound sessions that those same DSL customers were using. The main aim there was to determine how many customers it would be feasible to place behind a single SP-NAT device. The results of that can be found on the same web-page (just scroll up a bit).
Note that all these results are for a single ISP during a particular time period. It is very likely that other ISPs would see significantly different numbers depending on the profiles of the customers they tend to attract - warez monkeys vs Grandma, for instance.
Awesome, that saves me some effort. Can you reveal what the profile of this particular ISP is? My main interest, is exploring the relationship between end users doing peer-to-peer on IPv4, and end users who have IPv6 (Teredo, 6to4). Do you have any numbers around that, or is it very hard to extract some? I would say, looking out for IP protocol 41, and UDP port 3544 would do the trick. Teredo of course uses non-standard ports for peer-to-peer, but that doesn't matter so much as it's always sending bubble packets to the server to keep NAT state open. Your document talks about ports per session, do you have any data around ports per byte? I expect one would find that port 445 etc. drops down quite significantly in cases like that. What about session length, in seconds? I'm hoping to prove that with a bit of movement from a couple of application vendors (Skype, etc.) we can start doing SP-NAT and preserve end-to-end, but over IPv6 instead of IPv4. Azureus is already doing IPv6 over IPv6, uTorrent has it coming in 1.8. Ideally, I'd like to get some numbers around this IPv4 p2p / IPv6 p2p relationship regularly updated, as I suspect as more and more applications become IPv6 aware the numbers will change quite significantly. Next big thing to watch out for is uTorrent 1.8 - by default it turns on IPv6(Teredo,6to4,etc.) in Windows when it is installed. Azureus only uses IPv6 if it's enabled already. -- Nathan Ward
Responses can be found in-line. Nathan Ward wrote:
Awesome, that saves me some effort.
Can you reveal what the profile of this particular ISP is?
Not really, for two reasons - 1) I don't want to say anything that would give away the ISP in question and 2) I don't really know. I referred to profile as more of a comparative thing - we'd need to do similar studies for other ISPs before we could say where a particular ISP might fall on the scale.
My main interest, is exploring the relationship between end users doing peer-to-peer on IPv4, and end users who have IPv6 (Teredo, 6to4). Do you have any numbers around that, or is it very hard to extract some?
I would say, looking out for IP protocol 41, and UDP port 3544 would do the trick.
So you just want to look at doing a similar study, except just looking at users who have demonstrated IPv6 capability? If so, I don't think I have that sort of data at hand, but it wouldn't be too hard to get it. The biggest difficulty is that it takes quite a while to process the packet traces (although I'm guessing you wouldn't need to look at the full four days worth of traffic).
Teredo of course uses non-standard ports for peer-to-peer, but that doesn't matter so much as it's always sending bubble packets to the server to keep NAT state open.
Your document talks about ports per session, do you have any data around ports per byte? I expect one would find that port 445 etc. drops down quite significantly in cases like that. What about session length, in seconds?
Again, I don't think I've been tracking these things specifically for incoming connections. I tracked a whole heap of stuff for outgoing flows, because that was what the Alcatel project was primarily focused on. But I could modify the code and re-run the processing to get that sort of stuff, if required.
I'm hoping to prove that with a bit of movement from a couple of application vendors (Skype, etc.) we can start doing SP-NAT and preserve end-to-end, but over IPv6 instead of IPv4.
Azureus is already doing IPv6 over IPv6, uTorrent has it coming in 1.8.
Ideally, I'd like to get some numbers around this IPv4 p2p / IPv6 p2p relationship regularly updated, as I suspect as more and more applications become IPv6 aware the numbers will change quite significantly.
Being able to do this sort of thing on a regular basis would be pretty neat. The set of traces we've been using for this are starting to get a little out-of-date now (February 2007). It's mainly a matter of getting a box in an appropriate location within an ISP to get some decent up-to-date traces.
Next big thing to watch out for is uTorrent 1.8 - by default it turns on IPv6(Teredo,6to4,etc.) in Windows when it is installed. Azureus only uses IPv6 if it's enabled already.
-- Nathan Ward
Ultimately, we can probably give sort out some numbers for you if you don't mind waiting for the data to be processed. If you're keen, let me know and we can start working out the exact specifics of what you want measured. Shane Alcock WAND Network Research Group University of Waikato
I'll talk to you about the other details off-line, but wanted one response to go on-list. On 5/08/2008, at 5:13 PM, Shane Alcock wrote:
Being able to do this sort of thing on a regular basis would be pretty neat. The set of traces we've been using for this are starting to get a little out-of-date now (February 2007). It's mainly a matter of getting a box in an appropriate location within an ISP to get some decent up-to-date traces.
Right. Feb '07 is a bit old - Azureus got IPv6 support in August '07. I shall renew my call for help :-) -- Nathan Ward
Hi Nathan et al, Sorry I've been slack - travelling around the land of Eire at the moment with time-zones and flakey Internet causing some challenges. Alcatel-Lucent sponsored this research in association with WAND to investigate the SP-NAT option, and as Shane mentioned the results were quite revealing. One item we should mention is that the we did not correlate RADIUS Acct (or similar) with the captured traffic - this means we were unable to definitively identify unique individuals with a global ID (such as username), instead resorting to unique IP address. Second the source data may be now out of date. This really leaves the door open for more research - something that Alcatel-Lucent would be interested in discussing with ISP who are interested! We recognise the value of this data and I'm in complete agreement with Nathan on making it widely available - hence our presentation at APRICOT earlier this year on the subject. We have also been doing work with Bell Labs to look at the economics of SP-NAT, and of course we may have a trick or two up the proverbial sleeve which our development team are working on. The "net10" issue is by far the single biggest consideration for SP- NAT. Some ISP we have talked to are going to simply re-use an assignment many times within their organisation, some are looking towards IANA for a new assignment (see the drafts Brian posted), while for others even a /8 is not big enough. I expect both SP-NAT and approaches like dual-stack lite and sNAT (http://tools.ietf.org/html/draft-droms-softwires-snat-01 ) will be adopted. There is no "right" answer, but some options available to operators to continue limited IPv4 operations post exhaustion. One thing that is clear from the research - our expectations regarding the amount of unsolicited inbound traffic have been challenged. Both dual stack lite and SP-NAT/CGN all involve PAT, so the user experience will be identical (in our analysis). No amount of SP-NAT, Dual Stack Lite or tricks up sleeves will bring back the IPv4 experience we have today in which you and I have a global IPv4 address we can play with. IPv6 is clearly the only way forward and one we need to start the planning for now. Any interested parties please feel free to contact me off-list or via Shane if there is a research opportunity. Best Regards, David Miles (david.miles(a)alcatel-lucent.com) On 05/08/2008, at 6:13 AM, Shane Alcock wrote:
Responses can be found in-line.
Nathan Ward wrote:
Awesome, that saves me some effort.
Can you reveal what the profile of this particular ISP is?
Not really, for two reasons - 1) I don't want to say anything that would give away the ISP in question and 2) I don't really know. I referred to profile as more of a comparative thing - we'd need to do similar studies for other ISPs before we could say where a particular ISP might fall on the scale.
My main interest, is exploring the relationship between end users doing peer-to-peer on IPv4, and end users who have IPv6 (Teredo, 6to4). Do you have any numbers around that, or is it very hard to extract some?
I would say, looking out for IP protocol 41, and UDP port 3544 would do the trick.
So you just want to look at doing a similar study, except just looking at users who have demonstrated IPv6 capability? If so, I don't think I have that sort of data at hand, but it wouldn't be too hard to get it. The biggest difficulty is that it takes quite a while to process the packet traces (although I'm guessing you wouldn't need to look at the full four days worth of traffic).
Teredo of course uses non-standard ports for peer-to-peer, but that doesn't matter so much as it's always sending bubble packets to the server to keep NAT state open.
Your document talks about ports per session, do you have any data around ports per byte? I expect one would find that port 445 etc. drops down quite significantly in cases like that. What about session length, in seconds?
Again, I don't think I've been tracking these things specifically for incoming connections. I tracked a whole heap of stuff for outgoing flows, because that was what the Alcatel project was primarily focused on. But I could modify the code and re-run the processing to get that sort of stuff, if required.
I'm hoping to prove that with a bit of movement from a couple of application vendors (Skype, etc.) we can start doing SP-NAT and preserve end-to-end, but over IPv6 instead of IPv4.
Azureus is already doing IPv6 over IPv6, uTorrent has it coming in 1.8.
Ideally, I'd like to get some numbers around this IPv4 p2p / IPv6 p2p relationship regularly updated, as I suspect as more and more applications become IPv6 aware the numbers will change quite significantly.
Being able to do this sort of thing on a regular basis would be pretty neat. The set of traces we've been using for this are starting to get a little out-of-date now (February 2007). It's mainly a matter of getting a box in an appropriate location within an ISP to get some decent up-to-date traces.
Next big thing to watch out for is uTorrent 1.8 - by default it turns on IPv6(Teredo,6to4,etc.) in Windows when it is installed. Azureus only uses IPv6 if it's enabled already.
-- Nathan Ward
Ultimately, we can probably give sort out some numbers for you if you don't mind waiting for the data to be processed. If you're keen, let me know and we can start working out the exact specifics of what you want measured.
Shane Alcock
WAND Network Research Group University of Waikato _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
Morning List, I've been following this discussion with some interest. I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand. Are users advised that their data is being captured for analysis? What is the law regarding this sort of data capture? Are regulators/auditors involved in ensuring appropriate security of captured data? I'm not after a flame war on this issue, if it's already been discussed with respect to earlier projects I'd be interested in a link to the previous discussions. Cheers Don
On Wed, Aug 6, 2008 at 12:14 PM, Don Gould
Morning List,
I've been following this discussion with some interest.
I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand.
Are users advised that their data is being captured for analysis?
What is the law regarding this sort of data capture?
Are regulators/auditors involved in ensuring appropriate security of captured data?
I'm not after a flame war on this issue, if it's already been discussed with respect to earlier projects I'd be interested in a link to the previous discussions.
Cheers Don
WAND works typically with anonymised data and headers only. Regards Ian -- Web: http://wand.net.nz/~iam4/ Blog: http://iansblog.jandi.co.nz
On Wed, 2008-08-06 at 12:14 +1200, Don Gould wrote:
I've been following this discussion with some interest.
I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand.
Are users advised that their data is being captured for analysis?
What is the law regarding this sort of data capture?
Are regulators/auditors involved in ensuring appropriate security of captured data?
I'm not after a flame war on this issue, if it's already been discussed with respect to earlier projects I'd be interested in a link to the previous discussions.
While I can't comment on the approach actually being taken, since I have no idea (and IANAL), I'd have thought such concerns could easily be dealt with by just capturing aggregate statistics. In other words, "Between 2008-08-01 00:00:00 and 23:59:59, ISP X's DSL customers terminated X inbound TCP connections on port X, totalling X bytes." Of course the customer's data had to be processed by network hardware and/or servers in order to collect the above stats, but they're being processed by network hardware anyway in order to switch and route the traffic. As long as the output of the system doesn't include any identifying data, which the above example does not, I would have thought that there'd be no problem. I'd be interested to hear from someone more involved in this type of analysis on the legal issues involved... -jasper
On 6/08/2008, at 12:14 PM, Don Gould wrote:
Morning List,
I've been following this discussion with some interest.
I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand.
Are users advised that their data is being captured for analysis?
What is the law regarding this sort of data capture?
Are regulators/auditors involved in ensuring appropriate security of captured data?
I'm not after a flame war on this issue, if it's already been discussed with respect to earlier projects I'd be interested in a link to the previous discussions.
Is it any different to analysing traffic in order to, for example, detect and limit p2p file sharing? Generally, this sort of analysis is done with only packet headers. p2p file sharing detection/limiting stuff often looks at full packets. One could argue that that is more invasive. Plenty of other things do deep packet inspection as well - ddos detection, transparent http proxying, etc. Much more invasive than the simple header analysis that has been proposed and discussed here. -- Nathan Ward
If you were driving along the public road on a car, you can't object to people taking a picture of you, or taking notes about what you were wearing, number plate, etc. You effectively loose the right of privacy in public. And if someone saw your car on a public road, and decided to go and make an exact duplicate (with the exception of trademarks, and design marks) you couldn't do much about it. I would have thought that once your traffic is on the "public" Internet your right to privacy is somewhat diminished. I say "public", because the Internet is clearly not a private network. So I would have guessed that the capture of summary data (packets headers) should be fine. I would also guess that the exact duplication of that data (full packet capture) should also be okay. I'm more perplexed by copyright though. If I write a book and put it into a public library you don't have the right to make an exact copy. So if I email that same document across the Internet, does someone else have the right to make an exact duplicate with a packet capture? I'm kinda guessing the answer would be no. But then the same old argument explodes into web proxies, which also obviously make a copy of data that is being browsed. With regard to determining which traffic is p2p; I believe this would be futile with only packet headers. You would need at least the start of the payload. -----Original Message----- From: Nathan Ward [mailto:nznog(a)daork.net] ...
I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand.
Are users advised that their data is being captured for analysis? ... Is it any different to analysing traffic in order to, for example, detect and limit p2p file sharing?
Generally, this sort of analysis is done with only packet headers. p2p file sharing detection/limiting stuff often looks at full packets. One could argue that that is more invasive. Plenty of other things do deep packet inspection as well - ddos detection, transparent http proxying, etc. Much more invasive than the simple header analysis that has been proposed and discussed here. -- Nathan Ward _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
Philip D'Ath wrote:
If you were driving along the public road on a car, you can't object to people taking a picture of you, or taking notes about what you were wearing, number plate, etc. You effectively loose the right of privacy in public.
O RLY? http://www.stuff.co.nz/4642693a28.html -- Juha www.techsploder.com
If you were driving along the public road on a car, you can't object to people taking a picture of you, or taking notes about what you were wearing, number plate, etc. You effectively loose the right of
If this wasn't the case then paparazzi would have a terrible time, don't you think Juha? :-) -----Original Message----- From: Juha Saarinen [mailto:juha(a)saarinen.org] Sent: Wednesday, 6 August 2008 1:17 p.m. To: Philip D'Ath Cc: nznog(a)list.waikato.ac.nz Subject: Re: [nznog] Importance of end-to-end IPv4 - research - please help Philip D'Ath wrote: privacy
in public.
O RLY? http://www.stuff.co.nz/4642693a28.html -- Juha www.techsploder.com
On 6/08/2008, at 1:10 PM, Philip D'Ath wrote:
With regard to determining which traffic is p2p; I believe this would be futile with only packet headers. You would need at least the start of the payload.
That depends if you're looking for specific protocols, or traffic patterns that are common to p2p file sharing. The latter is much harder to hide - but is also probably much more computationally intensive. -- Nathan Ward
On Wed, Aug 6, 2008 at 1:10 PM, Philip D'Ath
If you were driving along the public road on a car, you can't object to people taking a picture of you, or taking notes about what you were wearing, number plate, etc. You effectively loose the right of privacy in public.
Etc etc In my humble opinion this argument is completely bogus. The PSTN is a public network, but if I make a call to my grandfather in England, I don't expect anyone to be able to listen to what I'm saying without a warrant. If I have a unicast HTTP session, the content of that session is between me and the webserver. No-one should be able to look at that without a warrant. Looking at IP/layer 4 headers is ok because you're not seeing the "communication content" and there is no personally identifiable information except your IP address. To correlate an IP address with me is something only my ISP can do, so releasing the info to a third party seems ok...as long as they can't correlate the IP address to a name/person. Jonathan
Ahh, but there is specific legislation in place that protects the privacy of the PSTN. You lose that protection if you make that same phone call over IP. I'm not aware of any act of parliament that offers you that same protection for Internet traffic From: Jonathan Woolley [mailto:nznog(a)jonathanwoolley.com] ... In my humble opinion this argument is completely bogus. The PSTN is a public network, but if I make a call to my grandfather in England, I don't expect anyone to be able to listen to what I'm saying without a warrant. If I have a unicast HTTP session, the content of that session is between me and the webserver. No-one should be able to look at that without a warrant. Looking at IP/layer 4 headers is ok because you're not seeing the "communication content" and there is no personally identifiable information except your IP address. To correlate an IP address with me is something only my ISP can do, so releasing the info to a third party seems ok...as long as they can't correlate the IP address to a name/person. Jonathan
Isn't the internet largely a collection of interconnecting private networks? Apart from a physical wiretap (which is likely to be illegal under the same laws for PSTN etc), the only people that should have means to wiretap/packet-sniff a packet travelling over a private network segment are the network owners and they should have policies limiting who might have access to perform such a task. Regan -----Original Message----- From: Philip D'Ath [mailto:pid(a)ifm.net.nz] Sent: Wednesday, 6 August 2008 1:11 p.m. To: Nathan Ward; nznog Subject: Re: [nznog] Importance of end-to-end IPv4 - research - please help <snip> I would have thought that once your traffic is on the "public" Internet your right to privacy is somewhat diminished. I say "public", because the Internet is clearly not a private network. So I would have guessed that the capture of summary data (packets headers) should be fine. I would also guess that the exact duplication of that data (full packet capture) should also be okay. <snip>
Regan Murphy wrote:
Isn't the internet largely a collection of interconnecting private networks? Apart from a physical wiretap (which is likely to be illegal under the same laws for PSTN etc), the only people that should have means to wiretap/packet-sniff a packet travelling over a private network segment are the network owners and they should have policies limiting who might have access to perform such a task.
Don't most ISPs have terms and conditions that state they may monitor, record, and intercept traffic on their network for their purposes as required? If you're an ISP and you don't, you may want to talk to your legal consultants about including this... for doing this sort of research, or even using any kind of flow exports, DPI, etc. aj [I think this has gone OT enough from Nathan's original request.]
My answer to any privacy issues is this (I agree with the network being a "public space" regardless of the physical ownership of the underlying networks), if you choose to use software and protocols that don't offer end to end encryption then you void the right to complain about being snooped. There are solutions it is a choice not to use them. In reference to standards about recording data in public/private settings I point you at telecomunication privacy commission code. http://www.privacy.org.nz/telecommunications-information-privacy-code/ Psychologists have been dealing with this issue for a long time, and there are some good transferable ethics approval processes in that discipline for approving collection of data for study. APA being the standard. -JoelW
From the small amount of reading I've done what WAND has done appears to be fine. However, capturing header + payload and disseminating it to a third
I'd just managed to find that code and read some of it before the below
email. It's basically telling you how to interpret the privacy act for
telecommunications networks. This includes the internet, not just the PSTN
as "Internet Service Provider" is included under the definition of an
"Agency" and communication is a phone-call or "any other telecommunication".
The underlying assumption I've picked up from previous posts to NZNOG is
that "if it's on the internet you can just do whatever you want with the
traffic including payload". This appears to be wrong from a quick glance of
some of the rules set out in the code. This is true with or without a
company policy saying you can/can't do x, y or z. This note in "Rule 1" is
quite pertinent:
"
Note: Except where it is itself a party to a communication, a
telecommunications agency will rarely have a lawful purpose to collect the
content of any telecommunication. Indeed, it is unlawful to intercept the
content of a private communication in most cases (Crimes Act 1961, Part 9A).
There are some limited exceptional circumstances relevant to
telecommunications agencies (e.g. where acting pursuant to an interception
warrant to assist the Police or SIS). Employees of network operators can, in
the course of their duties, intercept telecommunications for maintenance
purposes but it is an offence for an employee of a network operator to use
or disclose information so obtained for unauthorised purposes –
Telecommunications Act 2001, ss.114 and 115).
"
party without a warrant is ILLEGAL.
I leave further reading of the code and analysis of fringe cases etc as an
exercise to the reader, but previous posts leave me a bit worried about the
cavalier attitude of some NZNOG posters to privacy of the internet.
Jonathan
On Wed, Aug 6, 2008 at 1:51 PM, Joel Wiramu Pauling
My answer to any privacy issues is this (I agree with the network being a "public space" regardless of the physical ownership of the underlying networks), if you choose to use software and protocols that don't offer end to end encryption then you void the right to complain about being snooped. There are solutions it is a choice not to use them.
In reference to standards about recording data in public/private settings I point you at telecomunication privacy commission code.
http://www.privacy.org.nz/telecommunications-information-privacy-code/
Psychologists have been dealing with this issue for a long time, and there are some good transferable ethics approval processes in that discipline for approving collection of data for study. APA being the standard.
-JoelW _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
The law is a bit ambiguous. For example the crimes act ammendment in 2003 outlawed hacking and specifically allowed police interception of computer communications but is silent on interception by network operators or researchers. When we got a legal opinion on this (shock horror - consulting someone who knew what they were talking about ;-) the advice was that the exact meaning of the act could only be established through precedent in case law. However there hadn't been any cases. AFAIK there still haven't. In absence of a definitive case, our advisor said that the crucial point would be the intent of intercepting data. So legitimate activities such as network maintenance and operations and also network research would be Ok but snooping on users private activities would not. We "think" we are Ok legally. As Shane said, ethically we have to debate our activities with the University ethics committee. If they understood networks that would be a good thing but as it is it's hard, but we are making progress on coming to an understanding. Richard.
Jonathan Woolley wrote:
From the small amount of reading I've done what WAND has done appears to be fine.
I don' agree. From what I have read I currently think that what WAND is doing is unacceptable with out informed concent from anyone involved in the research.
However, capturing header + payload and disseminating it to a third party without a warrant is ILLEGAL.
That would have been my assumption.
posts leave me a bit worried about the cavalier attitude of some NZNOG posters to privacy of the internet.
I agree with you. Hence why I raised my initial concerns. Cheers Don
Wow - What a frenzy. I was going to start talking about some other traffic anomaly detection mechanisms, but I'm picking this has just stopped being the place. Dean
Folks, We all know that end-to-end IPv4 is only required in very particular instances, so this argument could go away if we'd just implement RFC3514. http://www.apps.ietf.org/rfc/rfc3514.html Then it would be a simple matter of counting how many users sending 0x0, compared to how many sending packets marked 0x1. There's no way anyone's privacy could be compromised just by counting packets. JB -----Original Message----- From: Don Gould [mailto:don(a)bowenvale.co.nz] Sent: Wednesday, 6 August 2008 4:01 p.m. To: Jonathan Woolley Cc: nznog Subject: Re: [nznog] Importance of end-to-end IPv4 - research - please help Jonathan Woolley wrote:
From the small amount of reading I've done what WAND has done appears to be fine.
I don' agree. From what I have read I currently think that what WAND is doing is unacceptable with out informed concent from anyone involved in the research.
However, capturing header + payload and disseminating it to a third party without a warrant is ILLEGAL.
That would have been my assumption.
posts leave me a bit worried about the cavalier attitude of some NZNOG posters to privacy of the internet.
I agree with you. Hence why I raised my initial concerns. Cheers Don _______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
On 2008-08-06 16:00, Don Gould wrote:
Jonathan Woolley wrote:
From the small amount of reading I've done what WAND has done appears to be fine.
I don' agree.
From what I have read I currently think that what WAND is doing is unacceptable with out informed concent from anyone involved in the research.
In that case, traffic surveys where someone sits at the side of the road with a clipboard are also unacceptable without informed consent from every driver that passes by. As long as traffic is anonymised, or only analysed statistically, there's definitely no ethical problem. Brian
I think this discussion has moved off technical topics and into legal ones. Since this is not a legal mail list but a technical ones that portion of the thread should probably stop. Network operators are of course reminded that they should obey the law whenever possible and that if the law is in doubt a lawyer should be consulted. Simon. Wearing moderator hat. -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
Simon, Simon Lyall wrote:
Since this is not a legal mail list but a technical ones that portion of the thread should probably stop.
Can you advise where the appropriate forum is for raising legal issues that relate to network operation in NZ? Appologies to anyone on the list who feels that the questions raised were not appropriate to the list. I did express that I was not interested in a flame war on the topic and agree that this may not be the correct forum for *further* debate on the issue. The questions I raised in my initial post have now largely been answered. Thank you to those who took the time to respond. Cheers Don
On Wed, 6 Aug 2008, Don Gould wrote:
Can you advise where the appropriate forum is for raising legal issues that relate to network operation in NZ?
Internetnz have some forums and this is right within their area. Also the TCF and ISPANZ might be interested as well. Further to that I'd guess some of the research agencies ( KAREN, Universities) will have done some work as to the legality of their work ( people have already posted this). -- Simon Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
From what I have read I currently think that what WAND is doing is unacceptable with out informed concent from anyone involved in the research.
It looks like the data collected is less than your ISP would normally record for billing purposes. Your ISP must identify who you are and whether your traffic is national, international, and sometimes even make a distinction to local traffic. I haven't seen anyone sue their ISP for illegally wiretapping them for the purposes of producing a monthly bill. Yet. -- Spiro Harvey Knossos Networks Ltd 021-295-1923 www.knossos.net.nz
Just wanted to put out there that I may be interested in joining/following a research effort for this when I have some time during the summer, it would be a good way to flex some statistical tools. Now for just a little bit of a rant.... (Sorry for the slightly off topic but I just think it has to be said)
posts leave me a bit worried about the cavalier attitude of some NZNOG posters to privacy of the> internet.
I don't see how anyone can see the "internet" as private at all. Who knows who's private network the traffic is going to traverse to get to it's destination. The core philosophy of the internet in some ways is, "Just get it there". Making sure you got what you want and protecting it from prying eyes is up to you. Yes network operators should do there best to minimize information leakage, and have policies that limit and oversee what information people have access to, and to make sure actions are ethically reasonable. But that is a far cry from having privacy. The people that set up snooping on tor end nodes is a good example of this, if you want payload privacy there is cryptography. Even been tempted to think of an internet pipe is a private communication channel is a very very bad idea from a security standpoint. I am not sure ever expecting to have privacy of your source and destination is reasonable, which scares me a lot because of the amount of information you can gather from these side channels. Things get real blurry real fast... What about if I collected this data by watching peers on a torrent tracker? How about things like transparent proxies? I am not sure that the "law" works to well in this domain in the large. It is good for localized enforcement, but it cannot be assumed to hold for all the destinations of your packets. If we wanted to help users with privacy we would give them end to end crypto. But I am not sure the people that want legal wiretaps would like that all that much... Sorry for the rant guys, security just strikes a nerve sometimes ;) Regards, Bry
I just want to clarify a few things about our trace capture and analysis methods. More details about the traces we used for this analysis can be found at http://www.wand.net.nz/wits/localisp/b/3/ . Important things to note are that we snap all captured packets just after the transport header (TCP, UDP etc) as soon as we pull them off the wire. The IP addresses for this trace-set are not anonymized (unlike most other captures that WAND does) but we have no way of mapping IP addresses back to customers anyway. There are a number of checks and balances to ensure the privacy of any user data that we might end up examining. Primarily, we have a non-disclosure agreement with the ISP itself that includes not publicly sharing any information that is specific to either the ISP or its users. The agreement also defined what we could capture and what had to be anonymized - in this case, we were allowed to keep those four bytes of payload and unanonymized IP addresses. Obviously, summary statistics such as those on the web-page I linked to yesterday are OK. On top of all that, the university has ethics committees and a code of conduct to govern this kind of thing. The trace capture had to pass through an ethics committee at Waikato before we could even commence capturing traffic. We're also very careful to ensure that no-one here at the University gets access to the traces without signing an NDA. But ultimately, given that we've thrown away (almost) all the payload after the headers, users don't have too much to fear from me inspecting the packet headers anyway. It isn't as though I can read their email or steal their plain-text passwords. Hopefully that allays a few fears, Shane Alcock WAND Network Research Group University of Waikato Don Gould wrote:
Morning List,
I've been following this discussion with some interest.
I'm sure the issue of ethics has been raised on this topic but I hadn't seen any mention in this thread and am unclear where users stand.
Are users advised that their data is being captured for analysis?
What is the law regarding this sort of data capture?
Are regulators/auditors involved in ensuring appropriate security of captured data?
I'm not after a flame war on this issue, if it's already been discussed with respect to earlier projects I'd be interested in a link to the previous discussions.
Cheers Don
_______________________________________________ NZNOG mailing list NZNOG(a)list.waikato.ac.nz http://list.waikato.ac.nz/mailman/listinfo/nznog
participants (21)
-
Alastair Johnson -
Brian E Carpenter -
Bry Ashman -
David Miles -
Dean Pemberton -
Don Gould -
Ian McDonald -
Jasper Bryant-Greene -
Joe Abley -
Joel Wiramu Pauling -
Jonathan Brewer -
Jonathan Woolley -
Juha Saarinen -
Nathan Ward -
Philip D'Ath -
Regan Murphy -
Richard Hector -
Richard Nelson -
Shane Alcock -
Simon Lyall -
Spiro Harvey, Knossos Networks Ltd