55MPS will give max ~ 28G << 100G thruput @prema+sdf+64B+ifg. In the other direction, at 100G, with ave 512B your max is 20-25Mpps, in any case 40G is fine and you save your firms $ on the 100G. Actually line capture at these speeds and thruputs have been doable for a good while now, mutli-Tbps is the challenge now. Capture/Filtering and processing are actually two different but interdependent things, as you know speed/thruput are normally qualified in terms of packet sizes and app, nic profile:sustained/peak, For capture, L3 ACLs segment work etc its just fixed and static filtering; different to triggers which need higher state protocol/flow processing, db lookups, markov etc, alot more smarts and CPU CS this lowers the upper limit. CPU stats will be very different.

regards

Brian
---------------------------- Original Message ----------------------------
Subject: Re: [nznog] Affordable TICSA LI Solution and Support
From: Joel Wir��mu Pauling <joel@aenertia.net>
Date: Sat, June 30, 2018 12:40 pm
To: brianparsons@subpico.com
Cc: "Shane Alcock" <shane.alcock@waikato.ac.nz>
"nznog" <nznog@list.waikato.ac.nz>
--------------------------------------------------------------------------

> You can achieve 55MPS from the latest generation of Smartnics - enough to
> do 100GBits ports, with Layer3 ACLs ; with almost no CPU overhead. This is
> running pretty much Mainline Linux (with Vendor Specific Smartnic offloads
> i.e ASAP2 (mellanox), or Netronome). You then obviously need to dump it
> somewhere but with some NVMEE Cards on x8 Bus you can do that too - before
> shuffling it to slow and cheap archival.
>
> So ; no you don't need any real hardware smarts to do this today, there is
> off the shelf bits you can add to off the shelf x86 hardware. Just a little
> planning of your Capture box hardware and knowing what NIC's to use.
>
> -Joel
>
> On 30 June 2018 at 11:39, <brianparsons@subpico.com> wrote:
>
>> Hi Shane,
>>
>> I appreciate your response and it sounds like you and the OpenLI team
>> making great progress, i wish the project and deployments all the best.
>> Writing packet capture isnt hard and anyone with a compiler can do it. Of
>> course a little coding b/gnd helps and depending on what your trying to do,
>> the level of hardware knowhow you need.
>>
>> Take 100% packet capture - 'shouldnt' be a problem at a constant 1 pps,
>> but it will be if you take 2 s to process. Lets say you optimize hw down .5
>> s with a DAG* (i worked their native's ~2004)* your still not going to
>> make it. Timing values are too ilustrate my point and my take away from you
>> response is OpenLI is basically targeting the general usecase here in NZ
>> where say GE or 10G ~50/60 pk. % link utilization is still pretty OK and
>> fit-for-purpose. I assume here your OpenLI app (NWO->IAF/MF
>> ASN.1Enc/MD->HI) is well optimized from your excellent experience too. One
>> suggestion maybe to bench it with XIA/Spirent STC for thruput vs
>> line-rate/packet sizes vs expected from NOC. I dont use cap libraries or
>> anything like that myself, my arch/interface is very different in exo/no
>> net_dev, zero intr,dis flow handling depending on the ICs i got on hand (i
>> have a french foreign legion policy for hardware - survive the bench and
>> your in ). LI started here in NZ but was taken to another level in the US
>> with massive linerates and volumes - vetting starts with corner case
>> burntests, sinking IPGs <9.7ns ..<.97ns (100G+) with 1 Target frame - and
>> if you can catchit send try to send that 1 frame, only after that you get
>> to do something usefull and process it. Sounds easy as you can appreciate
>> back in the day with north/south PCIE this was a massive feat, no SoCs
>> root/pcie qpi we have today.
>>
>> Obviously this level of performance is bit OTT for our general LI small
>> ISP market, but data volumes aint gonna go down - cheers for the heads up.
>> Again wish you and the OpenLI project team much success.
>>
>> regards
>>
>> Brian Parsons BE(E&EE) MIEE | CEO TFE | LI TICSA SIGINT
>>
>> ---------------------------- Original Message ----------------------------
>> Subject: Re: [nznog] Affordable TICSA LI Solution and Support
>>
From: "Shane Alcock" <shane.alcock@waikato.ac.nz>
>> Date: Fri, June 29, 2018 2:38 pm
>> To: brianparsons@subpico.com
>> Cc: nznog@list.waikato.ac.nz
>> --------------------------------------------------------------------------
>>
>> > Hi Brian, all,
>> >
>> > As the primary developer on the OpenLI project, I've been asked to
>> address
>> > some points that you've raised and correct any misconceptions people
>> might
>> > have about OpenLI as a result of this thread.
>> >
>> > For anyone concerned about a possible lack of experience in writing
>> packet
>> > capture software, most of my 14 years working here at the WAND network
>> > research group has been dedicated to both writing and running software
>> for
>> > high speed passive capture and analysis projects. I am one of the primary
>> > authors and maintainers of libtrace [1] [2], which is widely used both in
>> > academia and industry (as well as the OpenLI software itself). Libtrace
>> > supports a variety of capture methods and hardware, including DAG and
>> DPDK,
>> > and has a parallel API for writing multi-threaded packet capture
>> > applications, so I have a reasonable understanding of what is required of
>> > both hardware and software to do high-rate packet capture and encoding.
>> >
>> > OpenLI is not offering a "complete" solution to your TICSA problems. We
>> > provide software that can capture packets and convert them into ETSI
>> > records, wrapped in a centralised provisioning system for managing
>> > intercepts, along with a mediation function that will push the resulting
>> > records to the appropriate agency. Integration of OpenLI into your
>> network
>> > is a matter for the operator; however, we have been working closely with
>> > the organisations that responded to our initial requests for funding to
>> get
>> > their deployments up and running.
>> >
>> > Longer term, we are starting to have conversations about what support
>> > models we want to provide in the future, especially after the feedback I
>> > received at the NZIX AGM last night, so watch this space. WAND has a long
>> > history of maintaining and continually improving any open-source software
>> > that it has released to the public [3] even when it is no longer
>> > specifically funded by our research income (libtrace being a prime
>> > example), so if OpenLI is being used then you can anticipate that WAND
>> will
>> > continue to look after it.
>> >
>> > Finally, the core software components of OpenLI are getting close to
>> > complete. We have provisioning, mediation, multi-threaded and
>> > multi-input-interface distributable collectors. We have support for voice
>> > intercept (SIP + RTP + RTCP) and support for standard IP intercept
>> (RADIUS
>> > + IPv4), with IPv6 and static IP range intercepts not too far away. Most
>> of
>> > the upcoming effort is focused on getting test deployments working within
>> > our partner networks, throwing "real" traffic at them and squashing any
>> > issues that come up. At this stage, it's looking like another month or
>> two
>> > of work to finish the original set of requirements.
>> >
>> > Any other questions about the OpenLI project, feel free to get in touch
>> > with me directly.
>> >
>> > [1] S.Alcock, P. Lorier and R. Nelson, "Libtrace: a packet capture and
>> > analysis library", https://dl.acm.org/citation.cfm?id=2185382
>> > [2] Libtrace Team, "libtrace: C library for working with network packet
>> > traces", https://github.com/LibtraceTeam/libtrace
>> > [3] WAND network research group on GitHub, https://github.com/wanduow
>> >
>> > Thanks,
>> >
>> > Shane
>> >
>> > On Wed, Jun 27, 2018 at 11:49 PM, <brianparsons@subpico.com> wrote:
>> >
>> >> Hi Nathan,
>> >>
>> >> We dont work with the NSA, and you dont need them to decrypt mobile data
>> >> there are many keys used in a heirachy depending on the pairing mobile
>> >> nodes eNB/UE/MME/HSS etc, you just need to be on the right interface to
>> >> catch and process the exchange Control/User Plane (keys vary) is trivial
>> >> for LI vendors like myself who carry key derivation/enc/dec logic and
>> >> protocol sequencing to handle such an exchange. eg UE/HSS:CK,IK - a well
>> >> known mapping is your USIM/AuC:K. As far as knocking on doors, yes
>> >> absolutely probably me, i sell and support what I build and proud of
>> it, but
>> >> NSA lol, but we sell more off shore - why is that?
>> >>
>> >> My post was only intended to inform of a local option that could make a
>> >> material difference to operators trying to meet their obligations under
>> the
>> >> ACT.
>> >>
>> >> As an LI provider i have met a couple of your members stung badly after
>> >> signing offshore and we couldve helped them, also i was encouraged by
>> the
>> >> LEAs that speak at your conference - they obviously cant directly
>> endorse
>> >> but the spirit and intention of my post was not to offend just inform.
>> TFE
>> >> like most LI vendors do not advertise or market, and there are only a
>> >> handfull of us OEMs that build LI solutions the rest are reseller middle
>> >> men.
>> >>
>> >> As far as OpenLI free it is your developing, thats great but till then
>> >> compliance is real, but we wish you all the best. 2050 is when you'l
>> have
>> >> enough field experience in CU/FIbre/hardware with ASN.1 + ETSI/CALEA at
>> >> working at single thread code level. Lets say you do cobble something
>> >> together before then - once you can multithread your kernel (tuplet
>> flows)
>> >> linerate 200g (64B) stream + LI let me know, sorry lets make it easy -
>> >> start with GE @ 1.44 Mpps + LI and ASN enc over HI => garauntee you a
>> DoS
>> >> all day and dont even think about SLAs. But good luck with that, until
>> then
>> >> TICSA 2017
>> >>
>> >> The point is LI is not just about about Software. Hardware, firmware and
>> >> real world field experience is required - how do you handle FEXT/NEXT on
>> >> upstream active/passive TAPs can you run in rx only given the
>> requirements
>> >> and restrictions on spurious noise and traffic, there are many tech and
>> >> capability requirements of the probe/mf which ETSI Parts are you going
>> to
>> >> do first?. HA/asymmetric network deployments (non-collo paths), how will
>> >> you handle this situ? and TIER 4 support on commercial fabrics (Open
>> >> generally means zip support - so i geuss nothing is realtime). If you do
>> >> all this how will you be able to demonstrate at any point in time the
>> two
>> >> core aspects of TICSA 'format usability' and 'nw readiness'.
>> >>
>> >> We do LI not 'network security', ie. we build the 'device' and MF as
>> >> descirbed in the NZ Search and Surveillance Act 2012 and NZ TICSA 2017.
>> >>
>> >> regards
>> >>
>> >> Brian Parsons BE(E&EE) MIEE | CEO TFE | LI TICSA SIGINT
>> >>
>> >>
>> >>
>> >> cheers
>> >>
>> >>
>> >> ---------------------------- Original Message
>> ----------------------------
>> >> Subject: Re: [nznog] Affordable TICSA LI Solution and Support
>> >>
>>
From: "Nathan Ward" <nznog@daork.net>
>> >> Date: Wed, June 27, 2018 10:06 pm
>> >> To: brianparsons@subpico.com
>> >> Cc: nznog@list.waikato.ac.nz
>> >> ------------------------------------------------------------
>> --------------
>> >>
>> >> >
>> >> >> On 27/06/2018, at 11:45 AM, brianparsons@subpico.com wrote:
>> >> >>
>> >> >> We are Telecom Forensics, an OEM of LI solutions based on the Kapiti
>> >> Coast.
>> >> >>
>> >> >> Spam spam spam
>> >> >
>> >> > Are you the chap from Kapiti who a few years ago was knocking about
>> >> claiming that the NSA had given him some sort of secret codes to
>> magically
>> >> decrypt all mobile data, or was that someone else?
>> >> >
>> >> > Not too sure why you’d be pushing this here, when there’s a community
>> >> effort to develop something open source and free.
>> >>
>> >> I’m looking forward to where the OpenLI project goes, and I’d suggest
>> that
>> >> anyone looking for LI tools look towards the OpenLI project, which has
>> been
>> >> discussed on this before, and at the NZNOG conference, and I believe
>> will
>> >> be discussed tomorrow at the NZIX AGM.
>> >> > ps. It’s free, but, go support it with some $, it’ll still be cheaper
>> >> than commercial solutions.
>> >> >
>> >> > NZ: Highest per capita ETSI compliant LI implementors in the world.
>> >> >
>> >> > How long before Fincham pops up saying “oh also there’s an OpenLI
>> >> project you should look at”? Place your bets.
>> >> >
>> >> > Looking forward to laughing about this one over drinks tomorrow night.
>> >> >
>> >> > --
>> >> > Nathan Ward
>> >> >
>> >> >
>> >>
>> >> _______________________________________________
>> >> NZNOG mailing list
>> >> NZNOG@list.waikato.ac.nz
>> >> https://list.waikato.ac.nz/mailman/listinfo/nznog
>> >>
>> >>
>> >
>>
>>
>> _______________________________________________
>> NZNOG mailing list
>> NZNOG@list.waikato.ac.nz
>> https://list.waikato.ac.nz/mailman/listinfo/nznog
>>
>>
>