At one point or another we've all watched Trek. From Scotty yelling "I'm giving it all she's got cap'n" to Picard's "Make it so." We love the mix of humanity & technology. Like Scotty, most techies get excited by the thought of pushing their own skills & systems to their very limits. Is there a place though in IT for pushing the tools we use to their limit, and can we overload our systems? I believe there is a place, and when it's appropriate we should push our systems, and ourselves for that matter, to their limits so you can know for certain what we can truly expect. Computers these days have many safety systems to prevent us from frying them. Let's face it, any techie worth his salt has the smell of burnt Silicon stored in the neurons of their olfactory nerve, so let's try and avoid that. Yes, some of these systems can be bypassed, but today that's not what I'm suggesting.

So how do we perform network analytics or problem determination on 10GbE links operating at near Warp speed? The answer is simple, get the right hardware, and software. You wouldn't send a crew venturing off into deep space without at least a Warp 2 capable ship, so you shouldn't expect to capture packets at high data rates on a 10GbE link with built in hardware and Libpcap. Heck even my little $69 Raspberry Pi can be easily configured with Libpcap & Wireshark to analyze traffic.  No, to do real 10GbE performance monitoring and management you'll need the right network interface card, and software. Let's assume you're on a tight budget, and most folks are, my recommendation is an Emulex One Connect Network Xceleration (NX) adapter (P/N OCe12101DM-SNF2) which offers a single wire-rate 10GbE port WITH Myricom's FastStack Sniffer10G included, all for under $690 list price. Now you might ask yourself, why not use something else? True there are other adapters and other solutions, although I think you'd be hard pressed to find another lossless wire-rate solution for 10GbE for one for under $700.

So what's so special about this Emulex adapter and FastStack Sniffer10G? Several things: a transparent access model, lossless wire-rate 10GbE packet capture, multiple shared parallel memory buffers, two buffer creation strategies, adjustable hashing algorithm, user space or kernel mode, wire-rate injection, and a number of useful tools and sample code. Let's take a moment and look into each of these.

By a transparent access model, we simply mean a libpcap replacement library on Linux, no coding required! This replacement library supports multiple in memory buffers & strategies that are totally  transparent to your libpcap compliant application. Sniffer10G employs the use of environment variables for configuring in memory queues. This allows Sniffer10G to easily plug into programs like: Snort, Suricata, BRO-IDS, WireShark, TCPDump, etc…

Lossless 10GbE wire-rate packet capture requires three things: hardware designed for gargantuan packet rates, software architected to bypass the OS and quickly move packets into user space, and a tight marriage between the two. This Emulex adapter uses a processor made by Myricom capable of sustaining a packet handling rate of approximately 16 million packets per second (Mpps), combined send & receive.   A 10GbE link fully loaded with the smallest possible packets, 64 bytes, can sustain a wire-rate of 14.88Mpps. Often folks interested in packet capture typically don't retransmit packets on the same link they're capturing on so a ceiling of 16Mpps is a perfect match to catch 14.88Mpps of worst case inbound traffic.

Multiple shared memory buffers is a way to spread the work. FastStack Sniffer10G supports two buffer models: cloned & flow hashed. With cloning Sniffer10G will create multiple in-memory copies of the same in-bound traffic. This enables you to have multiple different applications running that each have their own independent in-memory copy of the packets being captured. At wire-rate Sniffer10G should be able to sustain up to three clones for a total of four in-memory queues of the same identical captured traffic. Sniffer10G can provide more clones, but after three at wire-rate we may begin to drop packets. Cloning brings up a significant problem though, each queue could potentially receive a new packet every 67 nanoseconds so your code that is working these queues has to be extremely tight.  Flow hashing solves this problem. With flow hashing Sniffer10G spreads traffic between up to 32 different queues (each queue should be bound to it's own core). This gives your code working the queue in a worst/best case up to two full microsecond (best meaning that the packet distribution is even across 32 buffers) to process inbound packets.

So what about flow hashing. Well, flow hashing is non-deterministic, meaning that it's not a simple round robin method that yields easily predictable results because frankly, most folks interested in packet capture really don't want this type of packet distribution between queues. Flow hashing allows you to maintain network flow affinity meaning that the same queue will always receive packets from the same source going to the same destination/port etc… so that when you do deep packet inspection for things that span multiple packets you'll have access to all those packets in the same queue that make up that traffic flow. Pretty cool huh? So flow hashing is handled by default by using the IP/TCP/UDP source and destination addresses. This can be altered by setting the Receive Side Scaling (RSS) flags prior to launching your code using an environment variable (SNF_RSS_FLAGS).

The remaining cool features, such as an optional kernel space Sniffer10G driver, wire-rate injection & samples will be addressed in part two of this series… Stay tuned for Part 2.

In the 1970's Intel brought us the CPU. During the 1990s we saw the evolution of graphics processors with Nvidia popularizing the term GPU in 1999. Now we're witnessing the dawning of the Network Processing Unit, NPU. Much like it's multiple core graphics cousins, the NPU is a parallel processing architecture, but it has been tuned for manipulating network traffic.

This market is still rapidly evolving, and several approaches are progressing in parallel paths. On one end we have Tilera, founded in 2005 with engineers from MIT and Broadcom, their approach is a many-core one, where the cores are interconnected by an on chip network to each other & substantial system I/O. In the middle we have Myricom founded in 1994 from Caltech, their architecture is multi-core where a two busses are used by an intimate collection of cores to share on chip memory, I/O & multiple network devices which are more tightly coupled to the wire. Finally, there is the FPGA approach advocated by Napatech, eNdace, and Solarflare, here the focus is on using the FPGAs to provide well defined filtering and packet processing. It's not exactly a processor, but an interesting transitional step. Each of these approaches has value, but over the coming decade the market will decide a winner, it always does. First let's take a deeper dive into the raw hardware.

Tilera attacks the NPU much more like a GPU than the other two strategies. They pack up to 72 cores onto a single chip. Then Tilera leverages a network mesh architecture to connect the cores to each other, and to multiple: DDR3 controllers, Ethernet SerDes (ports), PCIe busses, and a pair of MiCA encryption acceleration engines. On the high end these 64-bit cores are clocked at 1-1.2Ghz, and typically each has both an L1 (32KB) & L2 (256KB) cache along with three execution pipelines. The mesh fabric in aggregate has over 100Tbps of bandwidth, and utilizes a non-blocking cut-through routing with 1 clock cycle per hop. They then utilize four independent 72-bit DDR3 controllers for a total addressable real memory capacity of 1TB, and an access speed of 1,866 MT/s. On the ethernet side of this high end chip Tilera has eight 10GbE XAUI interfaces and 32 SGMII ports for legacy 10/100/1000 Ethernet. On the PCI Express side they utilize six integrated PCIe controllers each with four lanes, providing up to 96Gbps of throughput. Finally, there are two MiCA acceleration engines, these provide for encryption support for six popular protocols along with a public key accelerator supporting four additional protocols. This is an awesome amount of hardware packed into a single 45mm square chip. The one page data sheet doesn't cover the power it requires, but it's likely in the 60-80W range. How you might ask could I come to this conclusion? Tilera produces a quad-port 10GbE PCIe card utilizing the 36 core version of this chip, and it consumes 50W of power, it requires a secondary ATX power connector along with an active fan/heat sink.

As mentioned we have Myricom in the middle. Since 1994 Myricom has been producing a family of single core network processors for their line of programmable network adapters. In late 2005 they introduced their first 10GbE processor. Since then they've turned the crank once in early 2008 to produce their current Lanai Z8E. As mentioned this is a single core RISC processor clocked at 333+ Mhz with 2MB of on chip SRAM, two 10GbE XAUI interfaces, and single 8-lane PCIe controller supporting 8x, 4x, 1x modes. This summer it is expected that Myricom will deliver their first multi-core architecture chip, but no details have yet to be officially released. We do know though that it will be more similar to Intel's multi-core architecture than Tilera's many-core interconnected via a mesh network approach. We also know they're going beyond XAUI to more tightly connect to the wire to further reduce network latency, as this has been a key focus of theirs the past few years.

Finally, on the hardware side we have the FPGA crowd headed up by Napatech, eNdace & Solarflare. Solarflare is the newest entrant in this space, but they appear to have the clearest vision of where this technology can really go. While Napatech & eNdace, now an Emulex owned company, have been focused on packet capture & network analytics Solarflare is focused on the High Frequency Trading (HFT) market. Solarflare has taken their low latency 10GbE ASIC and back ended it with a powerful FPGA. Their intent is to keep fundamental packet processing decisions on the network adapter. Concepts like intelligent filtering and payload normalization. The filtering is based on some pattern of bytes within not only the header, but also potentially the payload. If a packet makes it past the filter code then they are working to normalize the payload contents so trade data coming from multiple source will all be formatted identically when passed up to the user space application.

So what challenges lie ahead for those interested in squeezing the most out of these NPUs? Programatic ones. Tilera's many-core approach, and the wealth of hardware devices on the die are  compelling, but frankly programming it is going to be like herding cats. The programmer will have to attach a pair of cores to each 10GbE port, one to handle receive and the other transmit. If we take the HFT problem described above that Solarflare is working on and use this as an example then they should utilize three additional layers of cores behind this. The first layer to handle filtering, say four cores to spread the filtering workload out. Each cores would then be inspecting key strings within payloads for applicable securities symbols. The second layer would be data normalization, here another four cores would line up directly with the first layer. These cores would then be altering packet payloads to conform to a predefined standard structure. The third layer would be a single core to collect all the packets from the second layer and steer them to the proper user space memory locations via a PCIe device. So in this simple feed handler example we've used two cores for the 10GbE link, one to interface with the PCIe and eight to handle feed processing, for a total of 11. For a single 10GbE link that's a lot of cats to herd. On the opposite side we have the FPGA guys, and we've all known for years that programming FPGAs is non-trivial. It's gotten better, but it's still more magic than engineering. Myricom being in the middle is still the wildcard, as they've not yet said publicly how open their processing architecture will be.

So the NPUs are here, how will you leverage them to yield a competitive advantage for your enterprise?

To all those High Frequency Trading (HFT) shops out there that believe you need Linux to secure the lowest latency, well that's no longer the case. Myricom & Emulex now bring you FastStack DBL for Windows Version 2.2 which boasts an impressive 4.9 microseconds for a half round trip. That's a round trip divided by two which represents a send plus a receive. Oh, and that's out of the box using a simple transparent mode wrapper (dblrun). If you want you can also code to the DBL API which now works for both UDP and TCP channels.

In April of 2011 at the HPC Linux for Wall Street show Myricom announced DBL 2.0 which included TCP, and Windows support. Since then we've learned a few tricks and have further reduced the latency in transparent mode to 4.9us. Initially to achieve out of the box transparent mode we leveraged a technique called Layered Service Provider (LSP). We realized though that this trick cost us 500ns, and after considerably more research and coding have developed a non-LSP solution that saves the 500ns and has some additional performance improvements.

If you need additional flexibility, or require that DBL be in your code path along with other 3rd party modules, then DBL also offers a rich Application Programming Interface (API).

Furthermore FastStack DBL now supports both UDP and TCP in transparent mode on both 32 bit and 64 bit run time environments in both transparent & API mode. So if you're an HFT shop that uses Windows in production, or has been considering using it, now you've no longer got an excuse not to give it a try. All you've got to loose is latency.

So I've gotten a little fancy and moved to Blogger for posting all my blog entries on both this site and Cyber Weapons. The fancy part is that I've removed the traditional inflexible component used to manage the 10GbE.net blog and replaced it with a PHP widget that drains the content dynamically from my 10GbE Blogger account. Now I can post once and it automatically shows up in both places. I then migrated all my old relevant posts from 10GbE.net into the 10GbE Blogger site. It's either cool, or I've confused you.

Sorry there's been nothing new this week yet on the 10GbE front, I was at RSA in San Francisco all week, and focused on Cyber Security.  I'll be posting something shortly on this topic, but I'm still completing the research so stay tuned, and thanks for reading.

In the 1990s the federal government pulled a page from da Vinci's book, and got behind a software acquisition strategy called COTS (Commercial Off The Shelf) expecting that it would dramatically reduce software development & deployment costs while improving overall quality. It didn't take long before this approach began to be applied to application specific custom computing solutions used throughout key government agencies. Today many of these customers rely on systems built using a Lego like approach to building systems from COTS parts readily available to the general public.

This article is also published on 10GbE.net.

# wget http://www.myricom.com/pub/10g-tools/myri-tools-linux.tar.gz 

# gunzip -c myri-tools-linux.tar.gz | tar xvf - 

# cd myri-tools-1.26-linux/bin

# mdio_rw -b 0 | grep RX 

RX power = 0.6199 mW

myri_info - Returns a number of hardware statistics on the adapter & PCIe bus. 

myri_pcie_conf - This confirms the PCIe payload sizes of the adapter and the system chipsets.

myrige-test.sh - Performs an adapter loopback self test.

This series of articles was also posted on 10GbE.net. 

How does this impact the video servers on the other side of the network providing your content? Well if they're still using Gigabit (GbE) links, and only serving up SD streams they can support about 150 customers per network port.  When they move to 10GbE that jumps up to 1960 streams (10GbE links are a bit more efficient) per port.  Most of the Content Delivery Networks (CDNs) have already moved to 10GbE for this initial boost.  It becomes even more affordable when they utilize dual port cards so they can provide nearly 4,000 customers service from a single server.