Network Monitoring's 'Inconvenient Truth'
"If you don't have every packet, then any analysis you do is pointless"
Quote from an Endace customer responsible for monitoring one of the world's largest financial services networks.
Most network monitoring tools - such as intrusion detection or prevention (IDS / IPS), quality of service measurement, analytics and other applications - rely in some way on captured network traffic data. It's critical these systems have visibility of every single packet on the wire otherwise they can't function reliably. A partial view of traffic is often worse than no visibility at all.
And that's where the problem lies... as network speeds and loads increase, the packet-capture technology underlying most of these monitoring solutions is simply not capturing everything.
Most current network monitoring solutions (and appliances) rely on standard network interface cards (NICs) to capture network packets off the wire. While this approach is fine at relatively low speeds, as line speeds start to increase above 1Gb/s, standard NICs simply can't handle the load and start to drop packets.
So how serious is the issue, really?
Our research suggests software-based systems can only be relied on to capture 100% of packets up to around 2Gb/s before they start to drop packets. Consequently many of the packet-based security, monitoring and measurement systems deployed across the globe today could be ‘missing’ anywhere between 25% and 40% of the traffic on their networks (depending on load and conditions). And often neither the organisations nor their vendors even know it is happening.
NIC versus DAG
The video below demonstrates the problem with NIC-based packet capture and how DAG overcomes the interrupt storm.
What's causing this problem?
Put simply, standard NICs are not purpose built for high-speed packet capture; they're designed to send and receive packets at moderate rates sufficient for the services or clients running on a single computer.
Every packet that is captured by a NIC requires a series of CPU interrupts to inform the CPU of the arrival and availability of the packet. The maximum speed at which traffic can be captured reliably is therefore directly related to the processor's clock speed.
NIC-based packet-capture systems are fine as long as the CPU's clock speed is faster than the interrupt rate the NIC requires that CPU to service. But if the CPU isn't fast enough, packets are dropped as the CPU becomes inundated by an interrupt storm.
With Intel and AMD's move to a multi-core CPU strategy, CPU clock speeds have effectively maxed out at around 4.0GHz, putting a cap on the maximum capture rate CPUs can support with traditional interrupt-driven, NIC-based packet capture.
The solution is DAG® - 100% capture, zero CPU load
Endace's ingenious DAG® I/O technology removes this bottleneck and enables DAG-powered systems to capture 100% of network traffic to 40Gb/s and beyond.
It does this by offloading packet-capture functionality to DAG hardware, thereby freeing up CPUs from interrupts and enabling them to be dedicated to running network monitoring and analysis applications instead.
DAG technology is at the heart of the Endace Platform powering our network monitoring and recording Endace Systems.
Learn more about the Genius of DAG.
Why 10Gb/s changes everything
In the last few years the world has changed. Speeds and data volumes have grown exponentially and continue to do so. This has been driven by the rapid adoption of cloud computing architectures and network-centric technologies such as VoIP and video. In order to keep pace with these changes organisations are either already upgrading to 10Gb/s networks or considering an upgrade. And even higher speeds are just around the corner.
Even at 1Gb/s, organisations are finding their existing tools are not able to cope with increasing speeds and loads, and they know those systems won't survive a move to 10Gb/s. So as they plan the move to 10Gb/s and beyond, organisations are also realising they need to replace their existing monitoring, security and analysis infrastructures.
In addition, other environmental factors - such as regulatory and compliance requirements like PCIDSS and Sarbanes-Oxley - mean accurately capturing all network data is increasingly becoming a mandatory requirement as well as a prerequisite for effectively monitoring, analysing and securing the network.
The shift from 1Gb/s networking to 10Gb/s is forcing organisations to re-evaluate how they monitor and measure their networks. In effect, the transition is a natural inflexion point where organisations have to choose between persisting with today's ‘point solution’ based approach, which is proving to be flawed on a number of different levels, or making a strategic investment in a Monitoring and Recording Fabric which any compatible software application can be run on top of. They're realising the only solution that will scale to 10Gb/s and beyond is one which - like our Endace Platform - separates the network traffic capture from the applications that consume that traffic.
The need for a new approach
Our experience working with telecommunications, government agency and financial services running some of the world's largest and most complex networks has shown us the only workable solution to providing 100% accurate network monitoring and recording at speeds above 1Gb/s is to separate the traffic capture from the monitoring and analysis functions.
Our Endace Platform provides this separation. Network packets are captured in hardware using DAG® I/O technology. Captured traffic can then be recorded and/or made available to all network monitoring, security, or other network applications requiring it.
The approach enables our customers to deploy a 'fabric' of connected Endace Systems across their network wherever monitoring or recording is required. This fabric can be centrally managed, and captured traffic from across the network can be used by a wide range of network monitoring, analysis, and measurement applications - either running on Endace Systems (in our Application Dock hosting environment) or on other machines.
This approach delivers a flexible, scalable infrastructure that can:
- Scale as network speeds increase to 10Gb/s and beyond
- Allow all network monitoring and analysis applications - such as security, analysis, traffic recording and quality of service and performance measurement tools - to leverage a single source of authoritative traffic data
- Enable traffic to be captured and stored to satisfy auditing and compliance requirements
- Provide the flexibility to choose the applications that best suit the needs of the customer - including in-house custom applications
- Allow network applications to be integrated - speeding the identification, investigation and resolution of network issues
- Reduce the cost of purchasing, deploying, managing and upgrading network monitoring infrastructure.