Our 5 Layer Probe Architecture
To truly provide organisations with the power to see all, we have spent ten years developing a multi-layered architecture that blends hardware, firmware, operating system, management tools and applications into a single coherent whole. We call it our 5 layer Probe Architecture.
To observe an entire network, it's vital that your chosen technology can support multiple Probes, working at varying speeds (depending on where they are deployed in the network) on any number of interfaces.
The network of Probes must be able to work together - forming what we call a packet capture fabric, capable of capturing all the network traffic and making it available to any application that might need it.
By separating capture from analysis, traffic can be used (and re-used) wherever and whenever it's needed - at the network edge (for local decision making) or aggregated and correlated centrally for sophisticated analysis and decision making based on multiple sources of data.
The illustration and explanations below illustrates how we enable the power to see all.
Layer One - Core
In layer 1, which is enabled through our hardware, we:
Dechannelise: SONET dechannelisation is a hardware-optimised process. By performing the function in hardware there is no impact on the CPU. This improves the performance of the system.
Timestamp: Accurate timestamping is becoming more critical as networks carry more and more latency-sensitive traffic. Highly accurate and synchronised timestamps enable you to correlate events right across your network e.g. DDOS attacks. Once every packet has an accurate time stamp you can process packets within applications in non-real time - which significantly simplifies the development of your applications.
Our timestamping technology provides precision time stamps to within 7 nanoseconds and system accuracy to within + or - 100 nanoseconds. Few other systems come even close to matching this level of precision and accuracy - which is why our Probes are selected by many financial institutions to measure network latency.
Capture to RAM: Leveraging our proprietary DAG® technology, we use DMA to offload the CPU and catch every packet - putting them directly into RAM and thus avoiding the interrupt issue. This leaves the CPUs fully available to process and analyse packets.
Write to disk: Getting high speed traffic onto disk without dropping a single packet is a significant challenge. It's a problem that can only be solved with the right mix of hardware and software. We've spent years getting this recipe right.
Our storage management system stores every packet in a rotating file system that can be scaled up to 32TB on a single Probe, giving you days or even weeks of traffic storage (up to 3 day's of storage at 100% loaded 1Gb/s throughput). Stored data gives you the ability to do back-in-time forensic analysis of events - which adds a whole new dimension to applications such as IDS.
Layer Two - Formatting
In Layer 2, which is enabled through our OSm software we let you format the traffic. This is important as it enables you to change the format of the traffic to meet the needs of layer 3 (the Transport layer).
Filter: Layer 2 capability lets applications (layer 4) perform filtering on the captured data every time a session is established. We support filtering using industry standard tcpdump filter syntax. This allows applications to be fed a cleaner set of data, eliminates unnecessary session traffic, improves security and application performance. When the source of the data is the on-board storage system you can specify a time range and filter traffic accordingly.
Convert: For SONET traffic, we remove the SONET information from the frame and replace it with an Ethernet header, which enables Ethernet-only applications to support SONET without changing the application.
Load balance: At the application layer (layer 4) we enable you to allocate the optimal number of CPU cores to each of your applications. When establishing sessions to these applications our user-programmable load balancing capability ensures the right amount of traffic gets to each core for each application to perform at its peak.
Summarise: By summarising the traffic at a low level we are able to provide up to per-packet netflow record generation enabling netflow-based apps such as "ntop" to receive highly granular information about network traffic.
Layer Three - Transport
In Layer 3, which is enabled through our OSm software, we:
Read from disk: Once the traffic is written to disk, Layer 3 lets you access, manipulate and extract the stored data. The rich range of Level 2 filters makes this process very efficient - even with huge trace files.
Forward packets to internal applications: Layer 3 enables you to connect any internal data source (such as a monitoring port or a capture file) to any of the applications running on the Probe. In addition, the forwarding capability lets you filter subsets of the stored traffic and write it back to disk in a separate file, thereby extracting a subset of the traffic for more permanent archiving or subsequent analysis by consuming applications.
Forward packets to external applications: Layer 3 enables you to connect any internal data source to applications residing anywhere on the network using the onboard NIC card or the transmit capability of the DAG cards in the Probe.
Replicate: Each data source can be replicated and sent to multiple applications simultaneously. When combined with our powerful filtering capability this lets you reuse overlapping sub-sets of traffic simultaneously.
Layer Four - Applications
Layer 4 runs the Endace and customer applications that require captured packets; applications such as Cyber Security Monitoring, Low Latency Monitoring, and Analytics. Our virtual machine capability enables any Compatible Application running on Probes to access shared captured network traffic. We include a suite of management tools which enable you to control, monitor, maintain and manage your packet capture fabric.
Layer Five - Eventing
In Layer 5 we facilitate communication between applications which enables your IDS to pass alerts and event workflow information to your Analytics tools, or your Latency Measurement application to pass event information to your Analytics tool. Events can be configured to automatically trigger specific actions such as changing traffic filters, or isolating and archiving a specific subset of traffic for later inspection. This dramatically enhancing the performance of the Applications and reducing your time to resolution on important issues.
Probe Management Console
The Probe Management Console provides powerful command and control for your individual Probes. With a single Probe deployment the Management Console runs as a software Application on the Probe itself. For multi-Probe deployments, a fabric manager, running on its own dedicated server (our Central Management Server) provides management of your entire fabric of Probes from a central location.