Analyzing 150+ Million Network Flows in Real-Time with nProbe and Elastic Sketch

How we efficiently processed heavy NetFlow data in C++ to extract its volumetric characteristics, at Georgia Tech

Photo by Markus Spiske on Unsplash

The Project

A TCP packet structure

The Challenge

The Pipeline

Sample output of flow lines from nProbe. IPs and AS numbers are fictitious.
ACM Sigcomm: The original Elastic Sketch Paper
JSON output with Elastic Sketch statistics — fictitious IPs (“heavy hitters”) and AS numbers (“keys”)
Source: Georgia Tech website

Special Thanks!

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store