Shoring Up the Castle: Using Machine Learning and Data Visualization to Improve Network Security

Todd Howe, Linux Systems Engineer

Traditionally, specialists in information security divide into two opposing camps — the Red Team and the Blue Team. The Red Team’s role is to simulate barbarians at the gate: probing network defences and acting like attackers in order to expose and report weaknesses. Blue Team is left to pick up the pieces, plug the holes, and plan for the next intrusion attempt in a continuing process to keep a step ahead of the bad guys.

Spare a thought for the defenders, watching from the battlements — sometimes it seems like the Red Team gets all the best toys! Toolkits honed and adapted from the latest exploits found in the wilder regions of the internet are their siege engines and battering rams. Worse yet, attackers employing stealth techniques can be inside the castle’s treasure room while the defenders are still watching the walls. The situation for the defenders can seem dire as they scramble to catch up with the state of the art. Sometimes all that’s needed is a more systematic approach to security fundamentals. Sometimes, new technology is the way forward. Like, say, robots. An army of robots. With laser beams.

Happily, there has been progress on making this happen. It may not be robots, exactly, but in the last few years machine learning has been advanced as a way to move the state of the art beyond simple walls and moats by introducing better visibility into networks. Why not, to stretch the metaphor, rig the castle with tripwires and cameras? This has traditionally been the role of Intrusion Detection Systems. IDS have been around in some form for decades but even today IDS relies heavily on static signature and rule-based approaches, both of which may be evaded by an advanced attacker. And when they work, IDS tend to generate a deluge of information which security analysts sift through in order to separate real alerts from false alarms.

Inevitably, analysts will succumb to alert fatigue and some alarms will be missed. This is where Machine Learning comes in by offering a way to pre-sort alerts by dynamically profiling network and endpoint traffic by borrowing techniques from the field of data science. These techniques can learn when traffic patterns look ‘right’ based on previous experience. Paired with the advice of experts in data visualization like Stephen Few they offer better ways to display these patterns to a human mind readily overwhelmed by too much information. The effective display of a machine-curated set of alert information on a large screen for ease of interpretation (known as a Security Information and Event Management or SIEM dashboard) holds the promise of cutting costs and improving security by freeing staff to do things humans are good at, like visual pattern recognition instead of reading lists.

While the attacking hordes belonging to the Red Team is often portrayed as having more fun, it’s an equally interesting and vital time to be on the Blue Team as defence adapts with exciting new tools and techniques. Aided by the boundless potential of machine learning, Blue Team may just yet beat back the Red Team from making off with the keys to the castle.