The connected world, massive data explosion, cloud computing, and the mounting number of clients to consider all combine to make cyber security a defining technology of our time. It has replaced growth, expense control, and profitability as the number one concern keeping the C-suite and boards awake at night. Each breach reported above the fold seems to one-up the previous breach. According to Gartner, Global IT security spending will reach $76.9B in 2015. In the government sector alone, according to the U.S Federal Cybersecurity Market Forecast 2015-2020, federal agencies spend more than the entire GDP of North Korea on cyber security.
Cyber attacks have grown by an estimated 35 percent in the last three years. More critically, the advanced nature of these attacks makes many traditional security tools ineffective in prevention or discovery. And when these passive security systems do detect a potential breach, it is just one of thousands of such alerts that more often than not go without diligent analysis. There simply are not enough skilled security professionals to chase down each lead and decipher the difference between a false alarm and an advanced attack. The number of truly gifted security professionals – tier-three network analysts – is woefully insufficient to support the needs of private and public entities, especially with the tools we often attempt to empower them with. We have a massive skills shortage given the way we currently attempt to protect our networks.
Several solutions could address this skills shortage, but the most immediate and scalable solution is to train software through machine learning to emulate the behaviors of tier-three network analysts and reverse engineers and execute in an automated, unsupervised way.
Become a Productive Hunter
Tier-three network analysts, ideally, are responsible for hunting. They query network traffic (network packets, sessions, logs, and/or metadata) looking for behavioral anomalies not traditionally detected by signature-based approaches. From their observations, these hunters change course, issue a responsive query, or set up a remediation action plan for a cyber threat, which often requires reverse engineering of malware. It is common for these advanced network analysts to have numerous areas of interrogation open at the same time.
When the results of a query catch the eye of the network analyst, subsequent queries are issued to triage and focus in on the behavior of a potential bad actor. To be clear, many such hunting excursions reveal legitimate and fully acceptable activity by internal or external clients. But each must be examined until they can be explained away or pursued as a threat. But with current technology constraints, there are too few skilled resources to do this effectively.
With machine learning, tier-three network analysts can offload much of the heavy lifting that helps them distinguish a threat worth pursuing from legitimate activity requiring no additional investigation. By allowing the machine to do this work, tier-three analysts can spend more time pursuing the machine-identified threats. The added benefit is that tier-two analysts learn from the machine decisions that were initialized by the advanced network analysts. Taking it one step further, even advanced network analysts make mistakes, especially when juggling numerous investigations simultaneously. Machines can be taught to consistently avoid these uniquely human traits. And, once the machine identifies a threat, it can quickly take action to mitigate risk and data loss in the smartest and most effective way.
Teaching the Machine
The intersection of machine learning and computer security is an area of extreme interest, but suffers from a lack of innovation in approach and goals. Research efforts to empower defenders and analysts through the force multiplier of automated statistical models traditionally have been focused on the identification of malicious code through static analysis prior to execution. There is no shortage of academic work that consistently trod over the same approaches, yielding common and unimpressive results.
While simple approaches can be interesting, production quality triage pipelines based around expert-reinforced feedback loops, graphical reputation models, and deep learning models based on dynamic and symbolic execution are far more effective than traditional shallow classification efforts or academic proofs of concept. Traditional supervised models suffer from issues of disparately sized training sets and a limitation of output that can be used for other purposes. While these models have limitations, success has been had in using them as part of a larger ensemble, weighting their classifications against expert feedback, as well as other competing models to achieve a unified result.
An effective machine learning system that can be reinforced by experts needs to deal with the skill shortage and bandwidth challenge of tier-three network hunters and reverse engineers. Labeled reinforcement learning models and request/response/request instrumentation, combined with unsupervised clustering algorithms, can radically improve the effectiveness of limited human resources. Modern dimensionality reduction techniques, as seen in deep learning networks, can divide a set of diverse features into a vector for clustering. As additional network hunter and reverse engineering results are ingested, known samples can be used as “seeds” in the cluster results, allowing analysts to measure relative distance between samples both within the working set and the historical repository.
Prescriptive Insights For Faster, Better Decisions
When patterns emerge, consistency and results follow. Machines cannot replace the intuitive nature of a highly skilled network hunter or reverse engineer. But machines can manage more data points, operate without human flaws, and eliminate the heavy lifting that precedes the activity for which unique talent takes over. If we assume that the work performed under a machine-taught process equates to 50 percent of the total time a network analyst or reverse engineer spends before realizing she has a potential threat or legitimate activity, we have immediately doubled the productivity of this high demand, low supply resource. In addition, we create an operational model for training the next crop of tier-three network analysts through prescriptive, machine taught processes. And finally, we establish a rewarding environment for the “good guys,” who are outnumbered and out-spent by those who have bad intentions.
Cyber attacks are increasing, dwarfed only by the massive amounts of digital information being generated daily. To minimize major cyber hacks and avoid frequent data breaches, the future must include machine learning, as organizations adopt algorithmic pattern and network behavior analytics into their defense arsenal. This adoption process should be done carefully and methodically so that companies and agencies can learn to trust the machine-generated outcomes, where the machines become a prosthetic, or technology resource, for tier-three network analysts and the entire security operations center.
Jim Cushman is responsible for product strategy, development and sales of the Novetta portfolio of products. Jim joined Novetta after serving as Vice President of Business Process and Decision Management for IBM. He previously served as IBM’s Director of Master Data Management. Jim has deep experience in data analytics, having been a key C-level executive during the success of Initiate Systems, an identity and entity resolution software company sold to IBM in 2010. Jim holds a BS in Management and Finance from Purdue University.
Subscribe to Data Informed for the latest information and news on big data and analytics for the enterprise.