IT systems have never been more powerful or accessible to businesses. However, the scope and scale of cyber crimes continues to outpace tech innovation.
For years, the challenge for internal IT and security teams has been to use existing company data to construct an integrated picture of oddities and unexpected actions on their network. Recent advancements in machine learning and behavior or anomaly-based analytics that leverage existing enterprise logs have provided security teams with much more accurate intelligence than ever before.
In the past, security expertise was embodied in signatures, representing particular and specific types of malware. In time, the experts couldn’t keep up, signatures were out of date or not installed quickly enough, and hackers began to take full advantage. An attack from an employee account is signature-less, making conventional security approaches that rely on blacklists ineffective.
Security experts quickly realized that pattern patching alone wouldn’t work, so they added rules, such as the correlation rules found in security information and event management (SIEM). For example, if an HR employee has been terminated and begins accessing sales data for the first time, something is likely wrong, and an alert will immediately be sounded.
Technology outpaces analysis
As the number of endpoints (i.e. mobile devices) skyrocketed, so did the volume of data to be analyzed by firms, making it more difficult for security experts to rely on cut-and-dried rules. Existing—not to mention expensive—intelligence tools, typically some form of SIEM, were supposed to predict and detect these types of threats, but were unable to keep up. This left companies at an all-time vulnerable state for both insider threats and hackers.
Experts predict a 4,300 percent increase in annual data production by 2020 and IDC anticipates that the “digital universe” of data will reach 180 zettabytes in 2025 (that’s 180 followed by 21 zeroes). Thankfully, open source big data systems have provided a way to collect, process and manage monstrous amounts of data.
Open source big data technologies such as HDFS and Elasticsearch enable solutions that handle petabytes of security data with ease. This not only allows firms to store a wide range of data sources, but also reduces overhead cost of data storage altogether, which can reach millions of dollars annually for large organizations, due to the cost of vendor data management hardware and vendor per-byte pricing models. Consequently open source big data frees up the budget to invest in stronger analytics.
Algorithms crunch data
Another major advancement that has fortified cybersecurity tools is machine learning. The method of analysis flips the expert approach on its head; instead of requiring expert rule-writers to guess at attacks that might come, machine learning algorithms analyze trends, create behavior baselines—on a per user basis—and can detect new types of attacks very quickly using baselines and statistical models. These systems are more flexible and effective than any pure expert-driven predecessors.
Technology options available to enterprises are at an all-time high, and so are the number of cyber crimes that are committed. Fortunately, as technology has advanced, so has the ability to seek out cyber criminals that may have been virtually invisible in the past. User and entity behavior analytics and machine learning technology continue to provide chief information security officers with the accurate insights they need to thwart attacks before severe damage is done.
This article originally appeared on ThirdCertainty.com and was written by Nir Polak.
