World of Metrics: Trick or Threat? How do you know for sure?
.jpg)
Today begins the „darker half“ of the year. The harvesting season has ended. The year ends as well (depending on how you count the days and mark the start of the year). People celebrate Samhain, Halloween, or other festive days. In information security there is always a harvest season, and there is no darker half of the year. 2020 is no exception despite the extraordinary situation given the SARS-CoV-2 outbreak. So how do you decide what exceptions look like? What is a trick? What’s the difference between a trick and a threat? If you supervise any kind of digital infrastructure or set of systems, then these questions are very important.
Metrics is a hot topic – an euphemism for a dirty word – in computer science. It is used in other fields as well. Interestingly physics does not use it. Physicists use the metric system, but the measurements itself rely on the work of sensors. Usually the process of measuring something is always a comparison between something you know and something you observe. Furthermore, there is more than one way of measuring an attribute of a system. Computer science often takes the easy approach by using any numbers you can find. This is especially true for information security. The power of computing hardware often leads to an amnesia of context (similar to 3D films where special effects and 3D are gained by sacrificing most parts of the plot). This is a very simplified view, but it serves to illustrate the low quality of measurements often found in security-related answers. The marketing term Big Data hasn’t helped to regain the context necessary for using results sensibly.
Most people know that measuring something is important for determining the state of your IT landscape. If you have a monitoring system telling you when which service is available or what resource is scarce, then you already have data. Before you can use this data for a meaningful security analysis, you have to have an idea of what you are looking for. Define key parameters that tell you about the state of your systems. Make sure they can be easily measured. You can start with the data from the monitoring system. Next step is to reduce the parameters. Tools producing colourful graphs and other visualisations are great, but if you are going for a reduction by using these tools why not start with the reduction right away. You can gather some inspiration from the many collections of indicators of compromise. Be warned that these collections change every once in a while. This should not happen to the set of parameters you select. Why? Because the security view of metrics need to cover at least a time period of 6 months or more. You will always need historical data to compare your current values against. If you keep changing the measured parameters (even the way you measure them), then your ability to compare suffers.
Don’t go for real-time features unless you can handle events in real-time as well. You wouldn’t set your alarm clock or reminders unless you plan to act on something. If you can’t spare the time, then integrate this into the way data is obtained, processed, and reported. In case you can’t keep up with the flow of information you have a good indicator to measure and report less, not more.
Speaking of reasons to do something: This article is a reminder to question the results you see when facing reports. Decades of experience in maintaining IT infrastructure suggest that the way your monitoring data comes into existence needs to be fully understood. Otherwise you just see a collection of events such as the 5 O’Clock Charlie from the M*A*S*H TV series. This doesn’t mean that these events are no threat, but most frequently they are tricks. Make sure you have a way to determine the difference.