How can you establish a baseline for cybersecurity metrics?

One thing I've found helpful in establishing a baseline for cybersecurity metrics is to first clearly define your cybersecurity objectives. This involves understanding what you aim to protect, the potential threats, and the specific outcomes you desire from your cybersecurity program. By setting these objectives, you create a framework that guides the selection of relevant metrics, ensuring they are aligned with your organization's overall security goals and business objectives. This approach not only provides clarity but also ensures that the metrics you track are meaningful and actionable.

There will be two distinct buckets of objectives: 1) What does the business need to know? 2) What do the control owners need to know? For the first set you'll need to speak to the business leaders and decision makers. What is most important to them? What questions do they want answers to? These are likely to be more qualitative in nature; lower volume/higher value insights. The second set will likely be more quantitative. What's the compliance rate? What's the exception rate? What's the failure rate? If these figures start to deviate from acceptable levels then course correction is needed.

Be careful that you don't create KPI based on aspirational goals. That's a great way to set teams up for failure. Gathering baseline data should be an open effort to "see where you are." Whiteboard exercises are a good way to find out what data you have now and what data you would like to be able to collect later. If you don't like what the data tells you, then you can discuss KPIs with the team(s) to understand what is achievable. Just like other areas of risk and security, metrics should not be a static list, but an ever-evolving display of what is happening in your security program.

Establishing a baseline for cybersecurity metrics begins with defining your objectives. Determine what you want to achieve with your cybersecurity program – this could include protecting sensitive data, ensuring system availability, or complying with regulatory standards. Based on these objectives, identify key metrics that accurately reflect your cybersecurity posture. These metrics might include the number of detected incidents, response times to security breaches, or user compliance rates with security policies. By aligning metrics with your cybersecurity goals, you create a relevant and focused baseline that can effectively measure and guide the progress and effectiveness of your cybersecurity efforts.

You've not yet defined your metrics, you perhaps don't know what you can measure or what you're measuring against. Start instead with a maturity assessment and threat modelling. What controls are key in protecting your critical assets? What controls are key in answering the leadership's questions? Are those controls' requirements well defined in policy? Are those controls formally designed and documented? Are those controls implemented according to the design and requirements? Are those controls operating consistently and effectively? Now you can start to measure those controls. Where are there deviations?

It’s important to think “what behavior will this encourage?” Metrics mean nothing if you don’t understand the downstream effects it will have on behavior. If you set a target that influences your analysts to close cases as quickly as possible, you could be encouraging them to not be thorough, rush and just look at time as a measurement of success. Be intentional and meaningful in your deliberation of what targets and metrics to use.

As part of defining targets, also define a sliding scale of tolerances/thresholds and action plans. resolving a minor deviation typically requires significantly less effort than recovering a material control failure. Any metric target should have an owner. Who is accountable for continuing operational effectiveness? Who will need to make a decision to act if a control is deemed to be no longer satisfactory? It can also help to identify stakeholders, escalation paths and mitigations before they are needed in earnest. Periodically revisit your targets. Can they be tightened up based on prior performance? Are they still aligned with the organisation's risk appetite?

I advocate for carefully choosing metrics that are aligned with your cybersecurity objectives. It's crucial to select metrics that are not only relevant and measurable, but also provide actionable insights into the effectiveness of your cybersecurity posture. This could include metrics on incident response times, the number of detected threats, system uptime, or user compliance rates.

Ensure your metrics are meaningful and not a convenient distraction. Meaningful metrics might be harder to define and capture, but the effort pays off in providing actionable insights. "How many phishing emails have we blocked?" is less valuable than "How many phishing emails got through our filters?". Similarly, "What's our simulation click rate?" is far less valuable than "How many live phishing emails have been actioned?". In each case, the latter requires investigation and effort to compile, but also provides more tangible correlation to risk exposure. If you only measure what's conveniently reported then you're likely to miss the instances where a security control has failed.

Assume that any report will sooner or later be distributed beyond your control. Accordingly, any message should be clear in isolation, without prior knowledge or additional context. Seek feedback from the recipients: What does this report mean to you? What are the take-away messages that you get from it? Make revisions accordingly: Are the responses aligned with the intended message? How can the message be made more clear? Repeat the above objective and target setting activities at least once a year to make sure your metrics continue to be relevant. Provide a mechanism by which your stakeholders can raise queries or request revisions.

Be prepared to uncover areas in your program where you can't adequate measure performance or can't improve without significant change. When it comes to data and metrics that usually means an investment of time or dollars.

Begin by conducting a comprehensive risk assessment to identify and prioritize potential threats. Define key performance indicators (KPIs) aligned with your organization's goals and industry standards. These metrics may include incident response times, patching cadence, and user awareness training effectiveness. Regularly assess and update your baseline to adapt to evolving threats. Collaborate with relevant stakeholders to ensure a comprehensive understanding of organizational priorities and risks. Implementing continuous monitoring and analysis of these metrics provides insights into the effectiveness of cybersecurity measures, allowing for timely adjustments to security strategies as needed.