Introduction
Building effective protective monitoring (ProMon) is hard. Every pentester has stories about how they carried out a test and the security operations centre (SOC) completely missed them. But whether you choose to outsource this to a third party, or to build your own SOC, one of the areas that often gets overlooked is how to effectively test the ProMon solution once it’s in place.
The most common approach is red teaming to see how effective the ProMon is at identifying and responding to a simulated attack. However, carrying out a single red teaming exercise once the ProMon capability has been fully built means that you have no assurance throughout the build process, and may provide little value if there are basic functional issues with the SOC that prevent it from detecting any of the red teaming activity.
With pentesting and many other areas of security there has been a big push in recent years to “shift left”, with testing being carried out much earlier in the development process, as it’s become increasingly clear that the approach of performing a single pentest right at the end of the process is not always effective. In the same way, we need to employ a mixture of different testing approaches throughout the entire lifecycle of building out the ProMon capability to provide effective assurance.
Initial Use-Case Modelling
Most ProMon solutions start with a set of standard use-cases that cover common services (such as Active Directory) and common attacks (such as brute-forcing or password spraying). While these are usually a good starting point, one of the key challenges is to build up specific use-cases that cover the specific applications and context of the environment that’s being monitored. This exercise is often left to the blue (defensive) team, but can be challenging for them to perform on their own, especially when they don’t have in-depth knowledge of the bespoke systems and services within the environment, or previous experience on the red (offensive) side. These issues are further exacerbated if the blue team sits outside of the primary team managing the environment, or in the absence of clear communication between teams.
Formal threat modelling can play a useful role in helping develop those use cases; but it can also be a long and expensive exercise that can be difficult for the blue team to directly action. A more informal use-case modelling approach can be useful as an interim step.
This is where someone from the red team sits down with the system owners, talks through the system to make sure that they understand it, and then produces a list of attacks or malicious activities that they might want to perform as an attacker. These activities will be based not just on their understanding of the system, but also their experience as an attacker, and their wider knowledge of the environment. Once produced, this list of activities can be reviewed by the blue team, and then use-cases developed to detect the activity.
This is particularly useful in the early stages of developing a ProMon capability for bespoke systems, or environments where there are unusual requirements around business logic or the system context that are unlikely to be included in the default sets of use-cases provided with many SIEM products. It won’t cover every possible malicious action, but it can quickly provide an initial baseline to cover off key areas without the cost and time investments of carrying out a formal in-depth threat modelling exercise.
Use-Case Validation
Once the SOC is up and running with an initial set of use-cases and alerts configured, it’s important to verify that they’re actually working and detecting the activity and behaviour that they should. Rather than jumping immediately into a full-blown red teaming exercise, it’s often better to start with a smaller and more focused protective monitoring validation exercise.
This is best performed from an entirely white-box perspective, with the tester given a list of the use-cases that are expected to be in place and functional (both built-in and bespoke). From these, they can select a sample and execute actions that would be expected to trigger an alert, while keeping a detailed log of the actions that they perform, with timestamps provided to assist in tracking activities through detective systems.
If everything is working perfectly, the SOC should receive an alert for each of these actions. But in any cases where they don’t, they’ll get some real-world logs to review, and can then improve their existing use-cases based on these. This should be a collaborative process with the tester, to allow repeated retesting and refinement.
It sounds basic, but this type of testing can identify all kinds of issues, such as broken log forwarding, incorrectly defined event IDs or detection rules that are functional, but don’t actually match the logs generated by malicious activity. Rather than taking weeks or months, this gives assurance of the effectiveness of the SOC in a couple of days - which is not only a lot cheaper than red teaming, but also allows faster iterations and improvements for a developing ProMon capability. It also provides an opportunity for the red and blue teams to work side-by-side, which helps build a collaborative approach in what can otherwise end up as a somewhat adversarial relationship.
Rather than choosing from a list of implemented use-cases, a more black-box approach can also be taken, as the ProMon capability matures. This involves the tester choosing and executing their activities based on the actions that they would expect a real-world attacker to perform, rather than specifically linked to the use-cases presented by the blue team. This type of protective monitoring testing is a step closer to full adversary simulation, and is useful to identify gaps in the coverage once the core functionality of the SOC has been proven.
The initial testing can be performed without the SOC knowing exactly which systems are being tested - although there should be communication between the red and blue teams throughout the processes to prevent unnecessary escalations. Once completed, the same activity timeline should be provided, to allow the red and blue teams to review and identify any activities that weren’t detected
Red Teaming
Finally, we come to red teaming. The broad scope of red team engagements means that they often run for weeks or months, making them significantly more expensive than other types of testing. However, they provide the highest level of assurance that the blue team will not only be able to identify attacks, but also that they can effectively respond to them, by simulating an attack by a real-world adversary. As such, they are most appropriate to test the effectiveness of a well-established and mature team.
Depending on the scope, rules of engagement, and the level of covertness required, it may also be beneficial to embed someone from the red team in with the blue team for some or all of the engagement. This may allow deeper collaboration between the two and to ensure that the any gaps in the protective monitoring are fully understood, and to help the blue team quickly adapt and improve.
Final Thoughts
Building an effective protective monitoring solution is really hard. Trying to do it blind, without providing the blue team regular assurance that what they’re building is working as it should is even harder.
Red teaming is seen as the gold standard for how to test a blue team, but unless the ProMon solution is already mature and established, it often ends up as an inappropriate and expensive exercise. Instead, it can be far more productive and cost effective to shift left, and to carry out regular collaborative assurance activities throughout the entire build and lifecycle of the SOC.