To catch a thief, you must think like a thief.
Last month I presented on Adversary Emulation at Security BSides London 2019. This blogpost will give a high level overview of what was discussed, including best practices for your organisation.
Adversary emulation is, as the name suggests, a way of mimicking attacker behaviour in a controlled/isolated production environment. Such emulations are usually run in a post-compromise setting, i.e. assuming the attacker has got some initial foothold in the environment tested. Called a ‘rising technology’ by Gartner about two years ago, more and more organisations are relying on a form of attacker emulation to quantify the effectiveness of their detection and prevention tools.
Whilst there is some overlap with more traditional pentesting, attacker emulation is different in that it is typically automated and usually part of a continuous threat detection and response development cycle. Attacker emulation allows you to test your own security controls using a realistic attacker model, ensuring your threat detection capability works as expected. It is even possible to define specific attacker group models, allowing you to test particular attacker tactics, techniques and behaviours in your environment. Because of its automated nature, attacks can be tested over and over with minimal effort. If you integrate this in your security development cycle, it can help track the effectiveness of your defences. If your organisation is not quite there in terms of maturity, it is worth looking into a compromise discovery assessment or behavioural rules subscription first.
Another advantage is that working with adversary emulation tools helps you better understand attackers. In order to define an appropriate attacker model, you will have to research your potential adversaries to get an understanding of what they are after, which techniques they use and how they behave. This should be reflected in your defences, whether this is on endpoint, network or even a social level. A threat intelligence subscription is often helpful in keeping up to date with threat actors your organisation might face, as well as the tactics and techniques used by them.
Finally, whether you are trying to convince someone within your organisation, trying to impress a client or just doing it for your own benefit, attacker emulation is a great way to show that what you are doing to defend your organisation works. Emulations should trigger various layers of detection and prevention and should demonstrate that you would pick up such behaviour. Although it’s not guaranteed to provide you with total assurance, it’s is a great indication of the state of your defences.
There are various tools available to help with adversary emulation. MITRE, the organisation behind the popular ATT&CK™ framework, have released their own open source adversary emulation tool called CALDERA. The tool comes with several out-of-the-box attacker techniques, including a few attacker profiles. However, this is still limited in scope and can be extended with your own techniques by writing small Python modules, which are interpreted by the tool’s backend and can be easily added to attacker profiles through its web interface.
To get started with CALDERA, you simply download and install the latest version on a separate machine, which will be the CALDERA server. This server will act as the command and control (C2) channel, sending instructions to one or more hosts taking part in the emulation. After installing CALDERA, you will need to install a CALDERA agent on at least one other machine, which can be used as the starting host for the emulation (remember adversary emulation assumes a post-compromise scenario).
The current version of CALDERA supports two modes, chain and adversary. The former chains individual, isolated attacker commands together in a user-defined order. Whilst this is great to quickly test certain adversarial techniques, it is less realistic, and due to the static order more predictable. The adversary mode requires more time to set up, but is more flexible, as it allows you to randomise commands, use a (more) non-deterministic execution order and use the output of one attacker technique as the input for another. For instance, one action might be to dump credentials, while a second action uses those credentials to move laterally.
On top of the powerful functions CALDERA’s adversary plugin provides out-of-the-box, we can go a step further. We’ll focus on three extensions that can help emulate adversarial behaviour in an even more realistic way:
LOLbins, or Living off the Land Binaries, are legitimate executables that are commonly present on machines, and can be tricked into executing unintended commands. The benefits of this approach include the fact that the attacker doesn’t have to bring their own tools, and that their actions are more likely to stay under the radar due to the use of legitimate binaries. Common examples of LOLbins include interpreters such as PowerShell and cscript/wscript, but there are plenty of other options: the Execution tactic in MITRE ATTACK™ framework documents numerous techniques that can be used to this end.
To implement this in CALDERA, we created a new Software class that can be used in CALDERA’s pre/post condition structure. This allows us to specify for certain actions that software is required (e.g. an action that sets up a webshell requires a webserver), or that the action can help downloading or installing software (e.g. defining a LOLbin action that uses certutil.exe to download software).
With this, an attack path in CALDERA might be:
If you define the pre and post conditions of these actions properly, CALDERA will figure out the above path for itself.
An example of the certutil.exe LOLbin action code, using the Software object as a pre and post condition.
Obfuscation of command line arguments is a technique that we still see being used, although such commands tend to stand out more than normal command line arguments. The goal of this technique is to rewrite the command line in such a way that it will still execute, but circumvent keyword-based detection tools.
For instance, you might have a rule that looks for the word mimikatz in command line parameters. Thus, running the command powershell.exe /c Invoke-Mimikatz should trigger an alert. However, depending on how advanced your rule evaluation system is, it might not pick up some of the following variants of the same command:
To apply this concept in CALDERA, we created a new CustomCommandLine class that takes arbitrary commands and applies obfuscation techniques at random.
Implementation of the ‘format string’ obfuscation function, which can be applied at random when using our CustomCommandLine.
Masquerading, finally, is another easy way to bypass rule-based systems. We have seen attackers use this technique in the wild, copying a legitimate binary to a temp folder using a different name. For instance, copying powershell.exe to C:/Windows/Temp/svchost.exe.
This approach has several advantages from an adversarial point of view. An attacker can still take advantage of legitimate binaries available on the victim’s machine, while they might bypass detection systems that look for specific combinations of process and command line. For instance, your EDR solution might be looking for executions of powershell.exe with the -ec switch in the command line. However, if the attacker now runs C:/Windows/Temp/svchost.exe –ec <command>, the command will still successfully run, while your EDR Solution probably won’t fire an alert. Even worse, it might tell you all is fine given that the process is a signed, trusted Microsoft process (because this fake svchost process is actually PowerShell).
Our implementation of masquerading in CALDERA takes any command, copies the main binary to the Windows temp folder using a random yet legitimate sounding name (e.g. MicrosoftHost.exe, VMwareSvc.exe) and executes the new copy using the original command line parameters. It works with any CALDERA action, as long as it is using our CustomCommandLine class.
Implementation of masquerading binaries in CALDERA using our CustomCommandLine class.
The full CALDERA implementation of these three features is available on GitHub.
Putting it all together, the following demo taken from my BSides talk shows what a CALDERA run with these three features in place might look like:
Adversary emulation is a powerful way to test the effectiveness of your defences in an automated manner. To get the most out of such emulations, it is key to make them as realistic as possible: emulation tools such as CALDERA can help with this. By creating representative adversary profiles and implementing relevant attacker techniques, you’ll be better able to evaluate your detection capabilities. To extend CALDERA’s usefulness even further, we presented three extensions to normal CALDERA, implementing techniques commonly used by attackers, making the emulations more random and even more realistic.
If you want to learn more about how PwC uses CALDERA for testing the behavioural rules set we offer our clients, or want to discuss what we can do to help you enhance your threat detection capability, please get in touch via the contact buttons below.
Endpoint Threat Detection and Response Analyst, PwC United Kingdom
Tel: +44 (0)7850 908221