This week, Amazon Web Services announced updates to Guard Duty findings to help reduce multiple alerts and false positives. Alert fatigue is one of the biggest complaints I hear about Guard Duty. It isn’t so much that Guard Duty is prone to large numbers of false positives, but that it can be quite prone to generating multiple alerts for the same attack. This is honestly a tough problem for any threat intelligence, especially in an environment like AWS where each “stream” of an attack might hit multiple targets or come from multiple simultaneous resources.
Think of it as job security for incident response.
But the question arises on the best way to make use of Guard Duty while still minimizing alert fatigue. I find three techniques can help:
- Understand the Guard Duty finding types. I’ve included my personal descriptions below but you should develop your own list. This helps you both categorize events and build out different alerting and response paths.
- Correlate findings with the environment type. One downside of Guard Duty is that you can’t turn off the various finding types. While you can whitelist and blacklist specific IP addresses, you can’t turn off specific kinds of findings. This is a problem for findings types like those in the behavioral category that will alarm all the time in a development environment, but shouldn’t trigger often in production. If you correlate the finding type to the environment, this opens up possibilities like our next suggestion-
- Use different handling paths for different finding types and environments. This is where Cloud Detection and Response (CDR) can come in handy. Basically you take a finding and then correlate it against its environment, which then tells you the environment type (dev/staging/prod). You use this information to determine severity and immediacy. Have a behavioral finding pop up in prod? Send an immediate alarm or even trigger a guardrail. The same finding in dev might just pop a notification over to the account owner or drop it altogether. A cryptocurrency miner in any environment? You probably want to just stop the instance before charges add up, and simultaneously alert someone to take a look.
- Batch findings by type and time. When you build your alerting using CDR or your SIEM try to generate the alert for the first finding of the type within a given time period, not all of them. You still record them someplace for investigation. I’ve also seen environments where multiple alerts within a timespan will increase the potential severity of an incident. And, again, these “rules” should be both finding type and environment specific.
It’s all about understanding the data and instead of trying to treat them all the same, handle them differently based on the environment.
Know Your Finding Types
Keep in mind that for most of these you will get repeat/repetitive alerts. That’s the most common complaint I hear. On the other hand I keep getting ongoing validation that Guard Duty does find things that would be otherwise difficult or impossible to catch on your own. As such I recommend turning it on in every single account and region and centralizing the events to your CDR or SIEM.
Amazon maintains a list of current finding types. Below are my personal notes from research and real-world practice:
Backdoor Finding Types: generally good quality. The spambot is the most likely to cause a false positive. The denial of service attack findings may be triggered by legitimate traffic if it is high volume and resembles a DoS.
Behavior Finding Types: Pretty likely to generate false positives, especially in development or other environments that change behavior frequently. It sets a baseline and looks for deviations (e.g. new port being communicated to) so you can imagine it isn’t as reliable as communicating with a known botnet. I’m not going to ask why CloudTrail Insights isn’t included here and is a separate product.
CryptoCurrency Finding Types: Pretty reliable.
PenTest Finding Types: Fine I suppose, but only looks for 3 types of pen testing tools and only if the connection headers aren’t modified.
Persistence Finding Types: Valuable, but also prone to false positives. It looks for new kinds of activities (e.g. changing network security groups) when someone has permission but hasn’t performed that activity before. As you can imagine the value of this finding will vary based heavily on the environment.
Policy Finding Types: These are interesting, especially the root account usage, except you should NEVER have root Access Keys in the first place. Should be low-noise findings.
PrivilegeEscalation Finding Types: Certain environments may generate pretty high false positives. Should be reliable for a low-change production environment.
Recon Finding Types: Real mix of reliability and this is the area where I’m asking around more to see how well they work in larger environments. I believe this group does generate a lot of noise… some real but this is the category you need to watch for alert fatigue. This also appears to be where AWS worked on reducing the number of findings, per the blog post we linked above.
ResourceConsumption Finding Types: One finding in this group, for someone/something launching an instance for the first time. I don’t have a lot of feedback on this one.
Stealth Finding Types: The CloudTrail ones are very valuable, although if you are using an org trail you shouldn’t see them. The server access logging and password changes… I don’t have a lot of input on them but they should be low noise.
Trojan Finding Types: These should be pretty reliable, but will generate duplicates.
Unauthorized Finding Types: This is the last finding type category, and also one with a lot to understand. A big one is UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration- this will alarm if role credentials were stolen AND USED OUTSIDE OF AWS. If the attacker uses them from an instance in another account it won’t alarm. It also only works for roles assigned to instances, not other resources. SSH brute force has the potential for a lot of false positives (if you are the subject). This one surprisingly triggers less than I expect in my environments, but it’s also a good finding since you shouldn’t see it unless you have port 22 exposed to the Internet… which shouldn’t happen.