Supercharging Security Hub: Part 4, Taking Action
In our last post, we walked through the console and highlighted making the most of the Security Hub console and some tips and tricks to make it more useful. Today I want to dive into one of the best parts of Security Hub — taking actions on events and findings. We’ll start by reviewing how Security Hub handles findings and creates events, which then trigger actions. Then I’ll show you different approaches for handling actions.
How Security Hub Turns Findings into Events
As we previously discussed, Security Hub is a central clearinghouse for various security tools to report findings in a consistent format (the Amazon Security Findings Format). These findings are then accessible in the console and via API calls. Security Hub is multi-account, yet only supports a single region. Thus findings can be consolidated across accounts, but only within the AWS region where Security Hub is running.
Here’s the important bit for taking action; every finding creates a CloudWatch Event, also known as an EventBridge Event (EventBridge is the new name for the feature but since it is still in the CloudWatch console we will use that name). For those of you that aren’t totally familiar with CloudWatch Events think of them as an ephemeral message. Events are created constantly in AWS, by every service. We can build Rules to respond to these events based on their source and content. If an event happens and you don’t have a rule to capture it, it isn’t logged anywhere and it just disappears into the ether. Events pop up, look around, and if no one expects them — they evaporate.
Security Hub, by default, fires off CloudWatch Events for every finding, regardless of the source. Security Hub doesn’t differentiate between an assessment result finding or a threat event, like a Guard Duty alert for a misused credential. Security Hub actually creates three types of CloudWatch Events:
- Findings: Individual findings when they are ingested. This includes both new and repeat/existing findings. Thus if you leave port 22 open on that security group for a month, that finding will create an event every time the assessment runs, but it will have different creation and update times. AWS calls these Security Hub Findings – Imported.
- Custom Actions: If you create custom actions and take a custom action on a finding what it really does behind the scenes is fire off a new CloudWatch Event with the finding and adds the Custom Action ID. We will talk about this more in a moment. AWS calls these Security Hub Findings – Custom Action.
- Insight Results: Insights are filtered views to manage collections of findings such as “EC2 instances with ports exposed to the Internet”. Insights support a different flavor of custom actions. When you take the action it creates a new CloudWatch Event with your custom action ID and a list of the resources that match that Insight (e.g. the list of the instances) plus the Insight’s ARN. AWS calls these Security Hub Insight Results.
The formats for these events are all documented here https://docs.aws.amazon.com/
Thus every finding (assessment result or security alert) in Security Hub fires a CloudWatch Event when it is ingested or created. If you click on the Take Action button for a finding or Insight this bundles up the data, creates a new CloudWatch Event, and then adds the Custom Action’s ARN. It’s your responsibility to listen for and act on these events, which is where I will focus the rest of this post.
Trigger Actions from Events
As mentioned, if you don’t capture CloudWatch (EventBridge) Events they are gone forever. Fortunately, we have CloudWatch (EventBridge) Rules. A Rule triggers anytime a matching event occurs. Rules are region-specific, but unlike CloudTrail if you consolidate Security Hub across accounts the Events will appear in your Master account and you can simply build your CloudWatch Rules there. You can build a rule for all Security Hub events or only certain matching events, using a pattern.
All events looks like this:
If you only want Imported Findings it looks like this:
You can further filter down to only trigger on specific findings from a Custom Action, which would look like this:
This would trigger only on the custom action named “escalate”.
From the rule, you then set a target. CloudWatch sends the context and content of the event to the target. AWS supports a bunch of targets, but for taking actions it’s usually one of four options:
- A Lambda function. This is the most flexible option since you can write code to do anything you want with the event, Most of the AWS automation examples send the event right to Lambda. This is also the best option if you want to move events across regions since you can write a function to send to a different region. At DisruptOps this is how we configure client environments so we can centrally collect all accounts and regions the customer wants to enable, without forcing them into any particular deployment model. Our Lambda function calls our API directly and sends the event over.
- An SNS topic. This is only good if you want a simple notification. That’s why they call it Simple Notification Service.
- An SQS queue. We don’t see this as much for Security Hub but it’s useful in some automation scenarios where Lambda doesn’t fit, such as using a tool inside a container.
- Amazon Kinesis. Kinesis is for large scale collection, but practically speaking if you need to send across accounts or regions you will need to hit Lambda first, then use that to send to Kinesis.
This is what a simple rule looks like in the console to send all Security Hub events to a lambda function.
This basic pattern of CloudWatch Rule to a Lambda function target is incredibly versatile, but leads to our next section…
Choose an Action Option
It’s decision time! Now that you understand the kinds of Events and how to build Rules, you need to choose what actions to take and how you want to take actions . There are three main handling patterns:
- Send all findings (events) to a single collector or application that uses its internal logic to make the actual decisions and perform the actions. For example, this is what we do here at DisruptOps and it’s what allows us to handle complex scenarios and more advanced filtering, such as routing findings to the team associated with that account via Slack, but also triggering automated remediations for production accounts and predetermined critical issues.
- Send custom action findings to a specific Lambda function or another target that acts only on those actions. Our Escalate rule from above is an example where you could select the custom action in the console to send a finding to an incident response team.
- Send filtered findings to either central collection or a specific target. Our examples showed all findings or custom action events, but you can just as easily select certain finding types by building more-complex CloudWatch Rules patterns. For example, you could send all GuardDuty findings to a notification topic, or build alerting on all High or Critical findings and then use a separate rule to send Medium and below findings to a log collector or SIEM.
And three categories of actions:
- Send a notification. Notifications aren’t limited to SNS, since you can send findings to other types of notification services. Amazon has an example where they use AWS Chatbot to send findings to Slack https://aws.amazon.com/blogs/
security/enabling-aws- security-hub-integration-with- aws-chatbot/.
- Execute a remediation action. This can be an external automation tool (like DisruptOps) or your own Lambda function. AWS has some examples here if you want to build it (and then support it) yourself https://aws.amazon.com/blogs/
security/automated-response- and-remediation-with-aws- security-hub/.
- A combination of both. This is the use case we focus on, where we send the finding to the designated team over Slack or Teams (or create a JIRA ticket) but enrich the notification with additional data and then package up various remediation actions you can execute directly from Slack. And yes, you can do this yourself, we just think we do it better 🙂
Using DisruptOps to remediate a GuardDuty alert via Security Hub and Slack
Fit Your Workflow
So far I’ve focused on the mechanics and possible design patterns, but it’s easy to get lost in the weeds and lose sight of the overall story.
There are innumerable ways to work with Security Hub findings; far more than I can describe here (and it isn’t as if I’ve seen everything). Here are a few examples to get you thinking as you look at taking actions yourself:
- Send all actions of a certain criticality to a notification channel like Slack or Team for an incident response team. This is easy to set up and will give you a good sense of what to expect from Security Hub. It’ll also whet your appetite to start acting on events. You’ll learn what to filter on, and how to handle differences in environments. For example, some findings in dev accounts will quickly create alert fatigue, but those same findings are critical to watch for in production accounts.
- If you use Security Hub itself as the console to manage findings, a Custom Action to escalate issues and send notifications is also pretty simple to build.
- Build some simple actions for remediation of findings. For something glaringly obvious like Port 22 open to 0.0.0.0/0 in a production account, you might point a rule for just that finding to a Lambda to automatically remediate it. For something less certain, you could use a Custom Action that instead of a notification runs a remediation Lambda for that finding type.
- If you have a SIEM and want to collect findings from all your AWS security tools, including integrations, Security Hub works as a great clearinghouse since all the findings and events from all the different services use the ASFF, which makes life much easier when you want to parse, classify, and analyze the data.
I’m biased here, but I really like how we handle DisruptOps adds value to Security Hub. Using rules you define in the platform, DisruptOps enrichs each finding you send with additional data, and then routes it with recommended remediation actions right to whoever has responsibility and authority to make changes in that AWS account.
Once the rules are set up (and we have defaults) we use the tool of your choice (e.g. Slack, Microsoft Teams, or JIRA) and package in 1-click actions so your employees can remediate issues without having to use another console. This provides the security team with visibility across the enterprise, but puts remediation in the hands of those who know the environment and can make the changes with the lowest risk of breaking the wrong things.
These examples of taking action are only a starting point. The combination of aggregating security data from a wide range of sources, normalizing everything to a single findings format, and then creating events to trigger actions makes Security Hub far more useful and powerful than just another compliance dashboard.