AWS infrastructure security

The Tragedy of Security Dies on the Crucible of DevOps

Security ain’t what it used to be. Or perhaps it’s always been this way and it merely seems different due to the slow degradation of my youthful idealism.

Security is bifurcated. Down one path we strive to keep our organizations secure. To stop bad actors, protect data and assets, and defend users from threats and even their own mistakes. Down the other path lies compliance. Ensuring the organization meets regulatory, contractual, and other standards. Across both paths we manage risks; the risks of breaches and downtime, or the risks of regulatory fines.

The tragedy of security is a reflection of The tragedy of the commons. From Wikipedia:

The tragedy of the commons is a situation in a shared-resource system where individual users, acting independently according to their own self-interest, behave contrary to the common good of all users by depleting or spoiling the shared resource through their collective action.

Most of security compliance is designed to reduce security risk. At least on paper. But over time we’ve seen standard after standard, regulation after regulation, decouple risk from compliance. The very nature compliance standards prevents organizations from tailoring security measures to organizational risks. I’m not saying this is always true, but it is largely true, especially as you scale into larger organizations. There is no evidence that requiring password resets every 90 days for users with MFA or other conditional restrictions now in common use improves security. The person who invented password complexity requirements literally regrets it and says they don’t work. There is neither evidence nor a sound scientific basis for requiring non-default encryption of all storage volumes in a cloud provider.

Security is a shared, finite resource. We only have so much money, so many security professionals, and so much time non-security staff can dedicate to security at the expense of their other goals. The more we pull towards compliance the less of this pool we have for security. The less compliance aligns with security, the fewer efforts reduce security risks.

This is the tragedy of security. We decoupled risk from compliance, making lack of compliance the risk itself. The greater real risks are decoupled from compliance, the more rigid the compliance regime, the fewer security resources are available for defense, and the less support for security efforts from other teams.

A great example came up in a conversation with Chris Farris. To paraphrase,

Developers do care about security. It’s compliance that is an irritation. Help them secure their application and they’ll support it. Tell them to take a 4 hour outage on a weekend to re-build their database with encryption because a compliance wonk demands it and you’re just pissing them off.

I’m not saying all compliance is dumb rules, but some compliance rules are dumb, and the misapplication of good rules decoupled from risks is really dumb.

DevOps becomes the crucible for the future of security because in the world of cloud and DevOps individual application teams become responsible for the entire stack, including much of security. New servers, networks, and firewalls are a mere git commit and some API calls away. This also places a larger burden on those teams to manage their own security and compliance. We still have centralized security and shared security resources, but when it comes to the tip of the spear we absolutely rely more on the DevOps teams. We can’t create a big DMZ for the cloud. The lines between internal and external networks can no longer be categorized into cookie-cutter zones. Many applications build on native cloud services don’t even have a network anymore and rely on IAM rules and resource policies written in JSON pushed by application teams through infrastructure as code.

Thus a fair few of my recent cloud and DevOps-focused compliance projects are more about doing good security first, and then figuring out how to write a report that makes it looks like it meets the letter of compliance even though it doesn’t, even when it’s more secure.

There are two potential solutions.

The first is to revise security standards to better fit cloud and DevOps and reduce the number of dumb or inapplicable rules. Some of this work is happening, but I’ve come to believe this requires a generational shift we don’t have time to wait for. Let’s not give up, but we don’t have to wait.

The other is to use automation to remove as much of the security and compliance burden from individuals as possible, while still allowing them the freedom and control to build fast. I’m not suggesting the overall pool of security becomes smaller, but that we use automation and other technologies to reduce the individual requirement to burn cycles on the less valuable things.

It’s about time and focus of limited resources. And really, what’s more valuable… a good penetration test or a compliance audit? Now look at how much you spend on penetration testing and how much you pay for an audit.


  1. Avatar
    David Mortman August 15, 2020 at 11:00 am - Reply

    I like what Chris Farris has to say about IAM vs crypto etc. But he overlooks/ignores an important thing. If your service is up and running, the data isn’t encrypted at rest anymore. If I can get access to a running system or if I find a sql injection (or any reasonable enumeration attack) it’s game over. Encrypting data at rest while important actually only protects you from three forms of attack:

    1) The disk drives being accessed by an unauthorized party which generally translates to a rogue insider or a drive/tape getting lost/stolen.(There’s a whole blog post here on the problems of getting usable data off a CSP’s drives).
    2) Protection from government subpoenas. (There’s another whole blog post here on the whys and wherefores of this problem and how to address it). For extra credit, re-read Chris’s post with an eye towards keeping governments/police agencies from reading your data w/o permission.
    3) The most likely/highest frequency “attack”. Auditors/regulators require it and in some cases can apply significant financial pressure to gain compliance. For some industries lack of encryption also raises reputation risk in addition to audit risk.

    And finally just to reiterate, if your service is up and running your data isn’t encrypted at rest.

  2. Rich
    Rich August 17, 2020 at 11:40 am - Reply

    Absolutely. Chris does mention that a bit by referencing an old TidBITS post of mine… keys are always exposed in memory someplace.

    But I still have my security blanket from when I was a kid. Just because I know monsters don’t exist doesn’t mean it isn’t comforting. Then again, pretty sure my parents only spent like $5 on it once and haven’t had to keep documentation of controls mapped to 37 different frameworks and standards up to date, so perhaps not the best analogy.

  3. Avatar
    Jeff Soehner August 21, 2020 at 6:53 am - Reply

    Long time listener; First time caller. (Happy BlackHat Anniversary BTW)

    It is refreshing to here your conclusions surrounding ‘how we get there from here’ and while it is clear that ‘shifting left’ will help solve the compliance burden, we may still be faced with the problem of context when learning to measure Infrastructure as Code without it.

    As a Appsec guy, we learned that static analysis produced more false positives that created noise so analysts needed to learn how to triage findings to be able to suppress them. Many of them needed to ask the right questions and gain enough context to feel confident enough to ‘look the other way’ and dismiss many of them.

    If we could take a similar approach of testing the recipe instead of waiting and ‘eating the cake’, won’t we be facing the same challenges as our SAST approach? How do we perfect automated parsing of orchestration templates to be sure we can prevent non compliant resources from being created in the first place?

Leave A Comment

7 + 15 =

About the Author: Rich Mogull

Rich Mogull
With twenty years of experience in information security, physical security, and risk management, Rich is one of the foremost experts on cloud security, having driven development of the Cloud Security Alliance’s V4 Guidance and the associated CCSK training curriculum. In addition to his role at D-OPS, Rich currently serves as Analyst & CEO of Securosis.

Sign-up for Updates!

  • This field is for validation purposes and should be left unchanged.