DevOps Security Best Practices or How Uber Lost 57M User Accounts

DevOps Security Best Practices

Security is of paramount importance in DevOps. According to Business Insider, over 1.2 billion user accounts got breached in the U.S last year. As a result, Facebook is still trying to remedy its reputation, and Google had to shut down its social network Google Plus altogether. These examples should be enough to spur a company of any caliber on to set up and follow DevOps security best practices to avoid a fiasco.

Why do DevOps security failures happen? Who should be responsible for them? What are the answers to these DevOps security challenges? And more importantly: Why should you care? We answer these and other DevOps security-related questions to help you deliver tamper-proof applications to your customers.

Disclosure: what this post is not going to be about is DevSecOps because DevSecOps is a nonsense buzzword (even without a dedicated page on Wikipedia) that merely means secure DevOps. It’s not like you can come to your CEO and ask: “Boss, what kind of Ops flavor you’d like: DevOps or DevSecOps or SecDevOps?” The answer is always DevOps, and always secure.

Why Is DevOps Security an Issue?

Let’s quickly recap what DevOps is — just to make sure we are on the same page. The definition of DevOps by software giant Amazon sounds pretty credible:

Or, in layman’s terms — DevOps is a project team’s collaborative mindset plus tools to automate development, tests, delivery, and maintenance of digital products.

Agile — Fast

DevOps is part of Agile — a hugely popular software development framework that promotes fast and frequent development iterations and puts a strong emphasis on collaboration between team members. Some companies deliver multiple updates to their digital products on a weekly basis, others — on a daily. Regardless, such high tempo leaves not much time for rigorous testing. And of course, vulnerabilities eventually creep their way into such projects.

Security — Slow?

There is a common misconception that more security checks cause longer delivery cycles and, as a result, missed deadlines. And while nobody wants additional delays on their projects, the damage a single security flaw can bring about may be devastating for a company that sacrifices security for the sake of agility.

Back in 2012, an iOS social app Path was immensely popular until someone discovered that the app was uploading users’ contacts to its servers over an insecure protocol and without permission. As a result, there is no more Path app despite a two-million userbase and all the hype about a long-anticipated mobile social network for close friends and family only.

Open-source & Containers — Scary

To expedite development, DevOps teams may leverage open-source solutions that often skip tests for security issues. Even a single library or a server component from an unverified source may cause a cascade of errors and eventually make your service or product unavailable to customers.

Containers represent another indispensable part of a modern DevOps setup, besides open-source code and tools. Developers, system administrators, and QA engineers use containers to almost instantly make a project available on a new machine, or to revert to a previous development version of a product. However, containers offer poor visibility into their contents and are frequently inadequately scanned. A report by Threat Stack revealed that 94 percent of respondents saw containers as a security threat.

Strategies to Secure DevOps

Get Everybody On Board

Companies run DevOps differently. Some have their software developers carry out all after-coding operations, e.g., building, testing, and deploying applications. Other companies have separate development and operations team. But for a secure DevOps environment, the best practice is to get absolutely everybody on board, regardless of the group they belong to. In fact, companies that have successfully established security policies in DevOps started from the very top — the management — while also actively involving their legal and IT departments into the conversation.

Work Out a Concise Security Policy

The point of having a security policy is to enable your team to quickly respond in case of an emergency. The rule of thumb is to prepare a WISP (written information security plan) and a DIRP (data incident response plan) that are brief and clear enough to take immediate actions when necessary. A good place to start would be the Common Weakness Enumeration — a community-developed list of common software security weaknesses. A decent policy should also stress secure coding patterns that leave no place for exploits.

Train Your Personnel

Of course, everybody on the project should be aware of the security policies and requirements depending on their role. For instance, software engineers need to avoid keeping credentials in the source code for convenience, QA team members do not necessarily need access to the development environment, and system administrators need to configure firewalls properly, among many other things. So make sure your onboarding process covers security basics.

Automate DevOps Security Processes and Tools

DevOps has always been about automation. Getting software products released at a fast pace requires properly setup continuous integration (CI) and continuous delivery (CD) pipelines — when new versions of products get compiled, checked, and monitored automatically. It’s vital to include automatic security verification into these DevOps processes.

A skillfully deployed DevOps toolchain helps you spot vulnerable code before it reaches the production environment. Such a toolchain will also monitor your product as it goes through test, integration, deployment, and release cycles providing you with insights about the potential security threats.

Take Control of Privileged Access Management

Roughly half of all security breaches happen as a result of poor privileged access management (PAM), according to a study by Beyond Trust. That’s not about an outside intruder, but rather your team members with good intentions that have more access rights than they should or that mismanage these rights. The three steps that companies can take to take PAM under control are as follows:

• Take stock of privileged accounts and assets

A DevOps environment can be a tangled mess sometimes. Still, you need to go at length and dig out all the privileged user accounts and all the places they have access to.

• Ban shared accounts and hard-coded passwords

DevOps engineers may sometimes hard-code credentials within a tool they are using for their convenience or use shared accounts, which complicates the audit.

• Implement the principle of least privilege

For example, an account with the tool that gathers metrics about bugs in a test environment probably does not need access to production and development environments.

Setup a Vulnerability Program

Vulnerability management can help you secure your products and customers by looking at your digital assets through a hacker’s eyes and then conducting penetration testing (or simply pen tests). There are two main variants of pen tests that companies use:

External penetration testing

When a company is looking for vulnerabilities on its outside defense perimeter: misconfigurations, default usernames and passwords, unpatched software, open ports, etc.

Internal penetration testing

When a company wants to check how much havoc a hacker or a rogue employee can wreak if they already have access to the system. Can they access the mission-critical parts of your product, including customer data?

Slice DevOps Environment into Secure Layers

The application’s logic and database do not need to live on the same server. In fact, from the security perspective, it’s better to run separate servers for each and group these elements and other DevOps assets into logical units that do not trust each other by default. Add a jump server with multi-factor authentication and adaptive access authorization to connect the units, then apply sessions — and you are all set. Now, if a hacker breaks into one segment, at least the rest of them will remain secure.

Is Your DevOps Secure Like Uber’s or Tesla’s?

If you sell a digital service or product, DevOps security should be your top priority. In that respect, no company is an exception. Companies like Tesla and Uber seem to be big and mature enough to fall prey to security breaches. Yet, the former let hackers use the company’s servers to mine a cryptocurrency, and the latter exposed personal information of about 57 million customers and drivers. The reason in both cases is misconfigured DevOps.

The good news is today one can easily find many consulting and development firms that can help businesses secure their DevOps processes, even if a business is only starting to implement DevOps. Contact us today!