A Guide for Multi-Cloud SSH Key Management

As more and more companies are moving their operations and data in multi-cloud environments, the questions around SSH key management are growing. Mark Thompson, Senior Vice President, Product Management, and Ryan Sanders, Product Marketing Manager, at Keyfactor, answered these questions during their joint session at the 2020 Critical Trust Virtual Summit.

In this blog, I’ll break down the core concepts covered in our session and provide best practices for multi-cloud SSH key management.

SSH seems to be everywhere. 84% of today’s enterprises use SSH protocol to some extent in their infrastructure.

Why? Because SSH a convenient way to authenticate and secure connections, not to mention it comes built into Linux and Unix-based systems. SSH is used widely by system admins to remotely access systems, and increasingly, in automated processes, such as secure copy and backup or secure file transfers.

Public-key authentication is the de facto choice for SSH access. Unlike passwords, SSH keys are not vulnerable to password brute-force attacks and require no user interaction to enable access, making them ideal for automated processes.

Because of their benefits, the proliferation of SSH keys is unparalleled. It is not uncommon to discover 50 to 200 keys per server, or to see 1 million keys in an organization.

Despite the high privilege access that they provide, they are not tightly controlled in many situations. This is creating a high-risk situation. With no management framework in place, more and more keys have fallen under the radar of auditors and security and risk management teams.

While most organizations are familiar with PKI and the management of X.509 certificates, the challenges related to SSH keys are entirely different.

For instance, most of the attention around certificates is focused on renewal before expiration (to prevent outages). However, SSH keys don’t expire, which creates the inverse risk of stale credentials and unauthorized access.

Below is a breakdown of the core differences:

The biggest risk enterprises are facing is SSH key sprawl. The root cause of SSH key sprawl is a lack of centralized key and access management, resulting from:

• Lack of control: Admins self-provision, copy and share keys without any restrictions
• No expiration: SSH keys do not expire and they are left unmanaged on systems for years
• Lack of policy: Root access is often granted even when it is not required
• Limited visibility: There is no centralized visibility of authorized and private keys
• Slow remediation: Security teams are hesitant to revoke or rotate keys.

Another cause of SSH key sprawl is personnel rotation. While there are standardized processes to remove departing personnel from the Active Directory, these processes often do not include their associated SSH keys. The complex web of trust relations between keys, users, and machines, makes the process of removing or rotating keys incredibly painful and risk-prone.

According to the findings of the 2020 Global Encryption Trends Study by nCipher and Ponemon, 57% of security teams admit that SSH keys are painful or very painful to manage.

Driven by business initiatives to migrate operations and data to the cloud, SSH keys are faced with a new norm. There are various reasons for the increased use of SSH keys across cloud platforms:

• 90% of all public cloud workloads today run on Linux (of which SSH is a core component)
• Multiple teams use SSH across clouds and configuration management tools (i.e. Ansible Control Nodes, Chef Infrastructure Servers, etc.)
• DevOps engineers use SSH to connect to Git repositories, remote workloads, etc.

As a sign of times, GitHub announced recently that they are moving away from using passwords, with SSH keys being a primary alternative. As developers make use of SSH keys in highly automated and distributed environments, the risks only continue to grow.

These developments underline that managing and protecting SSH keys in the cloud is critical for today’s enterprises. However, differences from cloud provider to cloud provider increase the complexity in managing SSH keys, especially as businesses adopt multi-cloud strategies.

• Google Cloud, AWS, and Azure all allow users to generate new keys or use an existing ssh keys
• SSH access is configured and managed differently in each cloud environment (see below)
• SSH keys are easy to generate, but hard to keep an up to date inventory of keys and trust relationships

It is important to remember that, even as some security responsibilities have shifted from the user to the cloud provider, it is always the user’s responsibility to manage and maintain their SSH keys. Ensuring consistency of control, policies, and visibility across all cloud platforms is critical.

Despite their importance, individuals are not handling SSH keys appropriately. Combined with cloud and workloads misconfigurations, an increased threat surface is created. Attacks involving the misuse of SSH and SSH keys have become commonplace. They’ve grown more sophisticated with each passing day, as known malware is modified to target SSH keys.

Here are just a few examples:

• Microsoft Azure disclosed a Linux-based attack that uses brute-force for password-based SSH logins to compromise VMs
• An attack against Exim servers (Linux email servers) downloads a shell script that adds an SSH key to the root account
• GoLang malware is a crypto-miner campaign that targets Linux servers and uses discovered SSH keys to target additional hosts and spread
• TrickBot malware updated its password grabber to target OpenSSH and OpenVPN
• FritzFrog botnet brute-forces access to millions of SSH IP addresses and inserts and attacker-owned public ssh key into the authorized keys file for persistent access
• Lemon_Duck malware uses port scanning to find machines listening on port 22 (SSH) and launches brute force attacks on the username root and a hardcoded list of passwords

The question that arises is how to mitigate these SSH-based attacks. Luckily, there are some easy to implement recommendations.

Disable root login

Root login is allowed on most Linux/Unix systems. While this recommendation seems counterintuitive, if password authentication and root password are enabled, the risk of attack is high.

Use public-key authentication

Password-based logins are vulnerable to brute-force attacks. Instead, key-based authentication is faster and more secure (so long as users are educated and keys are managed properly)

Use strong passphrase for keys

Private keys still need to be protected with a passphrase. Use best practices for passwords and protect the storage and use of private keys.

Set a custom SSH port

Port 22 is widely known by attackers and port scanning malware. Obscure SSH services by configuring a new port, when possible.

Keep SSH updated

Keep your server packages updated (this cannot be stressed enough). That includes OpenSSH patches against new vulnerabilities.

Additionally, a good recommendation is to limit the number of SSH key pairs per user. This is also critical since most users tend to reuse passphrases on their private SSH key. A single compromised key can be the gateway to all corporate critical resources.

The key (pun intended) to reducing your risk exposure is solid key management. This is critical, especially in multi-cloud environments. Not surprisingly, it starts with getting an inventory of what you have.

Discover the authorized_keys files on servers and map key-trust relationships to users and servers. This can be done effectively only by bringing into a centralized, automated inventory.

Based on the keys inventory, you can spot and analyze risks surfacing because of unauthorized SSH servers and user access, keys with root access, and weak or stale keys. Unused, unknown, and orphaned keys could be backdoor keys waiting to be exploited by adversaries.

To mitigate these risks, remove any unused and orphaned keys. Rotate or replace stale/weak keys, and ensure minimal levels of user access.

Deploy fresh keys in a controlled process. Enable self-service for admins and developers. Automate deployment of keys. Store private keys in a centralized location.

Stay attentive to ssh key rotation. Set rules for maximum key lifespans and alert users to rotate before “expiration”. Additionally, make it simple for users to rotate keys to minimize any friction.

Once you have set up a controlled environment it is easier to identify rogue keys created out of band. Regularly audit key types, rotation, and usage and ensure employee termination includes revocation and removal of respective SSH keys.

The Keyfactor SSH Key Manager is designed to address the challenges of SSH key management in modern multi-cloud and DevOps environments. The tool enables scalable and automated SSH key lifecycle management in even the most complex deployments.

Check out our eBook on 6 Steps to Take Back Control of Your SSH Keys.