Reign in Your Rogue Admins: How to Manage SSL Certificates

As organizations embrace digital transformation and cutting edge technologies, the scale and distribution of SSL/TLS certificates has grown rapidly, creating new challenges.

Sami Van Vliet, Principal Product Manager at Keyfactor, and Mrugesh Chandarana, Senior Product Manager at HID, discussed best practices to manage SSL certificates in their session at the Keyfactor Virtual Summit. This blog provides a recap of insights and recommendations covered in the session.

The proliferation of IoT devices in the enterprise ecosystem and the advent of virtualization and DevOps practices have skyrocketed the use cases of SSL/TLS certificates.

As organizations push for more rapid and efficient deployment of business applications, many DevOps teams deploy certificates without coordination with IT and security teams.

Lack of visibility and control over certificates and issuing CAs is a serious problem. 74% of the organizations don’t know how many keys and certificates their organization actually has. Having visibility into your certificates is the foundation for managing and protecting these valuable assets.

Despite their importance, SSL certificates are often left untracked and unmanaged. Nearly every enterprise has experienced an application outage due to an expired certificate. In fact, a recent Keyfactor and Ponemon report indicated that 87% of the respondents have experienced unplanned outages due to expired certificates.

Another trend we’ve witnessed is the shortening of SSL certificate validity. Since 2012, the CA/Browser (CA/B) forum has mandated the reduction of publicly-issued certificate lifespans from 60 months down to 27 months.

With the use of SSL/TLS certificates expanding and creating bigger threat surface, Google more recently proposed in 2019 a reduction to one-year lifespans, without success. Apple then strong-armed the CA/B forum by limiting certificate lifespans in its ecosystem to <398 days. These developments prompted Google Chrome and Mozilla to follow suit, forcing the CA/B Forum to set the new lifespan of certificates to just 13 months or 398 days.

In the years to come, we expect the trend to continue. Even so, it’s not uncommon today for organizations to issue certificates with a validity of only a few days or hours.

With the increased use of certificates and shortened lifespans, the workload for PKI and security teams has effectively doubled, yet their budget remains the same. This has created a number of new challenges for organizations:

• Lack of visibility. IT teams need to have complete visibility not just for the server certificates, but also for client authentication certificates.
• Manual processes. Certificate requests, renewals, and provisioning using manual methods and spreadsheets take hours or days. In addition to being time-consuming, it’s also prone to error.
• Shifting IT staff. Admins come and go, emails might also change, and as a result, expiration notifications are not received. Outages due to expired certificates are only one email away.
• No accountability. Many times, admins deploy “DIY” or built-in CAs without any security oversight.
• Slow response. When an expired certificate outage happens, identifying and replacing the expired certificate in all locations is a time-consuming process that might take many hours or even days.

Beyond the risk of outages, algorithm vulnerabilities and advances in computing have driven the need for crypto agility. The key reasons demanding organizations to be crypto agile are:

• Mass revocation: What happens if certificates are revoked by your CA on short notice (i.e. DigiCert revoked 50,000 in July 2020)?
• Algorithm deprecation: Cryptographic algorithms are constantly evaluated for vulnerabilities. Migration to SHA-2 has taken years and it won’t be the last change in algorithms.
• CA Compromise: If a CA is breached by an attacker, the entire chain of trust is broken. Certificates issued by the CA are rendered invalid and new certificates must be installed on all servers with certificates from the compromised CA.
• CA Agility: The ability to add/remove or switch from one CA to another is critical for business agility to avoid CA lock-in.
• Crypto-Library bugs: If a bug is found with the key-generation functions of a cryptographic library, then all keys generated since the bug was introduced must be replaced. Vulnerabilities like Heartbleed in OpenSSL expose certificates and private keys to risk.
• Quantum Crypto: advancements in quantum require organizations to prepare for the inevitable implementation of post-quantum algorithms. The time to start planning for quantum-safe crypto is today.

The requirement to be crypto agile makes automation a must-have to deal with all these challenges more effectively.

1.      Gain Visibility

To address TLS server certificate risks, organizations should establish and maintain clear visibility across all SSL/TLS certificates in their environment. The inventory should focus on:

• Certificate Authorities (CAs), which are the source of most certificates and synchronize with internal and public CAs.
• Network SSL/TLS Endpoints to identify where all certificates are installed across the network.
• Key and Certificate Stores to locate certificates that are not exposed to a port on the network.

The identification process of corporate SSL/TLS certificates should locate where all certificates are installed and create a centralized inventory of all issued certificates. Additionally, it should identify owners of all certificates and domains and understand associated web servers and apps.

2.      Remediate

Once all certificates have been identified, the next step is to remediate any weak points. This process should find vulnerabilities like weak keys or hashes and control the use of wildcard and self-signed certificates.

Expired certificates should be renewed, and private keys must always be protected. Finally, weak, or rogue certificates must be removed from the certificate ecosystem.

The recent hour-long outage of Spotify, affecting their streaming service due to an expired wildcard certificate, is a prime example of why visibility and remediation are critical.

The Spotify outage provided many useful lessons. Namely, wildcard certificates are useful in certain cases, but their usage should be limited and continuously monitored. Where wildcard certificates are required, generate them with different private keys to minimize the impact of an outage.

3.      Enforce Policy

Policy should be enforced at the enterprise level to cover all possible certificate owners and consumers, such as network admins, DevOps, security teams, PKI admins, and IT staff.

An enterprise-wide certificate management policy should define certificate owners and their roles and responsibilities. Certificate issuance and renewal must be standardized, and an early warning system must be set up to send expiration alerts to certificate owners. Finally, policies should be in place for certificate revocation and removal.

4.      Automate and Integrate

The broadening use of and reliance on SSL server certificates to secure important applications is rendering manual certificate management impractical. Automation has become important to handle in a cost-effective and efficient manner all certificate management issues.

Automation should be used wherever possible for the enrollment, installation, monitoring, and replacement of certificates. An automation solution should enable users to self-service their own certificates and should allow for integration with cloud and DevOps toolsets.

5.      Monitor and Report

To make sure an automation solution is working correctly and because of the broad use of certificates in all critical communications, operational or security failures related to SSL/TLS certificates can significantly impact the business operations. SSL/TLS certificates should be continuously monitored to prevent outages and security vulnerabilities.

The certificates should be monitored for impending expiration and for situations in which they are not operating, are not configured properly, or are vulnerable. Additionally, the scan should locate certificates generated outside of established procedures.

The monitoring process should check for any CRL/OCSP activity and regularly audit the certificate landscape for continuous compliance with certificate policies.

Managing SSL certificates often begins with manual processes (i.e. spreadsheets) and extends all the way to dynamic, zero-touch automation. Depending on your organizations needs, you’ll need to determine where you stand and how to improve your posture to prevent outages and stay ahead of the crypto-curve.

Download the Certificate Management Maturity Model to get started.