Demystifying the TLS Handshake: What it is and how it works

The Transport Layer Security (TLS) is designed to add security to network communications.  It is the difference between HTTP and HTTPS when browsing the Internet.

Using TLS creates additional work for the client and the server, but it has its benefits, including:

•     Confidentiality: TLS wraps traffic in an encrypted tunnel.  This makes it impossible for an eavesdropper to read or modify the traffic on its way to its destination.
•     Authentication: TLS proves the identity of the server to the client.  This is helpful in protecting against phishing sites.
•     Integrity: TLS includes protections that help with identifying if data has been modified or corrupted in transit.

All of these are valuable features when browsing the web.  This is why TLS is so popular and why most visits to a website begin with a TLS handshake.

Like a handshake in real life, the TLS handshake is an introduction.  It establishes that two computers want to talk to one another in a secure fashion.

A TLS handshake also defines some of the rules for this conversation.  Both the client and the server agree that they want the benefits of TLS, but they need to agree on the details.  A TLS handshake gets them from an initial “Hello” to the point where they can start talking privately.

Before diving into the details of the TLS handshake, it’s important to understand some key vocabulary.  TLS is a security-focused protocol, which means that it uses a lot of cryptography.  Some important terms to know when talking about TLS include:

• Asymmetric Encryption: Asymmetric or “public key” cryptography uses two related keys: a public and a private key.  Anything encrypted with a public key can be decrypted with the corresponding private key.  Similarly, a digital signature generated with a private key can be validated with the associated public key.
• Symmetric Encryption: Symmetric encryption uses the same key for both encryption and decryption.  This is useful because it is more efficient than encryption with asymmetric cryptography.  The TLS handshake is designed to set up a shared symmetric encryption key.
• Cipher Suites: A cipher suite is a combination of cryptographic algorithms used in the TLS protocol.  This includes an asymmetric encryption algorithm for the handshake, a symmetric encryption algorithm for encrypting the data sent over the connection, a digital signature algorithm, and a hash function used for verifying that data has not been corrupted in transit.
• Digital Certificate: A digital certificate proves the ownership of a public key.  Servers present a digital certificate during the TLS handshake so that the client knows that they’re communicating with the right person.

The goal of the TLS handshake is for the client and the server to agree on a shared symmetric encryption key in a secure fashion.  To do so, they use asymmetric encryption, which allows encrypted messages to be sent using only a public key.

The details of the TLS handshake depend on the asymmetric encryption algorithm used.  A client and a server using RSA go through the following steps:

*Image Source

1. Client Hello: The client kicks off the TLS handshake by sending configuration information to the server.  This includes data like the TLS version(s) supported by the client, which cipher suites it can use, and some random data called the “client random”.
2. Server Hello: The server responds with a message including its choices of TLS version and cipher suite, its digital certificate, and the “server random”.
3. Premaster Secret: The digital certificate provided by the server contains its public key, which can be used for encrypting messages sent to the server.  After verifying the validity of the certificate, the client generates another random value called the premaster secret.  Using the server’s public key, the client encrypts this value and sends it to the server.
4. Session Keys: The server uses its private key to decrypt the premaster secret.  At this point, both the client and the server have the client random, server random, and the premaster secret.  From these, they calculate a shared session key.  Since the premaster secret is a secret value – it was only sent in an encrypted form over the network – the session key is secret as well.
5. Client Finished: To complete the TLS handshake, the client sends a message to the server that is encrypted with the session key.  The server decrypts this message and verifies its correctness.  If it matches, the server knows that the client correctly calculated the session key.
6. Server Finished: The server also sends a message encrypted with the shared session key.  Decrypting and verifying this message proves to the client that the server correctly calculated the session key.

At this point, the client and server have a shared encryption key known only to them.  For the rest of the TLS session, all messages will be encrypted using this session key.

As mentioned above, the use of different asymmetric encryption algorithms can cause small changes in the steps.  For example, if the client and the server choose to use Diffie-Hellman instead of RSA, the contents of the messages sent in steps 2 and 3 are slightly modified:

• In step 2, the server will send a digitally-signed package containing the client random, server random, and its Diffie-Hellman parameter.
• After validating the digital signature, the client responds with a message containing its Diffie-Hellman parameter instead of a premaster secret.

Using these Diffie-Hellman parameters and the client and server randoms, it is possible for the client and server to calculate a shared, secret private key.  In the end, Diffie-Hellman and RSA produce the same result in slightly different ways.

The process described here is the basic TLS handshake, which is used in most TLS sessions.  However, it is possible to add additional steps that provide certain benefits.

For example, the basic TLS handshake uses a digital certificate to authenticate the server to the client but does not authenticate the client to the server.  A modified version of the TLS handshake can send a client’s digital certificate to the server as well, achieving mutual authentication.