Detailed explanation of HTTPS encryption protocol: TLS/SSL handshake process

1. Handshake and key negotiation process

The client authentication server based on RSA handshake and key exchange explains the TLS/SSL handshake process in detail as an example.

(1).client_hello

The client initiates a request and transmits the request information in clear text, including version information, cipher suite candidate list, compression algorithm candidate list, random numbers, extension fields and other information. The relevant information is as follows:

The highest supported TSL protocol version, from low to high, is SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2. Currently, versions lower than TLSv1 are basically no longer used;

List of cipher suites supported by the client. Each cipher suite corresponds to a combination of the four functions in the previous TLS principle: authentication algorithm Au (identity verification), key exchange algorithm KeyExchange (key negotiation), symmetric encryption algorithm Enc ( Information encryption) and information digest Mac (integrity verification);

A list of supported compression methods, used for subsequent information compression and transmission;

Random number random_C, used for subsequent key generation;

Extension field extensions support protocol and algorithm related parameters and other auxiliary information. The common SNI belongs to the extension field. The function of this field will be discussed separately later.

(2).server_hello+server_certificate+sever_hello_done

(a) server_hello, the server returns the negotiation information result, including the selected protocol version version, the selected cipher suite, the selected compression algorithm compression method, random number random_S, etc., where the random number is used for subsequent key negotiation ;

(b)server_certificates, configure the corresponding certificate chain on the server side for authentication and key exchange;

(3).Certificate verification

The client verifies the validity of the certificate. If the verification is passed, subsequent communication will be carried out. Otherwise, prompts and operations will be made based on different error conditions. Legality verification includes the following:

Trusted certificate path of the certificate chain, the method is as mentioned above;

Whether the certificate is revoked or not, there are two methods: offline CRL and online OCSP. Different clients will behave differently;

Validity expiry date, whether the certificate is within the valid time range;

Domain name domain, check whether the certificate domain name matches the current access domain name, and subsequent analysis of the matching rules;

(4).client_key_exchange+change_cipher_spec+encrypted_handshake_message

(a) client_key_exchange, after passing the legality verification, the client calculates and generates a random number Pre-master, encrypts it with the certificate public key, and sends it to the server;

(b) At this time, the client has obtained all the information needed to calculate the negotiation key: two plaintext random numbers random_C and random_S and the Pre-master generated by its own calculation, and the negotiation key is calculated;

enc_key=Fuc(random_C, random_S, Pre-Master)

(c) change_cipher_spec, the client notifies the server that subsequent communications will use the negotiated communication key and encryption algorithm for encrypted communication;

(d) encrypted_handshake_message, combines the hash values of all previous communication parameters and other related information to generate a piece of data, encrypts it using the negotiated key session secret and algorithm, and then sends it to the server for data and handshake verification;

(5).change_cipher_spec+encrypted_handshake_message

(a) The server decrypts the encrypted Pre-master data with the private key, and calculates the negotiation key based on the two plaintext random numbers random_C and random_S previously exchanged: enc_key=Fuc(random_C, random_S, Pre-Master);

(b) Calculate the hash value of all previously received messages, then decrypt the encrypted_handshake_message sent by the client, and verify the correctness of the data and key;

(c) change_cipher_spec, after passing the verification, the server also sends change_cipher_spec to inform the client that subsequent communications will use the negotiated key and algorithm for encrypted communications;

(d) encrypted_handshake_message, the server also combines all current communication parameter information to generate a piece of data, encrypts it using the negotiated key session secret and algorithm and sends it to the client;

(6). End of handshake

The client calculates the hash value of all received information, uses the negotiated key to decrypt the encrypted_handshake_message, and verifies the data and key sent by the server. If the verification is passed, the handshake is completed;

(7).Encrypted communication

Start encrypted communication using negotiated keys and algorithms.

Notice:

(a) The server can also require verification of the client, that is, two-way authentication. It can send client_certificate_request information in process 2. The client first sends client_certificate and certificate_verify_message information in process 4. The verification method of the certificate is basically the same. Certificate_verify_message uses the client's private A piece of key-encrypted data is obtained based on the negotiated communication information, and the server can use the corresponding public key to decrypt and verify it;

(b) Depending on the key exchange algorithm used, such as ECC, etc., the negotiation details are slightly different, but generally similar;

(c) The function of sever key exchange is that when the server certificate does not carry enough information, it is sent to the client to calculate the pre-master. For example, for DH-based certificates, the public key is not included in the certificate and needs to be sent separately;

(d) The change cipher spec can actually be used to notify the peer to change the currently used encrypted communication method. There is currently no in-depth analysis;

(e) alter message is used to indicate status changes or error messages during the handshake or communication process. Generally, the triggering conditions for alarm information are connection closure, receipt of illegal information, failure to decrypt information, user cancellation of operations, etc., receipt of alarm information Afterwards, the communication is disconnected or it is up to the recipient to decide whether to disconnect.

2. Session cache handshake process

In order to speed up the handshake establishment and reduce the performance degradation and resource consumption caused by the protocol (detailed analysis will be provided later), the TLS protocol has two types of session caching mechanisms: session identification session ID and session record session ticket.

Session ID is supported by the server and is a standard field in the protocol, so basically all servers support it. The server saves the session ID and negotiated communication information. 1M of memory in Nginx can save about 4000 session ID machine-related information, which takes up a lot of server resources. many;

Session ticket needs to be supported by both the server and the client. It is an extended field with a support range of about 60% (no reliable statistics and sources). The negotiated communication information is encrypted and sent to the client for storage. The key is only known by the server and occupies server resources. rare.

The main difference between the two is the location and method of saving negotiation information, which is similar to session and cookie in http.

When both exist, session_ticket (nginx implementation) takes precedence.

The handshake process is as follows:

Note: Although the handshake process takes 1.5 round-trips, the first piece of application data sent by the client to the server does not need to wait for the information returned by the server, so the handshake delay is 1*RTT.

(1).Session ID session ID

(a) If a connection has been established between the client and the server, the server will return the session ID after the handshake is successful and save the corresponding communication parameters in the server;

(b) If the client needs to establish a connection with the server again, the session ID in client_hello carries the recorded information and sends it to the server;

(c) The server retrieves the cache record based on the received session ID. If the retrieved cache record is not expired, the normal handshake process is followed;

(d) If the corresponding cache record is retrieved, change_cipher_spec and encrypted_handshake_message information are returned. The two information functions are similar. encrypted_handshake_message is the hash value of the current communication parameters and master_secret;

(f) If the client can verify the data encrypted by the server, the client also sends change_cipher_spec and encrypted_handshake_message information;

(g) If the server verifies that the data passes, the handshake is successfully established and normal encrypted data communication begins.

(2). Session recording session ticket

(a) If a connection has been established between the client and the server, the server will carry the encrypted session_ticket information in the new_session_ticket data and the client will save it;

(b) If the client needs to establish a connection with the server again, the encrypted information is carried in the extended field session_ticket in client_hello and sent to the server together;

(d) If the decryption is successful, change_cipher_spec and encrypted_handshake_message information will be returned. The two information functions are similar to those in session ID;

(f) If the client can verify the data encrypted by the server, the client also sends change_cipher_spec and encrypted_handshake_message information;

(g) If the server verifies that the data passes, the handshake is successfully established and normal encrypted data communication begins.

3. Reestablish the connection

Reestablishing the connection renegotiation is the process of giving up the TLS connection in use and re-carrying out identity authentication and key negotiation. The characteristic is that it can re-authenticate, update keys or algorithms without disconnecting the current data transmission, so server-side storage and caching information can be maintained. Both the client and the server can initiate the process of reestablishing the connection. Currently, Windows 2000 & XP and SSL 2.0 do not support this.

(1).The server reestablishes the connection

Server-side reconnection typically occurs when a client accesses protected data. The basic process is as follows:

(a) A valid TLS connection is established and communicates between the client and the server;

(b) Client access to protected information;

(d) After receiving the hello_request message, the client sends the client_hello message and starts to re-establish the connection.

(2).Client reestablishes connection

The client reestablishes the connection generally to update the communication key.

(a) A valid TLS connection is established and communicates between the client and the server;

(b) The client needs to update the key and proactively send client_hello information;

(c) After receiving the client_hello message, the server cannot immediately recognize that the message is not application data, so it will be submitted to the next step for processing. After processing, it will return a notification that the message requires reestablishing the connection;

(d) The server will not immediately stop sending data to the client before determining to reestablish the connection. It may happen at the same time or there may be cached data that needs to be sent to the client, but the client will not send any more information to the server;

(e) After the server recognizes the connection reestablishment request, it sends the server_hello message to the client;

(f) The client is also unable to immediately determine that the information is not application data, and will submit it to the next step for processing. After processing, it will return a notification that the information requires reestablishing the connection;

(g) The client and server start a new process of reestablishing the connection.

4. Key calculation

The previous section mentioned the two random numbers random_C and random_S transmitted in plain text and the Pre-master exchanged between the server and the client through encryption. The three parameters serve as the basis for key agreement. This section discusses the basic calculation process of key agreement and the use of keys during communication.

(1). Calculate Key

Involving the parameters random client and random server, Pre-master, Master secret, key material, when calculating the key, both the server and the client have these basic information. The exchange method is explained in the previous section. The calculation process is as follows:

(a) The client uses encryption algorithms such as RSA or Diffie-Hellman to generate Pre-master;

(b) Pre-master combines the two random numbers of random client and random server to calculate the Master secret through PseudoRandomFunction (PRF);

(c) Master secret combines the two random numbers of random client and random server to obtain the Key material through iterative calculation;

The following are some important records, which can solve the doubts of some friends who love in-depth research. I will share the copied materials with you:

(a) The first two bytes of the PreMaster secret are the TLS version number. This is a more important version number used to check the handshake data, because during the Client Hello phase, the client will send a list of cipher suites and currently supported The SSL/TLS version number is given to the server and is transmitted in clear text. If the handshake packet is cracked, the attacker is likely to modify the packet and select a less secure cipher suite and version for the server. , thereby cracking the data. Therefore, the server needs to compare the PreMaster version number decrypted in the ciphertext with the version number in the previous Client Hello stage. If the version number becomes lower, it means that it has been altered, and it will immediately stop sending any messages. (copy)

(b) Whether it is the client or the server, a random number is needed so that the generated key will not be the same every time. Since the certificate in the SSL protocol is static, it is necessary to introduce a random factor to ensure the randomness of the negotiated key.

For the RSA key exchange algorithm, the pre-master-key itself is a random number, plus the random number in the hello message, and the three random numbers are finally derived from a symmetric key through a key exporter.

The existence of pre master is that the SSL protocol does not trust that each host can generate completely random random numbers. If the random numbers are not random, then the pre master secret may be guessed, so it is not appropriate to only use the pre master secret as the key. Therefore, new random factors must be introduced, so the key generated by the client and server plus the three random numbers of pre master secret is not easy to guess. One pseudo-random may not be random at all, but three pseudo-random numbers Randomness is very close to randomness. For every additional degree of freedom, the randomness increases by more than one.

(2).Key usage

After 12 rounds of iterative calculations, Key will obtain 12 hash values, which are grouped into 6 elements. The list is as follows:

(a) mac key, encryption key and IV are a set of encryption elements, used by the client and server respectively, but both sets of elements are obtained by both parties at the same time;

(b) The client uses the client group element to encrypt the data, and the server uses the client element to decrypt the data; the server uses the server element to encrypt, and the client uses the server element to decrypt;

(c) Different keys are used in different directions of two-way communication, and it requires at least two attempts to crack the communication;

(d) encryption key is used to symmetrically encrypt data;

(e) IV is used as the initialization vector of many encryption algorithms. Specifically, symmetric encryption algorithms can be studied;

(f) Mac key is used for data integrity verification;

(3).Data encryption communication process

(a) Segment the application layer data into appropriate blocks;

(b) Number the fragmented data to prevent replay attacks;

(d) Calculate the MAC value and compress the data to form the transmission data;

(e) Use the client encryption key to encrypt the data and send it to the server;

(f) After receiving the data, the server uses the client encrytion key to decrypt, verify the data, decompress the data, and reassemble it.

Note: The calculation of MAC value includes two Hash values: client Mac key and Hash (number, packet type, length, compressed data).

5. Packet capture analysis

We will not analyze the packet capture in detail. According to the previous analysis, the basic situation can be matched. Based on the usual process of locating problems, I would like to mention some things that need attention:

(1). When capturing HTTP communication, you can clearly see the communication headers and the plain text of the information. However, HTTPS is encrypted communication and you cannot see the relevant headers of the HTTP protocol and the plain text information of the data.

(2). Packet capture HTTPS communication mainly includes three processes: TCP connection establishment, TLS handshake, TLS encrypted communication, mainly analyzing the handshake establishment and status information of HTTPS communication.

(3).client_hello

According to the version information, you can know the highest protocol version number supported by the client. If it is a lower version protocol such as SSL 3.0 or TLS 1.0, be very careful that some handshake failures may occur due to the low version;

Determine whether SNI is supported based on the server_name field in the extension field. If it exists, it is supported, otherwise it is not supported. It is very useful for locating handshake failure or certificate return error;

The session ID is part of the standard protocol. If no connection has been established, the corresponding value is empty. If it is not empty, it means that the corresponding connection has been established and cached before;

Session record session ticket t is an extended protocol part. The existence of this field indicates that the protocol supports session tickets. Otherwise, it does not support it. If it exists and the value is empty, it means that the connection has not been established and cached before. If it exists and the value is not empty, it means that there is a cached connection.

(4).server_hello

The highest version of the protocol supported by the server can be inferred based on the TLS version field. Different versions may cause handshake failure;

Determine the encryption protocol supported by the server based on cipher_suite information;

(5).ceritficate

The certificate chain configured and returned by the server is compared with the server configuration file based on the certificate information to determine whether the request is consistent with expectations. If not, whether to return the default certificate.

(6).alert

The alarm information alert will explain the reason for the failure to establish the connection, that is, the alarm type, which is very important for locating the problem.