TLS (v1.2) protocol

The full name is Transport Layer Security Protocol. The previous generation is Secure Sockets Layer (SSL, insecure), which has a wide range of uses. The most famous one is used for http, which upgrades http to https protocol. The latest version is TLSv1.3.
TLS ensures that the transmitted data is confidential .
TLS works between the transport layer protocol and the application layer protocol .
TLS protects against man-in-the-middle attacks by using certificates and trusted certificate authorities to authenticate the server (or client).

TLS Design Goals

Meet high efficiency, cross-platform, scalability and versatility.

The data transmitted at both ends of the communication should be safe and cannot be forged and tampered with.

The TLS/SSL protocol is standard, and any developer can implement the protocol based on the TLS/SSL RFC design specification, and developers can easily introduce the TLS/SSL protocol into their applications.

Cryptographic algorithms are constantly iterative. As time goes by, more secure algorithms will appear. In order to ensure continuous security, the TLS/SSL protocol allows dynamic introduction of new algorithms. Since the environment between the communication bodies is different, the protocol allows both parties to negotiate a cryptographic algorithm that both parties support.

The solution must be efficient. The TLS/SSL protocol involves the operation of many cryptographic algorithms, which increases communication delay and machine load. But now, some new technologies and solutions are gradually improving the efficiency of the TLS/SSL protocol.

cipher suite

A cipher suite generally refers to a combination of several cryptographic algorithms. It mainly includes authentication algorithm (via certificate), key agreement algorithm, encryption algorithm or encryption mode, encryption primitive of HMAC algorithm, encryption primitive of KDF, etc. The strength of the cipher suite determines the security of TLS. Web servers can usually configure cipher suites, such as nginx, so some https websites have banned the use of risky cipher suites.

TLS 1.3 supports several cipher suites: TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256, etc.

Use openssl to view the supported cipher suites below

openssl ciphers -V | column -t
#选取第一条结果，各列说明如下：
#第一列：数值代表密码套件的编号，每个密码套件的编号由IANA（互联网数字分配机构）定义。
#第二列：代表密码套件的名称，虽然密码套件编号是一致的，不同的TLS/SSL协议实现其使用的名称可能是不一样的。
#第三列：表示该密码套件适用于哪个TLS/SSL版本的协议。
#第四列：表示密钥协商算法。
#第五列：表示身份验证算法。
#第六列：表示加密算法、加密模式、密钥长度。
#第七列：表示HMAC算法。其中AEAD表示采用的是AEAD加密模式（比如AES128-GCM），无须HMAC算法
    1                    2                3       4       5         6              7
#0x13,0x02  -  TLS_AES_256_GCM_SHA384  TLSv1.3  Kx=any  Au=any Enc=AESGCM(256)  Mac=AEAD

key block

Generally speaking, to ensure the confidentiality and integrity of the database, TLS needs to use a symmetric encryption algorithm (faster than asymmetric encryption calculation) and a message authentication code (MAC), so a set of keys is required, and these keys are composed of The master key is generated using a key derivation function (KDF) . The number

of key blocks and the length depend on the negotiated symmetric encryption algorithm and HMAC algorithm, that is, determined by the cipher suite. For example, if the client and the server agree to use AES-256-CBC as the encryption algorithm and HMAC as the message authentication code, they must have the same AES key, HMAC key, initialization vector, etc., which are called key blocks. .

master key and pre-master key

The master key is generated by the preliminary master key, and the preliminary master key is also converted into the master key by using the password derivation function (KDF) , and the preliminary master key is obtained through negotiation (using RSA or DH (ECDH) algorithm) between the communication parties.

First use the preliminary master key to convert a fixed-length master key, and then convert the master key to any number of key blocks of any length.

Key Derivation Function (KDF)

Common KDFs include PBKDF2, PRF, etc.

The input parameters of KDF include the input value and salt. If the input value and salt are the same, the output is also the same. For both communication parties, the salt of each connection changes, but in order to ensure consistent output results, the salt value held by both parties must be the same each time.

TLS1.2 uses PRF for password derivation (TLS1.3 uses HKDF ), which requires three parameters (secret, label, seed), where: secret is the input, label is a fixed value, and seed is salt, which is a random value , the calculation process is roughly as follows:

P_hash(secret, seed) = HMAC_hash(secret, A(1) + seed) +
                             HMAC_hash(secret, A(2) + seed) +...
                             HMAC_hash(secret, A(i) + seed)
                             
PRF(secret, label, seed) = P_<hash>(secret, label + seed)

/*
 其中，A(0) = seed，A(i) = HMAC_hash(secret, A(i-1))
 hash指的是单向hash函数，如sha256等等。
 label是个固定字符串，ASCII码。
 以sha256为例，一次HMAC_sha256产生的数据为32字节。如果想得到80字节数据，则令i=3，这样会得到
 96字节数据，丢弃后16字节就可得到80字节。
*/

key agreement

In order to securely transmit key blocks for symmetric encryption and MAC, you cannot use another key to encrypt these key blocks, which will fall into an infinite loop of encryption, we need a dynamic key, which is different for each session , which is also forward secrecy.
At this time, the public key algorithm can be used.

Each client and server generate a preliminary , and the value is different each time.
After the session ends (after the connection is closed), the preliminary master key will be released automatically and will not be stored persistently.
The pre-master key must be kept confidential to ensure that an attacker cannot obtain the pre-master key by any means.

RSA negotiation

The steps are as follows:
1. The client initiates a connection request to the server, and the server sends the public key of the RSA key pair to the client.
2. The client generates a preliminary master key through a random number generator, encrypts it with the server's public key and sends it to the server.
3. The server decrypts the preliminary master key. If the decryption is successful, the preliminary master key will take effect.

Although the RSA negotiation has simple steps, it cannot guarantee forward security (referring to the decryption of historical data due to leaked keys).

DH negotiation

The steps are as follows:
1. The client initiates a connection request to the server.
2. The server generates an RSA key pair and sends the public key to the client.
3. The server generates the DH parameter and server DH key pair, signs the DH parameter and the server DH public key with the RSA private key, and finally sends the signature value, DH parameter, and server DH public key to the client.
4. The client verifies the signature through the server RSA public key, and obtains the DH parameters and the server DH public key.
5. The client generates the client's DH key pair through the DH parameters, and sends the client's DH public key to the server.
6. The client calculates the preliminary master key through the client DH private key and the server DH public key.
7. The server receives the client's DH public key and calculates the preliminary master key with its own DH private key.
Negotiations have since been completed.

Static DH

In each session, the DH parameter sent by the server is the same as the DH public key (saving calculation time). This is called a static DH algorithm, which cannot guarantee forward security.

Dynamic DH (DHE)

Each time the communication parties connect, the server DH key pair generated by the server through the DH parameter is different. After the session ends, the server DH key pair will also become invalid. Since the DH key is different for each session, this This method can provide forward security.

TLS protocol suite

TLS consists of several protocols, the two most important being the record protocol and the handshake protocol . The record protocol is a low-level protocol, and the handshake protocol is a high-level protocol. The record protocol defines a packet format that encapsulates data from a higher layer protocol and sends that data to another party. The handshake protocol is the key .

Handshake protocol (general name of TLS high-level protocol)

The handshake protocol is the core protocol of TLS, and it is the process in which the client and the server establish a session key to initiate secure communication. The handshake protocol has four parallel protocols: TLS Handshaking Protocols, Alert Protocol , Application Data Protocol, and Change Cipher Spec Protocol .

The main work of the handshake protocol is that the client and the server negotiate a cipher suite recognized by both parties , and negotiate encryption parameters based on the cipher suite. A message of the handshake protocol can be composed of multiple sub-messages, and multiple sub-messages are finally encapsulated into a message (or multiple messages) by the TLS record layer protocol and handed over to TCP for processing.

handshake agreement

In the TLS handshake process, the client and server play different roles, and the two parties negotiate. The client proposes some configuration (TLS version and cipher suites, precedence, etc.), and the server chooses which configuration to use. The server generally follows the client's preference, but may choose other configurations.
An example of the handshake process between client and server is as follows:

The client establishes a TLS connection with the server, and each TLS session generates a DH key pair, which is sent to the encryption algorithm supported by the server and the client DH public key, including 32-byte random numbers and optional information (additional parameters, etc. ).

The server generates a DH (Diffie-Hellman) key pair, and calculates the negotiated preliminary master key secret based on the server's DH private key and the sent client's DH public key, and uses KDF to derive the master key keys. The server responds with a ServerHello message, which includes: password used to encrypt TLS records, server DH public key, 32-byte random number, certificate, signature of all messages in ClientHello and ServerHello (using the private key corresponding to the certificate public key Calculation), MAC and signature of the same information. MAC is calculated using the public key.

When the client receives the ServerHello message, it first verifies the validity of the certificate, verifies the signature, calculates the DH session key and obtains the symmetric key from it, and verifies the MAC sent by the server. Once all verifications are done, the client is ready to send encrypted messages to the server.

work process

1. The client and server exchange hello sub-messages , which exchange random values ​​and supported cipher suite lists, negotiate cipher suites and corresponding algorithms, and check whether the session can be restored.
2. Exchange certificates and cryptographic information, allowing the server and client to verify each other's identities, divided into server authentication and client authentication, generally only verifying server certificates.
3. Exchange cryptographic parameters, and the client and server calculate the same preliminary . 4. Generate the master
key by preparing the master key and the random value of the server and client . 5. The handshake protocol provides encryption parameters to the record protocol . 6. The client and the server verify each other's Finished sub-message to avoid tampering with the message of the handshake protocol. Use Wireshark to capture a complete TLS v1.2 handshake process as follows:

sub message

The handshake protocol consists of many sub-messages (as shown in the figure above). The sub-messages in the handshake protocol must be sent in a specific order. For the client and server, if the messages in the specific a fatal error will occur. For example, after the client sends a ClientHello message, the next received message must be a ServerHello message sent by the server.
The sub-message structure is as follows:

//消息类型枚举
enum {
    hello_request(0), client_hello(1), server_hello(2),
    certificate(11), server_key_exchange (12),
    certificate_request(13), server_hello_done(14),
    certificate_verify(15), client_key_exchange(16),
    finished(20), (255)
} HandshakeType;

struct {
    //消息类型
    HandshakeType msg_type;
    //消息长度
    uint24 length;
    //消息内容
    select (HandshakeType) {
        case hello_request:       HelloRequest;
        case client_hello:        ClientHello;
        case server_hello:        ServerHello;
        case certificate:         Certificate;
        case server_key_exchange: ServerKeyExchange;
        case certificate_request: CertificateRequest;
        case server_hello_done:   ServerHelloDone;
        case certificate_verify:  CertificateVerify;
        case client_key_exchange: ClientKeyExchange;
        case finished:            Finished;
        //下面的子消息与扩展有关
        case certificate_url:     CertificateURL;
        case certificate_status:  CertificateStatus;
        case session_ticket:      NewSessionTicket;
        //......
    } body;
} Handshake;

Several important sub-messages

ClientHello

When the client connects to the server for the first time, the first message sent is ClientHello, which has the following structure:

struct {
    //客户端支持的TLS最高版本号，也就是客户端支持<=client_version的TLS版本
    ProtocolVersion client_version;
    //客户端的随机数，用来生成预备主密钥，计算主密钥和密钥块，校验完整的握手消息，防止重放攻击等
    Random random;
    /*
     和会话恢复有关。客户端和服务器完成一次握手，服务器会发送一个会话ID给客户端，
     下次连接的时候客户端会发送该会话ID给服务器，如果服务器端校验存在该会话ID，
     就会恢复上一次连接，从而减少握手过程，提升效率。
     对于一次全新的连接来说，客户端传递的session_id为空。
    */
    SessionID session_id;
    //客户端支持的密码套件列表,排在第一个的优先选择。
    CipherSuite cipher_suites<2..2^16-2>;
    //客户端支持的压缩算法，和TLS记录协议一样，一般不启用压缩算法。
    CompressionMethod compression_methods<1..2^8-1>;
    //TLS协议支持扩展，方便日后的升级。
    select (extensions_present) {
        case false:
          struct {};
        case true:
          //扩展内容
          Extension extensions<0..2^16-1>;
    };
} ClientHello;

ServerHello

After receiving the Clinet Hello message from the client, the server selects a cipher suite supported by both parties based on the cipher suite passed by the client and combined with the cipher suite configured on the server. If the match is incorrect, the handshake fails.

struct {
    //服务器根据客户端支持的版本选择一个双方都支持的版本
    ProtocolVersion Server_version;
    //服务器的随机数
    Random random;
    /*
     如果客户端传输的session_id不为空，则服务器会从缓存中寻找是否存在同样的session_id，
     如果找到表示可以进行会话恢复，可以复用上一个连接。如果没有找到，则进行一个完整的握手过程，
     传递一个新的session_id。
    */
    SessionID session_id;
    //服务器根据客户端支持的密码套件列表，选择一个双方都支持的密码套件。
    CipherSuite cipher_suite;
    //根据客户端支持的压缩算法，选择一个压缩算法，一般不启用压缩算法。
    CompressionMethod compression_method;
    //服务器根据客户端发来的扩展进行处理，客户端没有发送的扩展不能出现在此列表中。
    select (extensions_present) {
        case false:
          struct {};
        case true:
          //扩展内容
          Extension extensions<0..2^16-1>;
    };
} ServerHello;

Certificate

After the server sends the ServerHello message, it usually sends the Certificate sub-message immediately. The Certificate message is optional. According to the negotiated cipher suite, the server chooses whether to send the certificate message. In an HTTPS website, the server generally sends a certificate. If the negotiated cipher suite is DH_anon or ECDH_anon, the server does not send this sub-message.

The server generally sends the certificate for two purposes: one is identity verification, and the other is that the certificate contains the server's public key, which is combined with the key agreement algorithm of the cipher suite to negotiate a preliminary master key.
The message structure is as follows:

opaque ASN.1Cert<1..2^24-1>;
struct {
    ASN.1Cert certificate_list<0..2^24-1>;
} Certificate;

Each ASN.1Cert structure is a certificate, the first one is the certificate of the server entity, and the rest are intermediate certificates.
See the certificate structure: The HTTPS certificate

certificate contains the digital signature algorithm. When the client sent ClientHello before, it included the signature_algorithms extension, which contains the digital signature algorithm supported by the client. If the signature algorithm supported by the client does not include the signature algorithm of the certificate, then The handshake fails directly.

ServerKeyExchange

Optionally, this message needs to be sent if the certificate contains insufficient information for key exchange. Generally, DHE and ECDHE-related cipher suites need to send this kind of sub-message, because the client needs to send dynamic DH parameters and public key information (not in the certificate) every time it connects to the server, and the server private key is required to transmit these DH information ( and the server public key in the certificate are a pair) to sign.

However, DH_anon and ECDH_anon (without identity verification) need to send this sub-message because they do not have a Certificate sub-message, that is, no certificate carries DH information.
In addition, the RSA cipher suite does not need this sub-message, because the client can calculate the preliminary master key, encrypt it with the server's public key, and then decrypt it with the private key to obtain the preliminary master key.
DH_DSS and DH_RSA do not need this sub-message, and the Certificate sub-message already contains DH information.

The structure of the message is as follows:

struct {
    //判断密码套件中的密钥协商算法
    select (KeyExchangeAlgorithm) {
        //不需要身份验证，需要动态传递DH信息和DH公钥
        case dh_anon:
            ServerDHParams params;
        
        //动态DH
        case dhe_dss:
        case dhe_rsa:
            ServerDHParams params;
            //数字签名结构
            digitally-signed struct {
                //客户端随机数
                opaque client_random[32];
                //服务器随机数
                opaque server_random[32];
                //DH参数和公钥
                ServerDHParams params;
            } signed_params;
        
        //RSA、DH_DSS和DH_RSA无须传递Server Key Exchange子消息
        case rsa:
        case dh_dss:
        case dh_rsa:
        struct {} ;
        
        //如果是动态ECDH协商算法
        case ec_diffie_hellman:
            //ECC域信息
            ServerECDHParams   params;
            //对ServerECDHParams的签名
            Signature          signed_params;
    };
} ServerKeyExchange;
//服务器支持的密码套件，根据服务器配置的密码套件决定，这里举了个例子
enum {
    dhe_dss, dhe_rsa, dh_anon, rsa, dh_dss, dh_rsa, ec_diffie_hellman
} KeyExchangeAlgorithm;
//DH信息，包括参数和公钥
struct {
    opaque dh_p<1..2^16-1>;
    opaque dh_g<1..2^16-1>;
    opaque dh_Ys<1..2^16-1>;
} ServerDHParams;
//ECDHE参数和公钥
struct {
    //ECDH参数，ECC曲线
    ECParameters   curve_params;
    //ECC公钥
    ECPoint        public;
} ServerECDHParams;
//ECC曲线
struct {
    //曲线类型
    ECCurveType   curve_type;
    select (curve_type) {
        case explicit_prime:
        case explicit_char2:
        //使用定义好的命名曲线，选择客户端和服务器都支持的命名曲线
        case named_curve:
            NamedCurve namedcurve;
    };
} ECParameters;
//ECC公钥
struct {
    opaque point <1..2^8-1>;
} ECPoint;
//曲线类型
enum {
    explicit_prime (1),
    explicit_char2 (2),
    named_curve (3),
    reserved(248..255)
} ECCurveType;
//支持的命名曲线
enum {
    secp256k1 (22), secp256r1 (23), secp384r1 (24),
} NamedCurve;

ServerHelloDone

This message is sent after ServerHello, it is an empty message, the format is as follows:

//发送该消息的目的是通知客户端 服务器已经发送足够的消息，下面可以进行预备主密钥的协商。
//客户端收到该消息，可以进行证书校验、密钥协商等操作。
struct { } ServerHelloDone;

ClientKeyExchange

After the client receives the ServerHelloDone message from the server, it sends this message to negotiate and prepare the master key. There are two methods:
1. The client encrypts the pre-master key through RSA/ECDSP, sends it to the server, and the server decrypts it with the private key to obtain the pre-master key.
2. The dynamic DH parameters sent by the client server get the client DH public key and send it to the server. Both parties calculate the same preliminary master key based on the other party's DH public key and their own DH private key. It is necessary to choose a name supported by both parties. curve.

The message structure is as follows:

struct {
    //对应ServerKeyExchange中的KeyExchangeAlgorithm结构
    select (KeyExchangeAlgorithm) {
        case rsa:
            EncryptedPreMasterSecret;
        case dhe_dss:
        case dhe_rsa:
        case dh_dss:
        case dh_rsa:
        case dh_anon:
            ClientDiffieHellmanPublic;
        case ec_diffie_hellman:
            ClientECDiffieHellmanPublic;
        } exchange_keys;
  } ClientKeyExchange;
//加密后的预备主密钥，使用RSAES-PKCS1-v1_5加密方式。
struct {
    public-key-encrypted PreMasterSecret pre_master_secret;
} EncryptedPreMasterSecret;
//预备主密钥
struct {
    //应该传递客户端ClientHello消息传递的版本号，服务器需要校验此版本号等不等于ClientHello消息传递的版本号
    ProtocolVersion client_version;
    //一个46字节的随机数
    opaque random[46];
} PreMasterSecret;

//公钥编码方式
//implicit表示客户端DH公钥隐藏在证书中，explicit表示需要显式的传递公钥，传递的DH公钥没有任何加密处理。
enum { implicit, explicit } PublicValueEncoding;
//DH协商算法的公钥，如果客户端也有证书并把自己的证书发给服务器校验（implicit），那么该消息为空
struct {
    select (PublicValueEncoding) {
        //隐式
        case implicit:
        struct { };
        //显式，传递客户端DH公钥
        case explicit:
            opaque dh_Yc<1..2^16-1>;
    } dh_public;
} ClientDiffieHellmanPublic;

//ECDHE密钥协商算法的ECDH公钥
struct {
    select (PublicValueEncoding) {
        //隐式
        case implicit:
        struct { };
        //显式，传递客户端ECDH公钥
        case explicit:
            ECPoint ecdh_Yc;
    } ecdh_public;
} ClientECDiffieHellmanPublic;
//公钥结构
struct {
    opaque point <1..2^8-1>;
} ECPoint;

Finished

The Finished sub-message must be sent after the ChangeCipherSpec sub-message. If this order is not followed, a fatal error will be generated and the handshake will be interrupted. After the ChangeCipherSpec sub-message is sent, it indicates that it is ready for encryption, so the Finished sub-message sends encrypted data. The purpose of the Finished sub-message is to confirm that all handshake messages have not been tampered with.
The message structure is as follows:

//verify_data通过PRF函数计算
struct {
    opaque verify_data[verify_data_length];
} Finished;

//第一个参数master_secret是主密钥
//第二个参数labfinished_labelel，如果是客户端，则是"client finished"；如果是服务器，则是"server finished"
/*第三个参数handshake_messages是所有的握手协议消息。在TLS/SSL协议中，一般是客户端先发送
  Finished消息,服务器后发送。对于客户端来说，handshake_messages的内容包含所有发送的消息和接收到的消息，
  但不包括自己发送的Finished消息。对于服务器来说，handshake_messages的内容从Client Hello
  消息开始截止到Finished消息之前的所有消息，也包括客户端的Finished子消息。
  handshake_messages不包括Change Cipher Spec消息和Alert消息。
*/
verify_data = PRF(master_secret, finished_label, Hash(handshake_messages))
      [0..verify_data_length-1];

Compute the master key and key block

After the client and server negotiate a preliminary master key, they can calculate the master key. Before the ChangeCipherSpec protocol message is sent, the client and server need to calculate the master key .
The parameters for calculating the master key through the KDF function are as follows. After the calculation is completed, the client and server must clear the preliminary master key from the memory .

//pre_master_secret 预备主密钥
//master secret是三个参数（secret，label，seed）中的label，固定值
//ClientHello.random，ServerHello.random 客户端服务器的随机数，客户端随机数在前，服务器随机数在后
SecurityParameters.master_secret = PRF(pre_master_secret,
        "master secret",
        ClientHello.random + ServerHello.random);
//计算出的主密钥长度是48字节 [0..47]

After the master key is calculated, the key block needs to be calculated. The formula is as follows:

//第一个参数是主密钥
//第二个参数是label，固定为key expansion
//第三个参数是客户端和服务器的随机数，服务器随机数在前，客户端随机数在后
key_block = PRF(SecurityParameters.master_secret,
            "key expansion",
            SecurityParameters.server_random + SecurityParameters.client_random);
//密钥块可以拆分为密钥值，重要的有6个
//mac用
client_write_MAC_key[SecurityParameters.mac_key_length]
server_write_MAC_key[SecurityParameters.mac_key_length]
//对称加密用
client_write_key[SecurityParameters.enc_key_length]
server_write_key[SecurityParameters.enc_key_length]
//分组加密的初始化向量IV
client_write_IV[SecurityParameters.fixed_iv_length]
server_write_IV[SecurityParameters.fixed_iv_length]

warning protocol

After the client and server establish a connection, the key block will be negotiated through the handshake protocol. During the negotiation and authentication process, errors may occur. Error messages are handled by the warning protocol, which has multiple errors, and some errors may be fatal, directly terminating the connection between the client and the server.
The message format of the warning protocol consists of two parts: the warning error level and the detailed description information of the warning protocol .

// AlertLevel表示警告错误级别
enum { warning(1), fatal(2), (255) } AlertLevel;

// 警告协议的详细描述消息
enum {
    close_notify(0),
    unexpected_message(10),
    bad_record_mac(20),
    decryption_failed_RESERVED(21),
    record_overflow(22),
    decompression_failure(30),
    handshake_failure(40),
    no_certificate_RESERVED(41),
    bad_certificate(42),
    unsupported_certificate(43),
    certificate_revoked(44),
    certificate_expired(45),
    certificate_unknown(46),
    illegal_parameter(47),
    unknown_ca(48),
    access_denied(49),
    decode_error(50),
    decrypt_error(51),
    export_restriction_RESERVED(60),
    protocol_version(70),
    insufficient_security(71),
    internal_error(80),
    user_canceled(90),
    no_renegotiation(100),
    unsupported_extension(110),
    (255)
} AlertDescription;

// 警告协议包含两部分
struct {
    AlertLevel level;
    AlertDescription description;
} Alert;

Cipher switching protocol (obsolete in TLS1.3)

The role of this protocol is to inform the TLS record protocol that the key block required by its cryptography has been prepared. Once the client and server send the password . It is encrypted and protected.

When the TLS record protocol is processing the handshake protocol , password switching protocol , and warning protocol , because the key block is not ready at this time, these three sub-protocols are transmitted in plain text, and the TLS record protocol only adds message headers to the three sub-protocols.

The sub-message corresponding to the cipher switching protocol is Change Cipher Spec . Generally, the client sends this message first, and the format is as follows:

struct {
    enum { change_cipher_spec(1), (255) } type;
} ChangeCipherSpec;

application layer protocol

The application layer protocol is the upper layer protocol of the TLS/SSL record layer protocol, including HTTP, FTP, SMTP and other application layer protocols. The main information encrypted and protected by the TLS record protocol is the application layer protocol data, and the TLS record protocol will add MAC verification code data to the application layer protocol (depending on different encryption modes).

record agreement

All data is sent in a series of TLS records, called TLS packets. The TLS Record Protocol (Record Layer) is essentially a transport protocol and has nothing to do with the meaning of the transmitted data; the

TLS Record Protocol is used first and foremost for the data exchanged during the handshake. Once the handshake is complete and both parties share a session key, application data is split into chunks for transmission as part of a TLS record.

message structure

The data block of the TLS record protocol is up to 16KB, and the structure is as follows:
The first byte indicates the type of data sent , 22 (0x16) indicates handshake data, 23 (0x17) indicates encrypted data, 21 (0x15) indicates warning, 20 (0x14) Indicates a password switch .
● The value of the second byte and the third byte is called the protocol version (Protocol Version), for example, the two bytes of TLS v1.2 are 0x0303.
● The fourth byte and the fifth byte indicate the length of the data to be transmitted , and the length cannot be greater than bytes (16KB).
● The remaining bytes are the data to be transmitted (also known as the payload) whose length is equal to the value encoded by the fourth and fifth bytes of the record.

conversation

The client and the server will build a TCP connection, each connection is a session, the session has different states, and the state runs through the entire TLS/SSL protocol processing flow.

encryption parameters

That is SecurityParameters, the encryption parameter is the most important data structure in the TLS/SSL protocol . The client and the server discuss the cipher suite, and then negotiate all the encryption parameters based on the cipher suite. The most important encryption parameter is the master .

struct {
    //表示操作方是客户端或服务器
    ConnectionEnd         entity;
    //伪随机函数prf，用来将预备主密钥转换为主密钥，主密钥转位密钥块
    PRFAlgorithm          prf_algorithm;
    //加密函数，aes等
    BulkCipherAlgorithm   bulk_cipher_algorithm;
    //记录协议使用加密模式，三种
    CipherType            cipher_type;
    //加密算法密钥的长度
    uint8                 enc_key_length;
    //加密数据的长度
    uint8                 block_length;
    //盐长度
    uint8                 fixed_iv_length;
    uint8                 record_iv_length;
    //MAC算法
    MACAlgorithm          mac_algorithm;
    //MAC值的长度
    uint8                 mac_length;
    //MAC算法使用的密钥长度
    uint8                 mac_key_length;
    //记录协议的压缩算法，一般不启用
    CompressionMethod     compression_algorithm;
    //主密钥
    opaque                master_secret[48];
    //连接阶段，客户端向服务器传递的32字节随机数
    opaque                client_random[32];
    //连接阶段，服务器向客户端传递的32字节随机数
    opaque                server_random[32];
} SecurityParameters;

// 服务器端和客户端
enum { server, client } ConnectionEnd;
// 伪随机函数，在TLS v1.2 协议中，PRF函数默认加密基元是SHA256
enum { tls_prf_sha256 } PRFAlgorithm;
// 加密算法，比较流行的是aes算法
enum { null, rc4, 3des, aes } BulkCipherAlgorithm;
// 加密模式，aead是新型的加密模式，包含了消息验证码的处理
enum { stream, block, aead } CipherType;
// 消息验证码算法
enum { null, hmac_md5, hmac_sha1, hmac_sha256, hmac_sha384, hmac_sha512}
MACAlgorithm;
// 压缩方法
enum { null(0), (255) } CompressionMethod;

The value of each element of the encryption parameter is filled by the TLS/SSL handshake protocol. The client and server will negotiate a cipher suite , and fill the values ​​of each sub-element of the encryption . The TLS record protocol mainly encrypts and decrypts based on the value of encryption parameters , the most important one is the key block .
The client and server each have their own encryption parameters . The client encrypts the message with its own write MAC key, write encryption key, and write IV key block and sends it. After receiving the message, the server decrypts the message with the client's write MAC key, write encryption key, and write IV key block. For the client, the corresponding value of the server is used to decrypt the received message.

The key block contains 6 specific elements, namely:

//客户端MAC密钥、加密密钥、初始化向量
client write MAC key
client write encryption key
client write IV
//服务器MAC密钥、加密密钥、初始化向量
server write MAC key
server write encryption key
server write IV

Connection Status

Each TLS connection has four concepts of connection states, namely:
pending read states,
pending write states,
current read states, and
current write states
on the client side When initializing the connection with the server, the connection statuses of the client and the server are read and write statuses (the client and the server respectively maintain their own connection status). Once all encryption parameters are ready, the client enters the readable state, and the server enters the writable state. Only when the connection state is readable and writable, data .

After the client and the server send the cipher switch protocol message respectively, the connection state will be switched. Before the client and server send the password switching protocol message, all handshake messages are processed in plain text, without confidentiality and integrity protection.

For the TLS record protocol, each connection state consists of 4 parts.
compression state : Compression state, generally do not enable compression.
cipher state : the encryption algorithm used by each connection (three encryption modes) and the key block used by the encryption algorithm.
MAC key: MAC key per connection.
Sequence number : Each TLS record protocol message has a sequence number. The client and server each maintain a sequence number , and the sequence number starts from 0. The serial number only participates in MAC calculation . The role of the sequence number is to prevent replay attacks. When the client connects to the server, two variables ( client_send and client_recv )

are initialized in memory , client_send records the total amount of data blocks sent by all clients , client_recv records the total amount of data blocks received by all clients , and client For the first sending and receiving, the values ​​of client_send and client_recv are both 0. After that, each time the client sends a message, client_send+1, and each time a message is received, client_recv+1. ( The client message MAC calculation includes the value of client_send ) After the server receives the client connection, it also initializes two variables ( server_send and server_recv ) in memory. server_send records the total amount of data blocks sent by the server , and server_recv records The total amount of data blocks that all servers have received

. When the server receives and sends for the first time, the values ​​of server_recv and server_send are both 0. After that, each time the server sends a message, server_send+1, and each time a message is received, server_recv+1. ( The server message MAC calculation includes the value of server_send )

After the server receives the client message, it decrypts the client message, then calculates the MAC value, and performs integrity verification. It needs to use the value of the server_recv variable. If the verification is successful, it will respond A message to the client.

After the client receives the server message, it decrypts the server message, then calculates the value of the MAC, and performs an integrity check. It needs to use the value of client_recv. After the processing is successful, client_recv+1.

The client and the server continuously increment the serial numbers of themselves and each other. If handled correctly, the value of client_send is equal to the value of server_recv, and the value of client_recv is equal to the value of server_send .

Suppose the client_send=3 of the client (4 pieces of data are sent), and the server_recv of the server is also 3. At this time, an attacker wants to replay the 4th message (client_send=3), the server receives a new message, server_recv+=1, However, if the message client_send=3 is replayed by the attacker, an error will be reported when verifying the MAC, which prevents the replay attack.

work process

The recording protocol mainly has 4 parts of processing:
1. Data block.
2. Compression.
3. Encryption and integrity protection, mainly including three modes (stream encryption mode, packet mode, AEAD mode). 4. Add message headers and other information (depending on the encryption mode) before
the encrypted data .

data chunking

After the data of all upper-layer protocols enters the TLS record protocol, the message is first split into blocks. The maximum limit of each block is 16KB. The structure is as follows:

// 协商出的版本号
struct {
    //大版本号，如TLS v1.2 这里是1
    uint8 major;
    //小版本号，如TLS v1.2 这里是2
    uint8 minor;
} ProtocolVersion;

// TLS高层协议的4个协议，密码交换协议，警告协议，握手协议，应用层协议
enum {
    change_cipher_spec(20),
    alert(21),
    handshake(22),
    application_data(23)
} ContentType;

//数据分块后的数据结构
struct {
    //协议类型，4个中的一种
    ContentType type;
    //版本
    ProtocolVersion version;
    //载荷长度，fragment的长度
    uint16 length;
    //载荷
    opaque fragment[TLSPlaintext.length];
} TLSPlaintext;

compression

Compression is to convert TLSPlaintext to TLSCompressed, but it is not enabled in general protocols, and its compressed format is as follows:

//和TLSPlaintext结构一致，如果不压缩两者是一样的
struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    opaque fragment[TLSCompressed.length];
} TLSCompressed;

encrypted protection

After the data goes through the first few steps, it comes to the encryption layer and converts TLSCompressed to TLSCiphertext in the following format:

struct {
    ContentType type;
    ProtocolVersion version;
    //加密后fragment的长度
    uint16 length;
    //加密后的数据，包括MAC
    select (SecurityParameters.cipher_type) {
        //流密码
        case stream: GenericStreamCipher;
        //分组密码
        case block:  GenericBlockCipher;
        //AEAD
        case aead:   GenericAEADCipher;
    } fragment;
} TLSCiphertext;

stream cipher mode

It is less used now, and the structure of GenericStreamCipher is as follows:

stream-ciphered struct {
    opaque content[TLSCompressed.length];
    opaque MAC[SecurityParameters.mac_length];
} GenericStreamCipher;

Among them, the MAC value is obtained by performing HMAC operation on the serial number, message header, and TLSCompressed.fragment. Encrypt the TLSCompressed.fragment and MAC value to obtain the ciphertext, and then splicing the message to obtain the GenericStreamCipher. The whole is MAC first and then encryption .

//MAC的计算过程
//客户端和服务器各有一个MAC_write_key，MAC的长度由加密参数决定。
MAC(MAC_write_key, seq_num +//序列号
    TLSCompressed.type +TLSCompressed.version +TLSCompressed.length +//消息头
    TLSCompressed.fragment);//消息明文

block cipher mode

The block mode is a commonly used encryption mode. It needs to consider the initialization vector and padding value (the block cipher needs to be filled to a multiple of blocksize). The length of the initialization vector is determined by the record_iv_length parameter of the encryption random , otherwise Security attacks may be encountered.

struct {
    //初始化向量，长度一般和blocksize一致
    opaque IV[SecurityParameters.record_iv_length];
    //加密数据
    block-ciphered struct {
        //消息内容
        opaque content[TLSCompressed.length];
        //MAC值
        opaque MAC[SecurityParameters.mac_length];
        //填充值
        uint8 padding[GenericBlockCipher.padding_length];
        //填充长度
        uint8 padding_length;
    };
} GenericBlockCipher;

The process of generating GenericBlockCipher is to combine TLSCompressed.fragment, MAC, and padding value together and then encrypt, and finally add the message header and IV before encrypting the data . The method is also to encrypt first then MAC.
The MAC calculation process is as follows, which is consistent with the stream cipher mode

//MAC的计算过程
//客户端和服务器各有一个MAC_write_key，MAC的长度由加密参数决定。
MAC(MAC_write_key, seq_num +//序列号
    TLSCompressed.type +TLSCompressed.version +TLSCompressed.length +//消息头
    TLSCompressed.fragment);//消息明文

AEAD mode

AEAD implements encryption and integrity protection together, without additional consideration of HMAC algorithm, which has higher security . Examples of AEAD cipher suites are as follows:

#openssl v3.0查看TLS密码套件
openssl ciphers -V 

AEAD mode	Encryption Algorithm	cipher suite
GCM	AES-128-GCM	For example DHE-RSA-AES128-GCM-SHA256
ChaCha20-Ploy1305	ChaCha20-Ploy1305	For example ECDHE-ECDSA-CHACHA20-POLY1305

The encrypted structure is as follows:

struct {
    //nonce，长度取决于SecurityParameters.record_iv_length
    opaque nonce_explicit[SecurityParameters.record_iv_length];
    aead-ciphered struct {
        opaque content[TLSCompressed.length];
    };
} GenericAEADCipher;

AEAD encryption is obtained by using the serial number + message header as associated data, combining with TLSCompressed.fragment encryption, and then splicing the message .

//AEAD加密函数
//write_key MAC用，这里不需要
//nonce：随机数，明文传输
//plaintext,明文，=TLSCompressed.fragment
//关联数据，验证消息完整性，=序列号+消息头=seq_num +TLSCompressed.type +TLSCompressed.version +TLSCompressed.length
AEADEncrypted = AEAD-Encrypt(write_key, nonce, plaintext, additional_data)

During the decryption process, if the integrity check fails, an error will be reported

TLSCompressed.fragment = AEAD-Decrypt(write_key, nonce, AEADEncrypted,
    additional_data)

TLS protocol extension

Extensions are designed for the future development of TLS. Extensions are a pluggable unit in the protocol. Each extension is uniformly registered and managed by IANA.
The client can send multiple extensions to the server according to its needs, and the extension list message is included in the Client Hello message.
The server parses the extensions in the Client Hello message and returns the same type of extensions in the Server Hello message. Note that some extensions do not necessarily respond in the Server Hello message.

The client and server Hello messages contain extension information, and the extension is backward compatible, which means that the client and server are not strictly required to support a certain extension. For example, an extension is attached to the Client Hello message on the client side. If the server receives the extension, if it does not understand the meaning of the extension, it can ignore it and not process it, and it will not affect the subsequent handshake. If the server understands the meaning of the extension, it can Hello responds to the client Client Hello message with the same extension.
The extensions responded by the server must be a subset of the extensions requested by the client, and the extensions not sent by the client cannot appear in the Server Hello message, otherwise a fatal error will be generated.

extension type name	extension type number	RFC reference
server_name	0	RFC 6066
max_fragment_length	1	RFC 6066
client_certificate_url	2	RFC 6066
trusted_ca_keys	3	RFC 6066
truncated_hmac	4	RFC 6066
status_request	5	RFC 6066
supported_groups	10	RFC 7919
ec_point_formats	11	RFC-ietf-tls-rfc4492bis-17
signature_algorithms	13	RFC 5246
application_layer_protocol_negotiation（ALPN）	16	RFC 7301
signed_certificate_timestamp	18	RFC 6962
extended_master_secret	23	RFC 7627
SessionTicket TLS	35	RFC 4507
renegotiation_info	65281	RFC 5746

Extension

struct {
    //扩展类型，占2字节
    ExtensionType extension_type;
    //扩展数据
    opaque extension_data<0..2^16-1>;
} Extension;

enum {
    //类似于http中host请求头的作用
    server_name(0),
    max_fragment_length(1),
    client_certificate_url(2),
    trusted_ca_keys(3),
    truncated_hmac(4),
    //和OCSP有关
    status_request(5),
    //关于ECC的扩展
    supported_groups(10),
    //关于ECC的扩展
    ec_point_formats(11),
    //用于客户端向服务器列出自己支持的签名算法和摘要算法
    signature_algorithms(13),
    //使得客户端和服务器端能够协商出一个应用层协议，例如http1.1和http2，选择双方都支持的
    application_layer_protocol_negotiation(16),
    //和证书透明度有关，每一张服务器实体证书都可以由CA机构或服务器实体提交给CT日志服务器从而获得证书SCT信息。
    signed_certificate_timestamp(18),
    extended_master_secret(23),
    //用于会话恢复
    session_ticket(35),
    //和重新协商有关
    renegotiation_info(65281),
    (65535)
} ExtensionType;
//扩展列表
Extension extensions<0..2^16-1>;

TLS会话恢复

客户端和服务器握手成功后会建立一个TLS连接，只要客户端和服务器不主动关闭该连接，应用层数据请求就一直受该连接保护，一旦客户端和服务器关闭该连接，那么客户端下次访问服务器的时候就需要重新握手，建立一个新的连接。握手的过程非常消耗时间和CPU，所以会话恢复机制就是为了省去重新握手的时间和减少资源消耗。
目前TLS v1.2中有两种恢复机制，分别是Session ID 和SessionTicket TLS扩展。

Session

Session称为会话，它相当于TLS连接的状态，内容如下：
会话标识符（session identifier）：每个会话都有唯一编号。
证书（peer certificate）：对端的证书，一般为空。
压缩算法（compression method）：一般不启用。
密码套件（cipher spec）：客户端和服务器协商的密码套件。
主密钥（master secret）：每个会话会保存一个主密钥（master_secret）。
会话可恢复标识（is resumable）：表示某个会话是否可恢复。

SessionID

客户端发送Client Hello消息时，会传递一个Session ID参数，如果是一次完整的握手，不是会话恢复的TLS连接，那么这个Session ID为空，服务器在收到消息后，会生成一个Session ID，并在Sever Hello中传递给客户端，客户端仅在内存中保留Session ID。在双方发送Finished子消息后，会话结束，服务器将Session ID保存在Session Cache中，Session Cache相当于一个哈希表，键为Session ID，值为Session的信息。

会话恢复的过程：
客户端再次请求相同的网站，如果客户端内存中存在该网站对应的Session ID，则Client Hello消息附带该Session ID。服务器接收到该请求后，检查Session Cache是否能够匹配健值为Session ID的会话，如果没有或者不可恢复会话，则进行完整的握手协议，同时生成一个新的Session ID返回给客户端。如果能够恢复本次连接，则直接发送ChangeCipherSpec和Finished子消息，不进行密钥协商过程，因为主密钥存在于Session Cache中。最终客户端也发送ChangeCipherSpec和Finished子消息，表示会话恢复成功。

注意点：
在恢复的TLS连接中，虽然主密钥和前一次连接的一致， 但客户端和服务器最终生成的密钥块和先前的密钥块是不一致的， 原因在于通过PRF生成密钥块的时候，客户端和服务器的 随机数 不同于前一次连接，这也有效地增强了安全性。
在恢复会话完成后，也要校验客户端和服务器端的Finished消息，避免握手消息被篡改。
恢复会话的时候，本次连接协商出的密码套件必须和会话中的密码套件是一致的，否则就要进行完整的握手。是否恢复成功取决于客户端和服务器，即使存在可以恢复的会话，服务器也可以要求进行完整的握手。
会话中并不保存扩展信息，所以每个扩展必须充分考虑会话恢复的情况。
Session ID是明文传输的，服务器Session ID不应该包含隐私数据。

SessionTicket

SessionTicket的出现是为了解决Session ID的缺点，它由SessionTicket TLS扩展实现。
Session ID会话信息存储在服务器，对于访问量非常大的服务器，内存消耗非常大。
Session ID会话恢复应付不了负载均衡和分布式服务的情况，因为它不能跨主机访问。

工作方式：
服务器将会话信息加密后以票据（ticket）的方式发送给 客户端保存， 服务器本身不存储会话信息。
客户端接收到票据后将其存储到内存中（不解密），如果想恢复会话，则下一次连接需要将票据发送给服务器，服务器解密后，经过验证无误则可以进行会话恢复。

SessionTicket TLS扩展

对于一个新的连接，客户端ClientHello消息发送一个空的SessionTicket TLS扩展，表示想获得一个票据，服务器在SeverHello中响应一个空的SessionTicket TLS扩展。如果不想服务器支持此类会话恢复，则ClientHello消息不发送此扩展，服务器也不会处理。
如果服务器支持SessionTicket会话恢复，只要客户端发送了SessionTicket TLS扩展（无论是不是空），那么服务器会发送New Session Ticket子消息，该消息包含一个票据，客户端收到以后保存起来以备后用。

如果之前连接过一次，再次连接时想要进行会话恢复，则客户端ClientHello消息发送一个非空的SessionTicket TLS扩展，服务器收到扩展后，对票据进行解密、校验，确认无误后发送一个空的SessionTicket TLS扩展。接着发送一个New Session Ticket子消息 更新票据。然后客户端和服务器校验Finished子消息表示恢复会话的握手完成。

New Session Ticket子消息

由服务器发送该子消息，如果ServerHello子消息中包含SessionTicket TLS扩展，则必须发送此消息，如果不包含扩展，则不能发送。如果发送该消息则必须在Change Cipher Spec前发送。
如果服务器端成功校验客户端发送的票据，必须重新生成一个票据，然后通过New Session Ticket子消息发送新票据，客户端在下一次连接的时候应该发送新的票据。
消息结构如下：

struct {
    //票据的有效期
    uint32 ticket_lifetime_hint;
    //票据内容
    opaque ticket<0..2^16-1>;
} NewSessionTicket;
struct {
    //包含至少一组密钥，每组包含AES-128-CBC和HMAC-SH-256的密钥，其中有一组用于票据加密
    opaque key_name[16];
    //AES的IV向量
    opaque iv[16];
    //加密后的会话信息
    opaque encrypted_state<0..2^16-1>;
    //摘要值，对encrypted_state、key_name、iv进行MAC运算
    opaque mac[32];
} ticket;
//加密前的会话信息结构
struct {
    //TLS版本
    ProtocolVersion protocol_version;
    //密码套件
    CipherSuite cipher_suite;
    //压缩算法
    CompressionMethod compression_method;
    //主密钥
    opaque master_secret[48];
    //客户端的标识符
    ClientIdentity client_identity;
    //票据过期时间
    uint32 timestamp;
} StatePlaintext;

两种会话恢复机制共存

（1）如果服务器想使用SessionTicket机制，那么服务器Server Hello可以不发送Session ID。
（2）如果服务器不想使用SessionTicket机制，那么Server Hello消息中不包含SessionTicket扩展即可，此时应该生成Session ID发送给客户端。
（3）对于客户端来说，SessionTicket恢复的优先级应该更高，如果服务器同时发送了票据和Session ID，应该使用SessionTicket。
（4）如果服务器同时发送了票据和Session ID，为了方便切换两种会话恢复方式，客户端应该同时发送票据和Session ID。服务器接收后，如果在Session Cache中存在Session ID，则响应同样的Session ID给客户端，同时也发送票据给客户端。

---

参考：《深入浅出HTTPS：从原理到实战》虞卫东