The birth of HTTPS

background

With the enhancement of user security awareness, the network protocol has gradually transitioned from the Http protocol to the Https protocol in the past two years. At present, most APP/Web have begun to use the Https protocol. Here, the process of Http's gradual development into the Https protocol is oriented. The birth process of HTTPS.

Disadvantages of the HTTP protocol

• Communication use statement;
• not verify the identity of the correspondent;
• The integrity of the message cannot be verified;

Communication usage statement

The use of plaintext for communication means that the security is greatly reduced. When the communication process is eavesdropped, the transmitted data can be seen without additional investment. For example, using a packet capture tool, you can view any communication data using the HTTP protocol without any configuration;

Do not verify the identity of the correspondent

Failure to verify the identity of the communicating party will lead to the communication process being eavesdropped and may be disguised. For example, after using a packet capture tool to capture data, an HTTP request can be constructed according to the format of the data packet;

Unable to verify the integrity of the message

If the integrity of the message is not verified, the data may be tampered with during the transmission process.

solution

Communication usage statement

Since plaintext is not secure, ciphertext can be considered, that is, to encrypt communication data. Even if the data is eavesdropped, the other party still needs to spend a certain amount of investment to crack it, and this high cost indirectly improves the security level.

Do not verify the identity of the correspondent

Using the same algorithm as the server, a token is generated according to the network request parameters, and the identity of both parties is determined according to the token when requesting/response.

Unable to verify the integrity of the message

Use MD5/SHA1 and other algorithms for integrity verification. After receiving the data, the other party generates a hash value according to the same algorithm, and compares the hash value generated by the sender to verify the integrity of the data. Solving the above problems of HTTP, the HTTPS protocol was born:

Https = Http + 加密 + 认证 + 完整性验证

For the characteristics of encryption, authentication and integrity verification, the TCP/IP protocol suite has provided a protocol for secure data transmission - SSL (Secure Socket Layer) Secure Socket Layer. Although it is also based on TCP, SSL It is not an application layer protocol, but a protocol between the application layer and the transport layer. (For the theoretical knowledge related to the specific protocol, please refer to the theoretical knowledge of network programming .) Its purpose is to provide a secure transmission channel for the upper application layer protocol. . At this time, the Https protocol can be expressed in the following form:

Https = Http + SSL

In this way, HTTPS is not a separate application layer protocol, but only the process of Http using SSL channel for data transmission. Then for Https, you only need to understand Http (refer to HTTP detailed explanation ) and SSL protocol. The so-called HTTPS message is also an SSL message:

SSL protocol

SSL (Secure Socket Layer) is a network protocol between the application layer and the transport layer that provides security and integrity verification for network communications.

Compared with the TCP or HTTP protocol, the SSL protocol is much more complicated. Since it is also based on the TCP protocol, before using SSL to transmit data, a three-way handshake is required to establish a connection with the server. The specific process is shown in the figure:

The handshake process of the SSL protocol

1. The client first sends a message to the server. The content of the message includes: the encryption method supported by the client, the supported compression method, the SSL version number, the random number generated by the client, and the text content "Hello", etc.;
2. After the server receives the message, it also sends back a Hello, which carries the encryption method selected from the encryption methods supported by the client, the random number generated by the server, and the SSL version number of the server and other information;
3. Then the server sends a Certificate message to the client, which contains the public key certificate of the server;
4. Then the server sends Server Hello Done to the client, indicating that the initial negotiation handshake process is over;
5. After the client receives the message of the end of the handshake sent by the server, it responds with the Client Key Exchange. This message contains a random password string called Pre-master secret used in the communication encryption process, and uses the third The received public key certificate is encrypted;
6. Then the client sends a Change Cipher Spec message, which informs the server that all data after this step will be encrypted using the master secret generated in the fifth step (see the following introduction for the generation process of the master secret);
7. Then the client sends a Finish message, which contains the overall check value of all messages connected so far for integrity verification;
8. After the server receives the Change Cliper Spec message sent by the client, it also responds with the Change Cliper Spec message;
9. Then the server sends a Finish message to the client, indicating that the server has correctly parsed the overall check value sent by the client. At this point, the SSL handshake process ends.
10. Then start using the HTTP protocol to transmit data encrypted with the master secret.

illustrate

• The first two steps are the process of negotiating the encryption algorithm and transmitting the random numbers generated by them (preparing for the subsequent generation of the master secret);

• In the third step, the server sends its own certificate to the client. This certificate contains a digital signature (CA signature) and the public key of the server's CA certificate. The client hashes the server information contained in the certificate, and uses the receiving The obtained public key decrypts the digital certificate, obtains the Hash value, and compares it with the previously calculated Hash value to verify the validity (integrity & authenticity) of the certificate;

• When the server receives the Change Cipher Spec sent by the client (step 5), it will decrypt it with its own private key and obtain the Pre-master secret in the message. At this time, both parties in the communication have the other party's Random (generated in the first two steps). ), Pre-master secret, and its own Random, use the three numbers as seeds to generate the master secret through an algorithm, which is used to encrypt the data in the subsequent Http request process. The generation rules of the master secret are:

  master_secret = MD5(pre_master_secret + SHA('A' + pre_master_secret + ClientHello.random + ServerHello.random)) + MD5(pre_master_secret + SHA('BB' + pre_master_secret + ClientHello.random + ServerHello.random)) + MD5(pre_master_secret + SHA('CCC' + pre_master_secret + ClientHello.random + ServerHello.random));

For more details about the SSL protocol, please refer to: RFC 6101 SSL

Why is it so complicated

Consider two aspects: safety and efficiency. If the symmetric encryption method is directly used for encryption, it is of course safe if the key is not leaked, but the question is: how to pass the key to the other end? During the data transmission process, if the communication is eavesdropped, the key is stolen, Then encryption doesn't make any sense. Is it possible to use asymmetric encryption? It is theoretically possible. Before data transmission, the server only needs to transmit its own public key to the client. When the client transmits data, it uses the received public key for encryption, and the server uses the private key to decrypt the data after receiving the data. That's it. But there is a problem here that cannot be ignored: efficiency. Asymmetric encryption requires a lot of computation and will definitely take up a lot of hardware resources, so the efficiency is too low. So in order to solve the problem of security and efficiency, SSL uses a combination of symmetric encryption (encryption and decryption use the same key) and asymmetric encryption (public key encryption, private key decryption): using asymmetric encryption to transmit symmetric encryption Generate the seed of the key (pre master secret) [corresponding to the fifth step above], and then encrypt the communication data using symmetric encryption [corresponding to the tenth step above], which not only ensures the security of the key, but also improves the encryption speed.

Is HTTPS really safe?

Since Https is so secure, why can you still see the data inside with a packet capture tool? Here we need to make a brief introduction to the principle of the packet capture process. Packet capture is actually a man-in-the-middle attack. During the network request process, the packet capture tool acts as a client/server. When the client requests data, the packet capture tool Intercept the request, and then construct data (pretending to be a client) to initiate a request to the server. At this time, the server will return its own public key certificate, and then the packet capture tool will replace the server's public key certificate with its own public key certificate, and then Return data to the client (acting as a server at this time). After the client gets the public key certificate, it encrypts the Pre-master secret with the public key in the public key certificate, and sends it to the server. During the sending process, it is intercepted by the packet capture tool. The certificate public key generated by the packet capture tool, so the packet capture tool only needs to decrypt it with its own private key to get the real Pre-master secret (because the packet capture tool has obtained the random codes of both parties in the previous process, So at this step, the packet capture tool is equivalent to having obtained the key used for subsequent communication), and then the packet capture tool uses the public key in the public key certificate received from the server to encrypt the Pre-master secret, and then send it to the service. end. Although the subsequent communication process is encrypted, the packet capture tool has generated a master secret, so you can view the Https communication content. A picture to cover it up:

It can be seen from the principle of packet capture that to capture Https packets, a certificate must be installed on both the PC and the mobile phone.

Since it is so easy to be caught, will Https appear tasteless? In fact, no, you can capture the package, that is because you trust the package capture tool, and the corresponding certificate is installed on the mobile phone. If you want to install the certificate, you can try to grab one. Moreover, the subject of security is to seek development in attack and defense. There is no safest, only more secure. Therefore, the cost of attack is increased, and the goal of security is indirectly achieved.

How to capture Https

Here we take Charles4.0 as an example to capture Https:

1. Click on the menu Help -> SSL Proxying -> Install Charles Root Certificate to install the PC certificate;
2. Click the menu Help -> SSL Proxying -> Install Charles Root Certificate on a Mobile Device or Romote Browser in turn, a box will pop up, prompting you to go to chls.pro/ssl to download the certificate, and install it after the download is complete;
3. Click the menu Proxy -> SSL Proxying Settings in turn, select Enable SSL Proxying in the page, then click add below, and fill in *.* in the Host edit box in the pop-up box (meaning that all Https requests can be captured), Fill in 443 for the port and click OK;
4. The mobile phone sets the proxy (the IP address of your computer, the port defaults to 8888) to capture packets;

How to avoid being caught

Since the Https protocol is used, I just don't want others to see the data, so how to avoid being captured? There are 2 methods for reference:

• use a custom protocol;
• Use SSL Pinning;

Use the TCP protocol

Use a custom protocol, that is, based on TCP/UDP, use Socket for communication (for details, please refer to the theory of network programming ), this method can avoid Http type packet capture tools because it is not the Http/Https protocol used (such as Charles, Fiddler) to capture packets, but WireShark is inevitable, after all, it is too powerful to capture data at each layer in the network model.

Using SSL Pinning

It looks very unfamiliar, but it is not a new technology. You only need to put the public key certificate of the server into the APK, and compare it with the public key certificate of the server when requesting from the network, so as to avoid the packet capture tool using the construction certificate. The situation of capturing packets.

Summarize

The above is the whole process of transitioning from Http to Https. This protocol development-oriented approach may be more helpful to understand. Of course, this is just my personal understanding of the Https protocol. If there is anything inappropriate, please correct me.

Reference Books/Articles

Graphical HTTP

Detailed explanation of SSL/TLS principle