Top-down understanding of the network (3) - HTTPS articles

In the previous blog post of this series, we introduced a very commonly used application layer protocol HTTP. In daily work, the application of the HTTP protocol can be said to be ubiquitous. We browse the web and use mobile apps without HTTP. Through the previous introduction, we also understand that HTTP is essentially just a set of communication rules agreed by the client and the server, so that the client and the server can communicate with each other in an understanding. In the early stage of Internet popularization, web pages were mainly used to display public static content, and the HTTP protocol had no obvious defects. With the continuous development of the Internet, Internet applications have become more and more personalized and more complex, and some security issues have become very important. The evolution of HTTP to HTTPS has become the only way for Internet applications.

1. Security issues of the HTTP protocol

Before solving the problem, we first need to find the problem first. The HTTP protocol fundamentally has the following three security problems:

1. Data is not encrypted

Recalling the suggested HTTP server and client programs we wrote earlier, the data is actually transmitted in clear text, and the HTTP protocol itself does not involve the relevant definition of data encryption. Therefore, once our network communication message is intercepted, the user's privacy is completely exposed. In the Internet environment, it is very easy to intercept data. The operator can do it, the proxy server can do it, each routing node device in the data transmission can also do it, and even the malicious program of the client can do it. do it. Imagine if you are using an Internet application of online banking, your bank account password is clearly seen on the Internet, is it a very scary thing. Of course, if our client application is matched, we can also agree on an encryption algorithm on top of the HTTP protocol. When transmitting data, the server and client encrypt and decrypt according to the same rules to ensure data integrity (in fact, many mobile terminals The application's interface service does so). But there are still many problems. First, the browser is a third-party, and the private encryption protocol cannot be used in browser applications. Second, if the encryption algorithm of the client is cracked, the cracker can crack all the intercepted data in the same way. There are still security issues.

2. Unable to verify the identity of the other party

HTTP is an application layer protocol based on the TCP/IP protocol. The TCP protocol ensures that both parties transmit data correctly and completely, but cannot verify the identity of the other party. For example, when we visit the website of huishao.cc, any intermediate node can directly return the wrong data without forwarding the HTTP request after receiving the HTTP request. The terminal cannot judge who the other party of the returned data is at all, nor can it know whether the returned data is true. Corresponding to banking business, your balance information may be tampered with by any middleman, which is extremely harmful. Similarly, when the server receives the client's request, it cannot judge whether the client is normal or not. This may cause any pretender to intercept the client's request and continue to pretend to be the client and continue to do business with the server. Deeper communication.

3. Data may be tampered with

This problem is actually a problem of data integrity and consistency. Whether it is a client or a server, after the data is sent, it will pass through many intermediate nodes. Any node may change some parts of the data, and the receiver has no means to verify whether the data has been changed. For web applications, some malicious actors can add various annoying advertisements after intercepting the data. More serious, the interceptor may change some important data to make the client misunderstand the intention of the server, and lead the client to a more harmful attack website or download attack code.

2. Thinking of solutions for security problems

To sum up, the HTTP protocol has three fatal flaws in security. In practical applications, a new communication method that is more secure and fully compatible with the HTTP protocol is increasingly required. Let's first look at how to solve these security problems.

1. Use encryption to protect plaintext content

For the problem of transmitting data in plaintext, the simplest way we can think of is to encrypt the data, such as using a symmetric key to encrypt and decrypt, the process is as follows:

There are two obvious problems with this symmetric key encryption method. First, the client and the server use the same key. In principle, user A can obtain the key, and attacker B can also obtain the key. Lost the meaning of encryption. Secondly, even if the server assigns a different set of private key pairs to each client, it is very difficult to maintain, and the client environment is responsible. During the process of transmitting the private key, the attacker may also intercept the private key. Encryption is still meaningless.

Since symmetric encryption has such a problem, can we use asymmetric encryption? The asymmetric encryption process is as follows:

In asymmetric encryption, the key pair is divided into a public key and a private key. As the name implies, the public key itself can be made public and available to all clients, and the private key exists on the server. Compared with symmetric encryption, the security of the server is much higher than that of the client, and the private key is relatively safe. However, using this asymmetric encryption method to protect data also has limitations. As shown in the figure above, when the client sends data to the server, the attacker does not have the private key of the server and cannot decrypt the data at this time. However, when the server uses the private key to encrypt the information and returns it to the client, it may be intercepted by the attacker and decrypted using the public key.

Since neither asymmetric encryption nor symmetric encryption can perfectly protect the security of data, can they be combined? In the implementation of HTTPS, this is indeed the case. The process is as follows:

The main problem of asymmetric encryption is that the data encrypted with the private key can be decrypted by anyone with the public key. We only need to ensure that the server does not use the private key for encryption when returning data. As shown in the figure above, first, the client uses the encryption algorithm to be used by the public key. The symmetric key used is encrypted and sent to the server. The intermediate node does not have the private key of the server and cannot know what the key sent by the client is. Afterwards, if the server agrees to this negotiation, the client's symmetric key is used to encrypt the receipt data, and subsequent data communications are encrypted using this key, which cannot be decrypted by an attacker.

Well, the current encryption method seems to be very easy to use, but it seems that it is not easy to implement. How to correctly give the public key to the client is another tricky problem. Let's see below.

2. How to verify the identity of the other party

The reason why it is difficult to correctly give the public key to the client is that in network communication, neither party can verify the identity of the other party. For example, the server puts the public key on a public website for the client to download, but the client cannot determine whether the website is really the official website of the server, and the website itself may have been hijacked, and the client will download it to the attack. public key provided by the user. After that, all communication between the client and the attacker will be conducted directly, and the client will not know anything about it (the attack method of phishing websites). If the server sends the public key to the client first when the client initiates a session, it is also insecure. The intermediate node can directly intercept the public key of the server and replace it with the attacker's own public key and send it to the client. After that, the symmetric key sent by the client will be encrypted with the attacker's public key, and the attacker can get the symmetric key used for subsequent network communication, which is even more dangerous. Neither the client nor the server know what they sent. The data is first obtained by the attacker and decrypted successfully, and the attacker continues to communicate with both parties by pretending to be a client/server.

So, is there a way to securely obtain the public key of the server? This requires us to solve the problem of mutual authentication. To solve the problem of identity authentication, the essence is to find a way to bind the public key with the identity information and make it immutable. Our most direct idea is to also perform asymmetric encryption on the public key, but how to ensure that the public key used to decrypt the public key is correctly given to the client becomes a problem again, and it falls into an endless logical loop. Therefore, to break this cycle, we must recognize some secure public key and trust it. For another example, if you want to withdraw money from the bank, you must prove that you are you. You can say that my mother can prove that I am me, but you must prove that your mother is your mother. This will never prove your identity. But in real life, we will not encounter such a dilemma, because you can use your ID card to prove that you are you, and the ID card is issued by the state authority and recognized by all social institutions. To compare it to our network communication scenario, each end needs a certificate for self-certification, and this certificate is a certificate.

The workflow for the certificate is as follows:

As shown in the figure above, when the server applies for a certificate to a certificate authority, it needs to submit its own encryption public key. At this time, the certificate authority needs to ensure the identity of the applicant, that is, whether the applicant belongs to a certain domain name. The real operator, after the verification is correct (this directly affects the reputation of the certificate authority, that is, the bottom line of survival), uses its own key to encrypt the public key of the server, and then generates a certificate to the server. The certificate itself is a relatively complex technology, and its validity can be verified through the inheritance relationship. We will not introduce too much here. We only need to know that the public key used to decrypt the certificate has been integrated into the client device from the beginning. , of course, the client device can also add trusted certificates as required, and the certificates of these clients include domain name information and the corresponding public key. After that, whenever the client wants to initiate communication with the server, the server first sends the certificate to the client. If someone intercepts the tampering, but it does not have the private key encrypted by the certificate authority, the tampered certificate file cannot pass. The client's built-in public key is decrypted, the client will realize that it has been attacked, and the request can be interrupted. When the client receives the certificate, it finds the trusted paired certificate file locally, and decrypts it with the public key in the local certificate file. After the decryption is successful, the correct public key of the server can be obtained.

In this way, through certificates, we solve the problems of identity authentication and information tampering prevention. Next, we can understand the HTTPS protocol.

3. What exactly is the HTTPS protocol

Now, we have a theoretical solution to HTTP security. Saying that HTTPS is a protocol is actually a misnomer. The full name of HTTPS is Hyper Text Transfer Protocol over SecureSocket Layer, which is the HTTP protocol over the secure socket channel. Its essence is the SSL (SecureSocket Layer) or TLS (Transport Layer Security) protocol. The TLS protocol is an upgraded version of the SSL protocol. In this blog, we take TLS as an example to introduce. When the data format used for encrypted communication is the HTTP protocol format, this communication is called HTTPS. Similarly, Other network layer application protocols can also work on top of the TLS protocol, such as FPTS which uses TLS to make file transfers more secure. The relationship between HTTP, TLS and HTTPS is as follows:

Let's introduce the working method of the TLS protocol and understand the TLS protocol. HTTPS is only the format of the business data content it transmits is the HTTP protocol.

1. TLS workflow

According to our previous analysis, to communicate securely between the client and the server, three parts are required:

The client requests the authentication certificate from the server and verifies and obtains the public key of the server.
Use the server's public key to encrypt the conversation information to negotiate the communication algorithm and key.
Data communication is performed using the negotiated algorithm and key.

The first two steps are the TLS handshake process, and the last step is the communication process. Let's first look at the handshake process.

A total of 4 communications are required between the client and the server to complete the handshake process. The process is as follows:

During the handshake process, the client first initiates a ClientHello message, which contains the version number of the TLS protocol supported by the client, a random number generated by the client (used to generate the key later), the encryption algorithm supported by the client and the compression method, etc. We will introduce it in detail later.

After receiving the ClientHello message from the client, the server will respond with a SeverHello message. The main contents of the SeverHello message include the encrypted communication protocol version used for negotiation, an anytime generated by the server (used to generate the key later), the encryption method used for confirmation, and the certificate of the server.

After the client receives the SeverHello message from the server, it first verifies the certificate of the server. If the certificate is not issued by a trusted authority or the domain name in the certificate is inconsistent with the domain name actually accessed, or the certificate expires, the client needs to interrupt the communication ( Different browsers may implement it differently, and some may simply warn the user that it is not safe). If the certificate verification is passed, the client will extract the public key of the server. Send a response message to the server, the response message will contain a random number encrypted with the server's public key, and a notification of code change, indicating that subsequent communication with the server will use the encryption method and key agreed upon by both parties for communication. And the notification of the end of the client handshake. At this time, the client will send the Hash value of all the previously sent content to the server for the server to verify whether there is tampering in the middle. Of course, if the server also needs to verify your client's identity, the client will also send the certificate to the server in this step.

Finally, the server will send a response message to the client again. The response message includes an encoding change notification, indicating that information sent subsequently will be encrypted using the mutually agreed upon encryption method and key. The server will also send a server-side handshake end notification, and then the data communication between the client and the server will enter encrypted communication, and the format of the original data is completely consistent with HTTP.

2. About the content of the TLS protocol

The implementation of the TLS handshake process is mainly achieved through the following four types of sub-protocols:

TLS Handshaking protocols
Alert Protocol
Application Data Procotol Application Layer Data Protocol
Change Cipher Spec Protocol

Each TLS protocol message consists of a message header and a message body. The message header is fixed and has a length of 5 bytes and 40 bits.

The first byte of the message header describes the type of the message body, enumerated as follows:

The 2nd and 3rd bytes are used to describe the version number of TLS currently in use.

The 3rd and 5th bytes are used to describe the length of the current message, and the length will include header information.

After the end of the header information, it is the specific message body content, which is different according to different protocol types.

Handshake type protocol:

Typically, a successful TLS handshake requires two interactions between the client and server. The main function of the handshake is to negotiate any encryption methods and keys between the client and the server. The message of the handshake class consists of three parts, which are the subtype, the length of the submessage and the specific submessage body.

The sub-message type occupies 1 byte, and the enumeration is as follows:

| 类型 | 值 | | HELLO_REQUEST | 0x00 | | CLIENT_HELLO | 0x01 | | SERVER_HELLO | 0x02 | | CERTIFICATE | 0x0B | | SERVER_KEY_EXCHANGE | 0x0C | | SERVER_DONE | 0x0E | | SERVER_REQUEST | 0x0D | | CERTIFICATE_VERIFY | 0x0F | | CHIENT_KEY_EXCHANGE | 0x10 | | FINISHED | 0x14 |

During specific execution, multiple sub-messages may be encapsulated into the same TLS message and sent to the TCP layer for processing and transmission.

Warning type protocol:

Exceptions may occur during the handshake process. For example, when the client's encryption algorithm cannot be supported by the server, the sub-protocol of the warning type can be directly returned. The format of the warning type message is fixed and consists of two bytes. The first byte Describes the level of the warning, and the second byte describes the reason for the warning.

An enumeration of warning types is:

An enumeration of warning reasons is:

| 原因 | 值 | | close_notify | 0 | | unexpected_message | 10 | | bad_record_mac | 20 | | decryption_failed_RESERVED | 21 | | record_overflow | 22 | | decompression_failure | 30 | | no_certificate_RESERVED | 41 | | bad_certificate | 42 | | unsupported_certificate | 43 | | certificate_revoked | 44 | | certificate_expired | 45 | | certificate_unknown | 46 | | illegal_parameter | 47 | | unknown_ca | 48 | | access_denied | 49 | | decode_error | 50 | | decrypt_error | 51 | | export_restriction_RESERVED | 60 | | protocol_version | 70 | | insufficient_security | 71 | | internal_error | 80 | | user_canceled | 90 | | no_renegotiation | 100 | | unsupported_extension | 110 |

** Password switch protocol:**

The password confirmation protocol is the confirmation response information of the client and the server that we mentioned earlier. Once this message is sent, it means that subsequent communication can be carried out at the encryption layer.

Application layer data protocol:

The function of this protocol is to perform a layer of encryption protection on the upper layer application layer protocol data before transmission.

3. Detailed explanation of Client Hello sub-message

When the client wants to communicate with the server through TLS, the first thing sent is the client's Clent Hello message. This message contains the following information:

client_version: The highest TLS protocol version number supported by the client.

random: A random number generated by the client.

sessionID: The id of the session, which can be used to resume the session.

cipher_suites: The client sends a list of supported cipher suites.

compression_methods: Compression methods supported by the client.

extensions: extension content, TLS protocol supports easy extension, there can be multiple extension data.

We can use Wireshark to monitor the TLS protocol data. The structure of the Client Hello message is as follows:

4. Detailed explanation of Server Hello sub-message

The main function of the Server Hello message is to select a cipher suite that both parties are satisfied with according to the cipher suites supported by the client. Server Hello contains the following information:

server_version: The version number of the TLS protocol that the server finally chooses to use.

random: A random number generated by the server.

session_id: If the session can be resumed, it is the same as the session_id passed by the client. If not, create a new session_id to return.

ciper_suite: According to the cipher suite supported by the client, the server selects one supported by both parties to use.

compression_method: According to the compression method provided by the client, the server selects one supported by both parties to use.

extensions: According to the extension list passed by the client, the server will process it.

An example Server Hello message is shown below:

It can be seen that the main function of the Server Hello message is to select the cipher suite list and the compression method list provided by the client, and finally decide what to use for communication between the two parties.

5. Server Certificate，Server Key Exchange，Server Hello Done子消息

When the server sends the Server Hello message, it generally sends the Server Certificate message immediately. This message is optional, but generally sent by the server to verify the identity of the server to the client. And securely give the server's public key to the client. The information in the certificate message is a series of certificate chains. We will not describe too much about the details of the certificate here.

The Server Key Exchange sub-message is also sent conditionally, and some cipher suites need to send this message. Logically, after the server sends the Server Hello message, it will immediately send the Server Hello Done message. This message is an empty message, indicating that the server has sent all the things to be sent, and the client can perform certificate verification. The key agreement logic is gone.

These three messages are usually guaranteed to be sent in a TCP packet, as shown below:

Later, there are some messages that the client negotiates the key, the message that the handshake is completed, and so on. After these sub-messages are completed, it indicates that the entire handshake process is over, and the subsequent client and server begin to encrypt data.

After the complete handshake process is completed, subsequent business data communication will be encrypted, as shown in the following figure:

4. Ending

Now, we have a macro understanding of the logic of network communication at the application layer. The logic of the application data is clear, and more importantly, the data is safely and stably transmitted to the other party. This requires the introduction of transport layer protocols.

Focus on technology, love life, exchange technology, and be friends.

——Hunshao QQ: 316045346