background
HTTP can transfer many kinds of data, not only text, but also pictures, audio and video.
In the early days, texts and small pictures with a size of only a few K were basically transmitted on the Internet, but the situation is quite different now. The information contained in the webpage is really too much. The HTML of a random homepage may be hundreds of K, and high-quality pictures are all in M, not to mention those movies and TV series, which can be several gigabytes or tens of gigabytes. possible.
In contrast, the 100M fiber-optic fixed network or 4G mobile network has become a "small water pipe" under the pressure of these large files. Whether it is uploading or downloading, the network transmission link will be "full".
How to efficiently and quickly transfer these large files under limited bandwidth has become an important issue. This is like opening the refrigerator door (establishing a connection), how to stuff the elephant (file) in and then close the door (complete the transfer)?
Let's take a look at what methods in the HTTP protocol can solve this problem.
data compression
One of the most basic solutions is "data compression", turning the elephant into a piglet and putting it in the refrigerator.
Usually, when a browser sends a request, it will carry the "Accept-Encoding" header field, which contains a list of compression formats supported by the browser, such as gzip, deflate, br, etc., so that the server can choose a compression algorithm and put it in the " Content-Encoding" response header, and then compress the original data and send it to the browser.
If the compression rate can be 50%, that is to say, 100K data can be compressed into 50K size, then it is equivalent to doubling the network speed with the same bandwidth, and the acceleration effect is very obvious.
However, this solution also has a disadvantage. Compression algorithms such as gzip usually only have a good compression rate for text files, while multimedia data such as pictures, audio and video are already highly compressed, and will not become smaller after processing with gzip ( It might even grow a bit), so it fails.
However, the effect of data compression is still very good when processing text, so the servers of major websites will use this method as a "guarantee". For example, the "gzip on" directive is used in Nginx to enable compression of "text/html".
chunked transfer
In addition to data compression, is there any other way to solve the problem of large files?
Compression is to make a large file smaller as a whole. We can think in reverse. If a large file cannot be made smaller as a whole, then "break it up" and break it into multiple small pieces, and distribute these small pieces to the browser in batches. After the device is received, it will be assembled and restored.
In this way, the browser and the server do not need to store all the files in the memory, only send and receive a small part each time, the network will not be occupied by large files for a long time, and resources such as memory and bandwidth are also saved.
This idea of "breaking the whole into parts" is "chunked" chunked transfer encoding in the HTTP protocol, which is represented by the header field "Transfer-Encoding: chunked" in the response message, which means that the body part of the message is not It is sent at one time, but divided into many chunks (chunk) and sent one by one.
This is like using magic to turn elephants into "Lego building blocks", disassemble them one by one and put them in the refrigerator, and then cast spells to "resurrect with full blood" after arriving at the destination.
Chunked transmission can also be used for "streaming data", such as form pages dynamically generated by the database. In this case, the length of the body data is unknown, and the exact length cannot be given in the header field "Content-Length". Therefore, it can only be sent in chunks in chunked mode.
The two fields "Transfer-Encoding: chunked" and "Content-Length" are mutually exclusive, that is to say, these two fields cannot appear in the response message at the same time, and the transmission of a response message must either have a known length or It is an unknown length (chunked), which you must remember.
The encoding rules for chunked transmission are actually very simple. They also use plain text, which is very similar to the response header.
- Each block consists of two parts, the length header and the data block;
- The length header is a line of plain text ending with CRLF (carriage return line feed, ie \r\n), and the length is represented by hexadecimal numbers;
- The data block immediately follows the length header, and ends with CRLF at the end, but the data does not contain CRLF;
- Finally, a block of length 0 is used to indicate the end, that is, "0\r\n\r\n".
range request
With chunked transfer encoding, the server can easily send and receive large files, but there are still some issues to consider for very large files uploading G.
For example, if you are watching a time-travel drama that is currently on the air, you want to skip the opening title and watch the main movie directly, or there is a boring plot, and you want to drag the progress bar to fast-forward a few minutes. This is actually to get a large file. Fragment data, while block transfer does not have this capability.
In order to meet such requirements, the HTTP protocol proposes the concept of "range requests", allowing the client to use a dedicated field in the request header to indicate that only a part of the file is obtained, which is equivalent to the client's "dividing into parts".
The range request is not a necessary function of the web server, and it may or may not be implemented, so the server must use the field "Accept-Ranges: bytes" in the response header to clearly inform the client: "I support range requests".
What should I do if it is not supported? The server can send "Accept-Ranges: none", or simply not send
the "Accept-Ranges" field, so that the client thinks that the server has not implemented the range request function, and can only send and receive the entire file honestly.
The request header Range is a dedicated field for HTTP range requests, and the format is "bytes=xy", where x and y are
data ranges in bytes.
It should be noted that x and y represent "offset", and the range must be counted from 0, for example, the first 10 bytes are represented as "0-9",
the second 10 bytes are represented as "10-19", and " 0-10" are actually the first 11 bytes.
The format of Range is also very flexible. The starting point x and the ending point y can be omitted, which can conveniently represent the range of positive or reciprocal numbers. Assuming that the file is 100 bytes, then:
"0-" means from the beginning of the document to the end of the document, which is equivalent to "0-99", that is, the entire file;
"10-" is from the 10th byte to the end of the document, equivalent to "10-99";
"-1" is the last byte of the document, equivalent to "99-99";
"-10" is from the The last 10 bytes from the end of the document, equivalent to "90-99".
After the server receives the Range field, it needs to do four things.
First, it must check whether the range is legal. For example, the file is only 100 bytes, but the request is "200-300", which means the range is out of bounds. The server will return status code 416, which means "Your range request is wrong, I can't process it, please check again".
Second, if the range is correct, the server can calculate the offset according to the Range header, read the file fragment, and return the status code "206 Partial Content", which has the same meaning as 200, but it means that the body is only a part of the original data.
Third, the server needs to add a response header field Content-Range to tell the actual offset of the fragment and the total size of the resource. The format is "bytes xy/length". The difference from the Range header is that there is no "=", and there are more after the range total length. For
example, for a range request of "0-10", the value would be "bytes 0-10/100".
The last thing left is to send the data, directly send the segment to the client using TCP, and a range request is processed.
With the range request, it is easier for HTTP to process large files. When watching a video, you can calculate the range of the file according to the time point, without downloading the entire file, and directly and accurately obtain the data content of the segment.
Not only the drag and drop progress of the video requires a range request, but also the multi-segment download and resumable upload in common download tools are also implemented based on it. The main point is: first send a HEAD to see if the server supports range requests, and at the same time
get the size of the file ;
Open N threads, each thread uses the Range field to divide the fragments it is responsible for downloading, and sends a request to transfer data; the
download is not afraid of unexpected interruption, you don’t have to start all over again, just use the Range to request the rest according to the last download record That part is fine.
multi-segment data
The range request just mentioned only fetches one fragment at a time. In fact, it also supports using multiple "xy" in the Range header to obtain multiple fragment data at one time.
In this case, you need to use a special MIME type: "multipart/byteranges", indicating that the body of the message
is composed of multiple byte sequences, and a parameter "boundary=xxx" is also used to give the separation between segments mark
.
The format of multi-segment data is also similar to that of block transmission, but it needs to use the separation mark boundary to distinguish different segments, which can be compared through the figure.
Each segment must start with "- -boundary" (add two "-" in front), and then use "Content-Type"
and "Content-Range" to mark the type and range of this data, and then just like ordinary The response header of the response ends with a carriage
return and line feed, plus segment data, and finally uses a "- -boundary- -" (two "-" before and after) to indicate the
end of all segments.
summary
- Compressing text files such as HTML is the most basic method for transferring large files;
- Chunked transmission can send and receive data in a streaming manner, saving memory and bandwidth. It is represented by the response header field "Transfer-Encoding: chunked". The format of chunking is hexadecimal length header + data block;
- The range request can only obtain part of the data, that is, "block request", to achieve video drag or breakpoint resume, use the request header field "Range" and response header field "Content-Range", and the response status code must be 206;
- You can also request multiple ranges at one time. At this time, the data type of the response message is "multipart/byteranges", and
multiple parts in the body will be separated by boundary strings.