written in front
Although I have been in contact with programming for several years, I have always read other people's blog posts. At first, I was always afraid that what I wrote would be called garbage by the boss. But now that I think about it, I don't have to. I write my current understanding, the first is a record of my own growth, the second is that if I understand wrong, others can find my misunderstanding, and the third is to help those who leave later than me. Slow people, personally, I have read many blogs, and there are indeed many people who have helped me at different stages. In the future, I also want to write some blogs one after another. I hope this is a good start.
The problem we want to solve in this article is: in what format and in what way is our data sent to the server, and how to parse the returned data?
1. In what format: HTTP protocol
If you are familiar with HTTP, you can skip this section
1.1 Request
The data format of an HTTP network request is: protocol header + request header + blank line + request body . Each piece of data ends with "\r\n". Therefore, when we generate the request body, we need to send it to the server in this format, so that the server can parse and respond. For example, a normal GET request header looks like this.
If the GET request has parameters, the parameters will be concatenated in the format of ?a=1&b=2 after the path, and the request body will be a blank line. If the POST form request, then the data at the request body is a=1&b=2. There are many parameters in the request header, specifically one
Request Header Response Header Comparison Table View all request headers and request headers. What is more important to us is
Connection Host Accept Accept-Encoding Accept-Language。
Connection values include keep-alive (default) and close, which means whether the network request maintains a long connection. Host host address. Accept indicates what format data the client accepts, and Accept-Encoding indicates whether the client supports compression. A data of more than 30,000 bytes can be compressed to four to five kilobytes. The effect is very significant. It is also very effective for server pressure, reducing traffic, and speeding up transmission. Most browsers now support decompression. Accept-Language indicates what language format to receive, which requires server support.
1.2 Response
The HTTP response format is similar to the request. Response status + response headers + empty lines + data . Each piece of data ends with "\r\n". For example, a normal response header looks like this.
Of course, there are more than these response headers. Some non-essential but important request headers are missing here. Commonly used request headers in our use are:
Content-Type: Tell the client the data format and encoding type
Content-Length: Tell the client how many bytes of data (fixed data length)
Transfer-Encoding: Tell the client to transfer in the form of data blocks (data variable length)
Content-Encoding: Encoding type of data
We need to extract these data when parsing the response, and do different processing according to the data type.
Of course, this is just a rough introduction, not the focus of this article. For more detailed HTTP knowledge, please see here
2. In what way: TCP
To put it simply, initiating a network request is actually to request and respond in HTTP protocol format through Socket. The whole process is the operation of IO flow. The problem is given to JAVA network programming. Java's IO operations can be divided into two series: character stream and byte stream. The character stream mainly reads and writes characters, such as text files, with its own buffer, which has a cache for data, and the reading rate is relatively high. The byte stream mainly reads and writes bytes, such as some of our pictures or videos and other file operations.
In our application, it is easy to go astray if we only look at the response header, thinking about reading each line through the BufferReader of the character stream series to extract the response header, but the problem is that our data is likely to be not a character but a word section, and BufferReader and InputStream cannot be used interchangeably. The reason is that BufferReader has already read the data into its own cache, and if InputStream reads again, the data will be lost or cannot be read. In order to keep the purest data, we still use InputStream to read the data. There are also many knowledge points about Java IO. If you need to learn more, you can read this article to guide you to understand JAVA IO flow , or read other materials.
3. Get ready
The theoretical foundation is almost there, now we start to put our theory into practice
3.1 URL parsing
fun url(url:String):Request{
this.url=url
val urlReal = URL(url)
host=urlReal.host
port=urlReal.port
if (port==-1)
port=urlReal.defaultPort
api=urlReal.path
return this
}3.2 Create a default request header
/*添加默认头*/
private fun buildDefaultHeader(){
addHeader("Host", "$host:$port")
addHeader("Content-Type","application/x-www-form-urlencoded")
addHeader("User-Agent","PostmanRuntime/7.15.0")
addHeader("Accept","*/*")
addHeader("Cache-Control","no-cache")
addHeader("Accept-Encoding","br, deflate,gzip")
addHeader("Connection","keep-alive")
}
/*将请求头转字符串*/
private fun header2String(){
val stringBuilder =StringBuilder()
for (it in headerMap){
stringBuilder.append(it.key)
stringBuilder.append(": ")
stringBuilder.append(it.value)
stringBuilder.append("\r\n")
}
/*POST请求加上请求体长度*/
if (method=="POST"){
stringBuilder.append("Content-Length")
stringBuilder.append(": ")
stringBuilder.append(body.toString().length)
stringBuilder.append("\r\n")
}
headerString=stringBuilder.toString()
}3.3 Send data to the server
/*开启socket，以字节流形式发送给服务器拿到服务器返回的字节流准备处理*/
fun open(request:Request.Builder):Response{
socket= Socket(request.host,request.port)
val outputStream = socket.getOutputStream()
val inputStream = socket.getInputStream()
println(request.toString())
outputStream.write(request.toString().toByteArray())
val response = Response()
response.dealInput(inputStream)
socket.close()
return response
}
/*request的内部类，用于构建request*/
inner class Builder(val host: String,val port: Int,val head:String,val header:String,val body:String){
override fun toString(): String {
val stringBuilder =StringBuilder()
stringBuilder.append(head)
stringBuilder.append("\r\n")
stringBuilder.append(header)
stringBuilder.append("\r\n")
stringBuilder.append(body)
stringBuilder.append("\r\n")
return stringBuilder.toString()
}
}
3.4 Extract response header
Write our readline
private fun readLine(inputStream: InputStream):String{
val byteArray=ArrayList<Byte>()
while (inputStream.read().let {
if (it!=10&&it!=13){
byteArray.add(it.toByte())
}
it
}!=10){
}
return String(byteArray.toByteArray())
}3.5 Extract data (including GZIP decoding)
(We take the data in string or json format as an example, temporarily excluding file downloads) Here we need to deal with it according to the situation. If the response header returned by the server contains Content-Length, we can read the bytes of this length. . If the response header contains Transfer-Encoding , it will be ignored even if Content-Length is set. The data with Transfer-Encoding set will be sent in blocks. The first line in the format is the length of the first block, followed by the byte data of the corresponding length in another line. If more data is transmitted in the same format, until the last block length is 0, then the blank line ends. If the response header contains Content-Encoding: gzip , then our data cannot be directly converted to String, otherwise a bunch of garbled characters will come out, we need to perform gzip decoding, and then perform corresponding operations after obtaining the decoded data.
/*借鉴其它网友的解码方法*/
public static byte[] uncompress(byte[] bytes) {
if (bytes == null || bytes.length == 0) {
return null;
}
ByteArrayOutputStream out = new ByteArrayOutputStream();
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
try {
GZIPInputStream ungzip = new GZIPInputStream(in);
byte[] buffer = new byte[256];
int n;
while ((n = ungzip.read(buffer)) >= 0) {
out.write(buffer, 0, n);
}
} catch (Exception e) {
e.printStackTrace();
}
return out.toByteArray();
}Four, see the real chapter
4.1 Use our client to request Caiyun Weather (Content-Length version)
/*不带参的GET请求，GZIP加密 彩云key要自己申请，好像每天免费一万条*/
val request = Request()
val build = request.url("http://api.caiyunapp.com/v2.5/彩云key" +
"/121.6544,25.1552/weather.json").build()?:return
val open = Client().open(build)
println("body: "+open.string())The result of the request is as follows, the data is complete, because the number of decoded bytes is too large, so it is not truncated.
Content-Encoding: In the case of gzip, the bytes are only 4339. The reason is that we set support for GZIP in the request header. If our request header setting does not support gzip, it will be given to us in an unencoded format, which is about 30,000 bytes. .
4.2 Use our client to request variable-length data interface ( Transfer-Encoding version)
The request is the same as the others, just change the interface address to a test interface I wrote before, compare the two response headers and find that the following request has no Content-length and replaced it with Transfer-Encoding
http://59.110.212.105/PharmacyServer/0/selectMedicine.action
The result of the request is as follows:
4.3 Use our client to send a POST request with data in json format
The request comes with its own request body parameters, and the data is passed in a simple json format, which can be set as a POST request
val request = Request()
val formBody = FormBody()
formBody.addParam("userName","0001")
formBody.addParam("password","123456")
request.post(formBody)
request.addHeader("Content-Type","application/json")
request.setMethod("POST")
val build=request.url("http://59.110.212.105/PharmacyServer/signin.action").build()?:return
val open = Client().open(build)
println("body: "+open.string())The response result is as follows
5. Say something
How about it, does it feel familiar to use? The client has been able to satisfy basic GET\POST requests and extract the data we need. Here we actually use some of the questions that are often asked in junior and intermediate Java or Android interviews, HTTP protocol and IO. The current results are only a small stage, and there are still many things that need to be improved step by step. For example, as an Android development, network requests must not be placed on the main thread. We need to perform an active thread detection and throw our exception.
In the future, I plan to add interceptors, GSON analysis, and achieve request configuration, exception handling, performance improvement, etc. through custom annotations and interfaces like retrofit, and improve this system step by step. Of course, it is impossible to achieve the same effect as OKHttp, but the purpose is to express one's own understanding and gain while using it.
Looking forward to criticism and correction.