The so-called crawler is to simulate client sends the network request to obtain the network response, and parsing acquired data according to certain rules, and stored procedures. Python is bound to say the reptile around, but Requests library.
1 Introduction
Requests for library official document says so:
Requests only a non-transgenic Python HTTP library, humans can safely enjoy.
Warning: non-professional use of other HTTP libraries can cause dangerous side effects, including: safety deficiency, disease redundant code, re-invent the wheel disorders, eating disorders document, depression, headaches, and even death.
This presentation is quite vivid, they say no more. Installation Terminal command pip install requests.
2 Quick Start
2.1 transmission request
Import Requests module:
import requests
Get pages:
r = requests.get('http://xxx.xxx')
At this point, we get the Response object r, we can get the information by r. Requests simple API means that all types of HTTP requests are obvious, let's look at the use of common types of HTTP requests get, post, put, delete the examples:
r = requests.head('http://xxx.xxx/get')
r = requests.post('http://xxx.xxx/post', data = {'key':'value'})
r = requests.put('http://xxx.xxx/put', data = {'key':'value'})
r = requests.delete('http://xxx.xxx/delete')
Typically we set the request timeout, use Requests to set the timeout parameter, in seconds, for example:
r = requests.head('http://xxx.xxx/get', timeout=1)
2.2 parameter passing
When using the get way to send a request, we will form the key parameters on the back of a question mark in the URL, such as: http: //xxx.xxx/get key = val, Requests by params keyword to a string? Dictionary to provide these parameters. For example, to pass key1 = val1 and key2 = val2 to http://xxx.xxx/get, the following example:
pms = { 'key1' 'val1', 'key2' 'val2'}
r = requests.get("http://xxx.xxx/get", params=pms)
Requests also allows a list as the value passed:
pms = { 'key1' 'val1', 'key2': [ 'val2' 'val3']}
Note: None of the key value of the dictionary will not be added to the query string of the URL's.
2.3 response content
Let's get a response about the content server, for example https://api.github.com address here:
import requests
r = requests.get('https://api.github.com')
print(r.text)
# Output
# {"current_user_url":"https://api.github.com/user","current_user...
When accessing the r.text, Requests will use its presumed text encoding, we can use r.encoding view its encoding, you can modify the code, such as: r.encoding = 'GBK', when changing the code, visit again r when .text, Request r.encoding will use all of the new value. Which is good flow of the hospital in Wuxi http://www.wxbhnkyy120.com/
1) in response to the binary content
For example, when we want to get a picture data will fetch response data in a binary manner, for example:
from PIL import Image
from io import BytesIO
i = Image.open(BytesIO(r.content))
2) JSON response content
Requests have been built JSON decoder, so we can easily parse the JSON data, for example:
import requests
r = requests.get('https://api.github.com')
r.json()
Note: A successful call r.json () is not necessarily a successful response, some servers will contain a JSON object (such as HTTP Error details 500) in response to a failed, then we need to check the status of the response code r. status_code or r.raise_for_status (), when successful call r.status_code to 200, r.raise_for_status () to None.
2.4 custom request header
When we give the request to add headers, the headers parameter to simply pass a dictionary can be, for example:
url = 'http://xxx.xxx'
hds= {'user-agent': 'xxx'}
r = requests.get(url, headers=hds)
Note: Custom headers priority is lower than some specific information, such as: setting up user authentication information in the .netrc, the license headers setting will not take effect, and when set auth parameter, the setting .netrc invalid. All headers values must be string, bytestring or unicode, it is generally not recommended to use unicode.
2.5 Redirection and history
By default, Requests will be processed automatically redirect all except HEAD, you can use the response object's history attributes following the redirect return in response to the object list, this list is sorted in accordance with the request by the late arrival early look at example:
import requests
r = requests.get('http://github.com')
print(r.history)
# Output
# []
If you are using get, post, put, delete, options, patch allow_redirects parameters can be used to disable the redirect. Examples are as follows:
r = requests.get('http://xxx.xxx', allow_redirects=False)
2.6 errors and exceptions
When experiencing network problems (such as: DNS query fails, rejects the connection, etc.), ConnectionError Requests will throw exception; when the HTTP request returns an unsuccessful status code, Response.raise_for_status () will throw an exception HTTPError; request timeout , Timeout will throw an exception; request exceeds the maximum number of redirects set TooManyRedirects will throw an exception. All Requests explicitly throw exceptions inherit from requests.exceptions.RequestException.