Advanced Usage of requests reptiles

1. File Upload

import requests
 
files = {'file': open('favicon.ico', 'rb')}
r = requests.post("http://httpbin.org/post", files=files)
print(r.text)

{
  "args": {}, 
  "data": "", 
  "files": {
    "file": "data:application/octet-stream;base64,AAAAAA...="
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "6665", 
    "Content-Type": "multipart/form-data; boundary=809f80b1a2974132b133ade1a8e8e058", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.10.0"
  }, 
  "json": null, 
  "origin": "60.207.237.16", 
  "url": "http://httpbin.org/post"
}

View Code

2. Cookies

import requests
 
r = requests.get("https://www.baidu.com")
print(r.cookies)
for key, value in r.cookies.items():
    print(key + '=' + value)

<RequestsCookieJar[<Cookie BDORZ=27315 for .baidu.com/>, <Cookie __bsi=13533594356813414194_00_14_N_N_2_0303_C02F_N_N_N_0 for .www.baidu.com/>]>
BDORZ=27315
__bsi=13533594356813414194_00_14_N_N_2_0303_C02F_N_N_N_0

View Code

import requests
 
cookies = 'q_c1=31653b264a074fc9a57816d1ea93ed8b|1474273938000|1474273938000; d_c0="AGDAs254kAqPTr6NW1U3XTLFzKhMPQ6H_nc=|1474273938"; __utmv=51854390.100-1|2=registration_date=20130902=1^3=entry_date=20130902=1;a_t="2.0AACAfbwdAAAXAAAAso0QWAAAgH28HQAAAGDAs254kAoXAAAAYQJVTQ4FCVgA360us8BAklzLYNEHUd6kmHtRQX5a6hiZxKCynnycerLQ3gIkoJLOCQ==";z_c0=Mi4wQUFDQWZid2RBQUFBWU1DemJuaVFDaGNBQUFCaEFsVk5EZ1VKV0FEZnJTNnp3RUNTWE10ZzBRZFIzcVNZZTFGQmZn|1474887858|64b4d4234a21de774c42c837fe0b672fdb5763b0'
jar = requests.cookies.RequestsCookieJar()
headers = {
    'Host': 'www.zhihu.com',
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
}
for cookie in cookies.split(';'):
    key, value = cookie.split('=', 1)
    jar.set(key, value)
r = requests.get("http://www.zhihu.com", cookies=jar, headers=headers)
print(r.text)

cookie request

 

3. session to maintain

import requests
 
s = requests.Session()
s.get('http://httpbin.org/cookies/set/number/123456789')
r = s.get('http://httpbin.org/cookies')
print(r.text)

{
  "cookies": {
    "number": "123456789"
  }
}

View Code

import requests
 
requests.get('http://httpbin.org/cookies/set/number/123456789')
r = requests.get('http://httpbin.org/cookies')
print(r.text)

Comparison of the code can not be maintained

 

4. SSL certificate validation

import requests
 
response = requests.get('https://www.12306.cn')
print(response.status_code)

 

import requests
 
response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)

Close to verify

import requests
from requests.packages import urllib3
 
urllib3.disable_warnings()
response = requests.get('https://www.12306.cn', verify=False)
print(response.status_code)

Ssl ignored warnings not used

import requests
 
response = requests.get('https://www.12306.cn', cert=('/path/server.crt', '/path/key'))
print(response.status_code)

Specifies the local certificate

The proxy settings

import requests
 
proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "http://10.10.1.10:1080",
}
 
requests.get("https://www.taobao.com", proxies=proxies)

Of course, this example may not run directly, because the agency may not be valid, please replace their effective agent to test.

If the proxy requires HTTP Basic Auth, you can use something like http: // user: password @ host: port this syntax to set the proxy, for example:

import requests
 
proxies = {
    "http": "http://user:[email protected]:3128/",
}
requests.get("https://www.taobao.com", proxies=proxies)

In addition to the basic HTTP proxy, requests also supports SOCKS proxy protocol.

First, you need to install the library socks:

pip3 install 'requests[socks]'

 You can then use the SOCKS proxy protocol, for example:

import requests
 
proxies = {
    'http': 'socks5://user:password@host:port',
    'https': 'socks5://user:password@host:port'
}
requests.get("https://www.taobao.com", proxies=proxies)

6. timeout settings

import requests
 
r = requests.get("https://www.taobao.com", timeout = 1)
print(r.status_code)

 

In fact, the request is divided into two stages, i.e. connected (Connect) and read (read).

Provided above timeoutwill be used to read both the connection and the timeoutsum.

If you want to specify, respectively, you can pass a tuple:

r = requests.get('https://www.taobao.com', timeout=(5,11, 30))

7. Authentication

import requests

r = requests.post('http://localhost/Admin/Public/login.html', data=({'uid':'admin', 'pwd':'admin123'}))
print(r.status_code)
print(r.text)

8. Prepared Request

from requests import Request, Session
 
url = 'http://httpbin.org/post'
data = {
    'name': 'germey'
}
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36'
}
s = Session()
req = Request('POST', url, data=data, headers=headers)
prepped = s.prepare_request(req)
r = s.send(prepped)
print(r.text)

Here we introduce Request, then url, dataand headersparameters of a constructed Requestobject at this time to make a call Sessionto prepare_request()a method to convert it to a Prepared Request object and then calls the send()method to sent, results are as follows:

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "name": "germey"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "close", 
    "Content-Length": "11", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36"
  }, 
  "json": null, 
  "origin": "182.32.203.166", 
  "url": "http://httpbin.org/post"
}

View Code