Python web crawler 1

1 Prelude to web crawlers

1.1 Guide to the content of the web crawler course

1.2 Selection of Python language development tools

This lesson mainly uses the following 4 kinds:

IDLE of python is the default and commonly used entry-level authoring tool that comes with python

2 Rules of web crawlers

2.1 Getting started with the Requests library

2.1.1 How to install Requests

The Requests library (http://www.python-requests.org) is recognized as the best crawling third-party library. It has 2 characteristics:

1. simple
2. Very concise, even with one line of code you can get the corresponding resources from the web page

Installation method: Run command

test as an administrator :

2.1.2 The get() method of the Requests library

Python case sensitivity

The request method is used to encapsulate the
Requests library and provides a total of 7 common methods, the request method is the basic method, and the other 6 methods provide calling the request method to achieve


r.status_code: check the status code of the request. If it is 200, it means the return is successful.
r.headers: Get the header information of the page. The
Response object includes all the information returned by the server and the request information requested by the service




. The resources on the network are encoded. If there is no encoding, it will be unreadable.
r.apparent_encoding is more readable than r.encoding accurate

2.1.3 General code framework for crawling web pages

The
get can be crawled accurately and reliably , because the network connection is risky, and exception handling is very important.

Timeout: the exception of the entire process from sending the URL request to obtaining the content.
ConnectTimeout: just connecting to the remote server



common code framework makes the crawling more stable More effective and reliable

2.1.4 HTTP protocol and Requests library method

HTTP protocol

The user initiates a request and the server responds.
The stateless request is that there is no correlation between the first and the second. The
application layer protocol means that the protocol works on top of the tcp protocol (?)


, which is equivalent to the path of the file, except that it exists on the Internet that

is provided by the Requests library. Functions

The HTTP protocol locates resources through URLs, and manages resources through six commonly used methods. Each operation is independent and stateless.
In the world of HTTP protocol, network channels and servers are black boxes. What he can see is URL and corresponding operations on URL

Requests library methods (post(), put())

Obtain summary information of network resources with very little traffic

The type of request post() method is different, and the result obtained is different:
1.
2.

2.1.2 Analysis of the main methods of the Requests library

request

OPTIONS: Get some parameters from the service that the server can interact with the customer service. It is not directly related to obtaining resources, so it is less used.


Submitting data to the server

is also a common data submission

Custom HTTP header
Chrome/10: Represents Chrome's 10th This version
can simulate any browser to send access to the server,




hide the original IP address from which the user crawled the webpage

get

head

post

put

patch

delete

to sum up

Why is it designed like this? In the last six methods, because some methods need to use certain parameters, these parameters are put into the parameters as display-defined parameters, and the others are put into optional parameters. The
get method is most commonly used to crawl the content from the server. Without submitting content

2.1.3 Summary

Use get crawler
Use head to get a summary of larger resources

import requests
import time

def getHTMLText(url):
    try:
        r=requests.get(url)
        r.raise_for_status()
        r.encoding=r.apparent_encoding
    except:
        print("failed")


if __name__=="__main__":
    start=time.perf_counter()
    for i in range(100):
        getHTMLText("http://www.baidu.com")
    end=time.perf_counter()
    print("用时%.5f秒"%(end-start))

I did not consider whether it successfully crawled 100 times. . .

2.2 The "pirates can also be thefts" of web crawlers

2.2.1 Problems caused by web crawlers

2.2.2 Robots Protocol

"?" "html file in popm directory" is not allowed. . .

No robots protocol means that it can be crawled arbitrarily

2.2.3 How to comply with the Robots agreement

That is, the number of visits is small

2.3 Actual Combat of Requests Library Web Crawler (5 Examples)

Look at web content from the perspective of crawlers

2.3.1 Crawling of JD product pages

r.encoding: The encoding information can be parsed from the HTTP header, and JD.com provides the encoding

import requests
import time

def getHTMLText(url):
    try:
        r=requests.get(url)
        r.raise_for_status()
        r.encoding=r.apparent_encoding
        print(r.text[:1000])
    except:
        print("failed")


if __name__=="__main__":
    getHTMLText("https://item.jd.com/100006349587.html")

What actually jumped out was login

2.3.2 Crawling of Amazon product pages

Being able to crawl back things means that it’s not a network problem.
That is to say, the website rejected the crawler’s request. The
response object r contains the request request,
so the header information needs to be modified.

2.3.3 Baidu 360 search keyword submission

After analyzing the content, I will talk about it

2.3.4 Crawling and storage of network pictures

The picture is in binary form, how to save it as a file?

• Custom name
  
• Original name
  

2.3.5 Automatic query of IP address attribution

Try to constrain the size of r.text, such as r.text[-500:], otherwise it will affect the use of idle.

As long as you know the form of the URL submitted to the background after pressing the button, you can code simulation submission