1. What is the reptile
Program web crawlers (web crawler referred reptiles) is in accordance with certain rules to fetch information from the Internet, since it is a normal procedure and what is the difference that users access the page? The difference between reptiles and user access to the information is that: the user is slow, a small amount of access to information, and reptiles are a lot of access to information.
There is also to be noted that: patent reptile is not the Python language, Java, Js, C, PHP, Shell, Ruby language and so on can be achieved, why Python is so crawlers fire? I feel that compared to other reptiles Python language may be the perfect spot various libraries, easy to get started with everyone, naturally active community, and community activists led Python reptile gradually become mature, mature and encourage more users to use, so benign cycle, so Python reptile reptilian compared to other languages is more fire.
Here are some reptiles Python hello world level, which is equivalent to you in the Baidu search keywords: Python.
2. Why to learn web crawler
Our initial understanding of the web crawler, the web crawler but why should learn it? You know, the only clearly know the purpose of our study, to be able to learn better knowledge of this one, we will analyze the reasons for everyone to learn web crawler.
Of course, different people learn reptiles, may be different purposes, here we summarize four common causes of learning crawlers.
1) study reptiles, can customize a private search engine , and can be a deeper understanding of the data collection of how search engines work.
After a brief information, we learned to write reptiles, reptiles can use the information in the Internet automatically collect, store or collect the corresponding treatment back in the time required to retrieve some information, just in back of the acquisition retrieval, which realized the private search engine.
2) Big Data era, to data analysis, we must first have a data source, and learning reptiles , allows us to get more data sources, and these sources can be collected by our purpose, to remove a lot of irrelevant data.
When making big data analysis or data mining, data sources can provide statistical data obtained from certain sites, it can also be obtained from certain internal documents or information, the way to obtain these data, we sometimes difficult to meet demand data, while the manual from the Internet to find these data, the effort required is too large.
At this point you can use crawler technology automatically get the data we are interested in content from the Internet, and the data content crawling back, as our data source to perform data analysis deeper and get more valuable Information.
3) For many SEO practitioners, learning reptiles can be a deeper understanding of how search engine spiders, which can better search engine optimization.
Since it is a search engine optimization, then it must be on how search engines work is very clear, but also need to know how search engine spiders, so conducting a search engine optimization in order to know ourselves, know yourself.
4) From the point of view of employment, the engineers at present belong to the reptile shortage of personnel and salary generally high , so deep to master this skill, for employment, it is very beneficial.
Interested in Python or are studying a small partner, you can join us to learn Python buckle qun: 784758214, look at how seniors are learning! From basic web development python script to, reptiles, django, data mining and other projects to combat zero-based data are finishing. Given to every little python partner! Every day, Daniel explain the timing Python technology, to share some of the ways to learn and need to pay attention to small details, click on Join us python learner gathering
3. reptile four essential tool
NO.1 F12 Developer Tools
- See the source code: rapid positioning element
- Analysis xpath: 1, suggested here Google-based browser, you can look at the source code interface just right
NO.2 capture tool
- Recommended httpfox, plug-in Firefox browser, to be better than Google Firefox F12 system comes with tools, you can easily view information site for receiving packets of contract
NO.3 XPATH CHECKER (Firefox plug-in)
Very nice xpath testing tools, but there are a few small drawbacks:
- xpath checker generates absolute path, encounter some dynamically generated icon (next page has a list of common buttons, etc.), erratic absolute path is likely to cause an error, so here is recommended in real time analysis, only as a reference
- I remember the figure below xpath box of "x:" get rid of, looks like this is an early version of the xpath syntax, and now some modules are not compatible (such as scrapy), or deleting avoid error.
NO.4 regular expression testing tool
Online regular expression test, used to practice your hand, also aided analysis! There are many ready-regular expressions can be used, can also be referenced!