In the early days of the Internet, writing crawlers was a technical job. Generally speaking, crawler technology was an integral part of search engines.
With the development of Internet technology, the threshold for writing crawlers has dropped again and again. Some programming languages even directly provide crawler frameworks, such as Python's Scrapy framework, which makes writing crawlers accessible to "ordinary people's homes."
We have discovered that writing crawlers is a cool thing, but even so, there is still a certain technical threshold for learning crawlers.
The current mainstream crawler method is programming in Python. There is no doubt that Python is powerful, but it still takes a month or two for beginners to learn Python.
Are there any simpler ways to crawl data? The answer is yes.
Some visual crawler tools use strategies to crawl specific data. Although they are not as accurate as writing your own crawler, the learning cost is much lower. Here are some visual crawler tools.
domestic tools
Microsoft Excel
First, I will teach you how to use Excel to crawl data. The Microsoft Excel 2013 version is used here. Let’s start teaching step by step~
(1) Create a new Excel and open it, as shown in the figure below
(2) Click "Data" - "Self Website"
(3) Enter the target URL in the pop-up dialog box. Here we take the national real-time air quality website as an example, click Go, and then import
Select the import location and confirm;
(4) The result is as shown in the picture below. How does it look? Is it great?
(5) If you want to update the data in real time, you can set it in "Data" - "All Updates" - "Connection Properties" and enter the update frequency.
Octopus
https://www.bazhuayu.com/
A visual, programming-free web page collection software that can quickly extract standardized data from different websites, helping users realize automatic collection, editing and standardization of data, reducing work costs.
A collection software suitable for novice users to try. It has powerful cloud functions. Of course, experienced crawlers can also develop its advanced functions.
locomotive
http://www.locoy.com/
Locomotive is an Internet data capture, processing, analysis, and mining software with complete collection functions. It is not limited to web pages and content. Any file format can be downloaded. It claims to be able to capture 99% of web pages.
The positioning of the software is relatively professional and precise. Users need to have a basic HTML foundation and be able to understand web page source code and web page structure. However, the software provides corresponding tutorials so that novices can learn and get started.
**JiSouKe
**
http://www.gooseeker.com/index.html
A simple and easy-to-use web information grabbing software that can capture web text, charts, hyperlinks and other web elements.
The operation is relatively simple and suitable for beginner users. There are not many features in terms of functions and there are many subsequent payment requirements.
Archer Cloud Crawler
https://www.shenjian.io
A novel cloud-based online intelligent crawler/collector, based on the Archer distributed cloud crawler framework, helps users quickly obtain large amounts of standardized web page data.
Similar to a crawler system framework, specific collection requires users to write their own crawlers, which requires a code base.
Madman Collector
http://www.kuangren.cc/
A set of professional website content collection software that supports the collection of posts and replies in various forums, as well as the capture of website and blog article content. It is divided into three categories: forum collector, CMS collector and blog collector.
It focuses on capturing the text content of forums and blogs, but is not very versatile for collecting data from the entire network.
foreign tools
Google Sheet
google.cn/sheets/about/
Before using Google Sheet to crawl data, you must ensure three things: use the Chrome browser, have a Google account, and your computer has circumvented the wall. If these three conditions are met, let’s get started~
(1) Open the Google Sheet website:
(2) Click "Go to Google Sheets" on the homepage, then log in to your account. You will see the following interface, and then click "+" to create a new form.
The newly created table is as follows:
(3) Open the target website to be crawled, a national real-time air quality website pm25.in/rank. The table structure on the target website is as shown below:
(4) Return to the Google sheet page and use the function =IMPORTHTML(URL, query, index). "URL" is the target website to crawl the data. Enter "list" or "table" in "query", depending on the data. The specific structure type, "index" is filled with Arabic numerals, starting from 1, corresponding to which table or list defined in the website;
For the website we want to crawl, we enter the function =IMPORTHTML("pm25.in/rank","table",1) in cell A1 of the Google sheet, press Enter and the data will be crawled~
(5) Save the crawled table locally
you-get
This is a project developed by programmers based on python 3. It has been open sourced on github and supports 64 websites, including Youku, Tudou, iQiyi, Bilibili, Kugou Music, Xiami... In short, there are all the websites you can think of. !
There is also a black technology aspect. Even if it is a website that is not on the list, when you enter the link, the program will guess what you want to download and then download it for you.
Of course, you-get must be installed in a python3 environment. After installing it with pip, enter "you get + the link of the resource you want to download" in the terminal and wait to collect the resource.
Here is a you-get Chinese instruction manual, just follow the steps written in the instructions.
import.io
https://www.import.io
Import.io is a web-based data collection platform that allows users to generate an extractor without writing code. Compared with most domestic collection software, Import.io is more intelligent and can match and generate a list of similar elements. Users can also collect data with one click by entering a URL.
Import.io is intelligently developed and easy to collect, but its processing capabilities for some complex web page structures are relatively weak.
Octoparse
https://www.octoparse.com/
Octoparse is the overseas version of Octopus. The collection page design is simple and friendly, and the operation is completely visual. It is suitable for novice users.
Octoparse has complete functions, reasonable price, and can be applied to complex web page structures. If you want to access Amazon, Facebook, Twitter and other platforms directly without going through the firewall, Octoparse is an option.
visual web ripper
http://visualwebripper.com/
Visual Web Ripper is an automated web scraping tool that supports various functions.
It is suitable for some advanced and difficult to collect web page structures, and users need to have strong programming skills.
content Grabber
http://www.contentgrabber.com/
Content Grabber is one of the most powerful web scraping tools. It is more suitable for people with advanced programming skills and provides many powerful script editing and debugging interfaces. Allows users to write regular expressions instead of using built-in tools.
Content Grabber web page has strong applicability and powerful functions. It does not fully provide users with basic functions and is suitable for people with advanced programming skills.
Mozenda
https://mozenda.updatestar.com/
Mozenda is a cloud service-based data collection software that provides users with many practical functions including data cloud storage functions.
Suitable for people with basic crawler experience.
[For those who want to learn crawlers, I have compiled a lot of Python learning materials and uploaded them to the CSDN official. Friends in need can scan the QR code below to obtain them]
1. Study Outline
2. Development tools
3. Python basic materials
4. Practical data
