Web crawlers in many areas have a wide range of applications, its goal is to obtain new data from the site, and store them for easy access. The web crawler tool more and more well known, because it simplifies and automates the entire process of reptiles, so that everyone can easily access the network resource data.
1. Ochtoprse
Octoparse is a free and powerful site crawler tool to extract various types of data required from the site. It has two learning modes - wizard mode and advanced mode, so non-programmers can also be used. You can download almost all web content and save as EXCEL, TXT, HTML, or other databases in a structured format. Scheduled Cloud Extraction has a function, you can get the latest information on the website. Providing IP proxy server, so do not worry about being aggressive site detected.
In short, Octoparse users should be able to meet the most basic needs or grab the high-end, without any coding skills.
2. Cyotek WebCopy
WebCopy reptile is a free tool that allows you to copy a partial or complete site content to a local hard disk for offline reading. It will be the site content is downloaded to the hard drive before scanning the designated site and automatically re-map the site link images and other web resources to match their local route. There are other features such as download URL included in the copy, but you can not reptiles. You can also configure the domain name, user agent string, default documents.
However, WebCopy does not include virtual or JavaScript DOM parsing.
3. HTTrack
As a free website crawler software, HTTrack offer is great for the entire site to download from the Internet to your PC. It provides applies to Windows, Linux, Sun Solaris and other Unix system version. It can mirror one or more sites (share links). To determine the number of connections open simultaneously when the download page under "Settings option." You can be obtained from the entire catalog of photos, documents, HTML code to update the current mirror site and resume interrupted downloads.
In addition, HTTTrack provides proxy support to maximize speed and provides optional authentication.
4. Getleft
Getleft is a free and easy to use tool reptiles. Enter the URL after the start Getleft and select the file to be downloaded, and then start the download site addition, it offers multi-language support, currently Getleft supports 14 languages. However, it only provides limited Ftp support, it can download files without recursion.
Overall, Getleft reptiles should meet the basic needs of the user without the need for more sophisticated skills.
5. Scraper
Scraper tool is a Chrome extension, limited data extraction, but very useful for online research and export data to Google Spreadsheets. For beginners and experts you can easily copy data to the clipboard or use OAuth stored in electronic form. Crawl does not offer all-inclusive service, but for the novice can be considered friendly.
6. OutWit Hub
OutWit Hub is a Firefox plug-in, with dozens of data extraction functions to simplify web search. Information will be stored in a format suitable for the extraction of the browser page. You can create automatic agents to extract data and format it according to the settings.
It is one of the easiest reptiles tool, free to use, provides a convenient extract web page data without writing code.
7. ParseHub
Parsehub is an excellent tool reptiles, support the use of AJAX, JavaScript, cookies and other data get web pages. It's machine learning techniques can be read, analyzed and then converted to web documents related data. Parsehub desktop application support Windows, Mac OS X and Linux systems, or you can use the built-in Web browser application.
8.Visual Scraper
VisualScraper is another great free tool reptiles and non-coding, simply click interface can collect data from the network. Can obtain real-time data from multiple pages, the extracted data exported as CSV, XML, JSON or SQL file. In addition to SaaS, VisualScraper also provide web crawlers services such as data transmission services and create software extracts the service.
Visual Scraper enable users to run their projects at a particular time, you can also use it to get news.
9. Scrapinghub
Scrapinghub is a cloud-based data extraction tool that helps thousands of developers access to valuable data. It's open source visual crawlers crawl the web allows users without any programming knowledge.
Scrapinghub use Crawlera, this is an intelligent proxy rotator support mechanism to bypass the bot, bot easily grab a large number of protected sites. It enables a user to the crawler and a plurality of position by the simple IP HTTP API, without the need for management agent.
10. Dexi.io
As a web crawler browser-based tool, Dexi.io allows users to fetch data from any website, and offers three types of robots to create a crawl task - extractor, crawling and piping. The free software provides data anonymous Web proxy server, the extracted data is stored on the server before Dexi.io archived within two weeks, or extracted directly exported as JSON or CSV file. It provides payment services to meet the needs of real-time access to data.
11. Webhose.io
Real-time data Webhose.io enables users to sources from around the world line is converted to a variety of clean format. You can use multiple filters to cover a variety of sources to capture data, and further extract keywords in different languages.
Captured data can be saved as XML, JSON and RSS formats, but also access to historical data from the archive. In addition, webhose.io supports up to 80 languages and crawl data results. Users can easily structured data indexing and searching Webhose.io crawl.
Overall, Webhose.io reptile meet the basic needs of the user.
12. Import. I
Users simply introduced from a specific website and exporting data to a CSV own set of data can be formed.
You can easily grab thousands of pages in minutes, without writing any code, and build more than 1000 API according to your requirements. Public API provides a powerful and flexible functionality to programmatically control Import.io and gain automatic access to data, Import.io by integrating Web data into your own application or website, just a few clicks you can easy reptiles.
In order to better meet the needs of the user grab, it also provides for the Windows, Mac OS X and Linux free application to build data extractor and crawlers, download data and sync with an online account. In addition, users can weekly / daily / hourly schedule reptiles task.
13.80legs
80legs is a powerful web crawler can be configured according to customer requirements. 80legs provide high-performance Web crawler, you can quickly work and obtain the required data within a few seconds.
14. Spinn3r
Spinn3r allows you to get all the data from a blog, news and social media sites, and RSS and ATOM in. Spinn3r released a firewall API, 95% of the index management work. It offers advanced spam protection, can eliminate spam and inappropriate language, thereby improving data security.
Spinn3r index similar to Google's content, and save the extracted data in a JSON file.
15. Content Grabber
Content Graber is a ripping software for corporate networks. It allows you to create a separate web crawlers agent.
It is more suitable for people with advanced programming skills, because it offers many powerful script editor and debugger interface for those in need. It allows users to use C # or VB.NET programming debugging or write scripts to control the crawling process. For example, Content Grabber can be integrated with Visual Studio 2013, in order to provide the most powerful script editing, debugging and unit testing according to the specific needs of the user.
16. Helium Scraper
Helium Scraper is a network data visualization software reptiles, when the association between elements is small would be better. It is non-coding, non-configuration. Users can access online templates based on the needs of various reptiles.
It basically meet the needs of users in the early stages of reptiles.
17. Uifth
UiPath is an automated web crawler. It can automatically crawl the Web and desktop data from third-party applications. Uipath across multiple web forms and extract data based on patterns.
Uipath provides built-in tools for further crawlers. When dealing with complex UI, this method is very effective. Screen Scraping Tool text elements may be treated individually, text blocks and text group.
18. Scrape. it
Scrape.it is a cloud-based Web data extraction tool. It is designed for people with advanced programming skills to design, as it provides the public and private package in order to discover along with millions of developers worldwide use, update, and share code. Its powerful integration can help users build custom reptiles according to their needs.
19. WebHarvy
WebHarvy is designed for non-programmers. It can automatically take the crawling text, image, URL and e-mail from the website and save content crawling in various formats. It also provides built-in scheduler and proxy support, and to prevent anonymous crawling the Web server is blocked, can be selected via a proxy server or VPN to access the target site.
The current version WebHarvy Web Scraper allows users to fetch data export to XML, CSV, JSON or TSV file can be exported to an SQL database.
20. Connotate
Connotate is an automated Web crawler software, designed for enterprise-class Web Crawler, require enterprise-class solutions. Business users can easily create extract agent in a matter of minutes without any programming.
It can automatically extract more than 95% of the site, including JavaScript-based dynamic web technologies, such as Ajax.
In addition, Connotate also offers an integrated web and database content features, including content from SQL databases and database MongoDB extraction.
Recommended reading:
Basics zero Python's most detailed source of tutorials
2019 Python Reptile Learning Roadmap full version
Why Python can be firmly secured the first card AI Artificial Intelligence Language
Python rise, TIOBE list of programming languages a new high!