1: .Net open source cross-platform crawler framework DotnetSpider (Star:449)
Download address: http://www.17ky.net/soft/479.html
DotnetSpider is an open source .NET cross-platform data collection crawler framework.
2: The open source crawler xNet written by the Russian cattle (Star:121)
Download address: http://www.17ky.net/soft/756.html
This open source tool written by a Russian genius, why is it said that he is powerful, because he has implemented the bottom layer of all Http protocols again, what is the benefit of this? As long as you write a crawler, you will encounter a maddening problem, that is, you know that your Http request header is exactly the same as the browser, why can't you get the data you want
3: Open source .net crawler Abot (Star:1072)
Download address: http://www.17ky.net/soft/66.html
Abot is an open source .net crawler that is fast, easy to use and extend
4: C# Crawler Engine Kernel Version SmartSpider (Star:18)
Download address: http://www.17ky.net/soft/549.html
SmartSpider crawler engine kernel version, a new design concept, a real minimalist version
5: .Net open source super crawler Hawk (Star:1068)
Download address: http://www.17ky.net/soft/798.html
HAWK is a data collection and cleaning tool. It is open source according to the GPL protocol. It can flexibly and effectively collect data from web pages, databases, and files, and quickly perform operations such as generation, filtering, and conversion through visual drag and drop. The areas where its functions are most suitable are crawler and data cleaning
6: Simple and efficient website crawler based on C#.NET (Star:64)
Download address: http://www.17ky.net/soft/70470.html
Simple-Web-Crawler - A simple web crawler based on C#.NET, supports asynchronous concurrency, switching proxies, operating cookies, and Gzip acceleration.
7: Web crawler NWebCrawler
Download address: http://www.17ky.net/soft/9291.html
NWebCrawler is an open source C# web crawler program
8: Reptile Xiaoxin Sinawler
Download address: http://www.17ky.net/soft/34589.html
The first domestic crawler program for Weibo data! Formerly known as "Sina Weibo Crawler". After logging in, you can specify a user as the starting point, and use the user's followers and fans as clues to collect basic user information, Weibo data, and comment data through personal connections. The data obtained by this application can be used as data support for scientific research, research and development related to Sina Weibo, etc., but please do not use it for commercial purposes. The application is based on the .NET2.0 framework and requires SQL SER...
9: Multithreaded web crawler spidernet
Download address: http://www.17ky.net/soft/34598.html
Spidernet is a multi-threaded web crawler program modeled on recursive tree, which supports the acquisition of text/html resources. You can set the crawling depth, the maximum download byte limit, support gzip decoding, and support gbk (gb2312) and utf8 encoding Resource; stored in sqlite data file. TODO: tag in source code describes unfinished function, hope to submit you...
10: Web crawler tool NCrawler
Download address: http://www.17ky.net/soft/34609.html
NCrawler is a Web Crawler tool, which allows developers to easily develop applications with Web Crawler capabilities, and has extensible capabilities, allowing developers to expand its functions to support other types of resources (such as PDFs). /Word/Excel files or other sources). NCrawler uses multithreading...