1: .Net open source cross-platform crawler framework DotnetSpider Star: 430
DotnetSpider is a cross-platform, high-performance, lightweight crawler software open sourced by Chinese people, developed in C#. It is currently one of the best crawlers for .Net open source crawlers.
Please click here to enter image description
2: An open source crawler written by a Russian expert xNet Star: 117
This open source tool written by a Russian genius, why is it said that he is powerful, because he has implemented the bottom layer of all Http protocols again, what is the benefit of this? As long as you write a crawler, you will encounter a maddening problem, that is, you know that your Http request header is exactly the same as the browser, why can't you get the data you want
3: Open source .net crawler Abot Star: 1050
Abot is an open source .net crawler that is fast, easy to use and extend
4: C# locomotive-like open source data collector V5_DataCollection Star: 25
V5 data collector is a professional data acquisition software for personal and professional users. It is suitable for simple configuration operations, and also adapts to the ability to collect complex data. What you see can be collected. The unique proxy polling collection mechanism of V5 data collector can effectively solve the problem of website blocking and can be used for dynamic monitoring of Internet data. It is definitely your first choice.
5: C# crawler engine kernel version SmartSpider Star: 17
SmartSpider crawler engine kernel version, a new design concept, a real minimalist version.
6: .Net open source super crawler Hawk Star: 1039
HAWK is a data collection and cleaning tool. It is open source according to the GPL protocol. It can flexibly and effectively collect data from web pages, databases, and files, and quickly perform operations such as generation, filtering, and conversion through visual drag and drop. The areas where its functions are most suitable are crawler and data cleaning
7: Simple and efficient website crawler based on C#.NET Star:58
Simple-Web-Crawler - A simple web crawler based on C#.NET, supports asynchronous concurrency, switching proxies, operating cookies, and Gzip acceleration.
8: Website data collection software network miner collector (original soukey picking)
Soukey picking website data collection software is an open source software based on .Net platform, and it is also the only open source software in the type of website data collection software. Although Soukey picks open source, it will not affect the provision of software functions, even richer than the functions of some commercial software. The main functions currently provided by Soukey picking are as follows: 1. Multi-task and multi-thread data collection, support POST method;...
9: Website data collection software NETSpider Star: 94
NETSpider website data collection software is an open source software based on .Net platform. Some functions of the software are developed by basic Soukey software. This version is developed with VS2010+.NET3.5. The main functions currently provided by NETSpider picking are as follows: 1. Multitasking and multithreading data acquisition, support POST method (to be determined); 2. Can...
10: Web crawler NWebCrawler
NWebCrawler is an open source C# web crawler program
11: Web crawler tool NCrawler
NCrawler is a Web Crawler tool, which allows developers to easily develop applications with Web Crawler capabilities, and has extensible capabilities, allowing developers to expand its functions to support other types of resources (such as PDFs). /Word/Excel files or other sources). NCrawler uses multithreading...
12: Multithreaded web crawler spidernet
Spidernet is a multi-threaded web crawler program modeled on recursive tree, which supports the acquisition of text/html resources. You can set the crawling depth, the maximum download byte limit, support gzip decoding, and support gbk (gb2312) and utf8 encoding Resource; stored in sqlite data file. TODO: tag in source code describes unfinished function, hope to submit you...
13: Web crawler ScrapingSpider Star: 48
ScrapingSpider is a crawler developed in spare time that supports multi-threading, keyword filtering, and intelligent recognition of text content. The core implementation of the spider is in the ScrapingSpider.Core assembly. The crawler class is the Spider class. The crawling logic of the crawler is separated from the page processing logic through events. The two key events are AddUrlEvent and Data...
14: Reptile Xiaoxin Sinawler
The first domestic crawler program for Weibo data! Formerly known as "Sina Weibo Crawler". After logging in, you can specify a user as the starting point, and use the user's followers and fans as clues to collect basic user information, Weibo data, and comment data through personal connections. The data obtained by this application can be used as data support for scientific research, research and development related to Sina Weibo, etc., but please do not use it for commercial purposes. The application is based on the .NET2.0 framework and requires SQL SER...