In art patents are common in the recent 10,000 keywords, data acquisition, storage media, huge amounts of data, distributed a technology field hottest vocabulary. Wherein the data acquisition is the most mentioned word.
Data collection is a prerequisite for big data analysis is a necessary condition, occupy an important position in the overall data utilization process. Data collection divided into three types: log collection system method, network data acquisition method, and other data collection method. With the development of Web2.0, Web system covers the entire value of a large amount of data, the current data collection system for the Web is usually accomplished by a web crawler, this article will big data network and web crawler system is described.
What is big data network
Large data network, means a non-traditional data sources, such as different forms obtained by crawling search engine data. Large network data can also be purchased from data aggregators or search engines website data to improve target marketing. This type of data may be structured, or may be unstructured (more likely), it may be linked by a network, text data, table data, image, video and so on.
Network make up the bulk of our data available to today, according to many studies have found that unstructured data accounted for 80% of them. While these forms of data earlier been ignored, but increasing demand and the need for more data competition makes it necessary to use as much data sources.
Large data network can be used to do
Internet has billions of pages of data, big data network as a potential source of data for strategic business development of the industry, it has a huge potential for use.
The following examples illustrate the use of the value of big data network in different industries:
How to collect network data
Current collecting network data in two ways: one is the API, web crawler is another method. API also known as application programming interface, the site's manager for the convenience of users, a programming interface to write. Mainstream social media platforms such as Sina Weibo, Baidu Post Bar and Facebook etc provide services API, you can obtain the relevant DEMO on its official website open platform. But, after all, is limited by the API technology platform developers, in order to reduce the load website (platform), the general platform will interface calls are made daily limit restrictions, it gives us great inconvenience. To this end we usually adopt the second way - web crawler.
Using crawler technology to collect large data network
网络爬虫是指按照一定的规则自动地抓取万维网信息的程序或者脚本。该方法可以将非结构化数据从网页中抽取出来，将其存储为统一的本地数据文件，并以结构化的方式存储。它支持图片、音频、视频等文件或附件的采集，附件与正文可以自动关联。
在互联网时代，网络爬虫主要是为搜索引擎提供最全面和最新的数据。在大数据时代，网络爬虫更是从互联网上采集数据的有利工具。
网络爬虫原理
网络爬虫是一种按照一定的规则，自动地抓取网络信息的程序或者脚本。网络爬虫可以自动采集所有其能够访问到的页面内容，为搜索引擎和大数据分析提供数据来源。从功能上来讲，爬虫一般有网络数据采集、处理和存储 3 部分功能，如图所示：
网络爬虫采集
网络爬虫通过定义采集字段对网页中的文本信息、图片信息等进行爬取。并且在网页中还包含一些超链接信息，网络爬虫系统正是通过网页中的超链接信息不断获得网络上的其他网页。网络爬虫从一个或若干初始网页的 URL 开始，获得初始网页上的 URL，爬虫将网页中所需要提取的资源进行提取并保存，同时提取出网站中存在的其他网站链接，经过发送请求，接收网站响应以及再次解析页面，再将网页中所需资源进行提取......以此类推，通过网页爬虫便可将搜索引擎上的相关数据完全爬取出来。
数据处理
数据处理是对数据（包括数值的和非数值的）进行分析和加工的技术过程。网络爬虫爬取的初始数据是需要“清洗”的，在数据处理步骤，对各种原始数据的分析、整理、计算、编辑等的加工和处理，从大量的、可能是杂乱无章的、难以理解的数据中抽取并推导出有价值、有意义的数据。
数据中心
所谓的数据中心也就是数据储存，是指在获得所需的数据并将其分解为有用的组件之后，通过可扩展的方法来将所有提取和解析的数据存储在数据库或集群中，然后创建一个允许用户可及时查找相关数据集或提取的功能。
网络爬虫工作流程
如下图所示，网络爬虫的基本工作流程如下。首先选取一部分种子 URL。
- 将这些 URL 放入待抓取 URL 队列。
- 从待抓取 URL 队列中取出待抓取 URL，解析 DNS，得到主机的 IP 地址，并将 URL 对应的网页下载下来，存储到已下载网页库中。此外，将这些 URL 放进已抓取 URL 队列。
- 分析已抓取 URL 队列中的 URL，分析其中的其他 URL，并且将这些 URL 放入待抓取 URL 队列，从而进入下一个循环。
总结
当前，网络大数据在规模与复杂度上的快速增长对现有IT架构的处理和计算能力提出了挑战，据IDC发布的研究报告，预计到2020年，网络大数据总量将达到35ZB,网络大数据将成为行业数字化、信息化的重要推手。