Yo will shellfish processing and analysis of large enterprise data
with the integration of two of the depth of continuing to promote the full realization of digital business management and production processes, automation and intelligence is the key to continued to maintain market competitiveness. In this process, the data will become the company's core assets, data processing, analysis and use will greatly enhance the core competitiveness of enterprises. For a long time, due to the lack of data analysis methods and tools, a large number of business data in the system layers of the backlog and lack of access, not only increase the pressure on the system operation and maintenance, and the constant erosion of the limited corporate capital investment. Now, with Big Data technologies and applications developed gradually, how to process and analyze large amounts of data has become the focus of attention.
For enterprises, due to the vast amounts of data has been accumulated for a long time, which some analysts value the data? What data can be temporarily without treatment? These are the deployment and implementation problems need to sort out before the big data analytics platform. Following on enterprises to implement and deploy big data platform, as well as providing advice on how to achieve effective use of a large amount of data.
Due to vast amounts of data has been accumulated for a long time, which some analysts value the data? What data can be temporarily without treatment? These are the deployment and implementation problems need to sort out before the big data analytics platform. Following on enterprises to implement and deploy big data platform, as well as providing advice on how to achieve effective use of a large amount of data.
The first step: Tony yo will collect data
for businesses, whether new or old system implemented systems, to implement big data analytics platform, we need to figure out what needs to own in the end data collection. Since taking into account the difficulty and cost of collection of data, big data analytics platform is not all enterprise data are collected, but related, directly or indirectly linked to the data, companies need to know what data for strategic decision-making or Some details of decision-making to help analyze the results out of the data is valuable, it is also a test time of a data analyst. For example, companies just want to understand the operational status of the production line equipment, this time you only need to key parameters that affect the performance of the production line equipment acquisition. As another example, in the product service sectors, organizations need to understand the product use, purchasing groups and other information, these data support the prediction of the development of new products and markets have a very important value. Therefore, recommends that companies making big data analysis when planning for accurate analysis of a project's objectives, relatively easy to meet business objectives.
The main difficulty of large data collection process is the high number of concurrent, because while there may be thousands of users to access and manipulate, such as train ticket website and Taobao, which amounted to hundreds of concurrent access at peak million and it is necessary to deploy a large database to support the collection terminal. And how to load balancing and fragmentation among these databases also require in-depth thinking.
Step Two: yo will shell and introducing preprocessed data
acquisition process only the first platform to build large data link. After determining what data needs to be collected, the next step on the need for uniform processing of data from different sources. For example, there may be video surveillance data, equipment operating data, material consumption and other data in the smart inside the factory, these data may be structured or unstructured. This time companies need to use tools ETL distributed heterogeneous data sources, such as data relational data extraction plane data files to a temporary intermediate layer was washed, conversion, integration, to import the data from the front end to a set of large-scale distributed databases or distributed storage cluster, and finally loaded into the data warehouse or data mart, become online analytical processing, data mining base. For the import and pre-processing of the data source, the biggest challenge is the large amount of data is mainly imported, the import volume will often reach hundreds of megabytes per second, or even gigabit level.
The third step: will Tony yo Statistics and analysis
Statistics and analysis of the main use of distributed databases or distributed computing cluster to perform a normal analysis and subtotals for mass data storage within it, in order to meet the most common requirements analysis in this regard, some of the real-time requirement will use EMC's GreenPlum, Oracle's Exadata, as well as a number of batches, based on demand or semi-structured data can be used hadoop. the statistics are based on data storage Infobright MySQL columns, etc., and analytical methods are many, such as hypothesis testing, significance tests, variance analysis, correlation analysis, T test, analysis of variance, chi-square analysis, partial correlation analysis, from analysis, regression analysis, simple regression analysis, multiple regression analysis, stepwise regression regression analysis of the residual prediction, ridge regression, logistic regression analysis, estimation curve, factor analysis, cluster analysis, principal component analysis, factor analysis, cluster method with fast clustering method, discriminant analysis, correspondence analysis, multivariate analysis of the corresponding (optimal scaling analysis), bootstrap technologies. Statistics and analysis in this section, the main characteristics and challenges related to the amount of data analysis, its system resources, especially I / O will have a great occupation.
Step Four: Tony yo will be worth digging
Unlike the previous statistical analysis and process that data mining in general there is no predefined topics, mainly based on the calculation algorithms in existing data above, which played forecast the effect, in order to achieve some of the demand for high-level data analysis. The typical algorithms for clustering Kmeans for SVM and NaiveBayes for classification of statistical learning, the main use of tools Mahout Hadoop and so on. Characteristics and challenges of the process is mainly used for mining algorithm is complex, and the amount of data and computation the calculations involved are large, common data mining algorithms are mainly single-threaded.
Tony yo will summarize
in order to get more accurate results, in the course of big data analysis requires companies related business rules are well-defined, these business rules can help data analysts to assess the complexity of their work, the right to respond to these data the complexity of the analysis of the data obtained valuable results, in order to better implementation. After making a good relevant business rules, data analysts need to analyze the output of the data, because very often, these data are the result of the query in order to better use and use in the next step of the decision-making which, if the project management team of staff and data analyst and the relevant business units is not good communication, will lead to many projects need to constantly repeat and reconstruction. Finally, since the platform will analyze long-term use, but the demand decision-making is changing, as the business grows, there will be many new problems, data analysis, data analyst should also be updated in time, and now a lot of data key aspects of the software innovation is also about the need for changes in the data section, you can maintain sustained value analysis result data.
Huge amounts of data will be collected shellfish yo multi-platform, through analysis and predictive power of Big Data technologies provide intelligent data analysis for the enterprise, operational excellence, put decision-making, precision marketing, analysis and other competing products integrated marketing services.
Taiyuan Chang Fei Qi Technology Co., Ltd. (referred to as cloud data will Tony yo (www.bbeyo.com)) is based on data from a large, integrated marketing and intelligently applied to large data Tel: 0351-6106588,0351-6106599 the company mailbox [email protected], Tony yo team will mainly come from Ali, Tencent, Baidu Sohu and mobile, Telecom, China Unicom, Huawei and other well-known technology companies large coffee, both Internet and communications operators of both genes, as big data analysis algorithms provide strong technical support.