Agile big data, that is, in the agile philosophy guiding principles, common platform to build a series of tools, and a set of big data application lifecycle methodology to support the lighter, more flexible, lower the threshold of big data practice. This article explained the whole "agile big data" we understand from the theoretical level.
First, the concept of agile principles big data
Components of 1.1 / platform / product / localization
Componentized / platform: large data processing by modular link abstract form highly cohesive functional components of the plurality of platforms; assembly platform independent and can use existing integrated platform components, can be combined to solve more questions on different links.
Product / localization: a combination of components of different platforms, with the extracted abstract service logic model rules and algorithms model can be easily constructed in the field of products of a particular business solutions; Solutions product actual local landing process, including the data model adaptation / ruleset introducing / parameter adjustment algorithm model.
1.2 unified / open technology / control technology
Designed to simplify the complexity of the unified system, improve the management and control capabilities; designed to enhance the fitness of opening up, improve flexibility; two complement each other, need to find a reasonable balance, and yet the overall management and control of.
1.3 Standardization / of the interface / configuration of / Visualization
Standardization / interfaces of: in the large data processing chain, a series of standardized protocols, protocol namespace includes data / meta data and a data type specification protocol / data access interface protocol / Query Language protocol / data transfer protocol / data security protocols ; service interface and the way to provide queue interface between system interaction.
Configuration of / Visualization: Visualization and arranged to provide interactive manner.
1.4 self-help / automation / intelligent
Modern data applications require the ability to output, so users in the field in an environment subject to management and control can be more self-oriented use of platforms and data to achieve business needs; self-help routine operations can better support in an automated fashion; insights self-help can be to intelligently better support.
1.5 of engine-driven (Event Engine / Action Engine / rules engine)
By introducing advanced engine driving ability, so that big data applications can be more agile and quick, agile, proactive contact of external audiences, this time big data application itself has become a powerful engine driving the business.
Second, you can abstract the universal tool platform
Enterprise-class real-time data platform, for example, we are guided by the principles of the agile concept of big data, real-time data platform overall end-to-modularized segmentation, and a series of standardized protocols, and finally to determine the principle of a unified and open to which common platform and tool boundaries and interface specification development.
The figure is a conceptual block diagram architecture real-time data platform, in subsequent articles we will be real-time data platform as the starting point, elaborate abstraction and architecture derived from a common platform tools.
Third, large data applications throughout the life cycle
3.1 verification requirements analysis phase
In the requirements analysis stage, we need to be able to quickly develop data applications prototype POC, and after validated able to cover all the needs of rapid iteration points as early as possible.
Agile Big Data platform / configuration of / visualization capabilities to support business development staff needs iteration verified quickly by configuring and visually. Business developers only need to focus on the business problem itself, without too much attention to big data technology issues.
3.2 architecture design and selection phase
In practice, storage and compute engine selection process, to consider a lot of factors, in addition to data and scale to meet SLA requirements, but also had to suffer the limitations and problems of open source technology selection.
Standardized agile Big Data / Interface technology / unification / open and so the ability to provide a set of best practices architecture selection, not only greatly reduces the complexity of system design, shielding the open source technology is not compatible with the problem, select the flexibility to support different storage and computation engines.
3.3 Test embodiment tuning stage
Testing and tuning embodiment Big Data customized development is often a very time-consuming work, and as the variable length and diversity selection processing chain technology, further increasing the complexity of tuning tests.
Internet Agile large data / interface of / configuration of / visual / unification / control and so able to make the process of implementing the test tuning becomes an iterative process takes Visual configuration / test / verification, data processing chain through question is disposed of / visual masking, the problem is too much unified technology selection / control of the shield.
3.4 on-line migration phase deployment
Big Data customized development of on-line deployment step migration often complicated, error-prone, even if the script can be supported in a way, it may be due to non-uniform and non-intuitive bring potential problems. Platform agile big data / configuration of / visual / unified / control of / self-help and other capabilities allow on-line deployment migration easier, which have benefited from unified platform capabilities, and these capabilities open to a self-service way user.
3.5 Management operation and maintenance monitoring phase
Management operation and maintenance monitoring in the enterprise tend to be unified imputation management and control, agile big data platform / control of / self-help, also offer the appropriate capabilities, in addition, can also provide automated / intelligent and so the ability to further reduce operating workload; also can be integrated via an interface capabilities with existing environmental monitoring system operation and maintenance.
Fourth, practice practice agile big data
The figure is the relationship between the various components of our great agility summarized data:
Agility Agility Big Data Big Data platform concept + + stack agile methodologies → Big Data Big Data agile practices
In this paper, the definition of "agile big data", as well as the concept of agile big data, and briefly describe how to build the platform stack and learn how to practice the method based on this idea. In future articles, we will expand our detailed agile big data journey around specific agile big data experience.
Author: Lu Shanwei
Source: CreditEase Institute of Technology
Reproduced in: https: //juejin.im/post/5d0afac3f265da1b8466eae4