Source code of this article: GitHub·click here || GitEE·click here
1. Based on business
Data services usually have many business models, which leads to complex system architecture and business. Different businesses have their own capabilities and complexity. Data management itself is not an easy task, so in the early stage of system architecture, Business scenarios considering service capabilities:
API service: data service based on Http mode, which obtains data through request, such as risk control model, scoring, anti-fraud and other services;
Platform services: a comprehensive service capability integration system, with low customer demand for customized services, and complete process data service capabilities, such as an automated digital marketing platform that provides full process management capabilities for marketing;
Collection service: Usually, customers submit related click events in a way of burying points, and the collection system conducts summary analysis based on omni-channels and provides feedback to customers;
Visual analysis: This is divided into two major areas, data analysis and visualization. Data can be loaded into multiple data sources for joint analysis, and highly automated analysis based on front-end components, such as common data insight systems;
Tool privatization: Based on accumulated technical capabilities, directly sell data management systems to customers and deploy them on their own local services;
In the data service scenario, different businesses need data support in their respective scenarios, but different businesses need the same basic functions such as operations, settlement, orders, etc. To understand different business scenarios, it is very simple to find common points and differences. The idea: the similarities are developed in public services, and the business differences are developed in independent services to facilitate the continuous expansion and evolution of the system.
Two, business layer architecture
The biggest difference between different data service capabilities is that they need to rely on the support of core data. From a business perspective, the system architecture layer also requires complex public functions. These need to be considered at the early stage of the architecture, otherwise the business will develop rapidly. It is about to face the problem of reconstruction.
Customer operation: Each customer's access requires a complete set of procedures, service descriptions, billing rules, contract management, recharge, service activation and deactivation, billing and a series of supporting functions. There are usually two entrances: customer login End, the service side operation end.
Payment and settlement: The most complex system module that provides payment capabilities, such as aggregating multiple payment channels to solve the customer's recharge and refund, or the service party's own payment needs, and provide various settlement bill data output and reconciliation The ability to balance accounts.
Order management: each customer’s request, or the use of each service, requires detailed order records for billing actions that involve unit price, order number, and time. As the core basis for settlement, it is also the most concentrated business data. The place where the outbreak occurred.
Authority system: In the data service system, the authority system design focuses more on solving the needs of the main body of the company. Different business teams are responsible for different service operations, customer management, etc., so a clear and systematic authority management is required for different roles. Business personnel allocate reasonable permissions.
Log integration: In the detailed log system, normal business log data can be used for data completion analysis when the service is abnormal, and abnormal log data can be used for developers to analyze system problems and bottlenecks to continuously optimize service capabilities.
The division and design of services based on business scenarios, as well as the basic construction of public services, ensure that the structure of the business layer is reasonable and scalable. The basic consideration for whether it is reasonable is whether the continuous new business scenarios require drastic revisions of the system. If the service capacity is continuously enriched, the cost of system transformation is small, and the natural structure is reasonable.
3. Data Center
Different business service scenarios need to rely on core data capabilities. This is the selling point of the service. The core data supporting service capabilities are usually deployed separately and provide various service scenarios, which are usually understood as data centers. At the same time, business services themselves will also generate various data. , Here will be stored independently according to the deployment method of the service.
Service capability: As a public dependency of multiple businesses, the data center not only needs to provide data-based query capabilities, but also needs to provide certain scheduling and computer systems when processing massive data tasks.
Deployment method: According to data characteristics, it will usually be stored in clusters, sub-databases and tables, OLAP engines, data warehouses, etc., and provide unified service capabilities based on data characteristics and open to the business layer.
Data update: Data needs to be updated in real time or regularly. The source of data is usually various data calculated and processed by big data, and data that is incorrectly verified by the business layer, or data that is continuously optimized during use.
The independent architecture deployment of the data center is a very necessary operation. Most of the data is linked. The linkage processing between data does not need to be coupled to the business level. The flow of data, correction, security management, etc. can all be done in the data center. Unified management avoids a series of complex problems caused by data mixed deployment.
Fourth, the bottom layer of big data
The lowest level of data service capabilities requires massive data processing capabilities, so many big data component technologies are used to store, calculate, analyze, and move data.
Data storage: The most common storage at the bottom of big data is in the form of files, structured database storage, semi-structured log files, and some unstructured data.
Computing power: The processing of massive data needs to rely on a variety of parallel computing, offline tasks, real-time computing and other methods to achieve the purpose of rapid processing.
Data handling: After data processing is completed, service capabilities are not directly provided at the bottom layer. Data is usually synchronized to the upper data center to provide service capabilities for the business. The handling here can be data output or data input to be processed.
The underlying components of big data are the core capabilities of the system. Accurate calculation and analysis of data ensure service capabilities, and continuous automation and tool-based management of the existing architecture are very important. The process of massive data management is more manual. More means that the efficiency is lower, especially in the process of pushing data to the data center or receiving data at the bottom layer, it is necessary to agree on a strategy to ensure safe and stable automatic data transmission.
Five, overall consideration
For the design of a complex system, the most important thing is to clearly sort out the business model, analyze the business model, and make the system architecture based on the business characteristics to avoid many detours, such as the above data service system:
First of all, from a big perspective, the system splits the three major parts of business services, data centers, and big data underlying capabilities, and requires that there is no strong coupling relationship between each large module to ensure that the modules can be expanded independently;
Secondly, determine the core functions required by each module, the business layer guarantees the basic service capabilities, and then extract and encapsulate the basic functions required by each business down, split out business services and public services, and support business capabilities;
Then determine the mode of collaboration between the various modules, such as the communication capabilities between the business and the data center, interface standards, data security and other details, or the data handling mode between the data center and the underlying big data, to ensure data circulation capabilities;
Finally, the specific details of each module are implemented. What needs to be considered here is that according to the business model, if the same components and architecture methods can be selected, try to unify the architecture selection and component dependencies, and reduce the barriers between different modules;
The above-mentioned complete system architecture takes about seven months from the beginning of the establishment to the provision of stable service capabilities. During this period, it is constantly evolving and upgrading, and new service modules are continuously launched and system monitoring is carried out until the business services are relatively complete and systematic. relatively stable.
Six, source code address
GitHub地址：知了一笑
https://github.com/cicadasmile/spring-cloud-base
GitEE地址：知了一笑
https://gitee.com/cicadasmile/spring-cloud-base
Read the label
[ Java Foundation ] [ Design Pattern ] [ Structure and Algorithm ] [ Linux System ] [ Database ]
[ Distributed architecture ] [ micro service ] [ big data components ] [ SpringBoot Advanced ] [ Spring & Boot foundation ]
[ Data Analysis ] [ Technical Map ] [ Workplace ]
