background
Recently, the industry's data security frequent incidents related to the enterprise caused irreparable damage, the more weak data security conscious enterprises sounded the alarm. How to company data the highest concentration of data analysis, data services, data management, and other data products for permission control, data security has become the most important tasks in construction.
If the control to be divided from the perspective, then, the rights control functions can be divided into control and data-level permission level access control list. Early data security products use traditional permissions model, can only achieve functional level permissions control, data-level permissions can not control. Products based on higher data security requirements, we need to build a variety of products while meeting data security we need to build a variety of products while meeting data security platform.
To this end, the US group the user platform application development team not only designed to express and control a variety of permissions model complex relationships, but also for before, during, after the three scenarios, each design approval, authority to audit three subsystems data security complete closed-loop, and thus meet the data security requirements.
Permissions function application expression products, generally "has permission" and the relationship between the expression of more complex data products permission. For example, report data products, not only to express the user can access this report and the need to express the dimension which users can access reports, metrics, and the range of dimension values. These dimensions also need to inform the metrics database table from which to model, has permission to access and create reports.
Permissions model
The traditional model has permission ACL (Access Control List) Access Control Lists, RBAC (Role-Based Access Control) role-based access control and so on. The above model is more suitable for the type of product application access control list, and the data type of the product for information security requirements are higher, and the relationships between various resources and more complex, difficult to use the traditional model will be a clear expression of internal relations, so we on the basis of RBAC permissions model, extensible design a new permissions model.
As shown, the conventional privilege model 2:
- ACL Access Control directly linked lists, and user permissions, direct relationship between the user and the maintenance of a list of resources so as to achieve the purpose of access control list.
- RBAC model is associated roles and permissions, the user to the corresponding role and obtain the corresponding permission.
Why design a new permissions model?
- ACL model is the concept of establishing a direct relationship between users and resources, not roles. When some users need the same number of resource rights, empowerment becomes very complex operation, then this model is less adapted.
- RBAC model introduces the concept, role and resource role in building relationships. When some users need the same number of resource permissions, you only need to build a character and give permission to use these resources. When a user joins this role, you have all the privileges of the role. Solve the problem of empowerment complex operations.
However ACL model and the RBAC model, there are the following questions:
- Data products complex relationship between resources, can not express this complex relationship well. For example: There are multiple tabs next report, there are a number of components under tabs, there are multiple dimensions, such as a lower index components. At the same time, dimensions, metrics and from different data models, database tables and so on. There is a relationship between resources and the resources, to the administrator when a user gives permission to all or part of the report, under the sub-resources report required to obtain the corresponding privileges at the same time.
- There is no corresponding relationship between RBAC roles and role model. For example: organizational structure, organizational structure where employees are as follows: Eastern Region / sell a zone / sale of a group of employees has a role is the role of a sales group. When there is no relationship between the roles of employees if needed permission Eastern Region roles, you need to add to the role of the Eastern Region. And if having a subordinate relationship between roles and role, we can solve this problem.
The new permissions model is how to solve these problems above:
- Resource model design, having dependencies between resources and resources, and the resources permit multi-level, impressions in a tree structure. For example, the report is a parent resource, labels, components, sub-resource under the dimension indicators are statements that clearly demonstrate the relationship between resources and sub-resources report the following when this empowerment, in order to meet a variety of access control and authentication when empowerment requirements.
- Have dependencies between roles and role, such as employee in the Eastern Region / sale of a district / sales organization a group, between the Eastern Region / sell a zone / sale of a group of three characters each have a parent-child class affiliation when a group of employees in the sales department, the owner Eastern region, a district sales, sales of all rights of a group. When the authority does not conflict directly merge all rights, by the "principle of proximity" when covering the conflict.
3, the new privilege model comprises three parts, the center user, Resource Center, authority center.
User Center: user management, role management
- Into the role of individuals, organizations, custom three kinds, a user can have multiple roles at the same time, such as the user's default character corresponds to one individual, but also at the same time in the company's organizational structure has organized role in the organization of custom custom roles.
- The role of multi-level support to meet the express way between role permissions inheritance.
- User, department information Mafka (US group based on a distributed messaging middleware Kafka development of integrated solutions) real-time updates, ETL timing synchronization every day, to ensure that personnel entry, job transfer, transferred authority real-time synchronization.
Resource Center: Resource Management
- Support custom resource types, supporting custom resource access on the basis of the common resource type, to meet the unified control system various different resources.
- Resources to support multi-level, resources show the way of the tree structure to facilitate unified authentication empowerment resources; when to report a resource empowerment, hanging in a report, dimensions, metrics and other resources can be unified to obtain permission.
- Packaging simplify support resources empowerment process.
- Resource security security classification, resource person in charge, in accordance with the allocation of resources to support different templates approval authority buffet application.
Permissions center: the relationship between the expression of a variety of strategies and resources role
- Range strategy: platform dimensions such as dimension values in the report include the US group and the public comments, when empowerment, as required to support some or all of the user gives permission; when the authentication, in accordance with the rules of analytic dimension has a part or all of a person's rights .
- Expressions strategy: When the report to the user empowerment, setting the expression is limit 10, the user then represents the current limit on the basis of the report on other rights, only to return 10 before recording.
- Permissions automatically merge: a user has multiple roles, permissions automatically merge in accordance with the rules of multi-role authentication of the same resource; when parsing rules, permission to take when data collection is not a conflict, take the corresponding value in accordance with the priority conflict.
- Black list: support in accordance with specific rules, the highest priority for the comprehensive development of a person and a resource ban, black and white list policy, which is higher than white list blacklist.
challenge
In the process of building a data security platform, the main challenge facing the following:
- With increasing support line of business, we can not be customized to meet the needs of each business line common platform, the need to ensure a flexible and scalable system.
- Providing a common platform for data security, data security to meet most of the requirements to ensure the system's versatility.
- Rights system as a high-QPS system access, how to ensure high availability of the system.
Solutions
- Plugin provides flexible pluggable service, on the basis of common rights, to meet the various business lines flexible access control requirements.
- Provide a common data security platform to meet their basic rights, approval, audit of basic functions.
- Micro-service architecture, separating core and non-core services, data cache downgrade meet system availability.
solution
7, so that the general three points, internet content rights data, approval workflow platform, the audit log platform:
- Offering a variety of pluggable Plugin service, support, custom development on the basis on the basis of universal services.
- To provide basic services to meet a variety of common data security requirements.
- Provide management table, page enables administrators to manage and configure a variety of data and rules.
specific plan
Plugin service layer to ensure the system flexible and scalable
On the basis of satisfying the universal rights of the individual lines of business will inevitably be custom permission control needs, so the design authority Plugin module.
Universal service provides user management, resource management, authentication, authorization services, Plugin call basic services to achieve special permission control. Plugin applications and data management modules individually, calls for flexible pluggable universal service through the RPC method. Follow-up services Plugin modules support various applications access individual custom development.
8, the common services and service permissions Plugin is separated, a plurality of support services Plugin flexible pluggable:
- Universal service provides users, resources, authentication authorization and other general services, most systems can be realized based on the universal service authority control requirements.
- Plugin SDK-based service to provide both universal services to expand, each service separately Plugin deployed to ensure independent of each other between systems.
The final tiered control authority, the data is divided into core layer (users, resources, rights data), and an application layer. Data core data layer is managed by the General Services, to achieve a unified data management and control authority requirements. Plugin Application layer service access mode, data read and write permissions Plugin by external SDK common services layer, to achieve a custom control requirements. Each application layer data storage, control rules can be customized. Calls between the interfaces by BA certification authentication, ensure the security of calls between services.
Basic service level to ensure the system versatility
General Permissions System Architecture
Using a micro-service architecture, the system consists of the access layer, a service layer, a database layer, and an outer layer service. Mainly includes the following core services:
- User services: The main sectors containing user information and synchronization, role management.
- Resource Service: Contains resource registration, resource timing synchronization, security classification and administrators to manage resources, resource management package.
- Empowering service: self-application permissions, administrators empowerment.
- Authentication Services: Provides a variety of authentication SDK for consumer calls.
Figure 9:
- Access layer: all the system calls the external services through a unified SDK.
- Service layer: micro-service architecture, providing services to each other between the various services.
- The database layer: the rational use of caching, data demoted to ensure service availability.
- Public service integration company, to ensure the stable operation of the system.
Approval System Architecture
Provide universal approval services to provide multi-level approval template, select a template to start streaming approval, approval system, in accordance with the rules of analytical parameters to start automatically adapting the corresponding approval process when used. Reduced access procedure support for a key access.
Figure 10: Optimization of access approval process, to provide a common service approval, reduce development cost system access:
- A pre-development step requires approval feature 6, flow diagrams, and members of the group arranged approval, the configuration message notification, event mapping configuration, start approval flow, change status callback interface development.
- And we in the examination and approval service platform on the basis of the package, provides a common approval template, the approval system access only need to select a template to start the approval flow, and provides an interface to the callback. Able to meet most of the approval function.
General rules parsing engine, support approver, approval criteria, approval notification in accordance with the rules of matching dynamic analysis. Flexible automatic approval, more than multi-level approval, timed reminders, and other common functions.
Docking permission and audit system to ensure data security approval system:
- Docking rights system administrator privileges control.
- Docking audit system, the operating system facilitates subsequent audit data falls audit data.
Audit System Architecture
Audit services provide common data, the client logs Buried reported, the audit log stored by type Elasticsearch fall. Docking wishful visual reports the audit report, the docking system permissions control data access.
11: audit data model automatically extended support layer:
- Each application corresponds to a appkey, each appkey automatically create a template in accordance with sub-index date, support for automatic extension.
- Each type of audit logs corresponding to a type Elasticsearch index, when operating a new log, type is automatically created.
- Audit log corresponding to the field type of the field, the field is automatically extended when new.
Ensure system availability
Micro Services Architecture Services Separation
With more and more functions of the system module, a single architecture model is no longer suitable for agile development, increasing system startup module is slower, either module error the entire system services are not available.
In order to ensure high availability and scalability service, then the module micro split service architecture, and the core and non-core separation.
12:
- HTTP front-end access by the access layer, BA legality authentication check request by Nginx load balancing.
- Management console for unified management of various services by calling the service layer.
- Each module service layer abstraction system, each module is a micro-services, each micro services are deployed independently, can be deployed as needed depending on the size of each service.
- Client layer, Pigeon provide external uniform (inside the US group distributed RPC service communication framework) interfaces, the introduction of each service call service layer by POM.
Permission inheritance
Because resources to support multi-level, support permission inheritance model design rights, inheritance is turned on when empowerment, the user has full privileges to the following resources and all resources, only need to store the relationship between the ancestors of resources and user data storage. Greatly reducing the permission matrix size.
Data storage rights
The more access the system, the more resources and users. With the longer the system is running, the corresponding authority data will also grow rapidly. How to ensure performance and high availability interface while data growth.
Permissions Backup and Recovery
HBase reference design ideas and the version number of the MySQL Binlog, while empowering rights only to store the current user permissions latest data, historical data and operating rights records stored in a manner Elasticsearch version number. Only you need to query MySQL permissions data to the user authentication, to ensure the efficient authentication interface.
14:
- When empowering operation, data management authority by the version number, the version number after each operation plus 1, MySQL and Redis stores only the most recent data permissions.
- When data is stored by the historical rights of way version numbers to Elasticsearch, each view historical operating records or data recovery authority, according to the version number back to.
Authority expired cleanup
- By scheduling the timing Crane, notification rules according to the configuration of the scanning permission data about to expire, the renewal of the user send a message notifying permission.
- Scan data has expired rights, privileges cleaning up expired data in MySQL and Redis, and dumped to save Elasticsearch has permission for subsequent audit.
Separate read and write data, a cache, and a backup service fuse degraded
Each service using the MySQL database storage points, use Zebra (US group database access middleware layer) to read and write separation; and the rational use of backup data cache, and support services for blown downgrade to ensure high availability services.
Figure 15:
- Each service using the MySQL database storage points; core and non-core services separate service, on-demand services and database support elastic expansion.
- Roles, resources, and other hot spots to use Redis as cache data, and automatically sink to the MySQL query cache is not available at the time of Redis.
- Operating records and historical data does not fall into the active data Elasticsearch, in order to audit and data recovery.
- Support blown downgrade when service is not available, to ensure the availability of core services.
Rational use of message queues, task scheduling, thread pool, distributed lock
Using message queues, task scheduling, asynchronous thread pool, clipping, decoupling, reduce service response time, enhance the user experience. And the use of distributed lock to ensure data consistency.
Shown in Figure 16:
- Using the message queue processing user requests, the real-time operation returns success, the background processing according to the received asynchronous message MQ and modified state, the polling state display page or sending an elephant final result (US group internal communication tool) push message final result.
- You need to sync tasks performed by the timing schedule Crane distributed task scheduling platform.
- When approving the use of callback thread pool callback approval results and failure retry, less the cost of creating destroying threads.
- Distributed Lock to ensure that the same method can be executed on a thread of a machine in the same operation, or to avoid multi-user resubmit repeats the processing machine due to inconsistent data.
Outlook
As a common data security platform, a variety of customized needs of individual business lines can not all be met. Plugin now supports multiple pluggable service providers on the system architecture to implement custom access control list on the basis of universal services. Will follow the general order for permission, approval, audits Plugin development specifications, system access support custom development on the existing basis.
As shown in FIG 17:
- Follow-up will provide a unified Foreign Plugin development specifications, support the various access systems in the form square Plugin custom development services on the basis of platform services to meet their special privileges control requirements. Data products in order to achieve centralized management and control authority to ensure data security.
- The general order of rule separated out from the existing service, a general abstract rules engine service rules for implementing flexible and configurable.
END
Articles found an error, have questions about the content, give me a message oh ~
Eggs small welfare
Click on free access to Java study notes, interviews, documents, and video