Implementation of a configurable data masking and anti-masking framework

1.tony frame background

In the context of increasing business volume, a large amount of data is generated in various business activities. Data security control has always been an important part of governance, and data desensitization belongs to the category of security control. For Internet companies, data security has always been an extremely important and sensitive topic. Data desensitization refers to the transformation of certain sensitive information through desensitization rules to achieve reliable protection of sensitive private data. Personal information involving customer security data or some commercially sensitive data, such as ID number, mobile phone number, card number, customer number, etc., needs to be desensitized according to the regulations of relevant departments.
In the context of the existing microservice technology architecture, the use of sensitive data has the following pain points:
- Sensitive data interacts with PC and APP clients using plaintext data, and data security cannot be guaranteed.
- Sensitive data browsing lacks monitoring, and data governance cannot be targeted.
- The microservice architecture system has many and complex applications, and it is expected to introduce a general solution to solve sensitive data security issues in a unified manner.

Based on the above pain points, tony provides a complete, safe, transparent, and low transformation cost data desensitization integration solution.

2. Introduction to the tony framework

Tony is a sensitive data desensitization and browsing record monitoring tool, which can desensitize sensitive information in the system, provide traceability evidence in case of leakage, and provide security protection measures for enterprise data sharing, migration and distribution. The project is developed based on spring and provides the spring-boot-starter startup package. The access is simple, and the main functions are as follows:

Flexible desensitization program

The data of the business system is ever-changing. In order to ensure that all business data can be processed normally for sensitive information, data masking provides highly flexible custom masking configuration. Through the definition of rich and diverse desensitization rules, sensitive data processing can realize desensitization processing that is highly suitable for the business.

Unified plaintext data management

The framework has a built-in plaintext data management process corresponding to sensitive data, which caches and manages plaintext data in a unified manner, and provides a unified plaintext data access API.

Browsing history leaves traces

According to the browsing actions of different users on the sensitive information plaintext data, the access party can customize the log format template to record and output to the log file, so that the browsing of each piece of data can be traced.

Abnormal page view warning

The access party can set the threshold of sensitive browsing plaintext data volume in different scenarios by customizing the desensitization configuration. After the alarm is enabled, when the number of times the user accesses the plaintext data reaches the threshold, the early warning process will be triggered, and the access party can customize the early warning logic.

Tony's overall structure:

The tony-core module defines the implementation of the basic desensitization function, and the overall process uses the spi mechanism to reserve sufficient room for expansion. Based on tony-core, the existing car technology stack is integrated and the tony-transformers module is provided to expand the basic desensitization function: asynchronous cache event distribution, distributed plaintext data cache, and docking with the sensitive log traces of the security department of the group , Email alerts for abnormal page views.

tony architecture diagram.png

name	desc
MaskAnnotationProcessor	Business scenario proxy, by annotating the desensitization scenario method corresponding to the proxy.
IMaskProcess	Data Desensitization Processor
IMaskLogProcessor	Data Browsing Log Processor
IMaskCacheProcessor	Plaintext Data Cache Processor
IMaskLimitProcessor	Clear text data access quota management
IMaskWarningProcessor	Abnormal browsing warning processing
IMaskContent	Context Information Processor
IUnMaskProcessor	Plaintext Data Acquisition Processor
DefaultUnMaskProcessorAdapter	The plaintext data acquisition adapter provides the function of obtaining plaintext data from the cache.
DefaultMaskCacheProcess	Plaintext data cache processor, providing distributed cache function.
DefaultMaskContent	Context information processor, providing ThreadLocal implementation.
DefaultMaskWarningProcessor	Abnormal browsing warning processing, providing email warning implementation.
PlaintextProcess	Plaintext belongs to the acquisition interface constraints, and the extension of plaintext data acquisition can be customized.
MaskMonitorCacheHandle	Cache time consumption, asynchronous consumption of cache events, and storage of plaintext data.

Detailed explanation of tony's processing flow:

Tony and the business code are deployed together, and the access application implements the desensitization agent corresponding to the business scenario through the specification of desensitization scenarios and desensitization configuration items. Business codes can be used compatible only by configuring scene annotations. At this time, the desensitization of sensitive data is in charge of Tony, who will intercept the response data of the business scenario and desensitize the sensitive information.

Tony process detailed explanation.png

Detailed configuration of tony desensitization:

The desensitization configuration is mainly divided into four parts: desensitization context configuration, desensitization scene configuration, desensitization warning configuration, and log trace template. The details are shown in the figure below:

Introduction to tony desensitization configuration.png

key	name	desc
tony.appName	access system name
tony.maskSourceName	Desensitization Whitelist	The interface that controls the unified scene controls whether to desensitize data for different callers
tony.mask.XXX	desensitized scene	XXX is the key of the specific scene
tony.mask.XXX.enable	dynamic switch	Control whether desensitization is enabled
tony.mask.XXX.limits	Clear text browsing limit	Limit the number of plaintext views
tony.mask.XXX.cacheKey	Data cache parsing rules	Example: STATEMENT_BILL_PAGE:${ argItem.id }
tony.types.YYY	Desensitized field	YYY is the key of a specific field
tony.types.YYY.regex	Desensitization rule configuration	Desensitized regular expression example: (\d{3})\d{4}(\d{4})
tony.types.YYY.replacement	Desensitization rule configuration	Desensitized regular expression example: $1****$2
tony.types.YYY.logName	log field name

3. Detailed system design

3.1 Detailed design

3.1.1 Activity Diagram

Untitled Drawing - Page 14(1).png

3.1.2 Timing diagram

Untitled Drawing - Page 13(2).png

3.1.3 Class Diagram

Tony frame.png

Bag	responsibility
com.jd.car.tony.annoation	The annotation layer is responsible for declaring the business operations to be proxied.
com.jd.car.tony.limit	Responsible for recording the number of accesses to sensitive data and providing early warnings.
com.jd.car.tony.mask	Manage basic configurations, and mask and desensitize sensitive data based on configurations.
com.jd.car.tony.log	Record access operations of sensitive data and output log information in a unified format.
com.jd.car.tony.unmask	Provides the function of viewing plaintext information of sensitive data.
com.jd.car.tony.support	The util package is responsible for the support of global functions.

4. Access and use

4.1 Import coordinates

			<dependency>
                <groupId>com.jd.car</groupId>
                <artifactId>tony-spring-boot-starter</artifactId>
                <version>1.8-RELEASE</version>
            </dependency>

4.2 yaml configuration

tony:
  appName: "appName"
  #鉴权信息key
  sessionName: ""
  systemName: "systemName"
  warningName: "XXX"
  #脱敏白名单
  maskSourceName: ""
  types:
    name:
      regex: "([\\u4e00-\\u9fa5a-z0-9]{1})[\\u4e00-\\u9fa5a-z0-9]+"
      replacement: "$1**"
      logName: "accountName"
  mask:
    order:
      enable: true
      limits: 100
      limitError: false
      cacheKey: "order:${argItem.id}"
      menuName: "场景名称"
      name: "order"
      #日志记录操作类型
      maskLogOp: ""
      maskLogAccountType: 3
      
   #预警邮箱设置
   mail:
    mailHost: 
    mailPort: 
    mailUser: 
    mailPwd: 
    mailFrom: 
    mailTo: 
    copyto: 
    sendFlag: 
    
 #异步事件缓存明文数据配置，分布式缓存
 jd:
  cache:
    jimdb:
      enable: false
      url: ''
  event:
    enable: true
    queue:
      # 自定义queue名字，例如monitorQueue
      maskMonitorQueue:
        retryCount: 3
        maxBakSize: 1000
        # monitorHandle 处理事件的beanName
        handlerBean: maskMonitorCacheHandle

4.3 Log file configuration

<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds" debug="false">

    <contextName>demo</contextName>
    <!-- 日志最大的历史 单位：天 -->
    <property name="maxHistory" value="90"/>

    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS,GMT+8:00} -%5p ${PID:- } --- [%X{PFTID}][%t] %-40.40logger{39}[%L] : %m%n"/>

    <property name="LOG_CHARSET" value="UTF-8"/>

    <property name="LOG_DES_PATTERN" value="%msg%n" />

    <!-- 彩色日志依赖的渲染类 -->
    <conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter"/>
    <conversionRule conversionWord="wex"
                    converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter"/>
    <conversionRule conversionWord="wEx"
                    converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/>

    <property name="LOG_PATTERN"
              value="%d{yyyy-MM-dd HH:mm:ss.SSS,GMT+8:00} -%5p ${PID:- } --- [%X{PFTID}][%t] %-40.40logger{39}[%L] : %m%n"/>

    <!-- 彩色日志格式 -->
    <property name="CONSOLE_LOG_PATTERN"
              value="${CONSOLE_LOG_PATTERN:-%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%X{PFTID}][%15.15t]){faint} %clr(%-40.40logger{39}[%L]){cyan} %clr(:){faint} %m%n}"/>


    <appender name="mask" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${log.path}/today_log/mask.log</file>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>INFO</level>
        </filter>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}/history_logs/mask-%d{yyyy-MM-dd}.log</fileNamePattern>
            <maxHistory>30</maxHistory>
        </rollingPolicy>
        <encoder>
            <charset>${LOG_CHARSET}</charset>
            <pattern>${LOG_DES_PATTERN}</pattern>
        </encoder>
    </appender>
    <logger name="com.jd.car.tony.log.DefaultMaskLogProcessor" additivity="false">
        <level value="INFO" />
        <appender-ref ref="mask" />
    </logger>

</configuration>

4.3 Scene Access

 /**
     * 测试脱敏
     *
     * @return
     */
    @RequestMapping("test")
    @MaskMethod("order")
    public DemoResult test(AuthInfoBO authInfo) {
        return new DemoResult();
    }

    /**
     * 测试脱敏
     *
     * @return
     */
    @RequestMapping("test-list")
    @MaskMethod("order")
    public List<DemoResult> testList(AuthInfoBO authInfo) {
        List<DemoResult> objects = Lists.newArrayList();
        objects.add(new DemoResult());
        objects.add(new DemoResult());
        return objects;
    }

 		@Autowired
    private IUnMaskProcessor unMaskProcessorProxy;

    /**
     * 测试反脱敏
     *
     * @return
     */
    @RequestMapping("unmask")
    public Map testUnMask(@RequestBody UnMaskRequest unMaskRequest, AuthInfoBO authInfo) {
        return unMaskProcessorProxy.unMask(unMaskRequest, authInfo);
    }

4.3 Desensitization field configuration

@Data
public class DemoResult {

    @Mask(type = "name")
    private String userName = "jajajaasjcij";

    @Mask(type = "phone")
    private String userTel = "18911112222";

    private Long orderId = 1L;

}

4.4 Business Monitoring

business monitoring point	trigger logic
View sensitive data in clear text	When the number of single-account, single-day, and single-interface visits reaches the access limit, an email will be sent to the rights management personnel.
Sensitive data query log	After the sensitive data is desensitized, when the plaintext data is viewed, it is recorded separately in the log file according to the format required by the group.

Author: JD Retail Qiu Xinda

Source: JD Cloud Developer Community