Disconf - Distributed configuration management platform from Baidu

Summary

In order to better solve the problem of unified configuration management of multiple service instances in a distributed environment, this paper proposes a complete distributed configuration management solution (referred to as disconf[4], the same below). First of all, it realizes the unification of the configuration release of the homogeneous system, and provides a configuration service server, which can persistently manage the configuration and provide a restful interface to the outside world. On this basis, real-time push of configuration changes is realized based on zookeeper, and , provides a stable and effective disaster recovery solution, as well as a programming model and WEB user management interface with good user experience. Secondly, the configuration package management of heterogeneous systems is realized, and a global distributed consistency lock based on zookeeper is proposed to realize the unified deployment of active and standby systems and the automatic switching of active and standby systems when the system is abnormal. The practical results in Baidu's internal and external product lines show that this solution is effective and stable.

technical background

In a distributed environment, many instances of the same type of service are often deployed. These instances use some configuration, and in order to better maintain these configurations, the configuration management service is generated. Configuration issues for thousands or hundreds of service instances can be easily managed through this service.

Wang Ajing proposed the design and implementation of a configuration information storage scheme based on zooKeeper [1], which stores all configurations on zookeeper, which makes configuration management inconvenient, and they do not have relevant source code implementations. Taobao's diamond[2] is a system used by Taobao to manage persistent configuration. It has a complete open source source code implementation. It is characterized by simplicity, reliability, and ease of use. Most of Taobao's internal system configurations use diamond. Unified management. He stores the configuration in all configuration files in a scattered manner, only supports the KV structure, and the push of configuration updates is not real-time. Baidu's internal BJF configuration center service [3] adopts the implementation similar to Taobao diamond, which is also a decentralized configuration, only supports KV and non-real-time push.

Homogeneous systems are the mainstream of the market. In particular, in the case that deployment virtualization (such as JPAAS system, SAE, and BAE) is widely used in the industry, there will be more and more scenarios in which the same system uses the same deployment package. However, heterogeneous systems also have certain significance. For example, for multiple downstream instances in the "pull mode", only one downstream instance can be running at the same time. In this scenario, there is a problem that multiple instance machines have a "main-standby" mode. At present, there is no obvious domestic solution to solve this problem uniformly.

Functional features and design concept

Disconf is a complete unified solution for distributed configuration based on zookeeper.

Its functional characteristics are

Support distributed management of configuration (configuration item + configuration file)
- Configuration release unification
- Configuration release and update unification (cloud storage, release): The configuration is stored in the cloud system, and users publish and update the configuration on the platform in a unified manner.
- Configuration update automation: The user updates the configuration on the platform, and the system using the configuration will automatically discover the situation and apply the new configuration. In particular, if the user defines a callback function class for this configuration, this function class will be called automatically.
Configuring Heterogeneous System Management
- Unified deployment of heterogeneous packages: The heterogeneous system here refers to the situation where multiple deployment packages (jar or war) are required due to different configurations when a system deploys multiple instances (the same below). After using Disconf, only one deployment package is required for the deployment of heterogeneous systems, and the configurations of different instances are automatically allocated. In particular, in the case of a large number of deployment virtualization (such as JPAAS system, SAE, BAE) used in the industry, there will be more and more scenarios where the same system uses the same deployment package, and Disconf can naturally fit with him. Heterogeneous active/standby automatic switchover: If a heterogeneous system has active/standby machines, when the main machine hangs up, the standby machine can automatically obtain the main machine configuration and become the main machine.
- Heterogeneous active/standby machine Context sharing tool: In a heterogeneous system, the active/standby machine may need to share the context when switching. You can use the Context sharing tool to share the active and standby contexts.
Annotated programming, minimalist usage: we pursue a minimalist programming method with good user programming experience. Through simple annotation + very simple code writing, complex configuration distribution can be completed.
Requires Spring programming environment

Its design philosophy is:

Simple and good user experience:
- 摒弃了打散化配置的管理方式[2,3]，仍旧采用基于配置文件的编程方式，这和程序员以前的编程习惯（配置都是放在配置文件里）一致。特别的，为了支持较为小众的打散化配置功能，还特别支持了配置项。
- 采用了基于XML无代码侵入编程方式：只需要几行XML配置，即可实现配置文件发布更新统一化、自动化。
- 采用了基于注解式的弱代码侵入编程方式：通过编程规范，一个配置文件一个配置类，代码结构简单易懂。XML几乎没有任何更改，与原springXML配置一样。真正编程时，几乎感觉不到配置已经分布式化
可以托管任何类型的配置文件，这与[2,3]只能支持KV结构的功能有较大的改进。
配置更新实时推送
提供界面良好Web管理功能，可以非常方便的查看配置被哪些实例使用了。

详细设计

架构设计

disconf服务集群模式：

disconf的模块架构图：

每个模块的简单介绍如下：

Disconf-core
- 分布式通知模块：支持配置更新的实时化通知
- 路径管理模块：统一管理内部配置路径URL
Disconf-client
- 配置仓库容器模块：统一管理用户实例中本地配置文件和配置项的内存数据存储
- 配置reload模块：监控本地配置文件的变动，并自动reload到指定bean
- 扫描模块：支持扫描所有disconf注解的类和域
- 下载模块：restful风格的下载配置文件和配置项
- watch模块：监控远程配置文件和配置项的变化
- 主备分配模块：主备竞争结束后，统一管理主备分配与主备监控控制
- 主备竞争模块：支持分布式环境下的主备竞争
Disconf-web
- 配置存储模块：管理所有配置的存储和读取
- 配置管理模块：支持配置的上传、下载、更新
- 通知模块：当配置更新后，实时通知使用这些配置的所有实例
- 配置自检监控模块：自动定时校验实例本地配置与中心配置是否一致
- 权限控制：web的简单权限控制
Disconf-tools
- context共享模块：提供多实例间context的共享。

流程设计

运行流程详细介绍：

与2.0版本的主要区别是支持了：主备分配功能/主备切换事件。

启动事件A：以下按顺序发生。
- A3：扫描静态注解类数据，并注入到配置仓库里。
- A4+A2：根据仓库里的配置文件、配置项，去 disconf-web 平台里下载配置数据。这里会有主备竞争
- A5：将下载得到的配置数据值注入到仓库里。
- A6：根据仓库里的配置文件、配置项，去ZK上监控结点。
- A7+A2：根据XML配置定义，到 disconf-web 平台里下载配置文件，放在仓库里，并监控ZK结点。这里会有主备竞争。
- A8：A1-A6均是处理静态类数据。A7是处理动态类数据，包括：实例化配置的回调函数类；将配置的值注入到配置实体里。
更新配置事件B：以下按顺序发生。
- B1：管理员在 Disconf-web 平台上更新配置。
- B2：Disconf-web 平台发送配置更新消息给ZK指定的结点。
- B3：ZK通知 Disconf-cient 模块。
- B4：与A4一样。
- B5：与A5一样。
- B6：基本与A4一样，唯一的区别是，这里还会将配置的新值注入到配置实体里。
主备机切换事件C：以下按顺序发生。
- C1：发生主机挂机事件。
- C2：ZK通知所有被影响到的备机。
- C4：与A2一样。
- C5：与A4一样。
- C6：与A5一样。
- C7：与A6一样。

模块实现

disconf-web提供了前后端分离的web架构，具体可见：https://github.com/knightliao/disconf/tree/master/disconf-web

本部分会重点介绍disconf-client的实现方式。

注解式disconf实现

本实现会涉及到配置仓库容器模块、扫描模块、下载模块、watch模块，

使用AOP拦截的一个好处是可以比较轻松的实现配置控制，比如并发环境下的配置统一生效。关于这方面的讨论可以见这里。

In particular, the programming mode provided by this method is very simple. For example, when a program using the following configuration class uses it, it can be called directly by @Autowired. When using it, it is the same as using ordinary JavaBeans, but in fact it has been distributed melted. When the configuration is updated, the configuration class is automatically updated as well.

@Service
@DisconfFile(filename = &#34;redis.properties&#34;)
public class JedisConfig {

    // 代表连接地址
    private String host;

    // 代表连接port
    private int port;

    /**
     * 地址, 分布式文件配置
     * 
     * @return
     */
    @DisconfFileItem(name = &#34;redis.host&#34;, associateField = &#34;host&#34;)
    public String getHost() {
        return host;
    }

    public void setHost(String host) {
        this.host = host;
    }

    /**
     * 端口, 分布式文件配置
     * 
     * @return
     */
    @DisconfFileItem(name = &#34;redis.port&#34;, associateField = &#34;port&#34;)
    public int getPort() {
        return port;
    }

    public void setPort(int port) {
        this.port = port;
    }
}

XML-based configuration of disconf implementation

This implementation provides distributed configuration without any code intrusion.

The ReloadablePropertiesFactoryBean inherits the PropertiesFactoryBean class of the Spring Properties file and manages all configuration files to be reloaded when the configuration is updated. For each configuration file that is managed, it will be configured and obtained through the configuration repository container module, scan module, download module, and watch module to the configuration repository.

ReloadingPropertyPlaceholderConfigurer inherits the Spring Bean configuration value control class PropertyPlaceholderConfigurer. In the first scan of spring beans, disconf records which beans the configuration file's configuration is associated with.

ReloadConfigurationMonitor is a timed task that regularly checks whether the local configuration file is updated.

When the configuration of the configuration center is updated, the configuration file will be downloaded to the local instance, and the ReloadConfigurationMonitor will monitor this behavior and notify the ReloadingPropertyPlaceholderConfigurer to update the value of the related bean class.

In particular, this method cannot solve the problem that the configuration takes effect uniformly under concurrent conditions.

Active and standby allocation implementation

In the implementation, the concept of primary and secondary selection is provided for each configuration. User instances need to compete for global uniqueness before obtaining configuration values. Here, we use zookeeper-based global unique lock to achieve.

Comparisons

Taobao Diamond[2] Disconf Comparison

data persistence	stored on mysql	stored on mysql	Both are persisted to the database and are easy to manage
Push-pull model	Pull the model, pull the full data every 15s	Push model based on Zookeeper, real-time push	disconf基于分布式的Zookeeper来实时推送，在稳定性、实效性、易用性上均优于diamond
配置读写	支持实例对配置读写。支持某台实例写配置数据，并广播到其它实例上	只支持实例对配置读。通过在disconf-web上更新配置到达到广播写到所有应用实例	从目前的应用场景来看，实例对配置的写需求不是那么明显。disconf支持的中心化广播方案可能会与人性思考更加相似。
容灾	多级容灾模式，配置数据会dump在本地，避免中心服务挂机时无法使用	多级容灾模式，优先读取本地配置文件。	双方均支持在中心服务挂机时配置实例仍然可以使用
配置数据模型	只支持KV结构的数据，非配置文件模式	支持传统的配置文件模式（配置文件），亦支持KV结构数据(配置项)	使用配置文件的编程方式可能与程序员的编程习惯更为相似，更易于接受和使用。
编程模型	需要将配置文件拆成多个配置项，没有明显的编程模型	在使用配置文件的基础上，提供了注解式和基于XML的两种编程模型	无
并发性	多条配置要同时生效时，无法解决并发同时生效的问题	基于注解式的配置，可以解决并发性问题	无

</table>

http://www.cnblogs.com/duanxz/p/3568514.html