[Archaius Technology Topic] "Netflix Original Ecology" Dynamic Configuration Service Microservice Configuration Component Chameleon

Prerequisites

If you want to design and develop a microservice infrastructure, parameterized configuration is a very important point, and Netflix has also open sourced a configuration center client called Chameleon Archaius, and Archaius can be said to have more production levels than other clients. features and are more flexible.
* In the NetflixOSS microservice technology stack, almost all other components (such as Zuul, Hystrix, Eureka, Ribbon, etc.) depend on Archaius. It can be said that understanding Archaius is the basis for understanding and using other Netflix microservice components.

What is Archaius

Netflix Archaius is a configuration management library with a focus on dynamic properties from multiple configuration stores. It includes a set of Java configuration management APIs for Netflix. It is primarily implemented as an extension to the Apache Commons Configuration library. The main functions provided are:

Note that Netflix only open sourced the client part of its configuration center (that is, Archaius), and there is no open source server side. Archaius is actually independent of the implementation of the configuration source, and can be connected to various configuration centers as data sources. Later in this article, we will introduce how Archaius integrates with the Apollo configuration center.

The origin of the Archaius project

In a microservice environment, configuration often needs to be adjusted according to different contexts, or the configuration should be multi-dimensional. For example at Netflix, the contextual dimension includes environments (dev, test, and production).
Netflix hopes to be able to dynamically adjust service configurations according to the publishing environment and even the request context, so that the behavior and logic of Netflix's entire system can be dynamically deployed to adapt to the rapidly changing needs of Internet applications. To this end, the Netflix platform team has developed a configuration center product, which the team vividly calls the chameleon Archaius, because the chameleon is an animal that can dynamically adjust the color of its body according to its environment.

Archaius use case scenario at Netflix

Turn a feature on or off based on the request context.
A page displays 10 products by default. In some cases, the configuration can be adjusted through Archaius to only display 5 products.
Dynamically adjust the behavior of Hystrix circuit breakers.
Adjust the connection and request timeout parameters of the service call client.
If an online service generates an error alarm, you can dynamically adjust the log output level (the granularity can be as fine as the package or component level), so that you can troubleshoot problems through detailed logs. After the problem is located, restore the log output level to the default level.
For applications deployed in multiple regions or countries, through dynamic configuration, different functions can be enabled according to different regions and countries.
Some basic middleware configurations can be dynamically adjusted according to the user's actual access patterns, such as the TTL (Time To Live) of the cache.
The connection pool configuration of the database access client can be configured with different values for different services. For example, a service with a small request frequency RPS (Request Per Second) can be configured with a smaller number of connections, while a service with a high request frequency can be configured with a larger number of connections.
Runtime configuration changes can take effect in different dimensions, such as the dimension of a single instance in a cluster, the dimension of a region under multi-region deployment, the dimension of a service stack, or the dimension of an application cluster.
The Feature Flag is released. Although some features are online, they are not enabled immediately, but are enabled dynamically through configuration switches, so that an online feature can be flexibly enabled or disabled according to the situation.
Canary Release (Canary Release), when the new function is launched, let the old and new clusters coexist for a period of time, and gradually adjust the traffic to the old cluster to the new cluster through configuration. If the monitoring shows no abnormality, the new cluster will be launched , if abnormal, quickly switch back to the old cluster.

Archaius' technical foundation

archaius is netflix's open source dynamic attribute configuration framework, based on apache commons configuration, providing the function of obtaining configuration values at runtime.
At the heart of Archaius is the concept of composite configurations that can hold one or more configurations. Each configuration can be obtained from configuration sources such as JDBC, REST interface, xxx.properties file, etc. The configuration source can optionally be polled at runtime for dynamic changes,
The final value of a property depends on the topmost configuration that contains the property (since it is a composite configuration). That is, if a property exists in more than one configuration, the actual value seen by the application will be the value in the topmost slot in the configuration hierarchy, which of course is configurable.

Architecture design of Archaius

Archaius is actually an encapsulation and extension of Apache Common Configuration Library, providing a set of Java-based configuration APIs. The main features include:

Configuration can be dynamically adjusted: dynamic, type attributes
Configuration support types (Int, Long, Boolean, etc.).
High-performance and thread-safe: high-throughput and thread-safe configuration operations
Provides a framework for pulling configurations, which can dynamically pull changed configurations from configuration sources. (a polling framework that allows users to get property changes to configuration sources)
Support callback (callback) mechanism, which is called automatically when the configuration changes.
Support JMX MBean, you can view configuration and modify configuration through JConsole.

For applications (and most web applications) that are willing to use a convention-based property file location, an out-of-the-box composite configuration is provided (this is one of the powerful features), and an example diagram is given on the official website for conforming configurations as follows:

The core of Achaius is a concept called **Composite Configuration**, which can be simply understood as a hierarchical configuration with priorities. High priority level configurations will override low priority configurations. . Each level can obtain configuration from a configuration source, such as local configuration file, JDBC data source, remote REST API, etc. The configuration source can also dynamically pull changes at runtime. For example, in the above figure, the persistent database configuration (Persisted DB Configuration) refers to storing the configuration in a relational database, and the corresponding configuration source will periodically pull changes from the database). The final value of the configuration is determined by the top-level configuration. For example, if multiple levels contain a certain configuration item, the final value seen by the application is the topmost value in the configuration level. The order of configuration layers can be adjusted.

There are two ways to obtain configuration values through archaius:

One is to obtain the configuration center instance through the ConfigurationManager, then obtain the configuration value through the propName, and obtain the configuration through the ConfigurationManager
Another way is to obtain the DynamicProperty wrapper of the configuration item through DynamicPropertyFactory.

Implementation principle of Archaius

What is Archaius?

Archaius provides the function of dynamically modifying the configuration value. After modifying the configuration, there is no need to restart the application service. Its core idea is to poll the configuration source, each iteration, detect whether the configuration has changed, and re-update the configuration if there is a change.

The underlying archaius provides a concrete implementation of AbstractConfiguration that implements Apache-common-configuration.

ConcurrentMapConfiguration provides the function of maintaining configuration item configuration values in ConcurrentHashMap.
DynamicConfiguration provides the function of dynamically obtaining all configuration values from the data source, and updates the configuration values by polling the data source.
DynamicWatchedConfiguration also provides the function of dynamically updating the configuration. Unlike DynamicConfiguration, the configuration update is triggered when the data source changes.
DynamicConfiguration is a pull method, and DynamicWatchedConfiguration is a push method.

ConcurrentCompositeConfiguration uses a combination mode to combine different AbstractConfiguration implementations. For configuration centers with multiple configuration sources, ConcurrentCompositeConfiguration can be used. For the same configuration item, when multiple configuration sources have configuration values, the data of the first matching configuration source is taken.

How does DynamicPropertyFactory work?

DynamicPropertyFactory holds AbstractConfiguration instance.
When creating a DynamicProperty object, the DynamicProperty object will obtain the AbstractConfiguration instance held by the DynamicPropertyFactory.
The DynamicProperty object will register a DynamicPropertyListener with the AbstractConfiguration instance, and when the AbstractConfiguration has additions, deletions, modifications, and changes, it will notify the current DynamicProperty object.

When the number of DynamicProperty instances created is relatively large, there may be performance problems here. Every time any DynamicProperty is created, a listener will be added. At the same time, any configuration item changes will trigger the listener.

It may be considering that there will not be so many configuration item changes in the production environment. It is better to trigger the watcher only when the corresponding configuration item changes like zk-config.

How is AbstractConfiguration injected?

archaius only allows one AbstractConfiguration implementation class. If there are multiple configuration sources, different AbstractConfigurations can be combined using the ConcurrentCompositeConfiguration mentioned above.

There are several ways to inject AbstractConfiguration.

Configure archaius.default.configuration.class to specify the AbstractConfiguration implementation class
Configure archaius.default.configuration.factory to specify the AbstractConfiguration instance factory method class, and the factory method class needs to implement the getInstance method to return the AbstractConfiguration instance
Call the install(AbstractConfiguration config) method of ConfigurationManager
Call the initWithConfigurationSource(AbstractConfiguration config) method of DynamicPropertyFactory

The above methods are mutually exclusive, and only one of them can be used. After one takes effect, the others cannot be called.

A simple example:

Maven dependency configuration

<dependency>
    <groupId>com.netflix.archaiusgroupId>
    <artifactId>archaius-coreartifactId>
    <version>0.7.7version>
dependency>
复制代码

Get configuration source


public class DynamicConfigurationSource implements PolledConfigurationSource {
    
    

    public PollResult poll(boolean initial,Object checkPoint) throws Exception {
    
    
        Map map = new HashMap<>();
        map.put("test",UUID.randomUUID().toString());
        return PollResult.createFull(map);
    }
}
复制代码

Define the scheduler

AbstractPollingScheduler scheduler = new FixedDelayPollingScheduler(2000,2000,false);
复制代码

Define dynamic configuration

DynamicConfiguration configuration = new DynamicConfiguration(source,scheduler);
复制代码

simple unit test

.testng.annotations.Test
    public void testArchaius() throws Exception {
    
    
        PolledConfigurationSource source = new DynamicConfigurationSource();
        AbstractPollingScheduler scheduler = new FixedDelayPollingScheduler(2000,2000,false);
        DynamicConfiguration configuration = new DynamicConfiguration(source,scheduler);
        ConfigurationManager.install(configuration);
        final DynamicStringProperty stringProperty = DynamicPropertyFactory.getInstance().getStringProperty("test","nodata");
        Helpers.subscribePrint(Observable.interval(1,TimeUnit.SECONDS).take(20).doOnNext(new Action1() {
    
    

                    public void call(Long aLong) {
    
    
                        System.out.println(stringProperty.get());
                    }
                }),"test");
        TimeUnit.MINUTES.sleep(1);
    }
复制代码

accomplish

Start polling task

public synchronized void startPolling(PolledConfigurationSource source, AbstractPollingScheduler scheduler) {
    
    
    this.scheduler = scheduler;
    this.source = source;
    init(source, scheduler);
    scheduler.startPolling(source, this);
}
复制代码

Runnable and initialization for polling: the implementation is consistent

PollResult result = null;
    try {
    
    
       result = source.poll(false,getNextCheckPoint(checkPoint));
       checkPoint = result.getCheckPoint();
       fireEvent(EventType.POLL_SUCCESS, result, null);
       } catch (Throwable e) {
    
    
       log.error("Error getting result from polling source", e);
       fireEvent(EventType.POLL_FAILURE, null, e);
       return;
       }
       try {
    
    
          populateProperties(result, config);
       } catch (Throwable e) {
    
    
           log.error("Error occured applying properties", e);
      }
复制代码

Note that the source.poll method will be called, that is, polled of PolledConfigurationSource, the data source interface we implement, and you can customize the data source (jdbc, file, scm, etc.)

Summarize

In the process of in-depth understanding of Archaius, there is an unavoidable "obstacle" that is Apache Commons Configuration, because the former strongly relies on the latter for configuration management. As the saying goes, how much you know about Apache Commons Configuration determines how deeply you know Netflix Archaius.

share resources

Information sharing
To obtain the above resources, please visit the open source project and click to jump