Dubbo series (06-1) fault-tolerant cluster - Services dictionary

[Heel]

Spring Cloud Alibaba Series catalog - Dubbo articles

1. Background

This article will begin to analyze the source code of Dubbo cluster fault tolerance. Source fault-tolerant cluster consists of four parts, which are service catalog Directory, service routing Router, clustering and load balancing Cluster LoadBalance. These four interfaces are dubbo-clusterdefined in the project.

1.1 Directory Interface

public interface Directory<T> extends Node {
    // 1. 获取 serviceInterface
    Class<T> getInterface();
    
	// 2. 获取指定 serviceInterface 对应的服务接口实例
    List<Invoker<T>> list(Invocation invocation) throws RpcException;
}

Summary: Directory only manage a single instance of the corresponding serviceInterface. Here there were two core concepts Dubbo domain model, Invokerand Invocationcan be acquired by its executable invokers list of parameters invocation of the session, and then initiate a remote call.

Invoker Is the entity domain, it is Dubbo's core model, other models are scrambling to rely on it, or convert it, it represents an executable body, can invoke call to initiate it, it is possible that a local implementation, it may be a remote implementation is also possible to achieve a cluster.
Invocation It is a session field, which holds the calling procedure variables, such as the method name and parameters.
Protocol Service domain, which is the main entrance features Invoker exposure and reference, which is responsible Invoker life cycle management.

1.2 inheritance system

Services catalog currently has two built-in implementations, respectively StaticDirectory and RegistryDirectory, all of which are a subclass of AbstractDirectory. AbstractDirectory Directory implements the interface. Here we look at their inheritance hierarchy diagram.

Figure 1 Dubbo service catalog inheritance hierarchy diagram

Summary: 服务目录 Directory all instances of a single load management service interface corresponding. It has two implementations:

StaticDirectory As the name suggests, serviceInterface corresponding service provider is static, that is, read the service list information from the configuration file.
RegistryDirectory Dynamic access to designated serviceInterface corresponding service providers from the registry.

Directory Interface design principles Analysis:

DirectoryThe core method List<Invoker<T>> list(Invocation invocation), which only focus on the core functions, invocation example invokers get executed by calling parameters. Directory interface itself only read, not write function.
AbstractDirectory It defines some common implementation, an increase of routing information and subscriber consumerUrl of routerChain.
StaticDirectory/RegistryDirectoryWith write function. StaticDirectory is passed through the constructor can not be dynamically updated. The RegistryDirectory achieve a further NotifyListener interface when registering service information corresponding to the interface will change the callback notify(URL url, NotifyListener listener, List<URL> urls)method to notify RegistryDirectory Invoker update the list, with a dynamic write functions.

In addition, Directory inherited from the Node interface, this interface more Node successor, like the Registry, Monitor, Invoker etc inherited this interface. This interface includes a method of acquiring configuration information getUrl, the interface implementation class may provide configuration information outwardly.

StaticDirectory # getUrl typically null, URL corresponding to the following exemplary RegistryDirectory. Honestly in my opinion is essentially Dubbo URL is a configuration class, a variety of configuration information will be converted into a URL, the URL leads to understanding some difficulties, sometimes we really do not know the URL in the end what does it mean.

## RegistryDirectory#getUrl() -> Nacos 注册中心地址
registry://192.168.139.101:8848/org.apache.dubbo.registry.RegistryService?application=dubbo-consumer&dubbo=2.0.2&pid=24924&qos.port=33333&refer=application%3Ddubbo-consumer%26check%3Dfalse%26dubbo%3D2.0.2%26interface%3Dorg.apache.dubbo.demo.DemoService%26lazy%3Dfalse%26methods%3DsayHello%26pid%3D24924%26qos.port%3D33333%26register.ip%3D192.168.139.1%26side%3Dconsumer%26sticky%3Dfalse%26timestamp%3D1570940706766&registry=nacos&timestamp=1570940706897

StaticDirectory very simple, is not introduced, the following describes RegistryDirectory.

2. Source analysis

RegistryDirectory mentioned above, there are also two functions, one is reading function, invocation get Invoker executables list based on session parameters; the second is to write features, updates the list when Invoker registry service changes.

2.1 Service get

Figure 2 Dubbo service acquisition process

sequenceDiagram participant RegistryDirectory participant AbstractDirectory participant RouterChain participant Router RegistryDirectory ->> AbstractDirectory : list AbstractDirectory ->> RegistryDirectory : doList RegistryDirectory ->> RouterChain : route loop routers RouterChain ->> Router : route end

Summary: RegistryDirectory # List entrusted to doList method to obtain the list of services, doList filtered through routing rules will return a list of available enforcement body. RouterChain which holds all of the invokers, when calling notify -> refreshOverrideAndInvoker -> refreshInvoker -> routerChain.setInvokers(newInvokers)when updated invokers routerChain held.

@Override
public List<Invoker<T>> doList(Invocation invocation) {
    if (forbidden) {
        // 1. No service provider 2. Service providers are disabled
		...
    }
  
    List<Invoker<T>> invokers = null;
    try {
        // getConsumerUrl 返回服务订阅者的URL
        invokers = routerChain.route(getConsumerUrl(), invocation);
    } catch (Throwable t) {
    }
    return invokers == null ? Collections.emptyList() : invokers;
}

2.2 Service Update

RegistryDirectory addition to implementing Directory Services Interface to obtain a list of information, but also to achieve the NotifyListener interface dynamically update the list of services. Service with respect to acquisition, service updates are much more complex.

RegistryDirectory hold Registry registry instance, you need to subscribe to the specified service consumer url, so that when the service changes will call notify notify RegistryDirectory update the list of services.

Figure 3 Dubbo Service Update Process

sequenceDiagram participant RegistryProtocol participant RegistryDirectory participant Registry RegistryProtocol ->> RegistryDirectory : subscribe RegistryDirectory ->> Registry : subscribe alt 更新服务列表 Registry ->> RegistryDirectory : notify RegistryDirectory ->> RegistryDirectory : refreshOverrideAndInvoker end

Summary: RegistryDirectory will first subscription consumerUrl, so that when the service will notify a change notification updates the list of services.

2.2.1 initialization RegistryDirectory

RegistryDirectory initialization in DubboRegistryFactory, RegistryProtocol # doRefer will have to create, the former is Dubbo's own registry, registry-based memory, in dubbo-registry-default project, which is to integrate other existing registry . In general, the introduction of registry-based services are created through RegistryProtocol # doRefer. The following analysis is also a source of RegistryProtocol analysis.

/**
 * RegistryProtocol：创建 type 的远程代理 @Reference
 * @param registry 	注册中心实例
 * @param type		服务接口类型
 * @param url		注册中心地址
 */
private <T> Invoker<T> doRefer(Cluster cluster, Registry registry, Class<T> type, URL url) {
    // 创建 RegistryDirectory 实例。type是订阅的服务接口类型，url是注册中心地址。
    RegistryDirectory<T> directory = new RegistryDirectory<T>(type, url);
    // 设置注册中心和协议
    directory.setRegistry(registry);
    directory.setProtocol(protocol);

    // 生成服务消费者链接
    Map<String, String> parameters = new HashMap<String, String>(directory.getUrl().getParameters());
    URL subscribeUrl = new URL(CONSUMER_PROTOCOL, parameters.remove(REGISTER_IP_KEY),
                               0, type.getName(), parameters);
    // 设置服务策略
    directory.buildRouterChain(subscribeUrl);

    // 订阅 providers、configurators、routers 等节点数据
    directory.subscribe(subscribeUrl.addParameter(CATEGORY_KEY,PROVIDERS_CATEGORY + "," +
		CONFIGURATORS_CATEGORY + "," + ROUTERS_CATEGORY));
    ...
}

Summary: RegistryDirectory construction process the most important tasks:

Set Registry instance: used to obtain a list of registry services.registry.subscribe(url, this)
Setting Protocol example: for URL generation type interface based on service address of the remote agent.protocol.refer(serviceType, url)
Set RouterChain Examples: for service routing.routerChain.route(getConsumerUrl(), invocation)
Finally subscription service: used to obtain the list of servicesdirectory.subscribe(url)

// url指的是注册中心地址，serviceType是服务接口的类型
public RegistryDirectory(Class<T> serviceType, URL url) {
    super(url);
    this.serviceType = serviceType;			//订阅的服务接口类型
    this.serviceKey = url.getServiceKey();	//{group/}serivceInterface{:version}
    this.queryMap = StringUtils.parseQueryString(url.getParameterAndDecoded(REFER_KEY));
    this.overrideDirectoryUrl = this.directoryUrl = turnRegistryUrlToConsumerUrl(url);
    String group = directoryUrl.getParameter(GROUP_KEY, "");
    this.multiGroup = group != null && (ANY_VALUE.equals(group) || group.contains(","));
}

Consideration: RegistryDirectory main attributes are set by the set method, there is an overlap constructor arguments.

2.2.2 subscription service

Calls subscribe subscribe to the service list after construction RegistryDirectory, returns a list of corresponding service url.serviceInterface.

public void subscribe(URL url) {
    setConsumerUrl(url);
    CONSUMER_CONFIGURATION_LISTENER.addNotifyListener(this);
    serviceConfigurationListener = new ReferenceConfigurationListener(this, url);
    registry.subscribe(url, this);
}

2.2.3 Service Update

After the subscription service notice RegistryDirectory update the local list of services, but also the most complex part of RegistryDirectory when a service update. Update list generated according to invokerUrls invokers obtained from the registry. If it does not exist to create a new Invoker, if it already exists is ignored.

@Override
public synchronized void notify(List<URL> urls) {
    // 按 category 分类存储，服务提供者url，路由url，外部化配置url
    Map<String, List<URL>> categoryUrls = urls.stream()
        .filter(Objects::nonNull)
        .filter(this::isValidCategory)
        .filter(this::isNotCompatibleFor26x)
        .collect(Collectors.groupingBy(url -> {
            if (UrlUtils.isConfigurator(url)) {
                return CONFIGURATORS_CATEGORY;
            } else if (UrlUtils.isRoute(url)) {
                return ROUTERS_CATEGORY;
            } else if (UrlUtils.isProvider(url)) {
                return PROVIDERS_CATEGORY;
            }
            return "";
        }));

    // configuratorURLs
    List<URL> configuratorURLs = categoryUrls.getOrDefault(CONFIGURATORS_CATEGORY, Collections.emptyList());
    this.configurators = Configurator.toConfigurators(configuratorURLs).orElse(this.configurators);

    // routerURLs
    List<URL> routerURLs = categoryUrls.getOrDefault(ROUTERS_CATEGORY, Collections.emptyList());
    toRouters(routerURLs).ifPresent(this::addRouters);

    // providerURLs
    List<URL> providerURLs = categoryUrls.getOrDefault(PROVIDERS_CATEGORY, Collections.emptyList());
    refreshOverrideAndInvoker(providerURLs);
}

Summary: While updating the list of services is more complex, but the logic of this agency is very clear.

The list of services corresponding subscription serviceInterface classified by category. providers, routers, configurators.
The configuratorURLs into Configurator. Configurator of the external configuration having a higher priority. Stored in the variable configurators in.
The routerURLs into Router. By routerChain.addRouters(routers)setting the variable in routerChain.
The providerURLs into Invoker. Stored in the variable invokers in.

The first three steps are very simple, the main logic refreshOverrideAndInvoker are delegated to the refreshInvoker method.

2.3 refresh the Invoker list

2.3.1 refreshInvoker

refreshInvoker generate an updated list based on invokerUrls invokers obtained from the registry. If it does not exist to create a new Invoker, if it already exists is ignored.

private void refreshInvoker(List<URL> invokerUrls) {
    Assert.notNull(invokerUrls, "invokerUrls should not be null");
    if (invokerUrls.size() == 1 && invokerUrls.get(0) != null
        && EMPTY_PROTOCOL.equals(invokerUrls.get(0).getProtocol())) {
        // 1. invokerUrls 仅有一个元素，且 url 协议头为 empty，此时表示禁用所有服务
        // 设置 forbidden 为 true
        this.forbidden = true; // Forbid to access
        this.invokers = Collections.emptyList();
        routerChain.setInvokers(this.invokers);
        // 销毁所有 Invoker
        destroyAllInvokers(); // Close all invokers
    } else {
        // 2. 有可用的url
        this.forbidden = false; // Allow to access
        // 2.1 urlInvokerMap保存上一次的invokers
        Map<String, Invoker<T>> oldUrlInvokerMap = this.urlInvokerMap; // local reference
        if (invokerUrls == Collections.<URL>emptyList()) {
            invokerUrls = new ArrayList<>();
        }
        // 2.2 cachedInvokerUrls保存上一次的invokerUrls
        if (invokerUrls.isEmpty() && this.cachedInvokerUrls != null) {
            // 添加缓存 url 到 invokerUrls 中
            invokerUrls.addAll(this.cachedInvokerUrls);
        } else {
            // 缓存 invokerUrls
            this.cachedInvokerUrls = new HashSet<>();
            this.cachedInvokerUrls.addAll(invokerUrls);
        }
        if (invokerUrls.isEmpty()) {
            return;
        }
        // 2.3 核心方法：将 url 转成 Invoker
        Map<String, Invoker<T>> newUrlInvokerMap = toInvokers(invokerUrls);

        // 2.4 转换出错，直接打印异常，并返回
        if (CollectionUtils.isEmptyMap(newUrlInvokerMap)) {
            return;
        }

        // 2.5 更新routerChain中的invokers列表
        List<Invoker<T>> newInvokers = Collections.unmodifiableList(
            new ArrayList<>(newUrlInvokerMap.values()));
        routerChain.setInvokers(newInvokers);
        // 合并多个组的 Invoker， <methodName, Invoker> 列表映射关系
        this.invokers = multiGroup ? toMergeInvokerList(newInvokers) : newInvokers;
        this.urlInvokerMap = newUrlInvokerMap;

        try {
            // 2.6 销毁下线服务的 Invoker
            destroyUnusedInvokers(oldUrlInvokerMap, newUrlInvokerMap);
        } catch (Exception e) {
            logger.warn("destroyUnusedInvokers error. ", e);
        }
    }
}

Summary: refreshInvoker involves several collections, simple to explain: urlInvokerMapa list of services on the cache; cachedInvokerUrlsa cache on the URL.

invokerUrlsOnly one empty when the service agreement, indicating that at this time need to disable the service, to destroy all services upon return. At this time, forbidden = false.
Cache URL to cachedInvokerUrls set, when the registry returned service address list is empty, the service address directly the last cache.
The core of the method toInvokers, converting invokerUrls to Invoker.
Finally, there is updated routerChain, destroyed off the assembly line Invoker other clean-up work.

2.3.2 toInvokers

Convert providerUrls to Invoker object, the object is a return <URL#toFullString()，Invoker>of the Map. Wherein the core code is protocol.refer(serviceType, url)generated according to the object Invoker url. In addition, URL url=mergeUrl(providerUrl)we should also be concerned about, Principal externalize configuration.

private Map<String, Invoker<T>> toInvokers(List<URL> urls) {
    Map<String, Invoker<T>> newUrlInvokerMap = new HashMap<>();
    if (urls == null || urls.isEmpty()) {
        return newUrlInvokerMap;
    }
    Set<String> keys = new HashSet<>();
    // 获取消费端配置的协议
    String queryProtocols = this.queryMap.get(PROTOCOL_KEY);
    for (URL providerUrl : urls) {
        // 1.1 协议匹配，queryProtocols是消费者可接收的协议类型，可有多个，
        //    providerUrl.getProtocol()是服务者提供的协议类型
        if (queryProtocols != null && queryProtocols.length() > 0) {
            boolean accept = false;
            String[] acceptProtocols = queryProtocols.split(",");
            for (String acceptProtocol : acceptProtocols) {
                if (providerUrl.getProtocol().equals(acceptProtocol)) {
                    accept = true;
                    break;
                }
            }
            // providerUrl协议无法匹配，直接过滤掉
            if (!accept) {
                continue;
            }
        }
        // 1.2 empty 协议，也直接过滤
        if (EMPTY_PROTOCOL.equals(providerUrl.getProtocol())) {
            continue;
        }

        // 1.3 providerUrl.getProtocol() 不存在，也直接过滤
        if (!ExtensionLoader.getExtensionLoader(Protocol.class)
            	.hasExtension(providerUrl.getProtocol())) {
            continue;
        }
        
        // 2. 合并 url，参数配置
        URL url = mergeUrl(providerUrl);
        
        // 1.4 忽略重复 url，已经处理过了
        String key = url.toFullString(); // The parameter urls are sorted
        if (keys.contains(key)) { // Repeated url
            continue;
        }
        keys.add(key);

        // 3.1 匹配缓存中Invoker，如果命中直接添加到新集合newUrlInvokerMap中，
        //     未命中则生成新的Invoker后添加到新集合newUrlInvokerMap中
        Map<String, Invoker<T>> localUrlInvokerMap = this.urlInvokerMap;
        Invoker<T> invoker = localUrlInvokerMap == null ? null : localUrlInvokerMap.get(key);
        // 3.2 缓存未命中，真正将 providerUrl -> Invoker
        if (invoker == null) { // Not in the cache, refer again
            try {
                boolean enabled = true;
                // 匹配参数：disable或enable，是否允许生成代理
                if (url.hasParameter(DISABLED_KEY)) {
                    enabled = !url.getParameter(DISABLED_KEY, false);
                } else {
                    enabled = url.getParameter(ENABLED_KEY, true);
                }
                // * 核心方法：调用 refer 获取 Invoker
                if (enabled) {
                    invoker = new InvokerDelegate<>(protocol.refer(serviceType, url), url, providerUrl);
                }
            } catch (Throwable t) {
            }
            if (invoker != null) { // Put new invoker in cache
                // 将 invoker 存储到 newUrlInvokerMap 中
                newUrlInvokerMap.put(key, invoker);
            }
        } else {
        	// 3.2 缓存未命中，真正将 providerUrl -> Invoker
            // 将 invoker 存储到 newUrlInvokerMap 中
            newUrlInvokerMap.put(key, invoker);
        }
    }
    keys.clear();
    return newUrlInvokerMap;
}

Summary: toInvokers code is very long, core logic invoker = new InvokerDelegate<>(protocol.refer(serviceType, url), url, providerUrl) , as are the other main logic to determine whether to execute this code, create a new Invoker.

protocol agreement match. The consumer may receive protocol and providerUrl.getProtocol () comparison. Typically, consumers will not set this parameter, which is the default will match.
protocol agreement is valid. When empty agreement or the agreement does not exist, simply ignored.
ProviderUrl configuration parameters. mergeUrl (providerUrl), default: external Disposition configuratiors> consumerUrl> providerUrl.
If judged to have been processed. newUrlInvokerMap the key for the URL # toFullString (), if it already exists, simply ignored.
Determining whether the cache already exists. If urlInvokerMap hit in the cache, simply ignored.
ProviderUrl parameters determine whether to disable the service. If disabled, simply ignored.
If all pass, then call protocol.refer (serviceType, url) to generate a new invoker.

Most of the above logic is very simple, mainly to look at mergeUrl(providerUrl)methods, covering rule parameters.

2.3.3 mergeUrl

Profiles coverage rule: external configuration of priority, consumer priority.

private URL mergeUrl(URL providerUrl) {
    // 1. consumerUrl > providerUrl
    providerUrl = ClusterUtils.mergeUrl(providerUrl, queryMap);
    // 2. configuratorUrl > consumerUrl
    providerUrl = overrideWithConfigurator(providerUrl);
    providerUrl = providerUrl.addParameter(Constants.CHECK_KEY, String.valueOf(false)); 
   	...
    return providerUrl;
}

private URL overrideWithConfigurator(URL providerUrl) {
    // 1. configuratorUrl  "override://"
    providerUrl = overrideWithConfigurators(this.configurators, providerUrl);

    // 2. configuratorUrl from "app-name.configurators"。针对整个应用
    providerUrl = overrideWithConfigurators(CONSUMER_CONFIGURATION_LISTENER.getConfigurators(), providerUrl);

    // 3. configuratorUrl from "service-name.configurators"。针对应用中的某个服务接口
    if (serviceConfigurationListener != null) {
        providerUrl = overrideWithConfigurators(serviceConfigurationListener.getConfigurators(), providerUrl);
    }
    return providerUrl;
}

Summary: mergeUrl covering principles: external configuration of priority, consumer priority. As CONSUMER_CONFIGURATION_LISTENER and serviceConfigurationListener main content dubbo-configcenter of.

External Configuration Example of:

override://0.0.0.0/org.apache.dubbo.DemoService?category=configurators&dynamic=false&enable=true&application=dubbo-test&timeout=1000

override: Override agreement.
0.0.0.0: That is valid for all services, specific IP indicates only take effect for the specified IP. Required.
org.apache.dubbo.DemoService: That only works on specific services interface. Required.
category=configurators: Indicates that this parameter is a dynamic type of configuration. Required.
dynamic=false: False representation of persistent data, when the registrant exit data is still stored in the registry. Required.
enable=true: Indicates whether the rule to take effect, the default is true. Optional.
application=dubbo-test: Indicates that only take effect for the specified application. Optional.
timeout=1000&...: If the preceding rules in force, covering the appropriate configuration information.

2.4 Other methods described

2.4.1 toMergeInvokerList

this.invokers = multiGroup ? toMergeInvokerList(newInvokers) : newInvokers;

When a subscriber group toMergeInvokerList method multiGroup = true is disposed a plurality of time, by combination and invokers. Under normal circumstances, we will not set the group using dubbo, that is, this method will not go directly back invokers.

private List<Invoker<T>> toMergeInvokerList(List<Invoker<T>> invokers) {
    List<Invoker<T>> mergedInvokers = new ArrayList<>();
    Map<String, List<Invoker<T>>> groupMap = new HashMap<>();
    // group -> Invoker 列表
    for (Invoker<T> invoker : invokers) {
        String group = invoker.getUrl().getParameter(GROUP_KEY, "");
        groupMap.computeIfAbsent(group, k -> new ArrayList<>());
        groupMap.get(group).add(invoker);
    }

    if (groupMap.size() == 1) {
        // 1. 只有一个组，直接添加
        mergedInvokers.addAll(groupMap.values().iterator().next());
    } else if (groupMap.size() > 1) {
        // 2. 多个组，则需要使用 CLUSTER.join 将同组的 invoker 合并
        // {
        //     "dubbo": [invoker1, invoker2, invoker3, ...],
        //     "hello": [invoker4, invoker5, invoker6, ...]
        // }
        // 通过集群类合并每个分组对应的 Invoker 列表
        for (List<Invoker<T>> groupList : groupMap.values()) {
            StaticDirectory<T> staticDirectory = new StaticDirectory<>(groupList);
            staticDirectory.buildRouterChain();
            mergedInvokers.add(CLUSTER.join(staticDirectory));
        }
    } else {
        // 3. invokers.isEmpty()
        mergedInvokers = invokers;
    }
    return mergedInvokers;
}

Summary: The main logic is CLUSTER.join(staticDirectory)late and then look at this method.

The intentions of recording a little bit every day. Perhaps the content is not important, but the habit is very important!