1. Frontier
The service export and service reference are all finished. Next, we need to analyze the service invocation process, but there is a very important module in the service invocation process that needs to be studied first, namely the cluster fault-tolerant module. The cluster fault-tolerant module contains four parts. They are Service Directory, Service Routing Router, Cluster and Load Balance LoadBalance , which will be explained in four chapters below. , First we look at the service directory Directory.
Definition: The service directory directory is a collection of Invoker, and the elements in this collection will be dynamically adjusted as the registry changes
Some information related to the provider is stored in the service directory directory . Through the service directory, the consumer can obtain the information of the provider, such as ip, port, service protocol, etc. With this information, consumers can make remote calls through clients such as Netty . In a service cluster, the number of providers is not static. If a machine is added to the cluster, a provider record must be added to the directory accordingly. Or, if the configuration of the provider is modified, the original records in the directory will also need to be updated accordingly. Looking at it this way, the functions of the directory and the registry are indeed similar. In fact, after the directory obtains the service configuration information of the registry, it will generate an Invoker object for each configuration information and store the Invoker object. This Invoker is the object ultimately held by the directory. What is the use of Invoker? Just look at the name, this is an object with remote call function
2. Service directory structure
The overall structure of the service directory directory is as follows:
It can be seen from the above figure that there are only two specific implementation classes of Directory, namely StaticDirectory and RegistryDirectory , both of which inherit the AbstractDirectory class, and AbstractDirectory implements the Directory interface. The Directory interface defines a very important method, namely list( Invocation invocation), used to enumerate the Invoker list
The Directory interface inherits the Node interface. The Node interface defines a very important method, namely getUrl() to obtain the configuration information URL . The class that implements this interface can provide configuration information to the outside, such as Registry, Monitor, Invoker, etc. in dubbo Have inherited this interface
RegistryDirectory not only implements the Directory interface, but also implements the NotifyListener interface. In this way, when the registry node information changes, RegistryDirectory can obtain the change information through this interface method, and dynamically adjust the internal Invoker list according to the change information
Three, Directory source code
From the service directory structure, we only need to analyze three classes to analyze the Directory source code, namely AbstractDirectory (abstract class), StaticDirectory (implementation class) and RegistryDirectory (implementation class)
3.1 AbstractDirectory
AbstractDirectory encapsulates the Invoker enumeration process , and the specific enumeration logic is implemented by subclasses. This is a typical template pattern. Next, let's take a look at the source code of AbstractDirectory:
public AbstractDirectory(URL url, URL consumerUrl, RouterChain<T> routerChain) {
if (url == null) {
throw new IllegalArgumentException("url == null");
}
if (url.getProtocol().equals(REGISTRY_PROTOCOL)) {
// protocol 协议是 registry 时，解码获取 refer 参数值并设置到 url 中，同时移除 monitor 参数
Map<String, String> queryMap = StringUtils.parseQueryString(url.getParameterAndDecoded(REFER_KEY));
this.url = url.addParameters(queryMap).removeParameter(MONITOR_KEY);
} else {
this.url = url;
}
this.consumerUrl = consumerUrl;
// 设置路由链
setRouterChain(routerChain);
}
@Override
public List<Invoker<T>> list(Invocation invocation) throws RpcException {
if (destroyed) {
throw new RpcException("Directory already destroyed .url: " + getUrl());
}
// doList 是模板方法，由子类实现具体的业务逻辑
return doList(invocation);
}The logic is quite simple, so I won’t repeat it here, let’s look at the StaticDirectory source code.
3.2 StaticDirectory
StaticDirectory is a static service directory . As the name implies, the Invoker stored in it will not change . So, in theory, it is very similar to the immutable List function. Let's take a look at the specific source code of this class:
public class StaticDirectory<T> extends AbstractDirectory<T> {
private static final Logger logger = LoggerFactory.getLogger(StaticDirectory.class);
private final List<Invoker<T>> invokers;
public StaticDirectory(List<Invoker<T>> invokers) {
this(null, invokers, null);
}
public StaticDirectory(List<Invoker<T>> invokers, RouterChain<T> routerChain) {
this(null, invokers, routerChain);
}
public StaticDirectory(URL url, List<Invoker<T>> invokers) {
this(url, invokers, null);
}
public StaticDirectory(URL url, List<Invoker<T>> invokers, RouterChain<T> routerChain) {
super(url == null && CollectionUtils.isNotEmpty(invokers) ? invokers.get(0).getUrl() : url, routerChain);
if (CollectionUtils.isEmpty(invokers)) {
throw new IllegalArgumentException("invokers == null");
}
this.invokers = invokers;
}
@Override
public Class<T> getInterface() {
// 获取接口类
return invokers.get(0).getInterface();
}
// 检测 directory 是否可用
@Override
public boolean isAvailable() {
if (isDestroyed()) {
return false;
}
for (Invoker<T> invoker : invokers) {
if (invoker.isAvailable()) {
// 只要有一个 invoker 可用，就认为当前 directory 是可用的
return true;
}
}
return false;
}
// 销毁 directory 目录
@Override
public void destroy() {
if (isDestroyed()) {
return;
}
super.destroy();
// 遍历销毁 directory 目录下的所有 invoker
for (Invoker<T> invoker : invokers) {
invoker.destroy();
}
invokers.clear();
}
// 构建路由链
public void buildRouterChain() {
RouterChain<T> routerChain = RouterChain.buildChain(getUrl());
routerChain.setInvokers(invokers);
this.setRouterChain(routerChain);
}
@Override
protected List<Invoker<T>> doList(Invocation invocation) throws RpcException {
List<Invoker<T>> finalInvokers = invokers;
if (routerChain != null) {
try {
// 经过路由链过滤掉不需要的 invoker
finalInvokers = routerChain.route(getConsumerUrl(), invocation);
} catch (Throwable t) {
logger.error("Failed to execute router: " + getUrl() + ", cause: " + t.getMessage(), t);
}
}
// 返回 invoker 列表
return finalInvokers == null ? Collections.emptyList() : finalInvokers;
}
}There is no complicated logic in the StaticDirectory source code. The only thing that needs to be done is to filter the Invoker through the routing chain. This will be discussed in a later chapter. The other content is easy to understand, so I won’t go into too much detail.
3.3 RegistryDirectory
RegistryDirectory is a dynamic service directory that implements the NotifyListener interface. When the registry service configuration changes, RegistryDirectory can receive changes related to the current service. After receiving the change notification, RegistryDirectory can refresh the Invoker list according to the configuration change information . There are three more important logics in RegistryDirectory, as follows:
1), enumerate the Invoker list
2), receive notification of service configuration changes
3), refresh the stored Invoker list
Next, analyze the source code for everyone in the order of 1, 2, and 3.
3.3.1 Enumerate the Invoker list
The doList method implements the enumeration of the Invoker list. The source code is as follows:
@Override
public List<Invoker<T>> doList(Invocation invocation) {
if (forbidden) {
// 1. No service provider 2. Service providers are disabled
// provider 关闭或禁用了服务，此时抛出 No provider 异常
throw new RpcException(RpcException.FORBIDDEN_EXCEPTION, "No provider available from registry " +
getUrl().getAddress() + " for service " + getConsumerUrl().getServiceKey() + " on consumer " +
NetUtils.getLocalHost() + " use dubbo version " + Version.getVersion() +
", please check status of providers(disabled, not registered or in blacklist).");
}
if (multiGroup) {
// 多个 group 的话直接返回 invoker 列表
return this.invokers == null ? Collections.emptyList() : this.invokers;
}
List<Invoker<T>> invokers = null;
try {
// Get invokers from cache, only runtime routers will be executed.
// route 路由器职责链过滤满足条件的 Invoker 列表
invokers = routerChain.route(getConsumerUrl(), invocation);
} catch (Throwable t) {
logger.error("Failed to execute router: " + getUrl() + ", cause: " + t.getMessage(), t);
}
return invokers == null ? Collections.emptyList() : invokers;
}The main logic here is also to filter the list of invokers that meet the conditions through the route router chain, which is simple and easy to understand, so I won’t repeat it here.
3.3.2 Receiving notifications of service configuration changes.
RegistryDirectory is a dynamic service directory that will dynamically adjust with changes in the configuration of the registry . Because RegistryDirectory implements the NotifyListener interface, the registry change notification can be obtained through this interface. The specific implementation logic is in the notify method.
The source code of the notify method is as follows:
@Override
public synchronized void notify(List<URL> urls) {
// 定义三个 集合，分别为 provider 的 configurators URL、routers URL 和 providers URL
Map<String, List<URL>> categoryUrls = urls.stream()
.filter(Objects::nonNull)
.filter(this::isValidCategory)
.filter(this::isNotCompatibleFor26x)
.collect(Collectors.groupingBy(url -> {
if (UrlUtils.isConfigurator(url)) {
// configurators URL
return CONFIGURATORS_CATEGORY;
} else if (UrlUtils.isRoute(url)) {
// routers URL
return ROUTERS_CATEGORY;
} else if (UrlUtils.isProvider(url)) {
// providers URL
return PROVIDERS_CATEGORY;
}
return "";
}));
List<URL> configuratorURLs = categoryUrls.getOrDefault(CONFIGURATORS_CATEGORY, Collections.emptyList());
// configurators URL 转成 Configurator
this.configurators = Configurator.toConfigurators(configuratorURLs).orElse(this.configurators);
List<URL> routerURLs = categoryUrls.getOrDefault(ROUTERS_CATEGORY, Collections.emptyList());
// routes URL 转成 Router
toRouters(routerURLs).ifPresent(this::addRouters);
// providers
// 提供者 URL 列表
List<URL> providerURLs = categoryUrls.getOrDefault(PROVIDERS_CATEGORY, Collections.emptyList());
// 刷新 invoker 列表
refreshOverrideAndInvoker(providerURLs);
}The notify method mainly implements the following logic:
1) According to the protocol or category parameters in the URL, the URL is classified and stored
2) The toConfigurators and toRouters methods convert url into Configurator and Router respectively
3), refresh the invoker list according to the provider url list
The next focus is to analyze and refresh the source code of the invoker list
3.3.3 Refresh the Invoker List
The logic of refreshing the invoker list is mainly in the refreshOverrideAndInvoker method, which is the key to ensuring that the RegistryDirectory changes with the change of the registry. The source code of the whole process is as follows:
private void refreshOverrideAndInvoker(List<URL> urls) {
// mock zookeeper://xxx?mock=return null
// 获取配置 overrideDirectoryUrl 值
overrideDirectoryUrl();
// 刷新 invoker 列表
refreshInvoker(urls);
}
/**
* Convert the invokerURL list to the Invoker Map. The rules of the conversion are as follows:
* <ol>
* <li> If URL has been converted to invoker, it is no longer re-referenced and obtained directly from the cache,
* and notice that any parameter changes in the URL will be re-referenced.</li>
* <li>If the incoming invoker list is not empty, it means that it is the latest invoker list.</li>
* <li>If the list of incoming invokerUrl is empty, It means that the rule is only a override rule or a route
* rule, which needs to be re-contrasted to decide whether to re-reference.</li>
* </ol>
*
* @param invokerUrls this parameter can't be null
*/
// TODO: 2017/8/31 FIXME The thread pool should be used to refresh the address, otherwise the task may be accumulated.
private void refreshInvoker(List<URL> invokerUrls) {
Assert.notNull(invokerUrls, "invokerUrls should not be null");
// invokerUrls 中只有一个元素 && url 的协议头 Protocol 为 empty，表示禁用所有服务
if (invokerUrls.size() == 1
&& invokerUrls.get(0) != null
&& EMPTY_PROTOCOL.equals(invokerUrls.get(0).getProtocol())) {
// 禁用服务标识
this.forbidden = true; // Forbid to access
// 空 invoker 列表
this.invokers = Collections.emptyList();
routerChain.setInvokers(this.invokers);
// 销毁所有的 invoker
destroyAllInvokers(); // Close all invokers
} else {
// Allow to access
this.forbidden = false;
// local reference
// 原来的 invoker 映射map
Map<String, Invoker<T>> oldUrlInvokerMap = this.urlInvokerMap;
if (invokerUrls == Collections.<URL>emptyList()) {
invokerUrls = new ArrayList<>();
}
if (invokerUrls.isEmpty() && this.cachedInvokerUrls != null) {
// 添加缓存 url 到 invokerUrls 中
invokerUrls.addAll(this.cachedInvokerUrls);
} else {
this.cachedInvokerUrls = new HashSet<>();
//Cached invoker urls, convenient for comparison
// 缓存 invokerUrls
this.cachedInvokerUrls.addAll(invokerUrls);
}
if (invokerUrls.isEmpty()) {
return;
}
// Translate url list to Invoker map
// 将 url 转换成 url 到 invoker 映射 map，map 中 url 为 key，value 为 invoker
Map<String, Invoker<T>> newUrlInvokerMap = toInvokers(invokerUrls);
/**
* If the calculation is wrong, it is not processed.
*
* 1. The protocol configured by the client is inconsistent with the protocol of the server.
* eg: consumer protocol = dubbo, provider only has other protocol services(rest).
* 2. The registration center is not robust and pushes illegal specification data.
*
*/
if (CollectionUtils.isEmptyMap(newUrlInvokerMap)) {
logger.error(new IllegalStateException("urls to invokers error .invokerUrls.size :" + invokerUrls.size() + ", invoker.size :0. urls :" + invokerUrls
.toString()));
return;
}
List<Invoker<T>> newInvokers = Collections.unmodifiableList(new ArrayList<>(newUrlInvokerMap.values()));
// pre-route and build cache, notice that route cache should build on original Invoker list.
// toMergeMethodInvokerMap() will wrap some invokers having different groups, those wrapped invokers not should be routed.
routerChain.setInvokers(newInvokers);
// 多个 group 时需要合并 invoker
this.invokers = multiGroup ? toMergeInvokerList(newInvokers) : newInvokers;
this.urlInvokerMap = newUrlInvokerMap;
try {
// Close the unused Invoker
// 销毁不需要的 invoker
destroyUnusedInvokers(oldUrlInvokerMap, newUrlInvokerMap);
} catch (Exception e) {
logger.warn("destroyUnusedInvokers error. ", e);
}
}
}Refreshing the invoker list mainly implements the following logic:
1). Determine whether all services need to be disabled according to the number of incoming parameter InvokerUrls and the protocol header of the url. If necessary, set forbidden to true and destroy all Invokers
2). Convert url to invoker, and get the mapping relationship of <url, Invoker>
3). Combine the invokers of multiple groups and assign them to the invokers variable
4). Destroy useless invokers to prevent consumers from calling offline providers
Let's analyze the toInvokers method, that is, the process of converting url to invoker and obtaining the mapping relationship of <url, Invoker>. The source code is as follows:
/**
* Turn urls into invokers, and if url has been refer, will not re-reference.
*
* @param urls
* @return invokers
*/
private Map<String, Invoker<T>> toInvokers(List<URL> urls) {
Map<String, Invoker<T>> newUrlInvokerMap = new HashMap<>();
if (urls == null || urls.isEmpty()) {
// 列表为空，直接返回
return newUrlInvokerMap;
}
Set<String> keys = new HashSet<>();
// 获取 consumer 端配置的协议
String queryProtocols = this.queryMap.get(PROTOCOL_KEY);
for (URL providerUrl : urls) {
// If protocol is configured at the reference side, only the matching protocol is selected
// 如果在 consumer 端配置了协议，则只选择匹配的协议
if (queryProtocols != null && queryProtocols.length() > 0) {
boolean accept = false;
String[] acceptProtocols = queryProtocols.split(",");
for (String acceptProtocol : acceptProtocols) {
// provider 协议是否被 consumer 协议支持
if (providerUrl.getProtocol().equals(acceptProtocol)) {
accept = true;
break;
}
}
if (!accept) {
// provider 协议不被 consumer 协议支持，则忽略此 url
continue;
}
}
// 忽略 empty 协议
if (EMPTY_PROTOCOL.equals(providerUrl.getProtocol())) {
continue;
}
// 通过 SPI 检测 provider 端协议是否被 consumer 端支持，不支持则抛出异常
if (!ExtensionLoader.getExtensionLoader(Protocol.class).hasExtension(providerUrl.getProtocol())) {
logger.error(new IllegalStateException("Unsupported protocol " + providerUrl.getProtocol() +
" in notified url: " + providerUrl + " from registry " + getUrl().getAddress() +
" to consumer " + NetUtils.getLocalHost() + ", supported protocol: " +
ExtensionLoader.getExtensionLoader(Protocol.class).getSupportedExtensions()));
continue;
}
// 合并 url 参数，顺序是 : override > -D > Consumer > Provider
URL url = mergeUrl(providerUrl);
// The parameter urls are sorted
// url 参数拼接的字符串且被排序了，
// 例如：dubbo://192.168.1.247:20887/org.apache.dubbo.config.spring.api.DemoService?anyhost=true&application=service-class&
// bean.name=org.apache.dubbo.config.spring.api.DemoService&bind.ip=192.168.1.247&bind.port=20887&
// class=org.apache.dubbo.config.spring.impl.DemoServiceImpl&deprecated=false&dubbo=2.0.2&dynamic=true&generic=false&
// interface=org.apache.dubbo.config.spring.api.DemoService&methods=sayName,getBox&owner=world&pid=24316®ister=true&
// release=&side=provider×tamp=1572405725011
String key = url.toFullString();
if (keys.contains(key)) {
// Repeated url
// 忽略重复 key
continue;
}
keys.add(key);
// Cache key is url that does not merge with consumer side parameters, regardless of how the consumer combines parameters, if the server url changes, then refer again
// 原来本地缓存的 <url, Invoker>
Map<String, Invoker<T>> localUrlInvokerMap = this.urlInvokerMap; // local reference
// 获取现有的 url 对应的 invoker
Invoker<T> invoker = localUrlInvokerMap == null ? null : localUrlInvokerMap.get(key);
// Not in the cache, refer again
// 原来缓存没有或者provider配置变化了
if (invoker == null) {
try {
boolean enabled = true;
if (url.hasParameter(DISABLED_KEY)) {
// 获取 url 中 disable 配置值，取反，然后赋值给 enable 变量
enabled = !url.getParameter(DISABLED_KEY, false);
} else {
// 获取 url 中 enable 配置值
enabled = url.getParameter(ENABLED_KEY, true);
}
if (enabled) {
// 调用 Protocol 的 refer 方法构建获取 Invoker，
// 具体调用流程是 refer(ProtocolListenerWrapper) -> refer(ProtocolFilterWrapper) -> refer(AbstractProtocol)
invoker = new InvokerDelegate<>(protocol.refer(serviceType, url), url, providerUrl);
}
} catch (Throwable t) {
logger.error("Failed to refer invoker for interface:" + serviceType + ",url:(" + url + ")" + t.getMessage(), t);
}
if (invoker != null) {
// Put new invoker in cache
// 新 invoker 放入缓存
newUrlInvokerMap.put(key, invoker);
}
} else {
newUrlInvokerMap.put(key, invoker);
}
}
keys.clear();
return newUrlInvokerMap;
}The toInvokers method mainly implements the following logic:
1). Ignore the URLs of the empty protocol and the provider protocol that is not supported by the consumer
2), merge url parameters, the order is: override> -D> Consumer> Provider
3) Get the local cache invoker, if there is no cache or the provider configuration changes && provider is not disabled, build a new invoker and put it into the cache newUrlInvokerMap
Next, explain the toMergeInvokerList method, which is the logic of merging multiple groups of invokers. The source code is as follows:
private List<Invoker<T>> toMergeInvokerList(List<Invoker<T>> invokers) {
List<Invoker<T>> mergedInvokers = new ArrayList<>();
Map<String, List<Invoker<T>>> groupMap = new HashMap<>();
// 遍历 invoker 进行分组
for (Invoker<T> invoker : invokers) {
// url 中获取 group 配置
String group = invoker.getUrl().getParameter(GROUP_KEY, "");
groupMap.computeIfAbsent(group, k -> new ArrayList<>());
// invoker 放入同一组中
groupMap.get(group).add(invoker);
}
if (groupMap.size() == 1) {
// 只有一个分组，直接取出值
mergedInvokers.addAll(groupMap.values().iterator().next());
} else if (groupMap.size() > 1) {
// 多个分组时，遍历分组中的 invoker列表，并调用集群的 join 方法合并每个组的 invoker 列表
for (List<Invoker<T>> groupList : groupMap.values()) {
StaticDirectory<T> staticDirectory = new StaticDirectory<>(groupList);
staticDirectory.buildRouterChain();
// 通过集群类 的 join 方法合并每个分组对应的 Invoker 列表
mergedInvokers.add(CLUSTER.join(staticDirectory));
}
} else {
mergedInvokers = invokers;
}
return mergedInvokers;
}The logic of multiple combinations and the invoker list is very simple. The code comments are already very detailed, so I won’t repeat them here.
Finally, analyze the destroyUnusedInvokers method, which destroys useless invoices. The source code is as follows:
/**
* Check whether the invoker in the cache needs to be destroyed
* If set attribute of url: refer.autodestroy=false, the invokers will only increase without decreasing,there may be a refer leak
*
* @param oldUrlInvokerMap
* @param newUrlInvokerMap
*/
private void destroyUnusedInvokers(Map<String, Invoker<T>> oldUrlInvokerMap, Map<String, Invoker<T>> newUrlInvokerMap) {
if (newUrlInvokerMap == null || newUrlInvokerMap.size() == 0) {
// 新的 invoker 为空，表明禁用了所有的 provider，这里销毁所有的 invoker
destroyAllInvokers();
return;
}
// check deleted invoker
// 记录需要被销毁的 invoker 列表
List<String> deleted = null;
if (oldUrlInvokerMap != null) {
Collection<Invoker<T>> newInvokers = newUrlInvokerMap.values();
for (Map.Entry<String, Invoker<T>> entry : oldUrlInvokerMap.entrySet()) {
// 新的 invoker 列表中不包含原有的 invoker，则该 invoker 需要被销毁
if (!newInvokers.contains(entry.getValue())) {
if (deleted == null) {
deleted = new ArrayList<>();
}
// 该 invoker 添加到销毁列表中
deleted.add(entry.getKey());
}
}
}
// 销毁列表中有数据，需要销毁 invoker
if (deleted != null) {
for (String url : deleted) {
if (url != null) {
// 缓存中移除要销毁的 invoker
Invoker<T> invoker = oldUrlInvokerMap.remove(url);
if (invoker != null) {
try {
// 销毁 invoker
invoker.destroy();
if (logger.isDebugEnabled()) {
logger.debug("destroy invoker[" + invoker.getUrl() + "] success. ");
}
} catch (Exception e) {
logger.warn("destroy invoker[" + invoker.getUrl() + "] failed. " + e.getMessage(), e);
}
}
}
}
}
}So far we have analyzed the refresh logic of the Invoker list. Here is a brief summary of the whole process, as follows:
1), check whether the input parameter contains only one url, and the url protocol header is empty
2). If the first step is true, it means that all services are disabled, and all Invokers are destroyed at this time
3). If the detection result in the first step is false, then convert the url to an invoker, and get the mapping relationship of <url, Invoker>
4), merge multiple groups of Invoker
5), destroy useless Invoker
Four, summary
This article introduces the source code related to the service directory in detail. The logic is relatively simple, but there are still many things that need to be done to make the local service directory and the registry service configuration consistent. The service directory directory is a part of Dubbo cluster fault tolerance, and it is also a relatively basic part. It seems to be easier to understand. I hope everyone can grasp it.
reference:
https://dubbo.apache.org/zh-cn/docs/source_code_guide/directory.html