Talk skywalking of RemoteClientManager

sequence

In this paper, we look skywalking of RemoteClientManager

RemoteClientManager

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/client/RemoteClientManager.java

public class RemoteClientManager implements Service {

    private static final Logger logger = LoggerFactory.getLogger(RemoteClientManager.class);

    private final ModuleDefineHolder moduleDefineHolder;
    private ClusterNodesQuery clusterNodesQuery;
    private volatile List<RemoteClient> usingClients;
    private GaugeMetrics gauge;
    private int remoteTimeout;

    /**
     * Initial the manager for all remote communication clients.
     *
     * @param moduleDefineHolder for looking up other modules
     * @param remoteTimeout      for cluster internal communication, in second unit.
     */
    public RemoteClientManager(ModuleDefineHolder moduleDefineHolder, int remoteTimeout) {
        this.moduleDefineHolder = moduleDefineHolder;
        this.usingClients = ImmutableList.of();
        this.remoteTimeout = remoteTimeout;
    }

    public void start() {
        Executors.newSingleThreadScheduledExecutor().scheduleWithFixedDelay(this::refresh, 1, 5, TimeUnit.SECONDS);
    }

    /**
     * Query OAP server list from the cluster module and create a new connection for the new node. Make the OAP server
     * orderly because of each of the server will send stream data to each other by hash code.
     */
    void refresh() {
        if (gauge == null) {
            gauge = moduleDefineHolder.find(TelemetryModule.NAME).provider().getService(MetricsCreator.class)
                .createGauge("cluster_size", "Cluster size of current oap node",
                    MetricsTag.EMPTY_KEY, MetricsTag.EMPTY_VALUE);
        }
        try {
            if (Objects.isNull(clusterNodesQuery)) {
                synchronized (RemoteClientManager.class) {
                    if (Objects.isNull(clusterNodesQuery)) {
                        this.clusterNodesQuery = moduleDefineHolder.find(ClusterModule.NAME).provider().getService(ClusterNodesQuery.class);
                    }
                }
            }

            if (logger.isDebugEnabled()) {
                logger.debug("Refresh remote nodes collection.");
            }

            List<RemoteInstance> instanceList = clusterNodesQuery.queryRemoteNodes();
            instanceList = distinct(instanceList);
            Collections.sort(instanceList);

            gauge.setValue(instanceList.size());

            if (logger.isDebugEnabled()) {
                instanceList.forEach(instance -> logger.debug("Cluster instance: {}", instance.toString()));
            }

            if (!compare(instanceList)) {
                if (logger.isDebugEnabled()) {
                    logger.debug("ReBuilding remote clients.");
                }
                reBuildRemoteClients(instanceList);
            }

            printRemoteClientList();
        } catch (Throwable t) {
            logger.error(t.getMessage(), t);
        }
    }

    //......
}
复制代码

• RemoteClientManager provides a method of obtaining getRemoteClient usingClients, it provides a start method of a timed task register every 5 seconds to perform a refresh; refresh list by a method of acquiring instanceList clusterNodesQuery.queryRemoteNodes (), then according to the Address bit reordering weight, then compared with local RemoteClient list, if there is found to change the trigger reBuildRemoteClients operation, and finally execute printRemoteClientList

reBuildRemoteClients

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/client/RemoteClientManager.java

public class RemoteClientManager implements Service {

	//......

    private void reBuildRemoteClients(List<RemoteInstance> remoteInstances) {
        final Map<Address, RemoteClientAction> remoteClientCollection = this.usingClients.stream()
            .collect(Collectors.toMap(RemoteClient::getAddress, client -> new RemoteClientAction(client, Action.Close)));

        final Map<Address, RemoteClientAction> latestRemoteClients = remoteInstances.stream()
            .collect(Collectors.toMap(RemoteInstance::getAddress, remote -> new RemoteClientAction(null, Action.Create)));

        final Set<Address> unChangeAddresses = Sets.intersection(remoteClientCollection.keySet(), latestRemoteClients.keySet());

        unChangeAddresses.stream()
            .filter(remoteClientCollection::containsKey)
            .forEach(unChangeAddress -> remoteClientCollection.get(unChangeAddress).setAction(Action.Unchanged));

        // make the latestRemoteClients including the new clients only
        unChangeAddresses.forEach(latestRemoteClients::remove);
        remoteClientCollection.putAll(latestRemoteClients);

        final List<RemoteClient> newRemoteClients = new LinkedList<>();
        remoteClientCollection.forEach((address, clientAction) -> {
            switch (clientAction.getAction()) {
                case Unchanged:
                    newRemoteClients.add(clientAction.getRemoteClient());
                    break;
                case Create:
                    if (address.isSelf()) {
                        RemoteClient client = new SelfRemoteClient(moduleDefineHolder, address);
                        newRemoteClients.add(client);
                    } else {
                        RemoteClient client = new GRPCRemoteClient(moduleDefineHolder, address, 1, 3000, remoteTimeout);
                        client.connect();
                        newRemoteClients.add(client);
                    }
                    break;
            }
        });

        //for stable ordering for rolling selector
        Collections.sort(newRemoteClients);
        this.usingClients = ImmutableList.copyOf(newRemoteClients);

        remoteClientCollection.values()
            .stream()
            .filter(remoteClientAction -> remoteClientAction.getAction().equals(Action.Close))
            .forEach(remoteClientAction -> remoteClientAction.getRemoteClient().close());
    }

	//......
}
复制代码

• A method to construct and remoteClientCollection reBuildRemoteClients latestRemoteClients, then on the intersection obtained unChangeAddresses, then removed from the unChangeAddresses latestRemoteClients, then finally added to latestRemoteClients remoteClientCollection; after traversing remoteClientCollection, for action to the Create and divided into SelfRemoteClient GRPCRemoteClient, for further execution at GRPCRemoteClient connect operation; newRemoteClients last sort, and then re-assigned to usingClients; for the last action performed close to the close operation RemoteClient

RemoteSenderService

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/RemoteSenderService.java

public class RemoteSenderService implements Service {
    private static final Logger logger = LoggerFactory.getLogger(RemoteSenderService.class);

    private final ModuleManager moduleManager;
    private final HashCodeSelector hashCodeSelector;
    private final ForeverFirstSelector foreverFirstSelector;
    private final RollingSelector rollingSelector;

    public RemoteSenderService(ModuleManager moduleManager) {
        this.moduleManager = moduleManager;
        this.hashCodeSelector = new HashCodeSelector();
        this.foreverFirstSelector = new ForeverFirstSelector();
        this.rollingSelector = new RollingSelector();
    }

    public void send(String nextWorkName, StreamData streamData, Selector selector) {
        RemoteClientManager clientManager = moduleManager.find(CoreModule.NAME).provider().getService(RemoteClientManager.class);
        RemoteClient remoteClient = null;

        List<RemoteClient> clientList = clientManager.getRemoteClient();
        if (clientList.size() == 0) {
            logger.warn("There is no available remote server for now, ignore the streaming data until the cluster metadata initialized.");
            return;
        }
        switch (selector) {
            case HashCode:
                remoteClient = hashCodeSelector.select(clientList, streamData);
                break;
            case Rolling:
                remoteClient = rollingSelector.select(clientList, streamData);
                break;
            case ForeverFirst:
                remoteClient = foreverFirstSelector.select(clientList, streamData);
                break;
        }
        remoteClient.push(nextWorkName, streamData);
    }
}
复制代码

• RemoteSenderService provides a send method from obtaining clientList clientManager.getRemoteClient (), then choose one remoteClient execution remoteClient.push (nextWorkName, streamData) according to the type of selector

RemoteClientSelector

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/selector/RemoteClientSelector.java

public interface RemoteClientSelector {
    RemoteClient select(List<RemoteClient> clients, StreamData streamData);
}
复制代码

• RemoteClientSelector defines the select method

HashCodeSelector

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/selector/HashCodeSelector.java

public class HashCodeSelector implements RemoteClientSelector {

    @Override public RemoteClient select(List<RemoteClient> clients, StreamData streamData) {
        int size = clients.size();
        int selectIndex = Math.abs(streamData.remoteHashCode()) % size;
        return clients.get(selectIndex);
    }
}
复制代码

• HashCodeSelector realized RemoteClientSelector interface, by Math.abs(streamData.remoteHashCode()) % sizeselecting selectIndex to

RollingSelector

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/selector/RollingSelector.java

public class RollingSelector implements RemoteClientSelector {

    private int index = 0;

    @Override public RemoteClient select(List<RemoteClient> clients, StreamData streamData) {
        int size = clients.size();
        index++;
        int selectIndex = Math.abs(index) % size;

        if (index == Integer.MAX_VALUE) {
            index = 0;
        }
        return clients.get(selectIndex);
    }
}
复制代码

• RollingSelector realized RemoteClientSelector interface, which according to index and then incremented each time by Math.abs(index) % sizeselecting selectIndex

ForeverFirstSelector

skywalking-6.6.0/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/remote/selector/ForeverFirstSelector.java

public class ForeverFirstSelector implements RemoteClientSelector {

    private static final Logger logger = LoggerFactory.getLogger(ForeverFirstSelector.class);

    @Override public RemoteClient select(List<RemoteClient> clients, StreamData streamData) {
        if (logger.isDebugEnabled()) {
            logger.debug("clients size: {}", clients.size());
        }
        return clients.get(0);
    }
}
复制代码

• ForeverFirstSelector realized RemoteClientSelector interface, which always returns the first client

summary

RemoteClientManager provides a method of obtaining getRemoteClient usingClients, it provides a start method of a timed task register every 5 seconds to perform a refresh; refresh list by a method of acquiring instanceList clusterNodesQuery.queryRemoteNodes (), then according to the Address bit reordering weight, then compared with local RemoteClient list, if there is found to change the trigger reBuildRemoteClients operation, and finally execute printRemoteClientList

doc

• RemoteClientManager