Overview: Elasticsearch cluster version 7.3.3, number of servers: 140, number of nodes: 270, version 7.3.1, the total amount of data is 457.4TB, the monthly data volume of the platform is about 60T, single index storage capacity: 16T per month, daily 2T. There are about 50 million real-time interface calls per day, with an average of 462 requests per second, and one request is within 100ms.
This article attempts to explain the following internal mechanism problems of Elasticsearch Clients through source code analysis:
- How will the abnormality of the ES cluster server affect the client's request?
- Can the ES client automatically retry?
- How does the ES client manage multiple addresses, and how does it perceive new or deleted addresses?
- What strategy does ES client load balancing use?
- For the disconnection of a node in the ES cluster, how does the ES client perceive when sending a request, and what solution strategy will be adopted?
- Does the ES client request use a long connection, can the performance continue to be improved?
- ES sniff mechanism function?
These problems are common problems for business parties when using Elasticsearch Clients.
All types of Elasticsearch Clients can be found at: https://www.elastic.co/guide/en/elasticsearch/client/index.html. Since Java REST Client and Java API (TransportClient) are widely used in actual work, only the source codes of these two types of clients are analyzed here.
—Remarks: The Elasticsearch Clients version analyzed is 6.5.x.
1. Add multiple addresses
Both Elasticsearch Java REST Client and Elasticsearch Java API allow users to add multiple cluster access addresses when used.
1.1 Java REST Client
RestClient restClient = RestClient.builder(
new HttpHost("host1", 9200, "http"),
new HttpHost("host2", 9200, "http")).build();
Java API
TransportClient client = new PreBuiltTransportClient(Settings.EMPTY)
.addTransportAddress(new TransportAddress(InetAddress.getByName("host1"), 9300))
.addTransportAddress(new TransportAddress(InetAddress.getByName("host2"), 9300));
After adding, Elasticsearch saves it in a List list inside the Client. Among them, the Rest client is saved as a Node structure list; while the Transport client is internally converted to a DiscoveryNode list.
Java REST Client
...
List<Node> nodes = new ArrayList<>(hosts.length);
for (HttpHost host : hosts) {
nodes.add(new Node(host));
}
...
1.2 Java API
...
List<DiscoveryNode> builder = new ArrayList<>(listedNodes);
for (TransportAddress transportAddress : filtered) {
DiscoveryNode node = new DiscoveryNode("#transport#-" + tempNodeIdGenerator.incrementAndGet(),
transportAddress, Collections.emptyMap(), Collections.emptySet(), minCompatibilityVersion);
logger.debug("adding address [{}]", node);
builder.add(node);
}
listedNodes = Collections.unmodifiableList(builder);
...
2. Elasticsearch Clients detection mechanism
2.1 Java REST Client detection mechanism
Java REST Client does not have a timing detection mechanism, it combines the detection of node status with each user request. When sending a user request, if the response is normal, it indicates that the node status is ok, and if the request fails, the node is marked as abnormal. The main logic of RestClient processing requests is in the performRequestAsync() function
private void performRequestAsync(final long startTime, final NodeTuple<Iterator<Node>> nodeTuple, final HttpRequestBase request,
final Set<Integer> ignoreErrorCodes,
final HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
final FailureTrackingResponseListener listener) {
final Node node = nodeTuple.nodes.next();
...
client.execute(requestProducer, asyncResponseConsumer, context, new FutureCallback<HttpResponse>() {
@Override
public void completed(HttpResponse httpResponse) {
try {
RequestLogger.logResponse(logger, request, node.getHost(), httpResponse);
int statusCode = httpResponse.getStatusLine().getStatusCode();
Response response = new Response(request.getRequestLine(), node.getHost(), httpResponse);
if (isSuccessfulResponse(statusCode) || ignoreErrorCodes.contains(response.getStatusLine().getStatusCode())) {
onResponse(node);
if (strictDeprecationMode && response.hasWarnings()) {
listener.onDefinitiveFailure(new ResponseException(response));
} else {
listener.onSuccess(response);
}
} else {
ResponseException responseException = new ResponseException(response);
if (isRetryStatus(statusCode)) {
//mark host dead and retry against next one
onFailure(node);
retryIfPossible(responseException);
} else {
//mark host alive and don't retry, as the error should be a request problem
onResponse(node);
listener.onDefinitiveFailure(responseException);
}
}
} catch(Exception e) {
listener.onDefinitiveFailure(e);
}
}
...
}
The result processing function of the normal response is onResponse(), if the node is in the blacklist (blacklist), it will be removed:
/**
* Called after each successful request call.
* Receives as an argument the host that was used for the successful request.
*/
private void onResponse(Node node) {
DeadHostState removedHost = this.blacklist.remove(node.getHost());
if (logger.isDebugEnabled() && removedHost != null) {
logger.debug("removed [" + node + "] from blacklist");
}
}
In response to the exception handling function onFailure(), add the node to the blacklist:
/**
* Called after each failed attempt.
* Receives as an argument the host that was used for the failed attempt.
*/
private void onFailure(Node node) {
while(true) {
DeadHostState previousDeadHostState =
blacklist.putIfAbsent(node.getHost(), new DeadHostState(TimeSupplier.DEFAULT));
if (previousDeadHostState == null) {
if (logger.isDebugEnabled()) {
logger.debug("added [" + node + "] to blacklist");
}
break;
}
if (blacklist.replace(node.getHost(), previousDeadHostState,
new DeadHostState(previousDeadHostState))) {
if (logger.isDebugEnabled()) {
logger.debug("updated [" + node + "] already in blacklist");
}
break;
}
}
failureListener.onFailure(node);
}
However, nodes that have been added to the blacklist also have the opportunity to be re-enabled. RestClient mainly uses a retry timeout mechanism to control. Before executing each request, RestClient will judge whether there are nodes in the blacklist category that have met the retry requirements, and if so, remove them and add them to the candidate list for sending requests.
void performRequestAsyncNoCatch(Request request, ResponseListener listener) throws IOException {
...
performRequestAsync(startTime, nextNode(), httpRequest, ignoreErrorCodes,
request.getOptions().getHttpAsyncResponseConsumerFactory(), failureTrackingResponseListener);
}
private NodeTuple<Iterator<Node>> nextNode() throws IOException {
NodeTuple<List<Node>> nodeTuple = this.nodeTuple;
Iterable<Node> hosts = selectNodes(nodeTuple, blacklist, lastNodeIndex, nodeSelector);
return new NodeTuple<>(hosts.iterator(), nodeTuple.authCache);
}
static Iterable<Node> selectNodes(NodeTuple<List<Node>> nodeTuple, Map<HttpHost, DeadHostState> blacklist,
AtomicInteger lastNodeIndex, NodeSelector nodeSelector) throws IOException {
/*
* Sort the nodes into living and dead lists.
*/
List<Node> livingNodes = new ArrayList<>(nodeTuple.nodes.size() - blacklist.size());
List<DeadNode> deadNodes = new ArrayList<>(blacklist.size());
for (Node node : nodeTuple.nodes) {
DeadHostState deadness = blacklist.get(node.getHost());
if (deadness == null) {
livingNodes.add(node);
continue;
}
if (deadness.shallBeRetried()) {
livingNodes.add(node);
continue;
}
deadNodes.add(new DeadNode(node, deadness));
}
...
}
RestClient's retry timeout mechanism is judged according to the time interval, the minimum time interval is 1min, and then gradually increases until the longest interval is 30min.
/**
* Indicates whether it's time to retry to failed host or not.
*
* @return true if the host should be retried, false otherwise
*/
boolean shallBeRetried() {
return timeSupplier.nanoTime() - deadUntilNanos > 0;
}
DeadHostState(DeadHostState previousDeadHostState) {
long timeoutNanos = (long)Math.min(MIN_CONNECTION_TIMEOUT_NANOS * 2 * Math.pow(2, previousDeadHostState.failedAttempts * 0.5 - 1),
MAX_CONNECTION_TIMEOUT_NANOS);
this.deadUntilNanos = previousDeadHostState.timeSupplier.nanoTime() + timeoutNanos;
...
}
DeadHostState(TimeSupplier timeSupplier) {
...
this.deadUntilNanos = timeSupplier.nanoTime() + MIN_CONNECTION_TIMEOUT_NANOS;
...
}
2.2 Java API detection mechanism
The Java API (TransportClient) has a timing (sampling) detection mechanism. When the TransportClient is initialized, the TransportClientNodesService will be initialized. In its constructor, the sampling timer will be started:
TransportClientNodesService(Settings settings, TransportService transportService,
ThreadPool threadPool, TransportClient.HostFailureListener hostFailureListener) {
...
this.nodesSamplerInterval = TransportClient.CLIENT_TRANSPORT_NODES_SAMPLER_INTERVAL.get(this.settings);
...
if (TransportClient.CLIENT_TRANSPORT_SNIFF.get(this.settings)) {
this.nodesSampler = new SniffNodesSampler();
} else {
this.nodesSampler = new SimpleNodeSampler();
}
...
this.nodesSamplerFuture = threadPool.schedule(nodesSamplerInterval, ThreadPool.Names.GENERIC, new ScheduledNodeSampler());
}
The CLIENT_TRANSPORT_NODES_SAMPLER_INTERVAL configuration value defaults to 5s, that is, TransportClient detects all nodes every 5s. There are two types of Sampler, SimpleNodeSampler and SniffNodesSampler. Which one is used depends on the client CLIENT_TRANSPORT_SNIFF (client.transport.sniff) configuration. In addition, in addition to the detection triggered by timing, every time a new node is added (addTransportAddresses) or a node is removed (removeTransportAddress) will also trigger a sampling detection.
public TransportClientNodesService addTransportAddresses(TransportAddress... transportAddresses) {
...
nodesSampler.sample();
...
}
public TransportClientNodesService removeTransportAddress(TransportAddress transportAddress) {
...
nodes = Collections.unmodifiableList(nodesBuilder);
...
}
Look at the specific process of sampling:
class SimpleNodeSampler extends NodeSampler {
@Override
protected void doSample() {
HashSet<DiscoveryNode> newNodes = new HashSet<>();
ArrayList<DiscoveryNode> newFilteredNodes = new ArrayList<>();
for (DiscoveryNode listedNode : listedNodes) {
try (Transport.Connection connection = transportService.openConnection(listedNode, LISTED_NODES_PROFILE)){
final PlainTransportFuture<LivenessResponse> handler = new PlainTransportFuture<>(
new FutureTransportResponseHandler<LivenessResponse>() {
@Override
public LivenessResponse read(StreamInput in) throws IOException {
LivenessResponse response = new LivenessResponse();
response.readFrom(in);
return response;
}
});
transportService.sendRequest(connection, TransportLivenessAction.NAME, new LivenessRequest(),
TransportRequestOptions.builder().withType(TransportRequestOptions.Type.STATE).withTimeout(pingTimeout).build(),
handler);
final LivenessResponse livenessResponse = handler.txGet();
if (!ignoreClusterName && !clusterName.equals(livenessResponse.getClusterName())) {
logger.warn("node {} not part of the cluster {}, ignoring...", listedNode, clusterName);
newFilteredNodes.add(listedNode);
} else {
// use discovered information but do keep the original transport address,
// so people can control which address is exactly used.
DiscoveryNode nodeWithInfo = livenessResponse.getDiscoveryNode();
newNodes.add(new DiscoveryNode(nodeWithInfo.getName(), nodeWithInfo.getId(), nodeWithInfo.getEphemeralId(),
nodeWithInfo.getHostName(), nodeWithInfo.getHostAddress(), listedNode.getAddress(),
nodeWithInfo.getAttributes(), nodeWithInfo.getRoles(), nodeWithInfo.getVersion()));
}
} catch (ConnectTransportException e) {
logger.debug(() -> new ParameterizedMessage("failed to connect to node [{}], ignoring...", listedNode), e);
hostFailureListener.onNodeDisconnected(listedNode, e);
} catch (Exception e) {
logger.info(() -> new ParameterizedMessage("failed to get node info for {}, disconnecting...", listedNode), e);
}
}
nodes = establishNodeConnections(newNodes);
filteredNodes = Collections.unmodifiableList(newFilteredNodes);
}
}
In SimpleNodeSampler, the detection idea is to send a liveness request to all the nodes in the list (refer to the "adding multiple addresses" section). If the response is normal and the cluster name and other information can be checked, it is a normal node and added to the normal node Alternate in the list. If the connection/response is abnormal, output the debug log and do no other processing (hostFailureListener.onNodeDisconnected() currently does not do any processing). If the cluster name is incorrect, it will be added to the filtering list, which is not currently used. In addition, nodes added to the normal list will call establishNodeConnections() to establish connections for unconnected nodes, which is convenient for direct use when sending subsequent requests.
class SniffNodesSampler extends NodeSampler {
@Override
protected void doSample() {
...
@Override
protected void doRun() throws Exception {
...
transportService.sendRequest(pingConnection, ClusterStateAction.NAME,
Requests.clusterStateRequest().clear().nodes(true).local(true),
TransportRequestOptions.builder().withType(TransportRequestOptions.Type.STATE)
.withTimeout(pingTimeout).build(),
new TransportResponseHandler<ClusterStateResponse>() {
...
@Override
public void handleResponse(ClusterStateResponse response) {
clusterStateResponses.put(nodeToPing, response);
onDone();
}
...
}
HashSet<DiscoveryNode> newNodes = new HashSet<>();
HashSet<DiscoveryNode> newFilteredNodes = new HashSet<>();
for (Map.Entry<DiscoveryNode, ClusterStateResponse> entry : clusterStateResponses.entrySet()) {
if (!ignoreClusterName && !clusterName.equals(entry.getValue().getClusterName())) {
logger.warn("node {} not part of the cluster {}, ignoring...",
entry.getValue().getState().nodes().getLocalNode(), clusterName);
newFilteredNodes.add(entry.getKey());
continue;
}
for (ObjectCursor<DiscoveryNode> cursor : entry.getValue().getState().nodes().getDataNodes().values()) {
newNodes.add(cursor.value);
}
}
nodes = establishNodeConnections(newNodes);
filteredNodes = Collections.unmodifiableList(new ArrayList<>(newFilteredNodes));
}
}
The overall code of SniffNodesSampler is relatively long, so some less relevant codes are omitted here. In the overall process, SniffNodesSampler is similar to SimpleNodeSampler. However, when SniffNodesSampler sends a detection message, it sends a cluster state (ClusterStateAction) request, so the received response is the cluster state information, which includes all the node information in the cluster. In the subsequent for loop processing response information flow, SniffNodesSampler takes out all data type nodes and adds them to the available node candidate list, so subsequent users can directly connect to data type nodes.
3. Elasticsearch Clients request exception handling mechanism
3.1 Java REST Client request exception handling
The request processing entry and subsequent call process of Java REST Client are relatively intuitive, so I won’t go into details step by step. Let's directly look at the performRequestAsync() method of the RestClient class that processes the main logic of the request:
private void performRequestAsync(final long startTime, final NodeTuple<Iterator<Node>> nodeTuple, final HttpRequestBase request,
final Set<Integer> ignoreErrorCodes,
final HttpAsyncResponseConsumerFactory httpAsyncResponseConsumerFactory,
final FailureTrackingResponseListener listener) {
final Node node = nodeTuple.nodes.next();
//we stream the request body if the entity allows for it
final HttpAsyncRequestProducer requestProducer = HttpAsyncMethods.create(node.getHost(), request);
final HttpAsyncResponseConsumer<HttpResponse> asyncResponseConsumer =
httpAsyncResponseConsumerFactory.createHttpAsyncResponseConsumer();
final HttpClientContext context = HttpClientContext.create();
context.setAuthCache(nodeTuple.authCache);
client.execute(requestProducer, asyncResponseConsumer, context, new FutureCallback<HttpResponse>() {
@Override
public void completed(HttpResponse httpResponse) {
try {
RequestLogger.logResponse(logger, request, node.getHost(), httpResponse);
int statusCode = httpResponse.getStatusLine().getStatusCode();
Response response = new Response(request.getRequestLine(), node.getHost(), httpResponse);
if (isSuccessfulResponse(statusCode) || ignoreErrorCodes.contains(response.getStatusLine().getStatusCode())) {
onResponse(node);
if (strictDeprecationMode && response.hasWarnings()) {
listener.onDefinitiveFailure(new ResponseException(response));
} else {
listener.onSuccess(response);
}
} else {
ResponseException responseException = new ResponseException(response);
if (isRetryStatus(statusCode)) {
//mark host dead and retry against next one
onFailure(node);
retryIfPossible(responseException);
} else {
//mark host alive and don't retry, as the error should be a request problem
onResponse(node);
listener.onDefinitiveFailure(responseException);
}
}
} catch(Exception e) {
listener.onDefinitiveFailure(e);
}
}
@Override
public void failed(Exception failure) {
try {
RequestLogger.logFailedRequest(logger, request, node, failure);
onFailure(node);
retryIfPossible(failure);
} catch(Exception e) {
listener.onDefinitiveFailure(e);
}
}
private void retryIfPossible(Exception exception) {
if (nodeTuple.nodes.hasNext()) {
//in case we are retrying, check whether maxRetryTimeout has been reached
long timeElapsedMillis = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTime);
long timeout = maxRetryTimeoutMillis - timeElapsedMillis;
if (timeout <= 0) {
IOException retryTimeoutException = new IOException(
"request retries exceeded max retry timeout [" + maxRetryTimeoutMillis + "]");
listener.onDefinitiveFailure(retryTimeoutException);
} else {
listener.trackFailure(exception);
request.reset();
performRequestAsync(startTime, nodeTuple, request, ignoreErrorCodes, httpAsyncResponseConsumerFactory, listener);
}
} else {
listener.onDefinitiveFailure(exception);
}
}
@Override
public void cancelled() {
listener.onDefinitiveFailure(new ExecutionException("request was cancelled", null));
}
});
}
It can be seen that nodeTuple.nodes.next() first takes out the next node from the list, and then constructs and executes an Http asynchronous request. In the request response callback function, RestClient first judges the request response code. If the request response code is successful (isSuccessfulResponse(statusCode)) or in the ignore list specified by the user (ignoreErrorCodes.contains()), it indicates that the request is normal. Other request response codes indicate abnormal results. At this time, first judge whether it is a response code that needs to be retried (isRetryStatus(statusCode)), if not, return an error message to the user, and if so, perform a retry process: retryIfPossible(), and the logic of retryIfPossible() is also very Simple, except for the retry timeout judgment part, just call performRequestAsync() again in the method, and execute the above process again. The code of isRetryStatus is as follows:
private static boolean isRetryStatus(int statusCode) {
switch(statusCode) {
case 502:
case 503:
case 504:
return true;
}
return false;
}
Therefore, it can be seen from the above code analysis that in the Java REST Client, when sending a user request exception, as long as the server response code is not in the ignored response code list specified by the user and the response code is within the range of isRetryStatus (502/503/504), And if there is no retry timeout, RestClient will retry the request and keep retrying until all available nodes have retried.
Java API request exception handling
The entry point for executing requests in the Java API is the execute() method in AbstractClient.java:
/**
* This is the single execution point of *all* clients.
*/
@Override
public final <Request extends ActionRequest, Response extends ActionResponse, RequestBuilder extends ActionRequestBuilder<Request, Response, RequestBuilder>> void execute(
Action<Request, Response, RequestBuilder> action, Request request, ActionListener<Response> listener) {
listener = threadedWrapper.wrap(listener);
doExecute(action, request, listener);
}
Among them, doExecute() is an abstract method, which is implemented by the subclasses of AbstractClient (NodeClient / TransportClient, etc.). For TransportClient, in the specific implementation, it simply calls its proxy class TransportProxyClient object proxy to execute:
@Override
protected <Request extends ActionRequest, Response extends ActionResponse, RequestBuilder extends ActionRequestBuilder<Request, Response, RequestBuilder>> void doExecute(Action<Request, Response, RequestBuilder> action, Request request, ActionListener<Response> listener) {
proxy.execute(action, request, listener);
}
The specific code of the TransportProxyClient execute() method is as follows:
public <Request extends ActionRequest, Response extends ActionResponse, RequestBuilder extends
ActionRequestBuilder<Request, Response, RequestBuilder>> void execute(final Action<Request, Response, RequestBuilder> action,
final Request request, ActionListener<Response> listener) {
final TransportActionNodeProxy<Request, Response> proxy = proxies.get(action);
assert proxy != null : "no proxy found for action: " + action;
nodesService.execute((n, l) -> proxy.execute(n, request, l), listener);
}
Among them, the execute() of nodesService mainly does the preparatory work before the request, such as verifying the status of the connection node, obtaining the specific connection node, etc. The actual work of sending the request is performed in the proxy's execute(). One thing to note here is that the code that executes proxy.execute() is a Lambda expression, and its actual parameters are passed in nodesService.execute(), where the third parameter l is passed in as a RetryListener object . See the relevant code below:
public <Response> void execute(NodeListenerCallback<Response> callback, ActionListener<Response> listener) {
final List<DiscoveryNode> nodes = this.nodes;
if (closed) {
throw new IllegalStateException("transport client is closed");
}
ensureNodesAreAvailable(nodes);
int index = getNodeNumber();
RetryListener<Response> retryListener = new RetryListener<>(callback, listener, nodes, index, hostFailureListener);
DiscoveryNode node = retryListener.getNode(0);
try {
callback.doWithNode(node, retryListener);
} catch (Exception e) {
try {
//this exception can't come from the TransportService as it doesn't throw exception at all
listener.onFailure(e);
} finally {
retryListener.maybeNodeFailed(node, e);
}
}
}
In the Lambda expression proxy.execute(), the sendRequest of TransportService is called, and the last parameter is passed to ActionListenerResponseHandler:
public void execute(final DiscoveryNode node, final Request request, final ActionListener<Response> listener) {
ActionRequestValidationException validationException = request.validate();
if (validationException != null) {
listener.onFailure(validationException);
return;
}
transportService.sendRequest(node, action.name(), request, transportOptions,
new ActionListenerResponseHandler<>(listener, action::newResponse));
}
The sendRequest of TransportService finally calls sendRequestInternal(), and its calling relationship is bound in the TransportService object constructor:
...
this.asyncSender = interceptor.interceptSender(this::sendRequestInternal);
...
sendRequestInternal() finally sends the user request (some irrelevant code is omitted):
private <T extends TransportResponse> void sendRequestInternal(final Transport.Connection connection, final String action,
final TransportRequest request,
final TransportRequestOptions options,
TransportResponseHandler<T> handler) {
...
try {
...
connection.sendRequest(requestId, action, request, options); // local node optimization happens upstream
} catch (final Exception e) {
// usually happen either because we failed to connect to the node
// or because we failed serializing the message
...
@Override
protected void doRun() throws Exception {
contextToNotify.handler().handleException(sendRequestException);
}
});
...
}
Among them, contextToNotify.handler().handleException() is the place to handle the abnormal situation of request sending, so we only need to focus here. According to the previous analysis, the incoming handler is ActionListenerResponseHandler, so we look at the handleException() implementation in the ActionListenerResponseHandler class, which calls the listener's onFailure():
@Override
public void handleException(TransportException e) {
listener.onFailure(e);
}
As we have analyzed above, the listener passed to the third parameter of proxy execute is a RetryListener, and the onFailure code in RetryListener is as follows:
@Override
public void onFailure(Exception e) {
Throwable throwable = ExceptionsHelper.unwrapCause(e);
if (throwable instanceof ConnectTransportException) {
maybeNodeFailed(getNode(this.i), (ConnectTransportException) throwable);
int i = ++this.i;
if (i >= nodes.size()) {
listener.onFailure(new NoNodeAvailableException("None of the configured nodes were available: " + nodes, e));
} else {
try {
callback.doWithNode(getNode(i), this);
} catch(final Exception inner) {
inner.addSuppressed(e);
// this exception can't come from the TransportService as it doesn't throw exceptions at all
listener.onFailure(inner);
}
}
} else {
listener.onFailure(e);
}
}
Its main logic first determines whether the type of exception that occurs is ConnectTransportException. If not, call listener.onFailure(e) directly to return exception information to the user. If yes, execute as follows: maybeNodeFailed() (currently, es does not do any processing by default - initialize the lambda expression as an empty function); ++this.i means to increase the serial number and then take the next node, if i >= nodes .size indicates that all the nodes in nodes have been fetched once, and if there is still an error at this time, an exception will be thrown directly, indicating that no nodes are available. Otherwise, the request is sent again through the callback.doWithNode() call.
Therefore, it can be seen from the above code that in the Java API (TransportClient), when sending a user request exception, if the exception occurs is a node connection exception (ConnectTransportException), TransportClient will take the next node in order to retry and keep retrying Try until all available nodes have been tried. If an exception still occurs at this time, a warning of no available nodes will be issued. However, if the request exception is a non-node connection exception, TransportClient will not perform any retry or extra processing of the exception, and will directly return the information to the user through the callback interface.
4. Elasticsearch Clients client request load balancing mechanism
4.1 Java API
In the Java API (TransportClient), the es client load balancing strategy is similar to round robin (rr), which is implemented by an incremental serial number and modulo the number of nodes. send request node by node
private int getNodeNumber() {
int index = randomNodeGenerator.incrementAndGet();
if (index < 0) {
index = 0;
randomNodeGenerator.set(0);
}
return index;
}
final DiscoveryNode getNode(int i) {
return nodes.get((index + i) % nodes.size());
}
4.2 Java REST Client
The strategy is similar in the Java REST Client, but the implementation uses the rotate() method of Collections:
static Iterable<Node> selectNodes(NodeTuple<List<Node>> nodeTuple, Map<HttpHost, DeadHostState> blacklist,
AtomicInteger lastNodeIndex, NodeSelector nodeSelector) throws IOException {
...
if (false == livingNodes.isEmpty()) {
List<Node> selectedLivingNodes = new ArrayList<>(livingNodes);
nodeSelector.select(selectedLivingNodes);
if (false == selectedLivingNodes.isEmpty()) {
/*
* Rotate the list using a global counter as the distance so subsequent
* requests will try the nodes in a different order.
*/
Collections.rotate(selectedLivingNodes, lastNodeIndex.getAndIncrement());
return selectedLivingNodes;
}
}
}
5. Do Elasticsearch Clients use long connections?
As can be seen from the aforementioned liveness detection mechanism code, TransportClient will establish and maintain node connections in advance during liveness detection, so long connections are used.
Java REST Client uses the default CloseableHttpAsyncClient, you can see the build() code when it was created:
public CloseableHttpAsyncClient build() {
...
NHttpClientConnectionManager connManager = this.connManager;
if (connManager == null) {
...
final PoolingNHttpClientConnectionManager poolingmgr = new PoolingNHttpClientConnectionManager(
ioreactor,
RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", sslStrategy)
.build());
if (defaultConnectionConfig != null) {
poolingmgr.setDefaultConnectionConfig(defaultConnectionConfig);
}
if (systemProperties) {
String s = System.getProperty("http.keepAlive", "true");
if ("true".equalsIgnoreCase(s)) {
s = System.getProperty("http.maxConnections", "5");
final int max = Integer.parseInt(s);
poolingmgr.setDefaultMaxPerRoute(max);
poolingmgr.setMaxTotal(2 * max);
}
} else {
...
connManager = poolingmgr;
}
ConnectionReuseStrategy reuseStrategy = this.reuseStrategy;
if (reuseStrategy == null) {
if (systemProperties) {
final String s = System.getProperty("http.keepAlive", "true");
if ("true".equalsIgnoreCase(s)) {
reuseStrategy = DefaultConnectionReuseStrategy.INSTANCE;
} else {
reuseStrategy = NoConnectionReuseStrategy.INSTANCE;
}
} else {
reuseStrategy = DefaultConnectionReuseStrategy.INSTANCE;
}
}
ConnectionKeepAliveStrategy keepAliveStrategy = this.keepAliveStrategy;
if (keepAliveStrategy == null) {
keepAliveStrategy = DefaultConnectionKeepAliveStrategy.INSTANCE;
}
...
return new InternalHttpAsyncClient(
connManager,
reuseStrategy,
keepAliveStrategy,
...
}
It can be seen that the default connection manager is PoolingNHttpClientConnectionManager, and connection reuse and keepAlive are enabled by default, so long connections are also used.
6. Elasticsearch Clients sniff mechanism
By default, only the Java API (TransportClient) has a sniff mechanism, please refer to the code description in the TransportClient liveness detection section above. Its role can be used to automatically discover data type nodes and send client requests to data nodes at the same time in order to improve concurrency performance (usually clusters that have differentiated master/data/client role nodes are not encouraged to do so). Java REST Client needs to add additional dependencies to use sniff. By default, the sniff function is not provided, so it will not be described here.