There was an old line application, when the traffic growth, HttpClient throws BindException. StackTrace information part as follows:
java.net.BindException: Address already in use (Bind failed) at
java.net.PlainSocketImpl.socketBind(Native Method) ~[?:1.8.0_162] at
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) ~[?:1.8.0_162] at
java.net.Socket.bind(Socket.java:644) ~[?:1.8.0_162] at
sun.reflect.GeneratedMethodAccessor289.invoke(Unknown Source) ~[?:?] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_162] at
java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_162] at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:139) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) ~[commons-httpclient-3.1.jar:?] at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) ~[commons-httpclient-3.1.jar:?]`Ephemeral Port Exhausted
First Google, many people say is the operating system of temporary port exhausted. The idea is plausible, online services no connection pools, a large flow, HttpClient each create a connection will take up a temporary port number.
But I still have questions.
He said before the question briefly introduce temporary port number (Ephemeral Port).
A group consisting of a quaternary TCP connection identification:
{source_ip, source_port, destination_ip, destination_port}For HttpClient it, every time created as a source TCP connection, that destination_ip and destination_port is certain, just call the system calls connect, the operating system will automatically assign source_ip and source_port.
This allocation process is not only user HttpClient do not care, HttpClient developers do not care.
But temporary port to the operating system is a limited resource, there is a range of restrictions, too many simultaneous connections created, not good enough. Re-create a connection, it will error.
For example, following this nginx log, because the temporary port number is exhausted, Nginx can not create a connection to upstream of the:
2016/03/18 09:08:37 [crit] 1888#1888: *13 connect() to 10.2.2.77:8081 failed (99: Cannot assign requested address) while connecting to upstream, client: 10.2.2.42, server: , request: "GET / HTTP/1.1", upstream: "http://10.2.2.77:8081/", host: "10.2.2.77"This time my question here.
If the cause is temporary port number is exhausted, HttpClient BindException why throw it? As the founder of this party source TCP connection, only system calls connect, no need to bind system call ah.
If the cause is temporary port number is exhausted, like the above nginx log error that is reasonable, right?
HttpClient 3.1
Guess guess, give it up, but to go and see HttpClient code.
Application of the old old, older than the version with the tripartite library is also old. HttpClient or commons-httpclient-3.1.jar.
package org.apache.commons.httpclient.protocol;
public final class ReflectionSocketFactory:
public static Socket createSocket(
final String socketfactoryName,
final String host,
final int port,
final InetAddress localAddress,
final int localPort,
int timeout)
throws IOException, UnknownHostException, ConnectTimeoutException
{
if (REFLECTION_FAILED) {
//This is known to have failed before. Do not try it again
return null;
}
// This code uses reflection to essentially do the following:
//
// SocketFactory socketFactory = Class.forName(socketfactoryName).getDefault();
// Socket socket = socketFactory.createSocket();
// SocketAddress localaddr = new InetSocketAddress(localAddress, localPort);
// SocketAddress remoteaddr = new InetSocketAddress(host, port);
// socket.bind(localaddr);
// socket.connect(remoteaddr, timeout);
// return socket;
try {
Class socketfactoryClass = Class.forName(socketfactoryName);
Method method = socketfactoryClass.getMethod("getDefault",
new Class[] {});
Object socketfactory = method.invoke(null,
new Object[] {});
method = socketfactoryClass.getMethod("createSocket",
new Class[] {});
Socket socket = (Socket) method.invoke(socketfactory, new Object[] {});
if (INETSOCKETADDRESS_CONSTRUCTOR == null) {
Class addressClass = Class.forName("java.net.InetSocketAddress");
INETSOCKETADDRESS_CONSTRUCTOR = addressClass.getConstructor(
new Class[] { InetAddress.class, Integer.TYPE });
}
Object remoteaddr = INETSOCKETADDRESS_CONSTRUCTOR.newInstance(
new Object[] { InetAddress.getByName(host), new Integer(port)});
Object localaddr = INETSOCKETADDRESS_CONSTRUCTOR.newInstance(
new Object[] { localAddress, new Integer(localPort)});
if (SOCKETCONNECT_METHOD == null) {
SOCKETCONNECT_METHOD = Socket.class.getMethod("connect",
new Class[] {Class.forName("java.net.SocketAddress"), Integer.TYPE});
}
if (SOCKETBIND_METHOD == null) {
SOCKETBIND_METHOD = Socket.class.getMethod("bind",
new Class[] {Class.forName("java.net.SocketAddress")});
}
SOCKETBIND_METHOD.invoke(socket, new Object[] { localaddr});
SOCKETCONNECT_METHOD.invoke(socket, new Object[] { remoteaddr, new Integer(timeout)});
return socket;
}
catch (InvocationTargetException e) {
Throwable cause = e.getTargetException();
if (SOCKETTIMEOUTEXCEPTION_CLASS == null) {
try {
SOCKETTIMEOUTEXCEPTION_CLASS = Class.forName("java.net.SocketTimeoutException");
} catch (ClassNotFoundException ex) {
// At this point this should never happen. Really.
REFLECTION_FAILED = true;
return null;
}
}
if (SOCKETTIMEOUTEXCEPTION_CLASS.isInstance(cause)) {
throw new ConnectTimeoutException(
"The host did not accept the connection within timeout of "
+ timeout + " ms", cause);
}
if (cause instanceof IOException) {
throw (IOException)cause;
}
return null;
}
catch (Exception e) {
REFLECTION_FAILED = true;
return null;
}
}Focus is on these two sentences:
SOCKETBIND_METHOD.invoke(socket, new Object[] { localaddr});
SOCKETCONNECT_METHOD.invoke(socket, new Object[] { remoteaddr, new Integer(timeout)});HttpClient before the connect call bind, bind system call returned EADDRINUSE error:
EADDRINUSE
The given address is already in use.Then java.net.PlainSocketImpl.socketBind (Native Method) throws BindException.
In this case, indeed, is a temporary port number is exhausted, leading throws BindException, because of HttpClient prior to connect, first call the bind.
But, why did you bind it?
Bind before Connect
Prior to connect to bind, it is allowed, but there is no advantage, but has brought great harm.
Well, in fact, there may be little advantage in a particular case, where the first say harm, say the benefits later.
Said earlier, temporary port number is a limited resource, the number is limited. And the TCP connection four-tuples:
{source_ip, source_port, destination_ip, destination_port}If we called directly connect, to allocate temporary port by the operating system:
connect(socket, destination_addr, sizeof destination_addr);Then the operating system on a different destination_ip and destination_port, respectively maintains a temporary port number assigned.
Assuming that the number of temporary port number is N, then each combination of destination_ip and destination_port, can create N connections.
And if before the first call to connect bind:
bind(socket, source_addr, sizeof source_addr);
connect(socket, destination_addr, sizeof destination_addr);It has not yet been released source_port bind will not allow bind. Temporary port becomes a shared resource among different destination.
Assuming that the number of temporary port number is N, then the combination of all destination_ip and destination_port add up to a total of N can only create connections.
The reaction to HttpClient and java application, for example in terms of:
If your java application, it is necessary to use HttpClient access Baidu, but also to use HttpClient to access Google, but also to use HttpClient access Bing. Your operating system limit the number of temporary port number is 10000.
Less direct connect, Baidu, Google, Bing can exist 10,000 simultaneous connections, and no mutual influence between.
After the first bind connect, Baidu, Google, Bing can only create a combined total of 10,000 connections, and mutual influence between the need to connect the flow Baidu's big, multi-connection limit is exceeded, the need to connect Google and Bing will failure.
HttpClient 4.4
See here, the reason has been clear. Then we find whether there is a relatively new HttpCliet improved version of view. The following are HttpClient 4.4 connection to create the relevant code:
package org.apache.http.impl.pool;
public class BasicConnFactory implements ConnFactory<HttpHost, HttpClientConnection>:
@Override
public HttpClientConnection create(final HttpHost host) throws IOException {
final String scheme = host.getSchemeName();
Socket socket = null;
if ("http".equalsIgnoreCase(scheme)) {
socket = this.plainfactory != null ? this.plainfactory.createSocket() :
new Socket();
} if ("https".equalsIgnoreCase(scheme)) {
socket = (this.sslfactory != null ? this.sslfactory :
SSLSocketFactory.getDefault()).createSocket();
}
if (socket == null) {
throw new IOException(scheme + " scheme is not supported");
}
final String hostname = host.getHostName();
int port = host.getPort();
if (port == -1) {
if (host.getSchemeName().equalsIgnoreCase("http")) {
port = 80;
} else if (host.getSchemeName().equalsIgnoreCase("https")) {
port = 443;
}
}
socket.setSoTimeout(this.sconfig.getSoTimeout());
if (this.sconfig.getSndBufSize() > 0) {
socket.setSendBufferSize(this.sconfig.getSndBufSize());
}
if (this.sconfig.getRcvBufSize() > 0) {
socket.setReceiveBufferSize(this.sconfig.getRcvBufSize());
}
socket.setTcpNoDelay(this.sconfig.isTcpNoDelay());
final int linger = this.sconfig.getSoLinger();
if (linger >= 0) {
socket.setSoLinger(true, linger);
}
socket.setKeepAlive(this.sconfig.isSoKeepAlive());
socket.connect(new InetSocketAddress(hostname, port), this.connectTimeout);
return this.connFactory.createConnection(socket);
}Sure enough, get rid of, not before connect to bind up. Direct call connect:
socket.connect(new InetSocketAddress(hostname, port), this.connectTimeout);Conditional release or to actively upgrade the various libraries ah.
Connection pooling, blown downgrade
Like the old application of this, there is no limit on the occupation of the three parties rely on the resources, nor blown downgrade. Or indeed a very extensive.
First, there must be a connection pool, connection multiplexing to enhance the efficiency, and can limit the number of connections, the client-side service is good. HttpClient natively supports connection pooling.
Further tripartite fusion dependence have degraded, when a relying party or problems related to the large flow rate, the Step down, the fuse is blown, the control will affect the possible minimum. Fuse downgrade can hystrix.
Linux Ephemeral Port Range
The question on the investigation, summarized under the temporary port-related knowledge. Because each operating system, introduced here linux.
Temporary port number range:
# sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768 61000We assume that the business logic processing is very fast network or, from the establishment of a connection to close within 1ms, then a temporary port number that was assigned to the next available, just waiting for TCP connection can TIME_WAIT state ends.
Duration defined kernel code TIME_WAIT state $ KERNEL / include / net / tcp.h of:
#define TCP_TIMEWAIT_LEN (60*HZ)The default values are all above most of the linux kernel.
You can see, the default port number for temporary total of 61000-32768 = 28232. After a port number is used, a minimum of 60 seconds to release.
That is, if fixed source_ip, destination_ip, destination_port, only create up to 28,232 connections per minute, per second (61000-32768) /60=470.5 months.
Hundreds, a very small value. For large flow of business, it is easy to go wrong. Not to mention the above HttpClient first bind and then connect.
If you want to change this situation, increase the number of simultaneous connections can be created. There are several ways:
- 调大net.ipv4.ip_local_port_range
This range can turn up, but the maximum is not more than 65,536, the minimum can not be more than 1234
Example, can be adjusted in such a way:
sysctl net.ipv4.ip_local_port_range="1235 65000"This operation is no risk can be appropriately adjusted large.
- Port allows quick reuse
Is still in the TIME_WAIT state allows TCP connection occupied by local port, the other TCP connection. The default is not allowed.
Net.ipv4.tcp_tw_reuse can be configured at the system level:
sysctl net.ipv4.tcp_tw_reuse=1SO_REUSEADDR can also set options for a specific socket.
But TIME_WAIT state itself is meaningful, to ensure the reliability of TCP connections. Allow reuse TIME_WAIT state port number associated with that connection, although the utilization of resources provided, but may also bring hidden problems difficult to troubleshoot and resolve, we need to carefully open the related configuration.
As man ip (7) said:
A TCP local socket address that has been bound is unavailable for some time after closing, unless the SO_REUSEADDR flag has been set. Care should be taken when using this flag as it makes TCP less reliable.
- Using multiple source_ip
This comparison Tricky embodiment, as described above, fixed source_ip, destination_ip, destination_port, a fixed number of temporary port number.
If there are multiple source_ip, then the number of available temporary port number can be doubled.
How to use it, you need to use a system call bind characteristics of. If a bind when the specified source_ip, but source_port is set to 0, and set IP_BIND_ADDRESS_NO_PORT option socket.
tcp sockets before binding to a specific source ip with port 0 if you're going to use the socket for connect() rather then listen() this allows the kernel to delay allocating the source port until connect() time at which point it is much cheaper
In such a bind when the system does not assign a port number, but wait until connect redistribution, but specified source_ip.
Want to use this program, it must first bind and then connect up. This is the previously mentioned, bind before connect possible benefits.
This program is not practical in most cases, only one server available ip, this program is not take. Even I can use up too much trouble.
Reference
https://idea.popcount.org/2014-04-03-bind-before-connect/
https://www.nginx.com/blog/overcoming-ephemeral-port-exhaustion-nginx-plus/
https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux
https://github.com/torvalds/linux/blob/4ba9920e5e9c0e16b5ed24292d45322907bb9035/net/ipv4/inet_connection_sock.c#L118