Use of httpClient: database connection pool API

1. Why if the connection pool?

• Reduce the frequent setup time HTTP connection overhead, reducing waste TCP connection socket communication server resources when establishing and released;
• Support a higher number of concurrent;

2, conventional API connection pool HttpClient

As used herein, it is the latest version of HttpClient4.5.3, so the content below is based on the written version.

• PoolingHttpClientConnectionManagerConnection pool management implementation class
  PoolingHttpClientConnectionManagerHttpClient achieve a connection pool class, implementsHttpClientConnectionManager sumConnPoolControlinterface.
• Construction method:PoolingHttpClientConnectionManager(): Constructor with no arguments, you can see the source code from the method calls getDefaultRegistry () to register the protocol http and https protocols.
• Common methods:
  public void setMaxTotal(int max); The method defined in ConnPoolControl interface, represents the maximum number of connections is provided max.
  public void setDefaultMaxPerRoute(int max): This method is also defined in ConnPoolControl interface, it represents the maximum number of connections per default route to max
  public void setMaxPerRoute(HttpRoute route,int max) Set maximum number of connections a specified route, this configuration will overwrite the value of a route setDefaultMaxPerRoute of.
• Common methods:
  static RequestConfig.Builder Custom (): static methods for constructing Builder object, and then set the parameters;
  int getConnectionRequestTimeout (): Get the connection pool obtained from the maximum time, in milliseconds;
  int getConnectTimeout () : Get Create longest connection, in milliseconds;
  int getSocketTimeout (): Get the maximum time for data transmission, in milliseconds;
• RequestConfig has an internal static class Builder, for conveniently constructed RequestConfig object and parameter setting request, such conventional methods have the following:
  public RequestConfig.Builder setConnectionRequestTimeout (connectionRequestTimeout int): Get the maximum time setting of the connection, in milliseconds from the connection pool; .
  public RequestConfig.Builder setConnectTimeout (int connectTimeout): Create a set maximum time of connection, in milliseconds;
  public RequestConfig.Builder setSocketTimeout (int socketTimeout): setting the maximum time for data transmission, in milliseconds;.
• HttpRequestRetryHandler request retry Interface
  boolean retryRequest (IOException exception, int executionCount , org.apache.http.protocol.HttpContext context): implements the interface must implement this method, a method determines if the IO occurred during execution of the abnormality, should retry, retry executionCount times.

3, single-threaded - connection pool manager using HTTP requests

• Create an HTTP connection pool management object cm, as shown

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();

• And set the maximum number of connections for each route and the maximum number of default connection parameter, as follows

//将最大连接数增加到200
 cm.setMaxTotal(200);
//将每个路由的默认最大连接数增加到20
 cm.setDefaultMaxPerRoute(20);

• HttpGet HttpPost analog transmission request or a request to create HttpClients Note that when the object needs to obtain from the connection pool, i.e. to set the connection pool object, when executed IO abnormal need handling, to implement the interface HttpRequestRetryHandler; Note that this processed in response to the request can not be closed HttpClient object, if closing the connection pool will be destroyed. HttpClients object creates objects shown:

CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .setRetryHandler(retryHandler(5)).build();

• Can open thread to monitor the connection pool idle connections, and clean up invalid connection, you can set your own frequency monitor thread
• Code

package com.liangpj.develop.httpclient;
import org.apache.http.HttpEntityEnclosingRequest;
import org.apache.http.HttpRequest;
import org.apache.http.NoHttpResponseException;
import org.apache.http.client.HttpRequestRetryHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.conn.ConnectTimeoutException;
import org.apache.http.conn.HttpClientConnectionManager;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.protocol.HttpContext;
import org.apache.http.util.EntityUtils;
import javax.net.ssl.SSLException;
import javax.net.ssl.SSLHandshakeException;
import java.io.IOException;
import java.io.InterruptedIOException;
import java.net.UnknownHostException;

/**
 * 单线程-使用连接池管理HTTP请求
 * @author: liangpengju
 * @version: 1.0
 */
public class HttpConnectManager {

    public static void main(String[] args) throws Exception {
        //创建HTTP的连接池管理对象
        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
        //将最大连接数增加到200
        connectionManager.setMaxTotal(200);
        //将每个路由的默认最大连接数增加到20
        connectionManager.setDefaultMaxPerRoute(20);
        //将http://www.baidu.com:80的最大连接增加到50
        //HttpHost httpHost = new HttpHost("http://www.baidu.com",80);
        //connectionManager.setMaxPerRoute(new HttpRoute(httpHost),50);

        //发起3次GET请求
        String url ="https://www.baidu.com/s?word=java";
        long start = System.currentTimeMillis();
        for (int i=0;i<100;i++){
            doGet(connectionManager,url);
        }
        long end = System.currentTimeMillis();
        System.out.println("consume -> " + (end - start));

        //清理无效连接
        new IdleConnectionEvictor(connectionManager).start();
    }

    /**
     * 请求重试处理
     * @param tryTimes 重试次数
     * @return
     */
    public static HttpRequestRetryHandler retryHandler(final int tryTimes){

        HttpRequestRetryHandler httpRequestRetryHandler = new HttpRequestRetryHandler() {
            @Override
            public boolean retryRequest(IOException exception, int executionCount, HttpContext context) {
                // 如果已经重试了n次，就放弃
                if (executionCount >= tryTimes) {
                    return false;
                }
                // 如果服务器丢掉了连接，那么就重试
                if (exception instanceof NoHttpResponseException) {
                    return true;
                }
                // 不要重试SSL握手异常
                if (exception instanceof SSLHandshakeException) {
                    return false;
                }
                // 超时
                if (exception instanceof InterruptedIOException) {
                    return false;
                }
                // 目标服务器不可达
                if (exception instanceof UnknownHostException) {
                    return true;
                }
                // 连接被拒绝
                if (exception instanceof ConnectTimeoutException) {
                    return false;
                }
                // SSL握手异常
                if (exception instanceof SSLException) {
                    return false;
                }
                HttpClientContext clientContext = HttpClientContext .adapt(context);
                HttpRequest request = clientContext.getRequest();
                // 如果请求是幂等的，就再次尝试
                if (!(request instanceof HttpEntityEnclosingRequest)) {
                    return true;
                }
                return false;
            }
        };
        return httpRequestRetryHandler;
    }

    /**
     * doGet
     * @param url 请求地址
     * @param connectionManager
     * @throws Exception
     */
    public static void doGet(HttpClientConnectionManager connectionManager,String url) throws Exception {
        //从连接池中获取client对象，多例
        CloseableHttpClient httpClient = HttpClients.custom()
                .setConnectionManager(connectionManager)
                .setRetryHandler(retryHandler(5)).build();

        // 创建http GET请求
        HttpGet httpGet = new HttpGet(url);
        // 构建请求配置信息
        RequestConfig config = RequestConfig.custom().setConnectTimeout(1000) // 创建连接的最长时间
                .setConnectionRequestTimeout(500) // 从连接池中获取到连接的最长时间
                .setSocketTimeout(10 * 1000) // 数据传输的最长时间10s
                .setStaleConnectionCheckEnabled(true) // 提交请求前测试连接是否可用
                .build();
        // 设置请求配置信息
        httpGet.setConfig(config);

        CloseableHttpResponse response = null;
        try {
            // 执行请求
            response = httpClient.execute(httpGet);
            // 判断返回状态是否为200
            if (response.getStatusLine().getStatusCode() == 200) {
                String content = EntityUtils.toString(response.getEntity(), "UTF-8");
                System.out.println("内容长度：" + content.length());
            }
        } finally {
            if (response != null) {
                response.close();
            }
            // 此处不能关闭httpClient，如果关闭httpClient，连接池也会销毁
            // httpClient.close();
        }
    }

    /**
     * 监听连接池中空闲连接，清理无效连接
     */
    public static class IdleConnectionEvictor extends Thread {

        private final HttpClientConnectionManager connectionManager;

        private volatile boolean shutdown;

        public IdleConnectionEvictor(HttpClientConnectionManager connectionManager) {
            this.connectionManager = connectionManager;
        }

        @Override
        public void run() {
            try {
                while (!shutdown) {
                    synchronized (this) {
                        //3s检查一次
                        wait(3000);
                        // 关闭失效的连接
                        connectionManager.closeExpiredConnections();
                    }
                }
            } catch (InterruptedException ex) {
                // 结束
                ex.printStackTrace();
            }
        }

        public void shutdown() {
            shutdown = true;
            synchronized (this) {
                notifyAll();
            }
        }
    }
}

4, multi-threaded connection pool manager -HttpClient example HTTP request

• Create an HTTP connection pool management object cm, as shown

PoolingHttpClientConnectionManager cm = new PoolingHttpClientConnectionManager();

• And set the maximum number of connections for each route and the maximum number of default connection parameter, as follows

//将最大连接数增加到200
 cm.setMaxTotal(200);
//将每个路由的默认最大连接数增加到20
 cm.setDefaultMaxPerRoute(20);

• HttpClients objects created and provided connection pool object, HttpClients create an object shown in the object:

CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(connectionManager).build()

• Thread class inheritance to achieve a Get request execution thread class GetThread, overloaded run () method, the request execution HttpGet
• To request URI address is defined as an array, a GetThread create a thread for each URI, and start all threads

package com.liangpj.develop.httpclient;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.protocol.HttpContext;
import java.io.IOException;

/**
 * 多线程-HttpClient连接池管理HTTP请求实例
 */
public class MultiThreadHttpConnManager {
    public static void main(String[] args) {
        //连接池对象
        PoolingHttpClientConnectionManager connectionManager = new PoolingHttpClientConnectionManager();
      //将最大连接数增加到200
        connectionManager.setMaxTotal(200);
        //将每个路由的默认最大连接数增加到20
        connectionManager.setDefaultMaxPerRoute(20);
        //HttpClient对象
        CloseableHttpClient httpClient = HttpClients.custom().setConnectionManager(connectionManager).build();
        //URIs to DoGet
        String[] urisToGet = {
                "https://www.baidu.com/s?word=java",
                "https://www.baidu.com/s?word=java",
                "https://www.baidu.com/s?word=java",
                "https://www.baidu.com/s?word=java"
        };
        //为每一个URI创建一个线程
        GetThread[] threads = new GetThread[urisToGet.length];
        for (int i=0;i<threads.length;i++){
            HttpGet httpGet = new HttpGet(urisToGet[i]);
            threads[i] = new GetThread(httpClient,httpGet);
        }
        //启动线程
        for (int j=0;j<threads.length;j++){
            threads[j].start();
        }
        //join 线程
        for(int k=0;k<threads.length;k++){
            try {
                threads[k].join();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }

    /**
     * 执行Get请求线程
     */
    public static class GetThread extends Thread{
        private final CloseableHttpClient httpClient;
        private final HttpContext context;
        private final HttpGet httpget;
        public GetThread(CloseableHttpClient httpClient, HttpGet httpget) {
            this.httpClient = httpClient;
            this.context = HttpClientContext.create();
            this.httpget = httpget;
        }
        @Override
        public void run() {
            try {
                CloseableHttpResponse response = httpClient.execute(httpget,context);
                try {
                    HttpEntity entity = response.getEntity();
                }finally {
                    response.close();
                }
            }catch (ClientProtocolException ex){
                //处理客户端协议异常
            }catch (IOException ex){
                //处理客户端IO异常
            }
        }
    }
}