java之httpClient 3.x、AsyncHttpClient1.9.x使用总结

首先请大牛们见谅菜鸟重复造轮子的学习方式,本文适合新手看~

下面使用的同步http是HttpClient 3.X 的版本,不过早已 不在维护 ,如果刚开始使用http,建议大家都换成 4.X 版本,别看下面的有关同步http的部分了,4.x效率有质地提高,总结3.X只是因为无奈旧项目还在使用。后面再更新一篇有关4.x的,最新的HttpClient 4.X官方地址: http://hc.apache.org/httpcomponents-client-4.5.x/index.html

但鉴于可能有些旧的系统还是采用3.X版本的HttpClient,所以本文还是先记录下使用方法。

相反下面的异步http是Async Http Client 的 1.9.8 版本,这个版本还是挺好的。API请见: http://asynchttpclient.github.io/async-http-client/apidocs/com/ning/http/client/AsyncHttpClient.html

http使用场景很多,据以往经验,对于客户端来说,我们使用http一般会发出以下几种常见的场景:

  1. 以get方式请求服务器
    1. 不带任何参数
    2. 带上key-value对
  2. 以post方式请求服务器
    1. 不带任何参数
    2. 带上key-value对
    3. 带上字节数组
    4. 带上文件
    5. 带上文件+key-value对

以上的场景一般可以满足一般的需求,然后,我们可以在这基础上扩展一点点:假如遇到一个类似于报表的子系统,主系统要在关键的逻辑链路中“打点”,通过http调用报表子系统记录一些相关的信息时,那么如果我们使用同步http来请求报表子系统的话,一旦报表子系统挂了,那么肯定会影响到主系统的运行。

为了不影响到主系统的运行,我们可以采用“ 异步 ” 的方式通过http(AsyncHttpClient )请求报表子系统,那么即使子系统挂了,对主系统的关键链路的执行也不会产生多大的影响。所以,封装一个http组件,自然而然少不了封装异步http请求。而异步http所能够做的事情,也应该覆盖上面提到的几种场景。

再者,考虑到效率问题,除非有足够的理由,否则每次调用http接口,都创建立一个新的连接,是相当没效率的,所以MultiThreadedHttpConnectionManager 诞生了,HttpClient在内部维护一个 连接池 ,通过MultiThreadedHttpConnectionManager 我们可以设置“默认连接数”、“最大连接数”、“连接超时”、“读取数据超时”等等配置,从而来提高效率。

废话完了,怎么实现以上需求呢。

包的引用:

同步的http我使用的是org.apache.commons.httpclient的HttpClient的3.1版本。

maven配置为:

?
1
2
3
4
5
6
<!-- httpClient -->
<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version> 3.1 </version>
</dependency>

异步的http我使用的是com.ning.http.client的AsyncHttpClien的1.9.8版本。 注意 ,其需要依赖几个日志相关的组件、分别为log4j、slf4j、slf4j-log4j

maven配置为:

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<!-- slf4j-log4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version> 1.7 . 7 </version>
</dependency>
<!-- slf4j -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version> 1.7 . 5 </version>
</dependency>
<!-- log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version> 1.2 . 16 </version>
</dependency>
<!-- 异步IO -->
<dependency>
<groupId>com.ning</groupId>
<artifactId>async-http-client</artifactId>
<version> 1.9 . 8 </version>
</dependency>

为了实现连接池,我们通过一个工厂类来生成httpClient,为了上一层方便调用,我们定义了一个接口,规范了同步、异步http应该实现的方法。包结构如下:

一、同步的HttpClient 3.X

从工厂入手,工厂负责初始化httpClient的配置,包括“默认连接数”、“最大连接数”、“连接超时”、“读取数据超时”等等,不同的服务我们应该创建不同的manager,因为不可能我们调服务A和调服务B使用同一套配置是吧,比如超时时间,应该考虑会有所差异。初始化完配置后,把 manager传到实现类,在实现类中new HttpClient。

工厂代码如下:

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// 专门针对xx服务器的连接管理对象
   // 因为不同服务可能超时等参数不用,所以针对不同服务,把连接管理对象区分开来,这只是其中一个
   private static MultiThreadedHttpConnectionManager xxconnectionManager = new MultiThreadedHttpConnectionManager();
   static {
     // 专门针对xx服务器的连接参数
     xxconnectionManager = new MultiThreadedHttpConnectionManager();
     HttpConnectionManagerParams paramsSearch = new HttpConnectionManagerParams();
     paramsSearch.setDefaultMaxConnectionsPerHost( 1000 ); // 默认连接数
     paramsSearch.setMaxTotalConnections( 1000 );          // 最大连接数
     paramsSearch.setConnectionTimeout( 30000 );            // 连接超时
     paramsSearch.setSoTimeout( 20000 );                    // 读数据超时
     xxconnectionManager.setParams(paramsSearch);
   }
   /*
    * 返回针对XX服务的httpClient包装类
    */
   public static SyncHttpClientWapperImpl getXXSearchHttpClient() {
     return new SyncHttpClientWapperImpl(xxconnectionManager);
   }

注意 一点,这些连接数,超时等的配置,要做要调查工作之后再定夺,是根据访问服务的不同,我们自己的机器能有多少剩余的可用空间的不同而不同的,而不是随随便便就设置一个参数。

实现类的构造方法如下:

?
1
2
3
4
5
6
7
8
private HttpClient client; // httpClient
   private final static String CHARACTER  = "UTF-8" ;
   // 构造器,由工厂调用
   public SyncHttpClientWapperImpl(MultiThreadedHttpConnectionManager connectionManager) {
     client = new HttpClient(connectionManager);
     // 字符集
     client.getParams().setParameter(HttpMethodParams.HTTP_CONTENT_CHARSET, CHARACTER);
   }

这里有一个 挺困惑 的点:HttpClient有必要弄成静态的吗?即直接在工厂里面为每种服务生成一个静态的HttpClient,然后传到实现类?经测试,改成静态的效率并没有提高,在文件传输的测试中,甚至下降了,这个有点困惑,大家可以试一试一起讨论一下。

然后,在实现类中实现各种方法。

第一种,通过URL,以get方式请求服务器,返回字节数组。

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
public byte [] getWithQueryURL(String queryURL) throws HttpClientException {
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL is null." );
     }
     byte [] newbuf = executeByGet(queryURL);
     if ((newbuf == null ) || (newbuf.length == 0 )) {
       throw new HttpClientException( "Server response is null: " + queryURL);
     }
     return newbuf;
   }
private byte [] executeByGet(String url) throws HttpClientException {
     HttpMethod method = new GetMethod(url);
     // RequestHeader
       method.setRequestHeader( "Content-type" , "text/html; charset=UTF-8" );
     // 提交请求
     try {
       client.executeMethod(method);
     } catch (Exception e) {
       method.releaseConnection();
       throw new HttpClientException(url, e);
     }
     // 返回字节流
     byte [] responseBody = null ;
     try {
       responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
     } catch (IOException e) {
       throw new HttpClientException(e);
     } finally {
       method.releaseConnection();
     }
     return responseBody;
   }

接着,写一个通用的流解析方法,负责把返回的流解析成字节数组。
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
private byte [] getBytesFromInpuStream(InputStream instream) throws IOException {
   ByteArrayOutputStream outstream = new ByteArrayOutputStream();
   try {
     int length;
     byte [] tmp = new byte [ 8096 ];
     while ((length = instream.read(tmp)) != - 1 ) {
       outstream.write(tmp, 0 , length);
     }
     return outstream.toByteArray();
   } finally {
     instream.close();
     outstream.close();
   }
}

这样就完成了最简单的get请求的调用了。

第二种:通过URL和paramsMap参数,以post方式请求服务器,返回字节数组。

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
public byte [] postWithParamsMap( String queryURL, Map<String,String> paramsMap) throws HttpClientException{
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL is null." );
     }
     byte [] newbuf = executeByPostWithParamsMap(queryURL,paramsMap);
     if ((newbuf == null ) || (newbuf.length == 0 )) {
       throw new HttpClientException( "Server response is null: " + queryURL);
     }
     return newbuf;
   }
private byte [] executeByPostWithParamsMap(String URL, Map<String,String> paramsMap)  throws HttpClientException {
     PostMethod method = new PostMethod(URL);
     // 构造参数
     if (paramsMap != null ) {
       Set<Entry<String, String>> entrySet = paramsMap.entrySet();
       Iterator<Entry<String, String>> iterator = entrySet.iterator();
       NameValuePair[] nvps = new NameValuePair[paramsMap.size()];
       int i = 0 ;
       while (iterator.hasNext()) {
         Entry<String, String> entry = iterator.next();
         if (entry.getKey() != null ) {
           NameValuePair nvp = new NameValuePair(entry.getKey(),entry.getValue());
           nvps[i++] = nvp;
         }
       }
       method.setRequestBody(nvps);
     }
     // RequestHeader,key-value对的话,httpClient自动带上application/x-www-form-urlencoded
     method.setRequestHeader( "Content-type" , "application/x-www-form-urlencoded; charset=UTF-8" );
     // 提交请求
     try {
       client.executeMethod(method);
     } catch (Exception e) {
       method.releaseConnection();
       throw new HttpClientException(URL, e);
     }
     // 返回字节流
     byte [] responseBody = null ;
     try {
       responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
     } catch (IOException e) {
       throw new HttpClientException(e);
     } finally {
       method.releaseConnection();
     }
     return responseBody;
   }

第三种:通过URL和bytes参数,以post方式请求服务器,返回字节数组。
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
public byte [] postWithBytes(String queryURL , byte [] bytes) throws HttpClientException{
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL is null." );
     }
     byte [] newbuf = executeByPostWithBytes(queryURL,bytes);
     if ((newbuf == null ) || (newbuf.length == 0 )) {
       throw new HttpClientException( "Server response is null: " + queryURL);
     }
     return newbuf;
   }
private byte [] executeByPostWithBytes(String queryURL, byte [] bytes) throws HttpClientException {
     PostMethod method = new PostMethod(queryURL);
     RequestEntity requestEntity = new ByteArrayRequestEntity(bytes);
     method.setRequestEntity(requestEntity);
     // RequestHeader
     method.setRequestHeader( "Content-type" , "text/plain; charset=UTF-8" );
     // 提交请求
     try {
       client.executeMethod(method);
     } catch (Exception e) {
       method.releaseConnection();
       throw new HttpClientException(queryURL, e);
     }
     // 返回字节流
     byte [] responseBody = null ;
     try {
       responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
     } catch (IOException e) {
       throw new HttpClientException(e);
     } finally {
       method.releaseConnection();
     }
     return responseBody;
   }

第四种:通过URL、fileList、paramMap参数,以post方式请求服务器,返回字节数组。
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
public byte [] postWithFileListAndParamMap(String queryURL,List<File> fileList,Map<String,String> paramMap) throws HttpClientException, HttpException, IOException {
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL is null." );
     }
     if (fileList == null ) {
       throw new HttpClientException( "file is null." );
     }
     if (paramMap == null ){
       throw new HttpClientException( "paramMap is null." );
     }
     return executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
   }
private byte [] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramMap) throws HttpException, IOException, HttpClientException {
     if (queryURL != null && fileList != null && fileList.size() > 0 ) {
       // post方法
       PostMethod method = new PostMethod(queryURL);
       // Part[]
       Part[] parts = null ;
       if (paramMap != null ) {
         parts = new Part[fileList.size()+paramMap.size()];
       }
       else {
         parts = new Part[fileList.size()];
       }
       int i = 0 ;
       // FilePart
       for (File file : fileList){
         Part filePart = new FilePart(file.getName(),file);
         parts[i++] = filePart;
       }
       // StringPart
       if (paramMap != null ) {
         Set<Entry<String, String>> entrySet = paramMap.entrySet();
         Iterator<Entry<String, String>> it = entrySet.iterator();
         while (it.hasNext()) {
           Entry<String, String> entry = it.next();
           Part stringPart = new StringPart(entry.getKey(),entry.getValue());
           parts[i++] = stringPart;
         }
       }
       // Entity
       RequestEntity requestEntity = new MultipartRequestEntity(parts, method.getParams());
       method.setRequestEntity(requestEntity);
       // RequestHeader,文件的话,HttpClient自动加上multipart/form-data
//          method.setRequestHeader("Content-type" , "multipart/form-data; charset=UTF-8");
       // excute
       try {
         client.executeMethod(method);
       } catch (Exception e) {
         method.releaseConnection();
         throw new HttpClientException(queryURL, e);
       }
       // return 
       byte [] responseBody = null ;
       try {
         responseBody = getBytesFromInpuStream(method.getResponseBodyAsStream());
       } catch (IOException e) {
         throw new HttpClientException(e);
       } finally {
         method.releaseConnection();
       }
       return responseBody;
     }
     return null ;
   }

二、异步的AsyncHttpClient

同样的,按照这种思路,异步的AsyncHttpClient也有类似的实现,不过写法不同而已,在工厂中,AsyncHttpClient使用的是AsyncHttpClientConfig.Builder作为管理配置的类,也有类似连接超时,最大连接数等配置。

工厂类:

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
// 专门针对xx服务器的连接管理对象
   // 因为不同服务可能超时等参数不用,所以针对不同服务,把连接管理对象区分开来,这只是其中一个
   private static AsyncHttpClientConfig.Builder xxbuilder = new AsyncHttpClientConfig.Builder();
   static {
     xxbuilder.setConnectTimeout( 3000 );  // 连接超时
     xxbuilder.setReadTimeout( 2000 );     // 读取数据超时
     xxbuilder.setMaxConnections( 1000 );  // 最大连接数
   }
   /*
    * 返回针对XX服务的httpClient包装类
    */
   public static AsyncHttpClientWapperImpl getXXSearchHttpClient() {
     return new AsyncHttpClientWapperImpl(xxbuilder);
   }

其使用了builder 的设计模式,活生生的一个例子,值得学习。

实现类的构造方法:

?
1
2
3
4
5
private AsyncHttpClient client;
      
     public AsyncHttpClientWapperImpl(Builder xxbuilder) {
         client = new AsyncHttpClient(xxbuilder.build());
     }

这样,AsyncHttpClient对象就创建完毕了。接下来是各种场景的实现,感觉异步的AsyncHttpClient封装得比HttpClient 3.X更加容易使用,设计得更好。

第一种:通过URL,以get方式请求服务器,返回字节数组。

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
public byte [] getWithQueryURL(String queryURL)
     throws HttpClientException, HttpClientException {
   if (queryURL == null ) {
     throw new HttpClientException( "queryURL为空" );
   }
   byte [] newbuf = executeByGet(queryURL);
   if ((newbuf == null ) || (newbuf.length == 0 )) {
     throw new HttpClientException( "Server response is null: " + queryURL);
   }
   return newbuf;
}
private byte [] executeByGet(String queryURL) throws HttpClientException {
    byte [] responseBody = null ;
   try {
     Future<Response> f = client.prepareGet(queryURL).execute();  
     responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
   } catch (Exception e) {
     throw new HttpClientException(e);
   }
   return responseBody;
}

同样的,我们写了一个getBytesFromInputStream()方法解析服务端返回的流,我们发现,两个实现类里面都有一些共同的方法,这里可以考虑写一个父类,把这些方法提取出来。

第二种:通过URL和paramsMap参数,以post方式请求服务器,返回字节数组。

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
public byte [] postWithParamsMap(String queryURL,
       Map<String, String> paramsMap) throws HttpClientException {
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL为空" );
     }
     byte [] newbuf = executeByPostByParamMap(queryURL,paramsMap);
     if ((newbuf == null ) || (newbuf.length == 0 )) {
       throw new HttpClientException( "Server response is null: " + queryURL);
     }
     return newbuf;
   }
private byte [] executeByPostByParamMap(String queryURL,Map<String,String> paramsMap) throws HttpClientException {
     byte [] responseBody = null ;
     try {
       RequestBuilder requestBuilder = new RequestBuilder();
       // 添加 key-value参数
       if (paramsMap != null && paramsMap.size() > 0 ) {
         Set<Entry<String, String>> entrySet = paramsMap.entrySet();
         Iterator<Entry<String, String>> iterator = entrySet.iterator();
         while (iterator.hasNext()) {
           Entry<String, String> entry = iterator.next();
           if (entry.getKey() != null ) {
             requestBuilder.addFormParam(entry.getKey(), entry.getValue());
           }
         }
       }
       // 添加RequestHeader,key
       requestBuilder.addHeader( "Content-type" , "application/x-www-form-urlencoded; charset=UTF-8" );
       requestBuilder.setMethod( "POST" );
       // 添加URL
       requestBuilder.setUrl(queryURL);
       // request
       Request request = requestBuilder.build();
       // 提交
       ListenableFuture<Response> f = client.executeRequest(request);
       responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
     } catch (Exception e) {
       throw new HttpClientException(e);
     }
     return responseBody;
   }

第三种:通过URL和bytes参数,以post方式请求服务器,返回字节数组。
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
public byte [] postWithBytes(String queryURL, byte [] bytes)
   throws HttpClientException {
         if (queryURL == null ) {
   throw new HttpClientException( "queryURL is null." );
         }
         byte [] newbuf = executeByPostWithBytes(queryURL,bytes);
         if ((newbuf == null ) || (newbuf.length == 0 )) {
   throw new HttpClientException( "Server response is null: " + queryURL);
         }
         return newbuf;
     }
private byte [] executeByPostWithBytes(String queryURL, byte [] bytes) throws HttpClientException {
         byte [] responseBody = null ;
         try {
   RequestBuilder requestBuilder = new RequestBuilder();
   // 添加 bytes参数
   requestBuilder.setBody(bytes);
   // 添加RequestHeader,key
   requestBuilder.addHeader( "Content-type" , "text/plain; charset=UTF-8" );
   requestBuilder.setMethod( "POST" );
   // 添加URL
   requestBuilder.setUrl(queryURL);
   // request
   Request request = requestBuilder.build();
   // 提交
   ListenableFuture<Response> f = client.executeRequest(request);
   responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
         } catch (Exception e) {
   throw new HttpClientException(e);
         }
         return responseBody;
     }

第四种:通过URL、fileList、paramMap参数,以post方式请求服务器,返回字节数组。
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
public byte [] postWithFileListAndParamMap(String queryURL,
       List<File> fileList, Map<String, String> paramMap)
       throws HttpClientException, HttpException, IOException {
     if (queryURL == null ) {
       throw new HttpClientException( "queryURL is null." );
     }
     if (fileList == null || fileList.size() == 0 ) {
       throw new HttpClientException( "fileList is null." );
     }
     if (paramMap == null || paramMap.size() == 0 ) {
       throw new HttpClientException( "paramMap is null." );
     }
     return executeByPostWithFileListAndParamMap(queryURL, fileList, paramMap);
   }
private byte [] executeByPostWithFileListAndParamMap (String queryURL,List<File> fileList,Map<String,String> paramsMap) throws HttpException, IOException, HttpClientException {
     if (queryURL != null && fileList != null && fileList.size() > 0 ) {
       byte [] responseBody = null ;
       try {
         RequestBuilder requestBuilder = new RequestBuilder();
         // FilePart
         for (File file : fileList){
           Part filePart = new FilePart(file.getName(),file);
           requestBuilder.addBodyPart(filePart);
         }
         // StringPart
         if (paramsMap != null ) {
           Set<Entry<String, String>> entrySet = paramsMap.entrySet();
           Iterator<Entry<String, String>> it = entrySet.iterator();
           while (it.hasNext()) {
             Entry<String, String> entry = it.next();
             Part stringPart = new StringPart(entry.getKey(),entry.getValue());
             requestBuilder.addBodyPart(stringPart);
           }
         }
         // 添加RequestHeader,key
         requestBuilder.addHeader( "Content-type" , "multipart/form-data; charset=UTF-8" );
         requestBuilder.setMethod( "POST" );
         // 添加URL
         requestBuilder.setUrl(queryURL);
         // request
         Request request = requestBuilder.build();
         // 提交
         ListenableFuture<Response> f = client.executeRequest(request);
         responseBody = getBytesFromInpuStream(f.get().getResponseBodyAsStream());
       } catch (Exception e) {
         throw new HttpClientException(e);
       }
       return responseBody;
     }
     return null ;
   }

OK,入了个门后,更多的用法可以自己去看文档了,请不要局限以上几种常用的场景。

猜你喜欢

转载自blog.csdn.net/xuanyakushu/article/details/50491966