Article Directory
foreword
HttpClient was established in 2001 as a sub-project under the Apache Jakarta Commons project. It left Commons in 2004and was promoted to become a separate Jakarta project. In 2005, Jakarta created the HttpComponents project with the goal of developingthe successor to HttpClient 3.x. In 2007, the Commons project, which is the birthplace of the HttpClient project, left Jakarta and became a new top-level project. Soon after, HttpComponents also left Jakarta and became an independent top-level project responsible for maintaining HttpClient .
-
HttpClient provides an efficient, up-to-date, feature-rich client programming toolkit that supports the HTTP protocol, and supports the latest version of the HTTP protocol.
-
The HttpComponents project includes three modules : HttpClient , HttpCore , and AsyncClient , which provide better performance and greater flexibility.
-
HttpClient is dependent on HttpCore , the latest version of HttpClient is 5.2
-
HttpClient is separated by version 3.1, and there are many differences in usage between major versions
-
Latest document address: https://hc.apache.org/httpcomponents-client-5.2.x/index.html
-
Old document address: https://hc.apache.org/httpclient-legacy/userguide.html
-
github address: https://github.com/apache/httpcomponents-client
-
pom dependencies
<!-- 最新版本5 --> <dependency> <groupId>org.apache.httpcomponents.client5</groupId> <artifactId>httpclient5</artifactId> <version>5.2.1</version> </dependency> <!-- 版本4 --> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.5.13</version> </dependency> <!-- 旧版本3，07年后没更新 --> <dependency> <groupId>commons-httpclient</groupId> <artifactId>commons-httpclient</artifactId> <version>3.1</version> </dependency>
1. Easy to use
1.1 get request
String url = "http://httpbin.org/get";
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
final HttpGet httpget = new HttpGet(url);
// Create a custom response handler
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
final String responseBody = httpclient.execute(httpget, responseHandler);
System.out.println(responseBody);
}
1.2 post simple form request
String url = "http://httpbin.org/post";
String username = "root";
String loginPw = "";
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
final HttpPost httppost = new HttpPost(url);
final List<NameValuePair> params = new ArrayList<>();
params.add(new BasicNameValuePair("username", username));
params.add(new BasicNameValuePair("password", loginPw));
httppost.setEntity(new UrlEncodedFormEntity(params));
// Create a custom response handler
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
final String responseBody = httpclient.execute(httppost, responseHandler);
System.out.println(responseBody);
}
1.3 Form upload file
final HttpPost httppost = new HttpPost(url);
final MultipartEntityBuilder builder = MultipartEntityBuilder.create();
builder.addTextBody("username", username);
builder.addTextBody("password", password);
builder.addBinaryBody("file", new File("src/test/resources/test.txt"), ContentType.APPLICATION_OCTET_STREAM, "test.txt");
final HttpEntity multipart = builder.build();
httppost.setEntity(multipart);
1.4 upload json data
final HttpPost httppost = new HttpPost(url);
httppost.setHeader("Accept", "application/json");
httppost.setHeader("Content-type", "application/json");
final String json = "{\"id\":1,\"name\":\"John\"}";
final StringEntity stringEntity = new StringEntity(json);
httppost.setEntity(stringEntity);
2. Advanced usage
2.1 Timeouts and retries
Timeout control can be controlled by the RequestConfig class
String url = "http://httpbin.org/get";
RequestConfig requestConfig = RequestConfig.custom()
.setConnectionRequestTimeout(Timeout.ofSeconds(100L))//连接请求超时, 0为无限。默认值：3分钟。
.setResponseTimeout(Timeout.ofSeconds(600L)) // 响应超时时间，0为无限。带有消息复用的HTTP传输可能不支持响应超时
.build();
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
final HttpGet httpGet = new HttpGet(url);
httpGet.setConfig(requestConfig);
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
httpclient.execute(httpGet, responseHandler);
}
Retry, the default retry policy is the maximum number of times 1, and the retry interval is 1 second.
String url = "http://httpbin.org/get";
try (final CloseableHttpClient httpclient = HttpClients.custom()
.setRetryStrategy(new DefaultHttpRequestRetryStrategy(3, TimeValue.ofSeconds(20L)))
.build()) {
final HttpGet httpGet = new HttpGet(url);
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
httpclient.execute(httpGet, responseHandler);
}
2.2 Cookie
HttpClients.createDefault has a built-in default cookie manager that can be used to carry cookies for access
String url = "http://httpbin.org/cookies/set/foo/bar";
String url2 = "http://httpbin.org/cookies";
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
final HttpGet httpGet = new HttpGet(url);
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
final HttpGet httpGet2 = new HttpGet(url2);
String responseBody2 = httpclient.execute(httpGet2, responseHandler);
System.out.println(responseBody2);
final String responseBody = httpclient.execute(httpGet, responseHandler);
System.out.println(responseBody);
responseBody2 = httpclient.execute(httpGet2, responseHandler);
System.out.println(responseBody2);
}
You can also access cookie information bound by the local context
String url = "http://httpbin.org/cookies/set/foo/bar";
try (final CloseableHttpClient httpclient = HttpClients.createDefault()) {
// 创建一个本地的 Cookie 存储
final CookieStore cookieStore = new BasicCookieStore();
final HttpClientContext localContext = HttpClientContext.create();
// 绑定 cookieStore 到 localContext
localContext.setCookieStore(cookieStore);
final HttpGet httpget = new HttpGet(url);
final HttpClientResponseHandler<String> responseHandler = response -> {
final int status = response.getCode();
if (status >= HttpStatus.SC_SUCCESS && status < HttpStatus.SC_REDIRECTION) {
final HttpEntity entity = response.getEntity();
try {
final List<Cookie> cookies = cookieStore.getCookies();
for (Cookie cookie : cookies) {
System.out.println("Local cookie: " + cookie);
}
return entity != null ? EntityUtils.toString(entity) : null;
} catch (final ParseException ex) {
throw new ClientProtocolException(ex);
}
} else {
throw new ClientProtocolException("Unexpected response status: " + status);
}
};
String response = httpclient.execute(httpget, localContext, responseHandler);
System.out.println(response);
}
2.3 Interceptors
Httpclient supports certain processing of requests through interceptors. There are several ways to add interceptors
- addRequestInterceptorFirst
- addRequestInterceptorLast
- addResponseInterceptorFirst
- addResponseInterceptorLast
- addExecInterceptorFirst
- addExecInterceptorLast
- addExecInterceptorBefore
- addExecInterceptorAfter
The added interceptors can be divided into 3 types: request, response and exec, corresponding to request, response and execution. The name of Exec execution is in the enumeration ChainElement. In the source code of the HttpClientBuilder class, it can be found that except CACHING can be used through configuration, and the order in the enumeration is also the order of Exec execution, where MAIN_TRANSPORT execution includes request and response interceptor execution
ChainElement defines a set of elements that can be used to build HTTP request processing pipelines, and each element can implement specific functions, such as adding custom HTTP headers, adding authentication information, etc.
public enum ChainElement {
REDIRECT, COMPRESS, BACK_OFF, RETRY, CACHING, PROTOCOL, CONNECT, MAIN_TRANSPORT
}
The following is a modified code for the official interceptor example
AtomicLong count = new AtomicLong();
try (final CloseableHttpClient httpclient = HttpClients.custom()
.addExecInterceptorAfter(ChainElement.PROTOCOL.name(), "custom", (request, scope, chain) -> {
request.setHeader("request-id", Long.toString(count.incrementAndGet()));
return chain.proceed(request, scope);
})
.addExecInterceptorAfter("custom", "quit3rd", ((request, scope, chain) -> {
final Header idHeader = request.getFirstHeader("request-id");
if (idHeader != null && "3".equalsIgnoreCase(idHeader.getValue())) {
final ClassicHttpResponse response = new BasicClassicHttpResponse(HttpStatus.SC_NOT_FOUND, "Oppsie");
response.setEntity(new StringEntity("bad luck", ContentType.TEXT_PLAIN));
return response;
} else {
return chain.proceed(request, scope);
}
}))
.addExecInterceptorBefore(ChainElement.CONNECT.name(), "AAA", (request, scope, chain) -> {
System.out.println("AAA");
return chain.proceed(request, scope);
})
.addExecInterceptorBefore("AAA", "BBB", (request, scope, chain) -> {
System.out.println("BBB");
return chain.proceed(request, scope);
})
.addExecInterceptorAfter("AAA", "CCC", (request, scope, chain) -> {
System.out.println("CCC");
return chain.proceed(request, scope);
})
.addRequestInterceptorFirst((request, entity, context) -> {
System.out.println("第一个request first现在获取：" + context.getAttribute("foo"));
})
.addRequestInterceptorFirst((request, entity, context) -> {
System.out.println("第二个request first, 现在设置name");
context.setAttribute("foo", "bar");
})
.addRequestInterceptorLast((request, entity, context) -> {
System.out.println("第一个request last现在获取：" + context.getAttribute("foo"));
})
.build()) {
for (int i = 0; i < 5; i++) {
final HttpGet httpget = new HttpGet("http://httpbin.org/get");
System.out.println("Executing request " + httpget.getMethod() + " " + httpget.getUri());
httpclient.execute(httpget, response -> {
System.out.println("----------------------------------------");
System.out.println(httpget + "->" + new StatusLine(response));
EntityUtils.consume(response.getEntity());
return null;
});
}
}
The animation below shows the sequence of the execChain execution chain during debugging
The following are the request and response interceptors in the debugging process. From the name, we can see that except the Main class is a custom interceptor, the rest are self-contained, and the cookie processing is also implemented through the interceptor.
2.4 fluent API
HttpClienet version 4.5 and above supports fluent API, the advantage is that the code is more concise and thread-safe.
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>fluent-hc</artifactId>
<version>4.5.13</version>
</dependency>
String urlGet = "http://httpbin.org/get";
String urlPost = "http://httpbin.org/post";
String response = Request.Get(urlGet)
.addHeader("Authorization", "Bear:dw")
.execute()
.handleResponse(httpResponse -> {
int code = httpResponse.getStatusLine().getStatusCode();
if (code == HttpStatus.SC_SUCCESS) {
return org.apache.http.util.EntityUtils.toString(httpResponse.getEntity());
}
return null;
});
System.out.println(response);
String result = Request.Post(urlPost)
.bodyForm(Form.form().add("foo", "bar").build())
.execute()
.returnContent()
.asString();
System.out.println(result);
3. Use of the old version 3.1
3.1 Get request
String url = "http://httpbin.com";
HttpClient client = new HttpClient();
GetMethod method = new GetMethod(url);
method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
new DefaultHttpMethodRetryHandler(3, false));
try {
// Execute the method.
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
// Read the response body.
byte[] responseBody = method.getResponseBody();
// Deal with the response.
// Use caution: ensure correct character encoding and is not binary data
System.out.println(new String(responseBody));
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
// Release the connection.
method.releaseConnection();
}
3.2 Post request
String url = "http://httpbin.org/post";
HttpClient client = new HttpClient();
PostMethod method = new PostMethod(url);
NameValuePair[] data = {
new NameValuePair("user", "joe"),
new NameValuePair("password", "bloggs")
};
method.setRequestBody(data);
try {
int statusCode = client.executeMethod(method);
if (statusCode != HttpStatus.SC_OK) {
System.err.println("Method failed: " + method.getStatusLine());
}
byte[] responseBody = method.getResponseBody();
System.out.println(new String(responseBody));
} catch (HttpException e) {
System.err.println("Fatal protocol violation: " + e.getMessage());
e.printStackTrace();
} catch (IOException e) {
System.err.println("Fatal transport error: " + e.getMessage());
e.printStackTrace();
} finally {
method.releaseConnection();
}
Fourth, the use of asynchronous version
4.1 Basic request
final IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
.setSoTimeout(Timeout.ofSeconds(5))
.build();
final CloseableHttpAsyncClient client = HttpAsyncClients.custom()
.setIOReactorConfig(ioReactorConfig)
.build();
client.start();
final HttpHost target = new HttpHost("httpbin.org");
final String[] requestUris = new String[] {
"/", "/ip", "/user-agent", "/headers"};
for (final String requestUri: requestUris) {
final SimpleHttpRequest request = SimpleRequestBuilder.get()
.setHttpHost(target)
.setPath(requestUri)
.build();
System.out.println("请求url:" + requestUri);
final Future<SimpleHttpResponse> future = client.execute(
SimpleRequestProducer.create(request),
SimpleResponseConsumer.create(),
new FutureCallback<SimpleHttpResponse>() {
@Override
public void completed(final SimpleHttpResponse response) {
System.out.println(requestUri + " 返回状态码：" + response.getCode() + ",返回内容:" + response.getBodyText());
}
@Override
public void failed(final Exception ex) {
System.out.println(request + "->" + ex);
}
@Override
public void cancelled() {
System.out.println(request + " cancelled");
}
});
future.get();
}
System.out.println("Shutting down");
client.close(CloseMode.GRACEFUL);
4.2 Request pipeline execution
final MinimalHttpAsyncClient client = HttpAsyncClients.createMinimal(
H2Config.DEFAULT,
Http1Config.DEFAULT,
IOReactorConfig.DEFAULT,
PoolingAsyncClientConnectionManagerBuilder.create()
.setDefaultTlsConfig(TlsConfig.custom()
.setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_1)
.build())
.build());
client.start();
final HttpHost target = new HttpHost("httpbin.org");
final Future<AsyncClientEndpoint> leaseFuture = client.lease(target, null);
final AsyncClientEndpoint endpoint = leaseFuture.get(30, TimeUnit.SECONDS);
try {
final String[] requestUris = new String[] {
"/", "/ip", "/user-agent", "/headers"};
final CountDownLatch latch = new CountDownLatch(requestUris.length);
for (final String requestUri: requestUris) {
final SimpleHttpRequest request = SimpleRequestBuilder.get()
.setHttpHost(target)
.setPath(requestUri)
.build();
System.out.println("Executing request " + request);
endpoint.execute(
SimpleRequestProducer.create(request),
SimpleResponseConsumer.create(),
new FutureCallback<SimpleHttpResponse>() {
@Override
public void completed(final SimpleHttpResponse response) {
latch.countDown();
System.out.println(request + "->" + new StatusLine(response));
System.out.println(response.getBody());
}
@Override
public void failed(final Exception ex) {
latch.countDown();
System.out.println(request + "->" + ex);
}
@Override
public void cancelled() {
latch.countDown();
System.out.println(request + " cancelled");
}
});
}
latch.await();
} finally {
endpoint.releaseAndReuse();
}
System.out.println("Shutting down");
client.close(CloseMode.GRACEFUL);