SpringCloud large-scale series of courses are under production, welcome your attention and comments.
This article is one in a series
- 1. SpringCloud project basic engineering construction [SpringCloud Series 1]
- 2. SpringCloud integrates Nacos registration center [SpringCloud Series 2]
- 3. SpringCloud Feign remote call [SpringCloud Series 3]
- 4. SpringCloud Feign remote call public class extraction [SpringCloud Series 4]
- 5. SpringCloud integrates Gateway service gateway [SpringCloud Series 5]
- 6. Spring Cloud integrates Spring Security authentication [Spring Cloud Series 6]
- Spring Cloud Gateway Gateway Authentication [Spring Cloud Series 7]
- SpringCloud Gateway implements current limiting through redis [SpringCloud Series 8]
- SpringCloud Gateway integrates Ali Sentinel gateway flow control, fuse, downgrade [SpringCloud Series 9]
- SpringCloud Gateway integrates RSA to decrypt request parameters and encrypt response results [SpringCloud Series 10]
- The SpringCloud business management background uses FeignClient to call the oauth/token interface [SpringCloud Series 11]
- Spring Security OAuth2.0 multi-sign-on and single-sign-on [SpringCloud series 12]
- SpringCloud WeChat Mini Program Login Solution 1 [SpringCloud Series 13]
- SpringCloud WeChat applet to obtain mobile phone number and one-click login [SpringCloud series 14]
- SpringCloud WeChat applet to obtain user information and one-click login [SpringCloud Series 15]
This article implements the asynchronous batch saving of request logs in the thread pool, and the realization of saving log data in the database
log filter added
The first is to add a log filter to the gateway service
@Log4j2
public class LogFilter implements GlobalFilter, Ordered {
private static final String START_TIME = "startTime";
private static final List<HttpMessageReader<?>> messageReaders = HandlerStrategies.withDefaults().messageReaders();
@Resource
VisitRecordService visitRecordService;
@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
ServerHttpRequest request = exchange.getRequest();
// 请求方法
HttpMethod method = request.getMethod();
// 请求头
HttpHeaders headers = request.getHeaders();
// 设置startTime 用来计算响应的时间
exchange.getAttributes().put(START_TIME, System.currentTimeMillis());
// 构建日志记录
AccessRecord accessRecord = visitRecordService.build(exchange);
if (method != null) {
//设置请求方法
accessRecord.setMethod(method.name());
if (method == HttpMethod.GET) {
//获取get请求参数
MultiValueMap<String, String> formData = request.getQueryParams();
if (!formData.isEmpty()) {
//保存请求参数
accessRecord.setFormData(JSON.toJSONString(formData));
}
} else if (method == HttpMethod.POST) {
MediaType contentType = headers.getContentType();
if (contentType != null) {
Mono<Void> voidMono = null;
if (contentType.equals(MediaType.APPLICATION_JSON)) {
// JSON
voidMono = readBody(exchange, chain, accessRecord);
}
if (voidMono != null) {
//计算请求时间
cacueConsumTime(exchange);
return voidMono;
}
}
}
}
visitRecordService.put(exchange, accessRecord);
// 请求后执行保存
return chain.filter(exchange).then(saveRecord(exchange));
}
private Mono<Void> saveRecord(ServerWebExchange exchange) {
return Mono.fromRunnable(() -> {
cacueConsumTime(exchange);
});
}
/**
* 计算访问时间
*
* @param exchange
*/
private void cacueConsumTime(ServerWebExchange exchange) {
//请求开始时设置的自定义属性标识
Long startTime = exchange.getAttribute(START_TIME);
Long consumingTime = 0L;
if (startTime != null) {
consumingTime = System.currentTimeMillis() - startTime;
log.info(exchange.getRequest().getURI().getRawPath() + ": 耗时 " + consumingTime + "ms");
}
visitRecordService.add(exchange, consumingTime);
}
private Mono<Void> readBody(ServerWebExchange exchange, GatewayFilterChain chain, AccessRecord accessRecord) {
return DataBufferUtils.join(exchange.getRequest().getBody()).flatMap(dataBuffer -> {
byte[] bytes = new byte[dataBuffer.readableByteCount()];
dataBuffer.read(bytes);
DataBufferUtils.release(dataBuffer);
Flux<DataBuffer> cachedFlux = Flux.defer(() -> {
DataBuffer buffer = exchange.getResponse().bufferFactory().wrap(bytes);
DataBufferUtils.retain(buffer);
return Mono.just(buffer);
});
// 重写请求体,因为请求体数据只能被消费一次
ServerHttpRequest mutatedRequest = new ServerHttpRequestDecorator(exchange.getRequest()) {
@Override
public Flux<DataBuffer> getBody() {
return cachedFlux;
}
};
ServerWebExchange mutatedExchange = exchange.mutate().request(mutatedRequest).build();
return ServerRequest.create(mutatedExchange, messageReaders)
.bodyToMono(String.class)
.doOnNext(objectValue -> {
accessRecord.setBody(objectValue);
visitRecordService.put(exchange, accessRecord);
}).then(
chain.filter(mutatedExchange)
);
});
}
@Override
public int getOrder() {
return 2;
}
}
HttpMessageReader is a class for reading HTTP messages.
The ServerRequest.create method creates a new ServerRequest object that represents an HTTP request.
AccessRecord is a custom data model used to save access logs, the code is as follows:
import com.baomidou.mybatisplus.annotation.TableName;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.io.Serializable;
import java.net.URI;
import java.time.LocalDateTime;
@Data
@AllArgsConstructor
@NoArgsConstructor
@TableName(value = "access_recoder_log")
public class AccessRecord implements Serializable {
private String formData;
private URI targetUri;
private String method;
private String scheme;
private String path;
private String body;
private String ip;
private Integer status;
private Long userId;
private Long consumingTime;
private LocalDateTime createTime;
}
VisitRecordService is the implementation class for asynchronously saving logs defined here
VisitRecordService saves logs asynchronously
ServerWebExchange is an interface in Spring WebFlux used to represent the exchange of HTTP requests and responses. It provides methods to access requests and responses, and methods to access request attributes and response attributes. It can be used to process HTTP requests and responses, such as modifying request headers or response bodies, or forwarding requests to another handler.
Obtain the corresponding ServerWebExchange in the filter method of the filter, and then read the access information from it
@Slf4j
@Service
public class VisitRecordService {
//自定义的一个标识
private final String attributeKey = "visitRecord";
/**
* 构建一个 VisitRecord 实体类，但仅适用于获取 request 信息
*
* @param exchange gateway访问
* @return 访问信息
*/
public AccessRecord build(ServerWebExchange exchange) {
// 获取请求信息
ServerHttpRequest request = exchange.getRequest();
String ip = RequestUtils.getIpAddress(request);
// 请求路径
String path = request.getPath().pathWithinApplication().value();
// 请求schema: http/https
String scheme = request.getURI().getScheme();
// 请求方法
HttpMethod method = request.getMethod();
// 路由服务地址
URI targetUri = exchange.getAttribute(ServerWebExchangeUtils.GATEWAY_REQUEST_URL_ATTR);
// 请求头
HttpHeaders headers = request.getHeaders();
// 获取请求地址
InetSocketAddress remoteAddress = request.getRemoteAddress();
AccessRecord accessRecord = new AccessRecord();
accessRecord.setPath(path);
accessRecord.setScheme(scheme);
accessRecord.setTargetUri(targetUri);
accessRecord.setIp(ip);
accessRecord.setCreateTime(LocalDateTime.now());
return accessRecord;
}
/**
* 将访问信息存入 ServerWebExchange 当中，将会与当前请求关联起来，
* 以便于后续在任何地方均可获得
*
* @param exchange gateway访问合同
* @param visitRecord 访问信息
*/
public void put(ServerWebExchange exchange, AccessRecord visitRecord) {
Map<String, Object> attributes = exchange.getAttributes();
attributes.put(attributeKey, visitRecord);
}
}
Then save the log at the end of the request
@Slf4j
@Service
public class VisitRecordService {
/**
* 缓存，在插入数据库前先存入此。
* 为防止数据被重复插入，故使用Set，但不能确保100%不会被重复存储。
*/
private HashSet<AccessRecord> visitCache = new HashSet<>();
/**
* 保存访问记录
*
* @param exchange gateway访问
* @param consumingTime 访问耗时
*/
public void add(ServerWebExchange exchange, Long consumingTime) {
// 获取数据
ServerHttpResponse response = exchange.getResponse();
ServerHttpRequest request = exchange.getRequest();
//获取保存的日志记录体
AccessRecord visitRecord = getOrBuild(exchange);
//设置访问时间 单位毫秒
visitRecord.setConsumingTime(consumingTime);
// 设置访问状态
if (response.getStatusCode() != null) {
visitRecord.setStatus(response.getStatusCode().value());
}
//设置访问的用户ID 我这里是保存在请求头中
String userId = request.getHeaders().getFirst("userId");
if(StringUtils.isNotEmpty(userId)) {
visitRecord.setUserId(Long.parseLong(userId));
}
// 打印访问情况
log.info(visitRecord.toString());
// 添加记录到缓存中
visitCache.add(visitRecord);
// 执行任务，保存数据
doTask();
}
}
doTask here is to use the thread pool to asynchronously execute log saving
/**
* 信号量，用于标记当前是否有任务正在执行，{@code true}表示当前无任务进行。
*/
private volatile boolean taskFinish = true;
private final ThreadPoolExecutor threadPool = new ThreadPoolExecutor(1, 3, 15, TimeUnit.SECONDS, new ArrayBlockingQueue<>(10), namedThreadFactory, new ThreadPoolExecutor.AbortPolicy());
/**
* 单次批量插入的数据量
*/
private final int BATCH_SIZE = 500;
private void doTask() {
if (taskFinish) {
// 当前没有任务的情况下，加锁并执行任务
synchronized (this) {
if (taskFinish) {
taskFinish = false;
threadPool.execute(() -> {
try {
// 当数据量较小时，则等待一段时间再插入数据，从而做到将数据尽可能的批量插入数据库
if (visitCache.size() <= BATCH_SIZE) {
Thread.sleep(500);
}
//批量保存
batchSave();
} catch (InterruptedException e) {
log.error("睡眠时发生了异常: {}", e.getMessage());
} finally {
// 任务执行完毕后修改标志位
taskFinish = true;
}
});
}
}
}
}
ThreadPoolExecutor is a thread pool implementation in Java for managing and reusing threads to improve application performance and responsiveness.
It can control the number of threads, avoid resource waste and performance degradation caused by too many threads, and also avoid task waiting and response delays caused by insufficient threads.
Through ThreadPoolExecutor, we can submit tasks to the thread pool, and the threads in the thread pool will execute the tasks, thereby realizing asynchronous execution and concurrent processing of tasks.
Then the batchSave() method is the specific implementation of data saving
@Slf4j
@Service
public class VisitRecordService {
@Resource
VisitLogService visitLogService;
/**
* 缩减因子，每次更新缓存Set时缩小的倍数，对应HashSet的扩容倍数
*/
private final float REDUCE_FACTOR = 0.5f;
private void batchSave() {
log.debug("访问记录准备插入数据库，当前数据量：{}", visitCache.size());
if (visitCache.size() == 0) {
return;
}
// 构造新对象来存储数据，旧对象保存到数据库后不再使用
HashSet<AccessRecord> oldCache = visitCache;
visitCache = new HashSet<>((int) (oldCache.size() * REDUCE_FACTOR));
boolean isSave = false;
try {
//批量保存
isSave = visitLogService.saveBatch(oldCache, BATCH_SIZE);
} finally {
if (!isSave) {
// 如果插入失败，则重新添加所有数据
visitCache.addAll(oldCache);
}
}
}
}
VisitLogService is the normal data addition, deletion, modification and query category of mybatis. My definition here is as follows:
import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
import com.biglead.gateway.pojo.AccessRecord;
import com.biglead.gateway.mapper.VisitLogMapper;
import org.springframework.stereotype.Service;
/**
* 访问日志Service类
*/
@Service
public class VisitLogService extends ServiceImpl<VisitLogMapper, AccessRecord> {
}
VisitLogMapper is defined as follows
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.biglead.gateway.pojo.AccessRecord;
import org.apache.ibatis.annotations.Mapper;
@Mapper
public interface VisitLogMapper extends BaseMapper<AccessRecord> {
}
Then start the project, access the interface data, and it can be automatically recorded in the database. The
corresponding sql
CREATE TABLE `access_recoder_log` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`form_data` varchar(500) DEFAULT NULL,
`body` varchar(500) DEFAULT NULL,
`path` varchar(255) DEFAULT NULL,
`ip` varchar(255) DEFAULT NULL,
`status` int(11) DEFAULT NULL,
`user_id` bigint(20) DEFAULT NULL,
`scheme` varchar(255) DEFAULT NULL,
`method` varchar(255) DEFAULT NULL,
`consuming_time` bigint(20) DEFAULT NULL,
`create_time` datetime DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=25 DEFAULT CHARSET=utf8 COMMENT='日志记录表';
Finally, the source code:
SpringCloud source code of this project https://gitee.com/android.long/spring-cloud-biglead/tree/master/biglead-api-11-admin
This project management background web source code https://gitee. com/android.long/spring-cloud-biglead/tree/master/mall-admin-web-master
The source code of this project applet https://gitee.com/android.long/spring-cloud-biglead/tree/master/ mall-app-web-master
If you are interested, you can pay attention to the public account biglead, there will be java, Flutter, applet, js, English-related content sharing every week