In the distributed case, each log is distributed to the machine where the respective service is located. The collection and analysis of the log uses the original ancient model, which must be outdated. It is good that the cluster and service are small in scale, and the number is large. I think whether it is the operation and maintenance personnel. It's a headache for developers.
At present, the most popular intermediate suite to deal with this demand is ELK, which is the Java technology stack. It also meets the current needs of the company. The installation of ELK will not be described. If you are interested, you can check the official website or Baidu by yourself. There is still a lot of information.
The middleware for log collection and analysis is determined, and the remaining one is the log buried point and how to string the logs together. In the previous era of a single application, system-level logs could be resolved through aop. In the distributed case, for each independent service, its own log system is still solved by aop, and the only thing needed is how to string together the logs scattered to different applications. There is a high-level argument for this called business chain monitoring.
At present, the domestic open source products include the cat of Dianping, which is a complete set of business chain monitoring solutions. It is too heavy for my company at present. I already have elk in the log on my side, so there is no need to introduce cat. How to do it yourself.
Since it is a link, there is naturally an entrance and an exit. All we need to do is to generate a globally unique traceId at the entry and exit, and then pass the traceId to each service according to the business link. traceId is a line that connects the logs of each service. Note that the time of the service should be synchronized, because it is sorted according to the time of arrival.
For the generation of traceId, the simple solution can use uuid, and the snowflake algorithm of twiiter is recommended.
The transmission of traceId needs to be implemented according to the rpc framework. The dubbo framework is implemented using dubbo's fiter. The reference code is as follows:
// call process interception
public Result invoke(Invoker<?> invoker, Invocation invocation) throws RpcException {
//Get the serviceId asynchronously, if it is not obtained, no sampling will be performed
String serviceId = tracer.getServiceId(RpcContext.getContext().getUrl().getServiceInterface());
if (serviceId == null) {
Tracer.startTraceWork();
return invoker.invoke(invocation);
}
long start = System.currentTimeMillis();
RpcContext context = RpcContext.getContext();
boolean isConsumerSide = context.isConsumerSide();
Span span = null;
Endpoint endpoint = null;
try {
endpoint = tracer.newEndPoint();
// endpoint.setServiceName(serviceId);
endpoint.setIp(context.getLocalAddressString());
endpoint.setPort(context.getLocalPort());
if (context.isConsumerSide()) { //Is it a consumer
Span span1 = tracer.getParentSpan();
if (span1 == null) { // is rootSpan
span = tracer.newSpan(context.getMethodName(), endpoint, serviceId);//生成root Span
} else {
span = tracer.genSpan(span1.getTraceId(), span1.getId(), tracer.genSpanId(), context.getMethodName(), span1.isSample(), null);
}
} else if (context.isProviderSide()) {
Long traceId, parentId, spanId;
traceId = TracerUtils.getAttachmentLong(invocation.getAttachment(TracerUtils.TID));
parentId = TracerUtils.getAttachmentLong(invocation.getAttachment(TracerUtils.PID));
spanId = TracerUtils.getAttachmentLong(invocation.getAttachment(TracerUtils.SID));
boolean isSample = (traceId != null);
span = tracer.genSpan(traceId, parentId, spanId, context.getMethodName(), isSample, serviceId);
}
invokerBefore(invocation, span, endpoint, start);//记录annotation
RpcInvocation invocation1 = (RpcInvocation) invocation;
setAttachment(span, invocation1);//Set the parameters that need to be passed downstream
Result result = invoker.invoke(invocation);
if (result.getException() != null){
catchException(result.getException(), endpoint);
}
return result;
}catch (RpcException e) {
if (e.getCause() != null && e.getCause() instanceof TimeoutException){
catchTimeoutException(e, endpoint);
}else {
catchException(e, endpoint);
}
throw e;
}finally {
if (span != null) {
long end = System.currentTimeMillis();
invokerAfter(invocation, endpoint, span, end, isConsumerSide);//Record annotation after invocation
}
}
}
Dubbo passes the traceId between the consumer and the caller via invocation.setAttachmen.
If it is the RPC implemented by the http interface call, it is recommended to pass the traceId in the head of the request.
In the local service, the traceId is passed through the threadlocal variable.
If you want to print sql statements, you can implement it through the interceptor mechanism of the orm framework. The following is the reference code of mybatis
@Intercepts({ @Signature(type = Executor.class, method = "update", args = { MappedStatement.class, Object.class }),
@Signature(type = Executor.class, method = "query", args = { MappedStatement.class, Object.class,
RowBounds.class, ResultHandler.class }) })
public class MidaiLogMybatisPlugn implements Interceptor {
@Override
public Object intercept(Invocation invocation) throws Throwable {
Object result = null;
//Get trace from current thread
MidaiLogTrace trace = MidaiLogTraceService.getMidaiLogTrace();
if(trace !=null){
Object[] arguments = invocation.getArgs();
MidaiLogTraceService.traceSqlLog(trace.getTraceId(), getSqlStatement(arguments));
}
try {
result = invocation.proceed();
} catch (Exception e) {
throw e;
}
return result;
}
@Override
public Object plugin(Object target) {
return Plugin.wrap(target, this); // The wrapper class provided by mybatis
}
@Override
public void setProperties(Properties properties) {
}
private String getSqlStatement(Object[] arguments) {
MappedStatement mappedStatement = (MappedStatement) arguments[0];
Object parameter = null;
if (arguments.length > 1) {
parameter = arguments[1];
}
String sqlId = mappedStatement.getId();
BoundSql boundSql = mappedStatement.getBoundSql(parameter);
Configuration configuration = mappedStatement.getConfiguration();
String sql = showSql(configuration, boundSql);
StringBuilder str = new StringBuilder(100);
str.append(sqlId);
str.append(":");
str.append(sql);
str.append(":");
return str.toString();
}
public String showSql(Configuration configuration, BoundSql boundSql) {
Object parameterObject = boundSql.getParameterObject();
List<ParameterMapping> parameterMappings = boundSql.getParameterMappings();
String sql = boundSql.getSql().replaceAll("[\\s]+", " ");
if (parameterMappings.size() > 0 && parameterObject != null) {
TypeHandlerRegistry typeHandlerRegistry = configuration.getTypeHandlerRegistry();
if (typeHandlerRegistry.hasTypeHandler(parameterObject.getClass())) {
sql = sql.replaceFirst("\\?", getParameterValue(parameterObject));
} else {
MetaObject metaObject = configuration.newMetaObject(parameterObject);
for (ParameterMapping parameterMapping : parameterMappings) {
String propertyName = parameterMapping.getProperty();
if (metaObject.hasGetter(propertyName)) {
Object obj = metaObject.getValue(propertyName);
sql = sql.replaceFirst("\\?", getParameterValue(obj));
} else if (boundSql.hasAdditionalParameter(propertyName)) {
Object obj = boundSql.getAdditionalParameter(propertyName);
sql = sql.replaceFirst("\\?", getParameterValue(obj));
}
}
}
}
return sql;
}
private static String getParameterValue(Object obj) {
String value = null;
if (obj instanceof String) {
value = "‘" + obj.toString() + "‘";
} else if (obj instanceof Date) {
DateFormat formatter = DateFormat.getDateTimeInstance(DateFormat.DEFAULT, DateFormat.DEFAULT, Locale.CHINA);
value = "‘" + formatter.format(new Date()) + "‘";
} else {
if (obj != null) {
value = obj.toString();
} else {
value = "";
}
}
return value;
}
}
When the system concurrency reaches a certain order of magnitude, log4j log printing itself will become a bottleneck. At this time, mq is needed to decouple, instead of printing logs, mq messages are sent, which are processed by mq consumers. Because the current number of concurrent projects in the company is not enough to cause this problem, it has not been adopted yet.
After elk collects logs, kibana can provide search.
The last remaining workload is to provide a web interface to better analyze and display the data.