Use the file writing method provided by logback to quickly write text to about 100,000 data per second

The author has participated in an online project. The data storage capacity of this project is huge. About 10W of data are pushed every second. The project initially used mysql as storage, but with a traditional database, this data amount supports Before long, the use of sub-database and sub-table is only a temporary solution. In the end, the leader decided to use Hadoop + HBase for data storage. The final guarantee is the basis for the recovery of extreme problems in the future.

This level of data, written to a file, must have high requirements for IO. At that time, the code development was another colleague, which used the basic file API FileOutputStream provided by Java to write to the file. This method does not have any problem in normal writing, but when the amount of data is too large, the writing speed is extremely slow, causing the data in MQ to continuously squeeze and block.

At that time, the project team was discussing how to quickly write files to txt. After thinking, I found that the logs of our project, no matter how frequently they are accessed and how much traffic, the logs can always be written to the log in real time. In the file, the log framework is the logback that SpringBoot supports by default, so the author will use this method and propose it to the above. After discussion, I decided to use the API provided by logback to write the data to txt. Let me first introduce the data flow process and writing process

client data -> netty -> rabbit mq -> logback -> txt

That is, the client sends the data to the server's netty through a long connection. After netty receives the data, it is stored in the MQ. My logback program acts as a subscriber. After reading the data, it is classified and written into txt. Take a look at the code implementation, (only focus on the code written in txt, the author has made a lot of cuts in many places, the reader can know the meaning)

@Component
@RabbitListener
public class DataListener {

    private static final Logger logger = LoggerFactory.getLogger(DataListener.class);

    @RabbitHandler
    public void getMessage(@Payload Data data) {
        try {
            data.setUuid(UUID.randomUUID().toString().replace("-", ""));
            // 此处根据业务逻辑以数据的num后四位作为文件名称
            String str4id = "0000";
            if (data.getNum() != null && data.getNum().length() > 4) {
                str4id = data.getNum().substring(data.getNum().length() - 4, data.getNum().length());
            }
            data.setStr4id(10000 + Long.parseLong(str4id));
            String fileName = "data_" + data.getStr4id();
            // 文件名称可以看做是一个客户端对象，这个客户端对象会不停的发送数据过来， 对每个文件名称都做了一个DataLogWriter对象的缓存，有就使用已经创建的写入，没有则创建并保存到map缓存中后写入
            DataLogWriter dataSqlWriter = dataUtil.objMap.get(fileName);
            if (dataSqlWriter == null) {
                dataSqlWriter = new dataLogWriter(fileName);
                dataUtil.objMap.put(fileName, dataSqlWriter);
            }
            dataSqlWriter.outLog(dataUtil.getdataLog(data));
        } catch (Exception ex) {
            ex.printStackTrace();
            logger.error("gpswriter异常：", ex);
        }
    }

}

package cn.net.leadu.log;

import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.LoggerContext;
import ch.qos.logback.classic.encoder.PatternLayoutEncoder;
import ch.qos.logback.core.FileAppender;

import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.HashMap;
import java.util.Map;

public class DataLogWriter {

    private final Logger logbackLogger;
    LoggerContext loggerContext = new LoggerContext();
    public FileAppender fileAppender = new FileAppender();
    private String dayDate ; //当前天
    private String loggerName; //日志文件名称

    public void outLog(String msg){
        SimpleDateFormat yMd = new SimpleDateFormat("yyyy-MM-dd");
        Date date = new Date();
        String nowDate = yMd.format(date);
        //每次写入的时候，查看当前时间和对象中的天数是否相同，如果不相同则重置日志路径
        if(!nowDate.equals(dayDate)) {
            dayDate = nowDate;
            fileAppender.setFile("/data/data_backup/" + dayDate + "/" + loggerName + ".txt");
            fileAppender.start();
            logbackLogger.debug(msg);
        }else{
            logbackLogger.debug(msg);
        }
    }


    public DataLogWriter(String logfileName) {
        SimpleDateFormat yMd = new SimpleDateFormat("yyyy-MM-dd");
        Date date = new Date();
        loggerName = logfileName;
        fileAppender.setContext(loggerContext);
        fileAppender.setName("logfile");
        //文件路径 按照天 / 小时 / num后四位分文件
        dayDate = yMd.format(date);
        fileAppender.setFile("/data/data_backup/" + dayDate +"/" + loggerName + ".txt");
        PatternLayoutEncoder encoder = new PatternLayoutEncoder();
        encoder.setContext(loggerContext);
        encoder.setPattern("%msg");
        encoder.start();
        fileAppender.setEncoder(encoder);
        fileAppender.start();
        logbackLogger = loggerContext.getLogger("nelogger");
        logbackLogger.addAppender(fileAppender);
        logbackLogger.isAdditive();
    }

}

At that time, after writing like this, go to the service, open the cluster ~ found it is not possible ... MQ is still piled up or not down?
Why ... After thinking about it, I found that the reason is that when multiple services write to a file at the same time, it will find that the file occupies, waits for each other, and deadlocks, which causes MQ messages to be read and written. This involves The reader can know the file handle problem. The solution is to not open the cluster ~ Haha, only open one service. At
this time, the reading and writing speed is comparable to that of the rocket.

This service has been running for a long time online, and no problems have occurred. Of course, the operation and maintenance must pay attention to your server disks, and do not have various abnormal problems that occur when the disk is almost full. Big data is closely related to hardware storage ~

I will not discuss here what kind of optimization logback has made to file writing. If readers encounter the same problem, they can refer to this article and use logback to solve this problem ~

thanks for reading