Interview Questions | How to write a good and fast log library? (two)

Introduction

In the previous article, a highly scalable logging framework was built using the Chain of Responsibility pattern, and a high-performance I/O library was introduced to improve log writing performance.

Under this framework, "log write file" appears as an interceptor:

// 日志拦截器
class OkioLogInterceptor(private var dir: String) : LogInterceptor {
    private val handlerThread = HandlerThread("log_to_file_thread")
    private val handler: Handler
    private var bufferedSink: BufferedSink? = null
    private val dispatcher: CoroutineDispatcher

    init {
        // 开启线程串行地处理日志请求
        handlerThread.start()
        handler = Handler(handlerThread.looper, callback)
        dispatcher = handler.asCoroutineDispatcher("log_to_file_dispatcher")
    }

    override fun log(priority: Int, tag: String, log: String, chain: Chain) {
        GlobalScope.launch(dispatcher) {
            // 使用 Okio 将日志输出到文件
            checkSink().use {
                it.writeUtf8("[$tag] $log")
                it.writeUtf8("\n")
            }
        }
        chain.proceed(priority, tag, log)
    }
    
    // 构建缓冲输出流
    private fun checkSink(): BufferedSink {
        if (bufferedSink == null) {
            bufferedSink = logFile.appendingSink().buffer()
        }
        return bufferedSink!!
    }
}
复制代码

This is the schematic pseudo-code for writing a file log interceptor. The complete code can be clicked here . Interview questions | How to write a good and fast log library? (one)

In addition to high scalability and high performance I/O, where else can be optimized?

compressed log

Compressing the daily content of the log can not only further improve the I/O performance, reduce the traffic of client log uploading, but also save the company money (cloud storage is quite expensive).

Gzip is a commonly used compression format. Linux uses this format to compress files. In addition to file compression, Gzip is also used for network compression, and it is one of the three standard HTTP compression formats specified in RFC 2016.

A detailed introduction to the Gzip compression format can be found here .

With the help of the decorator pattern, the adapter pattern, and Kotlin's extension method syntax, adding Gzip functionality to the original output stream is simple.

The original code for constructing the output stream is as follows:

val bufferedSink = logFile.appendingSink().buffer()
复制代码

where appendingSink()and buffer()are both extension methods:

fun File.appendingSink(): Sink = FileOutputStream(this, true).sink()
fun OutputStream.sink(): Sink = OutputStreamSink(this, Timeout())
fun Sink.buffer(): BufferedSink = RealBufferedSink(this)
复制代码

That is to build FileOutputStream first, then adapt it to OutputStreamSink, and then decorate it as RealBufferedSink, and finally form RealBufferedSink( OutputStreamSink( FileOutputStream( File ) ) )such nesting doll structure.

The class in Okio that implements Gzip compression output is called GzipSink, and it also has a similar decorative constructor:

inline fun Sink.gzip() = GzipSink(this)
复制代码

Compression can be achieved simply by inserting gzip() in the original call chain:

val bufferedSink = logFile.appendingSink().gzip().buffer()
复制代码

The current nesting doll structure becomesRealBufferedSink( GzipSink( OutputStreamSink( FileOutputStream( File ) ) ) )

I ran the test program. The test method is to continuously output 10,000 long logs. When Gzip is not used, the log file size is 251 MB. After adding Gzip, it is only 2.1 MB. The full reduction is 100+ times .

把日志文件的后缀改成gz，这样从云端下载之后就能直接只用压缩软件解压看到原始日志。

为了对比加入压缩后 Okio 和 java.io 的速度性能差异，重写了 java.io 版压缩输出流的代码：

val outputStream = logFile.outputStream().gzip().writer().buffered()
复制代码

其中outputStream()和buffered()是系统预定的装饰流扩展方法：

 // 构造 FileOutputStream 并持有 File 实例
public inline fun File.outputStream(): FileOutputStream {
    return FileOutputStream(this)
}
// 构造 BufferedWriter 并持有 Writer 实例
public inline fun Writer.buffered(bufferSize: Int = DEFAULT_BUFFER_SIZE): BufferedWriter =
    if (this is BufferedWriter) this else BufferedWriter(this, bufferSize)

复制代码

而gzip()和writer()是自定义的装饰流扩展方法：

// 构建 GZIPOutputStream 并持有 OutputStream
fun OutputStream.gzip() = GZIPOutputStream(this)
// 构建 OutputStreamWriter 并持有 OutputStream
fun OutputStream.writer(charset: Charset = Charsets.UTF_8) = OutputStreamWriter(this, charset)
复制代码

GZIPOutputStream是针对 OutputStream 的，所以不得不使用 OutputStreamWrite 将 Writer 接口适配成 OuptStream 接口。

完整的 java.io 版压缩日志拦截器代码如下：

class FileWriterLogInterceptor private constructor(private var dir: String) : LogInterceptor {
    private val handlerThread = HandlerThread("log_to_file_thread")
    private val handler: Handler
    // 统计耗时起点
    private var startTime = System.currentTimeMillis()
    // 用于记录平均内存的列表
    private val memorys = mutableListOf<Long>()
    private var fileWriter: Writer? = null
    private var logFile = File(getFileName())

    val callback = Handler.Callback { message ->
        val sink = checkFileWriter()
        // 每来一条日志，记录此时的内存占用
        memorys.add(Runtime.getRuntime().totalMemory()/(1024*1024))
        when (message.what) {
            // 输出日志结束的标记
            TYPE_FLUSH -> {
                sink.use {
                    it.flush()
                    fileWriter = null
                }
                // 统计耗时即内存终点
                Log.v(
                    "test",
                    "fileWriter work is ok done=${System.currentTimeMillis() - startTime, memory=${memorys.average()}"
                )
            }
            // 正常写日志
            TYPE_LOG -> {
                val log = message.obj as String
                sink.write(log)
                sink.write("\n")
            }
        }
        false
    }

    companion object {
        private const val TYPE_FLUSH = -1
        private const val TYPE_LOG = 1
        // 若 300 ms 无日志请求，则进行冲刷
        private const val FLUSH_LOG_DELAY_MILLIS = 300L

        // 设计单例，防止启动多个写日志线程
        @Volatile
        private var INSTANCE: FileWriterLogInterceptor? = null
        fun getInstance(dir: String): FileWriterLogInterceptor =
            INSTANCE ?: synchronized(this) {
                INSTANCE ?: FileWriterLogInterceptor(dir).apply { INSTANCE = this }
            }
    }

    // 启动写日志线程
    init {
        handlerThread.start()
        handler = Handler(handlerThread.looper, callback)
    }

    override fun log(priority: Int, tag: String, log: String, chain: Chain) {
        if (!handlerThread.isAlive) handlerThread.start()
        handler.run {
            removeMessages(TYPE_FLUSH)
            obtainMessage(TYPE_LOG, "[$tag] $log").sendToTarget()
            val flushMessage = handler.obtainMessage(TYPE_FLUSH)
            // 倒计时，用于判断“已经无新日志”
            sendMessageDelayed(flushMessage, FLUSH_LOG_DELAY_MILLIS)
        }
        chain.proceed(priority, tag, log)
    }

    override fun enable(): Boolean { return true }
    
    // 以今天日期为文件名
    private fun getToday(): String = SimpleDateFormat("yyyy-MM-dd").format(Calendar.getInstance().time)
    private fun getFileName() = "$dir${File.separator}${getToday()}.log"

    // 构建 java.io 压缩文件输出流
    private fun checkFileWriter(): Writer {
        if (fileWriter == null) {
            fileWriter = logFile.outputStream().gzip().writer().buffered()
        }
        return fileWriter!!
    }
    
    // 自定义装饰流构造方法，以简化流构建代码
    private fun OutputStream.gzip() = GZIPOutputStream(this)
    private fun OutputStream.writer(charset: Charset = Charsets.UTF_8) = OutputStreamWriter(this, charset)
}
复制代码

关于其中每一个细节的讲解可以点击面试题 | 怎么写一个又好又快的日志库？（一）

下面是测试代码：

// 分别给 EasyLog 配置 Okio 日志拦截器和 FileWriter 日志拦截器
//EasyLog.addInterceptor(FileWriterLogInterceptor.getInstance(this.filesDir.absolutePath))
EasyLog.addInterceptor(OkioLogInterceptor.getInstance(this.filesDir.absolutePath))
MainScope().launch(Dispatchers.Default) {
    // 连续输出1万条日志并压缩
    repeat(10_000) {
        EasyLog.v(str4, "test")
    }
}
复制代码

输出日志如下：

fileWriter work is ok done=5130, memory=160.70305938812237
fileWriter work is ok done=5172, memory=157.5844831033793
fileWriter work is ok done=5155, memory=168.01649670065987

Okio work is ok done=4765, memory=130.96940611877625
Okio work is ok done=4752, memory=130.21985602879425
Okio work is ok done=4779, memory=135.28374325134973
复制代码

Okio 有 8% 左右的速度优势，及 20% 左右的内存优势。

总结

压缩日志是提升日志库性能的手段之一，常用的 Gzip 是压缩手段之一，Okio 和 java.io 都提供了对 Gzip 的支持，不过 Okio 在速度和内存上都稍好于 java.io。