Detailed distributed system logs
What is log
Log is added in chronological order, fully ordered sequence of records, in fact, a special kind of file format, the file is an array of bytes, and where log is a record of data, the file is only relative, where each record are arranged according to the relative order of time, the log can be said to be one of the simplest model of storage are generally read from left to right, such as a message queue, write the log file is typically linear, consumers sequence beginning at offset read.
Due to the inherent characteristics of the log, recording start insertion order from left to right, which means recording the left compared to the record to the right of "older", that we can not rely on the system clock, this feature for distributed The system is very important.
Application log
Application logs in the database
Logs had appeared have no way of knowing, may be conceptually very simple. In more log database art for a system crash, and the synchronization data when indexing, for example in the redo log MySQL, redo log is a disk-based data structures for the system hang time to ensure data correctness, completeness, also called write-ahead log, for example, during the execution of a thing, first write redo log, and then will apply the actual changes, so that when the system crash recovery can be performed according to weight redo log put thereby restoring data (in the initialization process, this time not yet client connections). Logs can also be used to synchronize databases between master and slave, because in essence, a database of all operational records have been written to the log, we just log synchronized to the slave, and the slave replay will be able to achieve master-slave synchronization, here you can also achieve a lot of other components needed, we can subscribe to in order to get all of the redo log changes in the database, enabling personalized business logic, such as audit, cache synchronization and so on.
Application logs in a distributed system
Is a state change on the nature of services on a distributed system, this can be understood as a state machine, two independent processes (not dependent on the external environment, such as the system clock, the external interface, etc.) consistent with the given input will produce consistent output and finally in a consistent state, and the log because of its inherently sequential does not depend on the system clock, just it can be used to solve the problem of orderly change.
We use this feature to implement distributed systems solve many problems encountered. The standby node RocketMQ e.g., primary broker receiving client requests, and logging, real-time synchronization and then to the salve, slave local playback, when hang master, slave can continue to process the request, a write request and continue as reject e.g. process read requests. Not only can record log data, the recording operation may be directly, for example, SQL statements.
Log data structure is the key to solving the problem of consistency, the log is like a sequence of operations, each record represents one instruction, such as widely used Paxos, Raft agreement, coherence protocols are based on the log build up.
Application log in Message Queue
Logs can easily be used between the inflow and outflow of data processing, each data source can generate its own log, where data sources can come from various aspects, such as an event stream (page hits, cache refresh reminders, database binlog change ), we can log into a centralized storage cluster, subscribers can read each record logs based on offset, according to the data in each record, change their operational applications.
这里的日志可以理解为消息队列，消息队列可以起到异步解耦、限流的作用。为什么说解耦呢？因为对于消费者、生产者来说，两个角色的职责都很清晰，就负责生产消息、消费消息，而不用关心下游、上游是谁，不管是来数据库的变更日志、某个事件也好，对于某一方来说我根本不需要关心，我只需要关注自己感兴趣的日志以及日志中的每条记录。
我们知道数据库的QPS是一定的，而上层应用一般可以横向扩容，这个时候如果到了双11这种请求突然的场景，数据库会吃不消，那么我们就可以引入消息队列，将每个队数据库的操作写到日志中，由另外一个应用专门负责消费这些日志记录并应用到数据库中，而且就算数据库挂了，当恢复的时候也可以从上次消息的位置继续处理(RocketMQ和Kafka都支持Exactly Once语义)，这里即使生产者的速度异于消费者的速度也不会有影响，日志在这里起到了缓冲的作用，它可以将所有的记录存储到日志中，并定时同步到slave节点，这样消息的积压能力能够得到很好的提升，因为写日志都是有master节点处理，读请求这里分为两种，一种是tail-read，就是说消费速度能够跟得上写入速度的，这种读可以直接走缓存，而另一种也就是落后于写入请求的消费者，这种可以从slave节点读取，这样通过IO隔离以及操作系统自带的一些文件策略，例如pagecache、缓存预读等，性能可以得到很大的提升。
分布式系统中可横向扩展是一个相当重要的特性，加机器能解决的问题都不是问题。那么如何实现一个能够实现横向扩展的消息队列呢? 假如我们有一个单机的消息队列，随着topic数目的上升，IO、CPU、带宽等都会逐渐成为瓶颈，性能会慢慢下降，那么这里如何进行性能优化呢?
1.topic / log fragmented, the message is essentially the topic written log, then with the greater the number of writes, single opportunity to slowly become a bottleneck, this time we can be a single topic into multiple sub-topic and assigned to each topic on different machines, in this way, for those messages with large amounts topic can be solved by adding machine, and for some messages may be assigned to a small amount of the same machine or partitioning
2.group commit, the producer client Kafka e.g., when the message is written, is written to a local memory queue, then the message for each partition, node summary, batch submitted for the server or broker terminal, also It can be used this way, the first written pagecache, retiming disk brush, brush the pan ways based on business decisions, such as financial services may take synchronous brush plate way.
3. To avoid unnecessary data copies
4.IO isolation
Epilogue
Log in distributed systems play a very important role, is the key to understanding the various components of a distributed system, with the deepening of understanding, we found that many distributed middleware are based on log construction, for example Zookeeper, HDFS, Kafka , RocketMQ, Google Spanner and so on, and even databases such as Redis, MySQL, etc., which are based on master-slave log synchronized manner, dependent on shared logging system, we can achieve a lot of system: data synchronization between nodes, concurrent updates data sequence problem (consistency), (system crash can continue to provide services through other nodes) persistence, distributed lock service and so on, we believe that through practice slowly, and after reading a lot of papers, there will be deeper understanding.
