We use etcd to store metadata in actual production. At first, when the scale of the cluster is small, there is not much metadata information and no problems are found. As the size of the cluster grows, the problem is gradually exposed
Some actual configurations still need to be researched and determined during initialization
1. --auto-compaction-retention
Since ETCD data stores multi-version data, the historical version needs to be cleaned regularly as the primary key written increases. The default historical data will not be cleaned, and the data cannot be written when the data reaches 2G, and the historical data must be cleaned and compressed to continue writing;
Therefore, according to business requirements, it is determined in advance how often the historical data will be compressed before going to the production environment; our production environment now compresses data once an hour by default after the upgrade. This can greatly ensure the stability of the cluster and reduce memory and disk usage.
2.--max-request-bytes
The maximum number of bytes of etcd Raft messages, etcd defaults to 1.5M; but many business scenarios find that 1.5M cannot meet the requirements when synchronizing data, so it is very important to determine the initial value in advance; Because 1.5M leads to our online business The problem of not being able to write metadata,
After the emergency upgrade, we changed the value to the default 32M, but the official recommendation is 10M, you can adjust it according to the business situation
3.--quota-backend-bytes
ETCDdb data size, the default is 2G, when the data reaches 2G, it is not allowed to write, and the historical data must be compressed before continuing to write; as mentioned in Part 1, we should determine the size in advance when starting, the official recommendation is 8G, here we also use the 8G configuration
# start command
/usr/bin/etcd --auto-compaction-retention ' 1 ' --max-request-bytes ' 33554432 ' --quota-backend-bytes ' 8589934592 '