1. Problems caused by out-of-synchronization of Linux cluster clocks
The clock here refers to the server time. If the clocks of each server in the cluster are inconsistent, it will inevitably lead to a series of problems. Just imagine that "a cluster is a team where each server works together. If everyone does not work at the same point, it will be a mess! "
To give an example, if an order is added in the business of an e-commerce website, a record will inevitably be added to the order table. There should be a field such as "order time" in this record. Often we will Get the current system time in the program and insert it into the database or get the time directly from the database server. Then our order subsystem is clustered deployment, or our database is also clustered deployment of sub-database and sub-table, but their system clocks are inconsistent. For example, the time of one server is yesterday, then this time If the order time becomes yesterday, then our data will be messed up! as follows
2. Cluster clock synchronization configuration
-
Cluster Clock Synchronization Ideas
-
Each server node in the distributed cluster can connect to the Internet
Scenario 1: Each node can access the Internet, and each synchronizes with the national time service center/time server
How to operate:
#使⽤ ntpdate 网络时间同步命令 ntpdate -u ntp.api.bz #从⼀个时间服务器同步时间Windows has scheduled tasks
Linux also has scheduled tasks, crond, you can use the scheduled tasks of Linux to execute the ntpdate command every 10 minutes
-
A certain server node in the distributed cluster can access the Internet or all nodes cannot access the Internet
Scenario 2 and Scenario 3: Use node1 as the time server in the LAN, and other servers synchronize time from node1
How to operate:
1) Select a server node A (172.17.0.17) in the cluster as the time server (the entire cluster time is synchronized from this server, if this server can access the Internet, you can keep this server and network time synchronized, if not just manually set a time)
- First set the time of A
- Configure A as a time server (modify the /etc/ntp.conf file)
-
1、如果有 restrict default ignore，注释掉它
2、添加如下⼏⾏内容
restrict 172.17.0.0 mask 255.255.255.0 nomodify notrap # 放开局域网同步功能,172.17.0.0是你的局域网网段
server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
3、重启⽣效并配置ntpd服务开机⾃启动
service ntpd restart
chkconfig ntpd on
-
Other nodes in the cluster can synchronize time from server A
ntpdate 172.17.0.17
3. Use Chronyd to synchronize time
Chrony is an implementation of the Network Time Protocol (NTP), a replacement for NTP clients and servers on Unix-like systems. The Chrony client can synchronize the system time with NTP servers, it can also synchronize with a reference clock (for example: GPS receiving device), and it can also synchronize with the time entered manually. Similarly, Chrony can also serve as an NTPv4 (RFC 5905) server to provide time synchronization services for other computers.
Chrony can synchronize the system clock faster, has better clock accuracy, and it is very helpful for systems that are not always online. Chrony's typical accuracy between two machines synchronized on the Internet is within milliseconds, and between machines on a LAN is within tens of microseconds. chronyd is smaller and more energy efficient, it uses less memory and it only wakes up the CPU when needed. It works well even when the network is congested for extended periods of time. It supports hardware time stamping on Linux, allowing extremely accurate synchronization over a local network. Chrony is free and open source, and supports GNU/Linux and BSD derivatives (such as FreeBSD, NetBSD), macOS and Solaris, etc.
3.1 Comparison with ntpd
(1) chronyd is better than ntpd.
chronyd can work in an environment where the reference time source is accessed intermittently, while ntpd needs to periodically poll the reference time source to work normally.
Even with longer periods of network congestion, chronyd can run well.
chronyd generally synchronizes clocks faster and more accurately.
chronyd adapts quickly to sudden changes in clock frequency caused by crystal oscillator temperature changes, whereas ntpd can take a long time to stabilize.
In the default configuration, chronyd never performs time-stepping after the system starts to synchronize the clocks, so as not to affect other running programs. ntpd can also be configured to never step the time, but it must use a different method to adjust the clock, which has some disadvantages, including a negative impact on clock accuracy.
chronyd can adjust the clock frequency on a Linux system over a greater range, which allows it to run even on machines with corrupted or unstable clocks. For example, on some virtual machines.
chronyd is smaller, uses less memory, and only wakes up the CPU when necessary, which is better for power savings.
(2) What chronyd can do but ntpd can't do
. Chronyd provides support for isolated networks. In isolated networks, the only way to correct time is to manually input. For example, the clock is viewed by an administrator. chronyd can examine bugs fixed in different updates to estimate how quickly the computer increases or decreases time, and then uses this estimate to adjust the computer clock.
chronyd can calculate the gain and loss rates of RTC clocks (clocks that keep time when the computer is turned off). It can use these calculated data at system startup, along with the time adjustment value obtained from the RTC clock, to set the system time. The RTC clock device is currently only available on Linux systems.
Supports hardware time stamping on Linux, allowing very precise synchronization over a local network.
(3) What ntpd can do but chronyd can't do,
ntpd supports all synchronization modes of NTP v4 (RFC 5905), including broadcast, multicast and manycast clients and servers modes. Note that broadcast and multicast modes (even with authentication) are less precise and less secure than normal servers and clients mode and should generally be avoided.
ntpd supports the Autokey protocol (RFC 5906) using public key cryptography for server authentication. Note that this protocol has proven to be insecure and may be replaced by Network Time Security (NTS).
ntpd includes drivers for many reference time sources, while chronyd relies on other programs (such as gpsd) to access data from time reference sources using shared memory (SHM) or Unix domain sockets (SOCK).
(4) chronyd and ntpd, how to choose
Chrony should be preferred for other systems except systems managed or monitored by tools that do not support Chrony, or systems with hardware reference clocks that cannot be used with Chrony.
Systems that need to authenticate packets using the Autokey protocol can only use ntpd because chronyd does not support this protocol. The Autokey protocol has serious security issues and should be avoided. Use symmetric keys for authentication instead of Autokey, which is supported by both chronyd and ntpd. Chrony supports stronger hash functions such as SHA256 and SHA512, while ntpd can only use MD5 and SHA1.
3.2 Server installation
Centos Linux 8.1 has chronyd service installed by default. We just need to configure it.
profile configuration
The red part requires server configuration, which is the basic configuration, mainly to realize basic functions.
vi /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# pool 2.centos.pool.ntp.org iburst
# 同步外部服务器上的时间
server ntp.api.bz iburst
# Record the rate at which the system clock gains/losses time.
driftfile /var/lib/chrony/drift
# Allow the system clock to be stepped in the first three updates
# if its offset is larger than 1 second.
makestep 1.0 3
# Enable kernel synchronization of the real-time clock (RTC).
rtcsync
# Allow NTP client access from local network.
# allow 192.168.0.0/16
# 允许哪个网段的客户端同步时间，133这台是服务器。
allow 192.168.94.0/24
# Serve time even if not synchronized to a time source.
# 字面翻译，提供时间即使没有同步一个时间源。
local stratum 10
# Specify file containing keys for NTP authentication.
keyfile /etc/chrony.keys
# Get TAI-UTC offset and leap seconds from the system tz database.
leapsectz right/UTC
Start chronyd and set it to start automatically at boot
[root@node1 ~]# systemctl start chronyd
[root@node1 ~]# systemctl enable chronyd
Check the time synchronization situation, the local can be synchronized to allow other nodes to synchronize
timedatectl is a time configuration tool for setting time, time-related configuration, etc.
[root@node1 ~]# timedatectl
Local time: 五 2020-05-29 11:17:46 EDT
Universal time: 五 2020-05-29 15:17:46 UTC
RTC time: 五 2020-05-29 15:17:46
Time zone: America/New_York (EDT, -0400)
# 当此值为yes代表同步成功，如果为no就代表未同步，或同步失败。
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
[root@filesystem ~]#
View where the current time is synchronized from, here is the host name. does not affect.
[root@node1 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* node1 10 7 377 537 +11us[ +25us] +/- 30us
[root@filesystem ~]#
firewall configuration
The firewall must be able to release this service, close the firewall or release the corresponding ports and services.
3.3 Client Configuration
profile configuration
vi /etc/chrony.conf
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
# pool 2.centos.pool.ntp.org iburst
server 192.168.94.133 iburst
Configure autostart
[root@node2 ~]# systemctl restart chronyd
[root@node2 ~]# systemctl enable chronyd
View synchronization status and information
[root@node2 ~]# timedatectl
Local time: 五 2020-05-29 11:28:12 EDT
Universal time: 五 2020-05-29 15:28:12 UTC
RTC time: 五 2020-05-29 15:28:12
Time zone: America/New_York (EDT, -0400)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
[root@node1 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* node1 11 6 17 34 -520ns[+7955ns] +/- 15ms
[root@node2 ~]#
We see yes, but how to judge whether it is the original time or synchronized. After all, many systems are not synchronized and the time is also approximate.