DolphinDB provides high-availability solutions for data, metadata, and clients, so that when a database node fails, the database can still operate normally, ensuring that business will not be interrupted.
Unlike other time series databases, the high availability of DolphinDB ensures strong consistency.
1 Overview
DolphinDB uses a multi-copy mechanism, and copies of the same data block are stored on different data nodes. Even if one or more data nodes in the cluster are down, as long as there is at least one copy available in the cluster, the database can provide services.
Metadata is stored on the control node. In order to ensure the high availability of metadata, DolphinDB uses the Raft protocol to form a Raft group by building multiple control nodes. As long as less than half of the control nodes are down, the cluster can still provide services.
DolphinDB API provides an automatic reconnection and switching mechanism. If the currently connected data node goes down, the API will try to reconnect. If the reconnection fails, it will automatically switch to other data nodes to perform tasks. Switching the data node is transparent to the user, and the user will not perceive that the currently connected node has been switched.
If you want to use high-availability features, please deploy a DolphinDB cluster first. The high-availability function is only supported in a cluster, not in a single instance. For cluster deployment, please refer to the multi-server cluster deployment tutorial .
2. High data availability
In order to ensure data security and high availability, DolphinDB supports storing multiple data copies on different servers, and maintains strong consistency between data copies. Even if the data on one machine is damaged, you can still access the copy data on other machines to ensure uninterrupted data service.
The number of copies can be set by the dfsReplicationFactor parameter in controller.cfg. For example, set the number of copies to 2:
dfsReplicationFactor=2By default, DolphinDB allows copies of the same data block to be distributed on the same machine. In order to ensure high data availability, copies of the same data block need to be distributed on different machines. The following configuration items can be added in controller.cfg:
dfsReplicaReliabilityLevel=1The following is an example to intuitively explain the high availability of DolphinDB data. First, execute the following script on the data nodes of the cluster to create the database:
n=1000000
date=rand(2018.08.01..2018.08.03,n)
sym=rand(`AAPL`MS`C`YHOO,n)
qty=rand(1..1000,n)
price=rand(100.0,n)
t=table(date,sym,qty,price)
if(existsDatabase("dfs://db1")){
dropDatabase("dfs://db1")
}
db=database("dfs://db1",VALUE,2018.08.01..2018.08.03)
trades=db.createPartitionedTable(t,`trades,`date).append!(t)The distributed table trades is divided into 3 partitions, and each date represents a partition. DolphinDB's Web cluster management interface provides DFS Explorer, you can easily view the data distribution. The distribution of each partition of the trades table is shown in the following figure:
Take the 20180801 partition as an example, the Sites column shows that the data with date=2018.08.01 is distributed on 18104 datanode and 18103 datanode. Even if the 18104 datanode is down, as long as the 18103 datanode is normal, the user still performs read and write operations on the data of date=2018.08.01.
3. High availability of metadata
Metadata is generated when data is stored, such as information such as where each data block is stored on which data node and where. If the metadata cannot be used, even if the data block is complete, the system cannot access the data normally.
Metadata is stored in the control node. We can deploy multiple control nodes in a cluster to ensure uninterrupted metadata services through metadata redundancy. All control nodes in a cluster form a Raft group. There is only one leader in the Raft group, and the others are followers. The metadata on the leader and the follower maintain strong consistency. Data nodes can only interact with Leader. If the current leader is not available, the system will immediately elect a new leader to provide metadata services. The Raft group can tolerate less than half of the control node downtime. For example, a cluster containing three control nodes can tolerate the failure of one control node; a cluster containing five control nodes can tolerate the failure of two control nodes. To set metadata high availability, the number of control nodes is at least 3, and data high availability needs to be set, that is, the number of copies must be greater than 1.
The following example is used to introduce how to enable metadata high availability for an existing cluster. Assuming that the control node of the existing cluster is located on the P1 machine, now two control nodes need to be added and deployed on the P2 and P3 machines respectively. Their intranet addresses are as follows:
P1: 10.1.1.1
P2: 10.1.1.3
P3: 10.1.1.5(1) Modify the configuration file of an existing control node
Add the following parameters to the controller.cfg file of P1: dfsReplicationFactor=2, dfsReplicaReliabilityLevel=1, dfsHAMode=Raft. The modified controller.cfg is as follows:
localSite=10.1.1.1:8900:controller1
dfsReplicationFactor=2
dfsReplicaReliabilityLevel=1
dfsHAMode=Raft(2) Deploy two new control nodes
Download the DolphinDB server package on P2 and P3, and unzip it, for example, unzip it to the /DolphinDB directory.
Create a config directory under the /DolphinDB/server directory. Create the controller.cfg file in the config directory and fill in the following:
P2
localSite=10.1.1.3:8900:controller2
dfsReplicationFactor=2
dfsReplicaReliabilityLevel=1
dfsHAMode=RaftP3
localSite=10.1.1.5:8900:controller3
dfsReplicationFactor=2
dfsReplicaReliabilityLevel=1
dfsHAMode=Raft(3) Modify the configuration file of an existing agent node
Add the sites parameter to the existing agent.cfg file, which represents the local area network information of the machine's agent node and all control nodes. The agent node information must be before all the control node information. For example, the modified content of P1's agent.cfg is as follows:
localSite=10.1.1.1:8901:agent1
controllerSite=10.1.1.1:8900:controller1
sites=10.1.1.1:8901:agent1:agent,10.1.1.1:8900:controller1:controller,10.1.1.3:8900:controller2:controller,10.1.1.5:8900:controller3:controllerIf there are multiple agent nodes, the configuration file of each agent node needs to be modified.
(4) Modify the cluster member configuration file of the existing control node
Increase the local area network information of the control node on the cluster.nodes of P1. For example, the modified content of cluster.nodes of P1 is as follows:
localSite,mode
10.1.1.1:8900:controller1,controller
10.1.1.2:8900:controller2,controller
10.1.1.3:8900:controller3,controller
10.1.1.1:8901:agent1,agent
10.1.1.1:8911:datanode1,datanode
10.1.1.1:8912:datanode2,datanode(5) Add cluster member configuration files and node configuration files for the new control node
Cluster.nodes and cluster.cfg are needed to control node startup. Copy the cluster.nodes and cluster.cfg on P1 to the config directories of P2 and P3.
(6) Start a highly available cluster
- Start the control node
Execute the following commands on the machine where each control node is located:
nohup ./dolphindb -console 0 -mode controller -home data -config config/controller.cfg -clusterConfig config/cluster.cfg -logFile log/controller.log -nodesFile config/cluster.nodes &- Start the agent node
Execute the following command on the machine where the agent node is deployed:
nohup ./dolphindb -console 0 -mode agent -home data -config config/agent.cfg -logFile log/agent.log &Starting and shutting down data nodes and modifying node configuration can only be operated on the Leader's cluster management interface.
- How to determine which control node is the leader
Enter the IP address and port number of any control node in the browser address bar to open the cluster management interface, such as 10.1.1.1:8900, and click the control node alias controller1 in the Node column to enter DolphinDB Notebook.
Execute the getActiveMaster()function, which returns the alias of Leader.
Enter the IP address and port number of the Leader in the address bar of the browser to open the cluster management interface of the Leader.
4. Client High Availability
When using the API to interact with the data node of the DolphinDB server, if the connected data node is down, the API will try to reconnect, and if the reconnection fails, it will automatically switch to other available data nodes. This is transparent to users. Currently, only Java and Python APIs support high availability.
The connect method of the API is as follows:
connect(host,port,username,password,startup,highAvailability)When using the connect method to connect to a data node, you only need to specify the highAvailability parameter as true.
The following example sets the Java API to be highly available:
import com.xxdb;
DBConnection conn = new DBConnection();
boolean success = conn.connect("10.1.1.1", 8911,"admin","123456","",true);If the data node 10.1.1.1:8911 goes down, the API will automatically connect to other available data nodes.
5. Dynamically add data nodes
Users can use addNodecommands to add data nodes online without restarting the cluster.
The following example shows how to add a new data node datanode3 on the new server P4 (intranet IP is 10.1.1.7), the port number is 8911.
To add a data node to a new physical server, you need to deploy an agent node to start the data node on the server. The port of the agent node of P4 is 8901, and the alias is agent2.
Download the DolphinDB package on P4 and unzip it to the specified directory, such as /DolphinDB.
Go to the /DolphinDB/server directory and create a config directory.
Create an agent.cfg file in the config directory and fill in the following content:
localSite=10.1.1.7:8901:agent2
controllerSite=10.1.1.1:8900:controller1
sites=10.1.1.7:8901:agent2:agent,10.1.1.1:8900:controller1:controller,10.1.1.3:8900:controller2:controller,10.1.1.5:8900:controller3:controllerCreate a cluster.nodes file in the config directory and fill in the following content:
localSite,mode
10.1.1.1:8900:controller1,controller
10.1.1.2:8900:controller2,controller
10.1.1.3:8900:controller3,controller
10.1.1.1:8901:agent1,agent
10.1.1.7:8901:agent2,agent
10.1.1.1:8911:datanode1,datanode
10.1.1.1:8912:datanode2,datanodeModify cluster.nodes on P1, P2 and P3 to be the same as cluster.nodes on P4.
Execute the following Linux command to start the agent node on P4:
nohup ./dolphindb -console 0 -mode agent -home data -config config/agent.cfg -logFile log/agent.log &Execute the following command on any data node:
addNode("10.1.1.7",8911,"datanode3")After executing the above script, refresh the Web cluster management interface, you can find that the newly added data node already exists, but it is in the closed state, you need to manually start the new data node.
6. Summary
By ensuring that data, metadata services, and API connections are not interrupted, DolphinDB can meet the 24-hour uninterrupted service needs in the Internet of Things, finance and other fields.