Today, I will introduce IoTDB deployment, operation and maintenance related knowledge to help you play IoTDB!
The text is 1859 words, and the estimated reading time is 5 minutes.
Out of the box
The concept of IoTDB is to simple system operation and maintenance, one-click startup, and out-of-the-box use. Let's start from the start, you need to install jdk8 or jdk11, download the release version, http://iotdb.apache.org/Download/, the directory structure after decompression:
The memory is in conf/iotdb-env.sh, configure MAX_HEAP_SIZE, and remove the previous comment. All other configurations are in conf/iotdb-engine.properties.
You may need to add execution permissions to the script, and then start the server in the background:
nohup sbin/start-server.sh >/dev/null 2>&1 &
Directory Structure
After starting with the default configuration and writing data, the project root directory will generate the following folders, all under the data directory:
It mainly includes 3 parts, data file data/data, system file data/system, and pre-write log data/wal.
Data directory: data/data
The directory organization method is sequential/out-of-order data -> storage group -> partition number (currently one partition by default) -> data file .tsfile and index file .resource.
The data file TsFile (Timeseries File) is a column storage file format designed by us. It mainly stores the raw data of each time series. TsFile has a separate API and can be used as an independent tool, just like Parquet and ORC. The data file can be configured with multiple directories, through the data_dirs parameter configuration.
System directory: data/system
There are several important documents.
mlog.txt: Metadata log, which records all metadata operations by means of appending, including adding and deleting storage groups, adding and deleting time series, etc. When restarting, the log in this file will be redone, and errors will be skipped.
system.properties: system properties, records some parameters that cannot be changed after startup, such as partition granularity, time precision, etc.
tlog.txt: label and attribute information. If not created, this file is empty.
Pre-write log directory: data/wal
The written data will be recorded in the pre-write log first and then written to the memory. After the data in the memory is flushed, the pre-write log will be cleared. When you find that you have written some data, the data directory is still empty. At this time, the data is in the pre-write log and memory.
Pre-write logs are organized according to storage groups, and one pre-write log file corresponds to one TsFile file.
If you want to forcibly flush the data in the memory, you can execute the flush command through the CLI. Will persist memory data, and write the log clearly.
System log: data/logs
In addition to the three most important ones above, there is also the system log. The log files are archived according to the date. The ones without a date are today's.
If there is a problem, you can come here to copy the log and send it to us.
Restarted
After testing some, I want to start again. It is very convenient. Delete all the three directories above and restart it. Still the most beautiful IoTDB in the whole street.
data migration
Before migration, it is best to use the CLI console to execute a flush command to persist the memory data (so that there is no pre-write log). Restarting IoTDB once has a similar effect. The migrated IoTDB version must be the same.
Introduce several scenarios:
1. Migrate the IoTDB on machine A to machine B (debugging, data backup, etc.)
You can copy the entire data directory (including data directory and system directory), configure the root directory on machine B, and start the IoTDB of machine B.
2. Migrate the data of part of the storage group of IoTDB on machine A to machine B
To migrate data, metadata must be migrated. First copy the data/system directory of machine A to machine B, then copy part of the storage group directory in the data directory of machine A to machine B, and configure the corresponding directory on machine B. Just start the IoTDB of machine B.
3. Migrate the metadata of IoTDB on machine A to machine B, but don’t need the data
Copy the data/system directory of machine A to the corresponding location of machine B, and start the IoTDB of machine B.
System monitoring
The memory and CPU of the monitoring system are still very cool. Let everyone be refreshed. The following applies to versions after 0.9.2, mainly using jvisualvm, and jdk8 is required locally.
If the machine is running IoTDB, directly enter jvisualvm on the command line of the machine to connect to the machine's IoTDB process.
If you want to run IoTDB on the server, you need to modify the configuration file conf/iotdb-env.sh first,
JMX_LOCAL="false"
JMX_IP="the_real_iotdb_server_ip" # 填写实际IoTDB的IP地址
Check conf/jmx.password. The default JMX user and password are recorded here, which can be modified. The permissions of each user are in conf/jmx.access.
Then establish a remote connection, the default port is 31999, you can see it after connecting:
to sum up
Today I introduced the directory structure of IoTDB, revisited and migration methods, and finally introduced the system monitoring method. Welcome to follow and forward!