foreword
At present, we use various storage services for most projects, such as oss, cos, minio, etc. Of course, due to various reasons, it may be necessary to migrate data between different storage services, so today I will introduce a more general data migration tool Rclone.
Tip: The following is the main text of this article, the following cases are for reference
1. What is Rclone?
Rclone is a command line program for managing files on cloud storage. It's a feature-rich alternative to cloud providers' web storage interfaces. Over 40 cloud storage products support rclone, including S3 object storage, business and consumer file storage services, and standard transfer protocols. Detailed recommendation to learn directly from the official website: English official website, Chinese website
2. What can Rclone do?
Backup (and encrypt) files to cloud storage.
Restore (and decrypt) files from cloud storage.
Mirror cloud data to other cloud services or on-premises.
Migrate data to the cloud, or between cloud storage providers.
Mount multiple encrypted, cached or diverse cloud storages as disks.
3. Use steps
1. Install Rclone
curl https://rclone.org/install.sh | sudo bash2. Generate a configuration file
The configuration can be selected at will. After the selection is completed, you can re-modify it. There will be modification methods later in the article.
rclone config3. View the generated configuration file
In the path /root/.config/rclone/rclone.conf (rclone.conf is the name configured when the configuration file is generated)
cd /root/.config/rclone4. Modify the configuration file
vim /root/.config/rclone/rclone.confModify the configuration as follows, please modify some parameters according to your own service configuration
[minio-zwy]
type = s3
env_auth = false
provider = Minio
region = cn-east-1
access_key_id = minioadmin
secret_access_key = minioadmin
endpoint = http://127.0.0.1:8000
[minio-lin]
type = s3
env_auth = false
provider = Minio
region = cn-east-1
access_key_id = minioadmin
secret_access_key = minioadmin
endpoint = http://127.0.0.99:80005. Data synchronization
zwy sync to lin
rclone sync minio-zwy:test minio-lin:testor
lin sync to zwy
rclone sync minio-lin:test minio-zwy:testrclone sync 源（配置文件名称）: 源数据Bucket 目标源名称：目标bucket6. Common commands
rclone config - 以控制会话的形式添加rclone的配置，配置保存在.rclone.conf文件中。
rclone copy - 将文件从源复制到目的地址，跳过已复制完成的。
rclone sync - 将源数据同步到目的地址，只更新目的地址的数据。
rclone move - 将源数据移动到目的地址。
rclone delete - 删除指定路径下的文件内容。
rclone purge - 清空指定路径下所有文件数据。
rclone mkdir - 创建一个新目录。
rclone rmdir - 删除空目录。
rclone check - 检查源和目的地址数据是否匹配。
rclone ls - 列出指定路径下所有的文件以及文件大小和路径。
rclone lsd - 列出指定路径下所有的目录/容器/桶。
rclone lsl - 列出指定路径下所有文件以及修改时间、文件大小和路径。
rclone md5sum - 为指定路径下的所有文件产生一个md5sum文件。
rclone sha1sum - 为指定路径下的所有文件产生一个sha1sum文件。
rclone size - 获取指定路径下，文件内容的总大小。.
rclone version - 查看当前版本。
rclone cleanup - 清空remote。
rclone dedupe - 交互式查找重复文件，进行删除/重命名操作。
7. Common operations
**rclone lsd:** List all directories, containers, and buckets under the specified path.
# remote is [name] in the configuration file
rclone lsd remote:pathrclone copy: Copy files from source to destination, skipping the already copied ones.
# `rclone copy` Copy the file in the specified path
rclone copy source:sourcepath dest:destpath**rclone sync:** The data in the path directory is always synchronized (empty directories will not be synchronized), not the path directory. When synchronizing data, the data of the destination address may be deleted; it is recommended to use the --dry-run flag to check the data to be copied and deleted first. When there is an error in synchronizing data, the data of any destination address will not be deleted.
rclone sync source:path dest:path**rclone move:** When synchronizing data, the data of the destination address may be deleted; it is recommended to use the --dry-run flag to check the data to be copied and deleted.
rclone move source:path dest:path**rclone purge:** Empty the path directory and data.
rclone purge remote:path**rclone mkdir:** creates the path directory.
rclone mkdir remote:pathSummarize
The above is all the content introduced today. The rclone sync method introduced here cannot achieve real-time synchronization during the synchronization process. That is, during the synchronization process, if a new file is uploaded to the synchronized file, the file will be lost. Of course, Rclone also provides other methods to solve this problem. Interested friends can go to the official website to learn English official website and Chinese website directly. You can also refer to this for learning: reference blog.