Practical tools|Teach you how to use the backup artifact Rclone, hand-in-hand nanny-level tutorial

Table of contents

What is Rclone

Function

What is Rclone

Rclone is a command-line program for managing files on cloud storage. It is a feature-rich alternative to cloud providers' web storage interfaces. More than 40 cloud storage products support rclone, including S3 object storage, commercial and consumer file storage services, and standard transfer protocols.

Rclone has powerful cloud functions comparable to unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm and cat. Rclone's familiar syntax includes shell pipe support and --dry-run protection. It is used on the command line, in scripts or through its API.

Users called rclone the "Swiss Army knife of cloud storage" where "technology and magic are indistinguishable".

Rclone really takes care of your data. It always preserves timestamps and verifies checksums. Transfers over limited bandwidth; intermittent or quota-limited connections can restart from the last good file transfer. You can check the integrity of the file. Where possible, rclone employs server-side transports to minimize this.

Tip: The above is translated from the official website.

Function

Rclone supports but is not limited to the functions listed below:

Backup (and encrypt) files to cloud storage
Recover (and decrypt) files from cloud storage
Mirror cloud data to other cloud services or locally
Migrate data to the cloud, or between cloud storage providers
Mount multiple, encrypted, cached or diverse cloud storage as disk
Analyze and illustrate data held on cloud storage using lsf, ljson, size, ncdu
Union File System presents multiple local and/or cloud file systems together as one

Tip: The above is translated from the official website.

characteristic

Pick-up service
- Always check MD5, SHA1 hashes to ensure file integrity
- Timestamps are preserved in the file
- Operation can be restarted at any time
- Can go in and out of the network, e.g. two different cloud providers
- Can download to local disk using multithreading
Copy new or changed files to cloud storage
Synchronize (one way) to make directories the same
Move files to cloud storage delete local after verification
Check hashes and missing/extra files
Mount your cloud storage as a network disk
Serve local or remote files via HTTP/WebDav/FTP/SFTP/DLNA
Experimental web-based GUI

Tip: The above is translated from the official website.

Supported Providers

virtual provider

use

The official documentation is written in great detail. Here we mainly write a few application scenarios to see how to use it. The following uses ubuntu as an example to demonstrate:

Install

Rclone is a binary file package, which can be directly downloaded and set to use environment variables. Here, the official script is used to install directly.

curl -O https://downloads.rclone.org/rclone-current-linux-amd64.zip
unzip rclone-current-linux-amd64.zip
cd rclone-*-linux-amd64

sudo cp rclone /usr/bin/
sudo chown root:root /usr/bin/rclone
sudo chmod 755 /usr/bin/rclone

Direct execution can check the version number and the installation is successful. rclone version

configuration

Configuration can be done by directly adding a configuration file or by entering an interactive configuration session command to complete the configuration step by step.

After the default configuration is completed, the configuration files are saved in the: /root/.config/rclone/rclone.conf directory.

Here I directly give the configuration file of Tencent COS.

[tencent-cos] # 自定义的名称
type = s3  # 存储类型，参考官方文档所有支持的类型
provider = TencentCOS  # 提供商，参考官方文档或者全部
env_auth = false   # 不通过环境变量配置认证
access_key_id = AKxxxxxxxx  # 腾讯云后台生成的密钥key
secret_access_key = Secretxxxxxxx # 腾讯云后台生成的密钥secret
endpoint = cos.ap-chengdu.myqcloud.com # 腾讯云cos所在的地区，看你所在存储桶的公网地址

grammar

# 本地到网盘
rclone [功能选项] <本地路径> <配置名称:路径> [参数] [参数]
# 网盘到本地
rclone [功能选项] <配置名称:路径> <本地路径> [参数] [参数]
# 网盘到网盘
rclone [功能选项] <配置名称:路径> <配置名称:路径> [参数] [参数]

# [参数]为可选项

Example:

# 同步本地/data/file的文件夹内容到tencent-cos存储下的/beifen文件夹中,并且排除/root/excludes.txt中指定的文件内容
rclone sync /data/file tencent-cos:/beifen --exclude-from '/root/excludes.txt'

# 两个网盘文件同步
rclone copy 配置网盘名称1:网盘路径 配置网盘名称2:网盘路径

command list

Use rclone --help to view all commands, only commonly used commands are listed here, and other commands can be found in official documents for details:

Order	illustrate
rclone copy	copy
rclone move	Move, if you want to delete the empty source directory after moving, add --delete-empty-src-dirs parameter
rclone mount	mount
rclone sync	Synchronization: Synchronize the source directory to the target directory, only the target directory is changed
rclone size	Check the file size of the network disk
rclone delete	Delete the contents of the file under the path
rclone purge	Delete a path and all its file contents
rclone mkdir	Create a directory
rclone rmdir	delete directory
rclone rmdirs	Delete empty directories under the specified environment. If the --leave-root parameter is added, the root directory will not be deleted
rclone check	Check if source and destination address data match
rclone ls	List all files and file sizes and paths under the specified path
rclone lsl	One more display upload time than above
rclone lsd	List the directories under the specified path
rclone lsf	List the directories and files under the specified path

Common parameters

Order	illustrate
-n = --dry-run	Test run to see what operations rclone will perform in actual operation
-P = --progress	Display real-time transmission progress, refresh once every 500 ms, otherwise refresh once every minute by default
--cache-chunk-size 5M	The size of the block, the default is 5M. The larger the upload, the faster the upload, and the more memory it takes up. If it is too large, the process may be interrupted
--onedrive-chunk-size 100M	Improve OneDrive upload speed for G-mount broadband servers
--drive-chunk-size 64M	Improve Google Drive upload speed for G-mount broadband servers
--cache-chunk-total-size SizeSuffix	The total size a block can occupy on local disk
--transfers=N	The number of parallel files, the default is 4. It is recommended to reduce this parameter on a VPS with relatively small memory, for example, it is recommended to set it to 1 on a 128M chick.
--config string	Specify the configuration file path, string is the configuration file path
--ignore-errors	skip error
--size-only	Check according to file size, do not check hash
--drive-server-side-across-configs	server-to-server transfer

log

rclone has 4 levels of logging, ERROR, , NOTICE, INFO and DEBUG. By default, rclone will generate ERROR and NOTICE level messages.

-q - rclone will only generate ERROR messages.
-v - rclone will generate ERROR, NOTICE and INFO message, recommending this .
-vv - rclone will generate ERROR, NOTICE, INFOand DEBUG messages.
--log-level LEVEL The - flag controls the log level.

filter

If we need to filter out some unimportant files during synchronization, such as program startup logs and sensitive files.

Order	illustrate
--exclude-from	Exclude list of files or directories from a file specified
--include-from	Include files or directories from a file specified
--filter-from	File filtering rules come from a file specified
--exclude	Exclude files or directories
--include	包含文件或目录
--filter	文件过滤规则，相当于上面两个选项的其它使用方式。包含规则以+开头，排除规则以-开头

文件类型过滤

比如 --exclude "*.bak"，表示排除所有 .bak 结尾的文件，也可以写作 --filter "- *.bak"

比如 --include "*.{png,jpg}"，包含所有 png 和 jpg 文件，排除其他文件，也可以写作 --filter "+ *.{png,jpg}"

--delete-excluded 删除排除的文件。需配合过滤参数使用，否则无效。
目录过滤

目录过滤需要在目录名称后面加上 /，否则会被当做文件进行匹配。以 / 开头只会匹配根目录（指定目录下），否则匹配所目录。这同样适用于文件。

--exclude ".git/" 排除所有目录下的.git 目录。

--exclude "/.git/" 只排除根目录下的.git 目录。

--exclude "{Video,Software}/" 排除所有目录下的 Video 和 Software 目录。

--exclude "/{Video,Software}/" 只排除根目录下的 Video 和 Software 目录。

--include "/{Video,Software}/**" 仅包含根目录下的 Video 和 Software 目录的所有内容。
大小过滤

默认大小单位为 kBytes ，但可以使用 k ，M 或 G 后缀。

--min-size 过滤小于指定大小的文件。比如 --min-size 50 表示不会传输小于 50k 的文件。

--max-size 过滤大于指定大小的文件。比如 --max-size 1G 表示不会传输大于 1G 的文件。
过滤规则来自某个文件

--exclude-from <规则文件> ：从文件添加包含规则

--include-from <规则文件>：从文件添加删除规则

--filter-from <规则文件> ：从文件添加包含 / 排除规则。比如 --filter-from filter-file.txt。

filter-file.txt 示例：
```
- secret*.jpg
+ *.jpg
+ *.png
+ file2.avi
- /dir/Trash/**
+ /dir/**
- *
```

环境变量

rclone 中的每个选项都可以通过环境变量设置。环境变量的名称可以通过长选项名称进行转换，删除 -- 前缀，更改 - 为 _，大写并添加前缀 RCLONE_。环境变量的优先级会低于命令行选项，即通过命令行追加相应的选项时会覆盖环境变量设定的值。

比如设置最小上传大小 --min-size 50，使用环境变量是 RCLONE_MIN_SIZE=50。当环境变量设置后，在命令行中使用 --min-size 100，那么此时环境变量的值就会命令行覆盖，最终生效的是 --min-size 100。

命令	说明
RCLONE_CONFIG	自定义配置文件路径
RCLONE_CONFIG_PASS	若 rclone 进行了加密设置，把此环境变量设置为密码，可自动解密配置文件
RCLONE_RETRIES	上传失败重试次数，默认 3 次
RCLONE_RETRIES_SLEEP	上传失败重试等待时间，默认禁用，单位s、m、h分别代表秒、分钟、小时
CLONE_TRANSFERS	并行上传文件数
RCLONE_CACHE_CHUNK_SIZE	块的大小
RCLONE_CACHE_CHUNK_TOTAL_SIZE	块可以在本地磁盘上占用的总大小
RCLONE_IGNORE_ERRORS=true	跳过错误

最佳实践

备份本地 /data/file 目录到腾讯云cos存储中，并设置定时任务每晚自动同步，这样可以达到定时备份重要文件上云的效果。

目录 /data/file 下创建 excludes.txt ，里面填写需要排除的文件列表。

logs/**  # 排除logs下所有文件
*.bak # 排除.bak结尾的文件
*.zip # 排除.zip结尾的文件

配置腾讯云cos在此忽略，直接编写 rclone 命令，测试执行是否同步。
```
rclone sync -v /data/file tencent-cos:/beifen --exclude-from '/data/file/excludes.txt' >> /root/rclone.log 2>&1
```
手动执行一把，然后查看 tail -f /root/rclone.log 是否输出日志了，只要没报错则成功。
配置定时任务，每天自动执行

执行 crontab -e # 打开定时任务列表，跳到最后新起一行添加步骤2.成功的命令并保存，内容如下：（表示每晚3点30分进行同步）
```
30 4 * * * rclone sync -v /data/file tencent-cos:/beifen --exclude-from '/data/file/excludes.txt' >> /root/rclone.log 2>&1
```
执行systemctl restart cron.service 重启定时任务生效
达到执行时间后检查 /root/rclone.log 下的日志保证没问题。