Image batch migration tool: image-transfer

Overview

In the process of cloud migration or cloud migration, user services need to be migrated in batches. Based on this background, the Tencent Cloud container expert team developed an image batch migration tool: image-transfer. This tool supports batch migration between mirror warehouses of multiple cloud vendors, and also supports one-click full migration of Tencent Cloud Mirror Warehouse TCR Personal Edition CCR to Tencent Cloud Mirror Warehouse Enterprise TCR.
This article first introduces the pain points of image migration in the process of business cloud/migration. Subsequently, the design ideas, functional modules and best practices of the image-transfer tool image-transfer are introduced in detail.
The tool is officially open source, the project address: https://github.com/tkestack/image-transfer

Binary package download address: https://github.com/tkestack/image-transfer/releases

How to migrate business to the cloud?

There are several scenarios for business cloud migration, one is self-built idc to cloud, the second is third-party cloud migration, and the third is remote disaster recovery, that is, hybrid cloud. In these scenarios, whether it is cloud migration or cloud migration, the migration process mainly includes the following steps.

Network Planning. vpc configuration, subnetting, etc.
data migration. Object storage, file storage, etc.
Application migration. Business, configuration, etc.

In the process of data migration, if a large number of business uses containerized deployment, batch image migration is required. At present, the mirror warehouse services of most cloud vendors do not provide the ability to migrate mirror warehouses in batches. If you want to migrate a business, you can only download the original image locally, modify the tag, and upload it to the destination image warehouse. This process has the following disadvantages:

Time-consuming and labor-intensive. Only for a single image migration, three steps are required, and you need to keep an eye on it. If a failure occurs, further processing is required.
Completely human operation, easy to make mistakes. When modifying the tag of the original image, it is very easy to make mistakes due to manual modification, and the tag modification is wrong.
When the amount of mirroring is large, the cost increases and the progress of cloud migration is slow. If there are hundreds or thousands of mirrors, such a human operation, the migration progress will be very slow, and the migration cost will be greatly increased.

The principle of image-transfer

Design goals

For the pain points of mirror migration. We hope to develop an image batch migration tool. For users, it only needs simple configuration to realize batch migration of images without manual intervention, which improves the progress of business cloud/migration and reduces costs. Specifically, the tool has the following design goals.

The configuration is simple and no complicated input is required . We hope that this tool is simple and easy to use for users. You only need to configure the source and destination mirror addresses and authentication to complete the batch migration.
Migrate massive images quickly and efficiently, reducing migration costs . In response to a large number of mirror migration needs, we hope that the tool can be efficiently completed, and with the help of some high-concurrency technologies, to achieve rapid migration.
Certain fault tolerance, reducing manual intervention and maintenance . During the migration process, we hope that the tool can identify and automatically repair some errors, minimize manual maintenance, and improve operation and maintenance efficiency.
There is no dependence on the operating environment, and the universality of tools is improved . We hope that the tool can run in any linux, mac os, windows operating system, without relying on docker and other programs.
Support Tencent Cloud CCR one-click full migration mode . At present, Tencent Cloud Container Image Service TCR Enterprise Edition is officially launched, and Tencent Cloud TCR Personal Edition (CCR) will gradually reduce maintenance in the future until it goes offline. Therefore, the tool needs to support the one-click full migration of the CCR warehouse to the TCR Enterprise Edition.
Support custom qps speed limit. When migrating images in batches, frequent calls to the mirror warehouse interface may cause the mirror warehouse to crash, so qps needs to be restricted.

Architecture

Image-transfer is composed of multiple modules. The following figure shows the image-transfer architecture.

Image batch migration tool: image-transfer

General mode input (default) : used to accept the mirroring migration task issued by the user. Including mirror migration configuration files and authentication configuration files. This mode is used to implement image migration between cloud vendors.
CCR one-click migration mode input : you need to add --ccrToTcr=true to the tool input parameter. This mode is used for one-click migration of CCR warehouse to TCR Enterprise Edition. In addition to adding the --ccrToTcr=true parameter, authentication configuration files and Tencent Cloud secret configuration files are also required.
pipeline : This module is the core of the tool to handle image migration. Responsible for processing mirror migration tasks issued by users, including processing the synchronization rules of the mirror warehouse according to the migration configuration file, and performing hierarchical pull and transfer tasks of mirrors. The module uses a highly concurrent pipeline model to improve the migration speed.
Retry task: This stage will retry the failed transfer task in the pipeline. The number of retries can be determined according to user input parameters, and the default is 2 times.

Pipeline

Since the tool is written in golang language, the core processing module uses go's pipeline high concurrency model. The entire pipeline module is divided into three small modules.

Multi-coroutine processing mirror warehouse synchronization rules

Here is the processing and analysis of the mirror migration configuration file input by the user to obtain each source mirror address (including repo and tag) that needs to be transferred, and the corresponding destination mirror address. Then form a job for each source mirror address and destination mirror address. If the source mirror tag is not specified in the mirror configuration file, all tags under the repo will be pulled, and then a job will be formed for each tag. This process uses golang's multi-coroutine approach to increase processing speed. The number of coroutines can be specified by the user in the input parameter --proc, the default is 5. After each job is composed, it will be put into the task channel, waiting to be consumed.

Task transmission channel

The task channel can be regarded as a simple middleware, implemented by the golang channel. After each job is produced, it will be put into the channel and wait to be consumed. This design can ensure that the job produced by the producer will be immediately put into the consumption line, and once the consumer end is free, consumption processing can be carried out. Improve job processing efficiency.

Multi-coroutine processing task

These coroutines are the consumer end of the job. After getting the job, it will first pull the manifest of the source address in the job to determine whether it is a multi-manifest mirror, then pull each blob, then transfer the blob to the destination address, and finally transfer the manifest to the destination address. The process is to use the cache, the data is not placed on the disk, and the efficiency is improved. This process uses golang's multi-coroutine approach to increase processing speed. The number of coroutines can be specified by the user in the input parameter --routines, the default is 5.

Best practices for image migration

This section will introduce how to use the image-transfer tool to implement batch image migration in different scenarios. The scenarios are as follows:

Image migration between different cloud vendors. For example, migrating from ACR, Alibaba Cloud Mirror Warehouse, to TCR, Tencent Cloud Mirror Warehouse.
The open source/self-built mirror warehouse is migrated to the cloud. For example, migrating from the harbor mirror warehouse to the Tencent Cloud mirror warehouse TCR.
Tencent Cloud TCR Personal Edition (CCR) is migrated to Tencent Cloud Mirror Warehouse Enterprise Edition TCR with one click.

Tool installation:

First, download and compile the tool, there are two ways, one is to directly obtain the binary file, and the second is to download the source code and compile.

Download address of binary release package:

https://github.com/tkestack/image-transfer/releases

The example uses the downloaded source code to compile for demonstration:

The compilation process is also very simple, enter the source code directory and directly make.

git clone https://github.com/tkestack/image-transfer.git
cd ./image-transfer
make

After the compilation is complete, the image-transfer binary file will be generated in the current directory. Ready to use. Next, perform a best practice demonstration.

Best practice scenario 1: Image migration between different cloud vendors

Take the example of migrating from ACR Mirror Warehouse of Alibaba Cloud to TCR Mirror Warehouse of Tencent Cloud.

1. Prepare the access credential information file for Tencent Cloud Mirror Warehouse TCR and Aliyun Mirror Warehouse ACR: auth.json

   grant-test.tencentcloudcr.com:
   username: xxx
   password: xxx
   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   registry.cn-hangzhou.aliyuncs.com:
   username: xxx
   password: xxx
   ccr.ccs.tencentyun.com:
   username: xxx
   password: xxx
   registry.hub.docker.com:
   username: xxx
   password: xxx

The configuration is very simple. Enter the address of the source mirror warehouse and the address of the destination mirror warehouse. And enter the user name and password corresponding to the mirror warehouse.

Whereinsecure indicates whether the registry is an http service, if it is, the secure field needs to be true, the default is false, and it is optional.

The user of the destination mirror warehouse needs to have push and create warehouse permissions. If not provided, anonymous access will be made by default.

The access credentials for Tencent Cloud TCR are obtained as follows:

The access credentials for ACR Mirror Warehouse ACR are obtained as follows:

2. Prepare the mirroring rule file to be migrated: rule.yaml

registry.cn-hangzhou.aliyuncs.com/grantzhao/sichenzhao:xx": "grant-test.tencentcloudcr.com/grantzhao/sichenzhao

This file is the source image and destination image to be transferred. The file rules are: source mirror address: destination mirror address,
where the source mirror address can be specified, or no tag, or multiple tags can be specified.
When specifying a single tag: the destination address may or may not contain the tag. If the tag is not included, the source mirror tag is used.
When tag is not specified: the destination address must contain tag.
When multiple tags are specified: separate multiple tags with English commas, such as grant-test.tencentcloudcr.com/grantzhao/sichenzhao:1.0,2.0,3.0. At this time, the destination address cannot contain the tag, and the tag of the source address is used by default.

3. Run the tool

   ./image-transfer --routines=5 --securityFile=./security.yaml --ruleFile=./rule.yaml --ns=default \
   --registry=grant-test.tencentcloudcr.com --retry=2 --qps=100

Parameter explanation:

--ns specifies a default ns, if the ns of the destination warehouse is empty, it will be replaced by the default ns.

--registry specifies a default registry. If the registry of the destination warehouse is empty, it will be replaced by the default registry.

--routines=5, which means to set the number of concurrent to 5. The default is 5.

--retry=2 means the number of retries after failure is 2, and the default is 2.

--securityFile, specify the authentication file.

--ruleFile, specify the mirror warehouse configuration file.

--qps, limit the requested qps not higher than 100/s.

4. Run results

Image batch migration tool: image-transfer
the last line

################# Finished, 0 transfer jobs failed, 0 jobs generate failed #################

Indicates successful operation.

Best practice scenario 2: Open source/self-built image warehouse migration to the cloud

Take the migration from the open source mirror warehouse docker hub to the Tencent Cloud mirror warehouse TCR as an example.

1. Prepare docker hub and the access credential information file of Tencent Cloud Image Warehouse TCR: security.yaml

   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   registry.hub.docker.com:
   username: xxx
   password: xxx

2. Prepare the mirroring rule file to be migrated: image.json

   sichenzhao/private-test:xxx": "grant-test2.tencentcloudcr.com/grantzhao/sichenzhao

3. Run the tool

   ./image-transfer --routines=5 --securityFile=./security.yaml --ruleFile=./rule.yaml --ns=default \
   --registry=grant-test.tencentcloudcr.com --retry=2

4. Run results

Image batch migration tool: image-transfer
the last line

################# Finished, 0 transfer jobs failed, 0 jobs generate failed #################

Indicates successful operation.

Best practice scenario 3: Tencent Cloud TCR Personal Edition (CCR) one-click migration to Tencent Cloud Mirror Warehouse Enterprise Edition TCR

The usage in this scenario is slightly different from the above two scenarios. Mainly manifested as changes in input parameters.

1. Prepare the image authentication configuration file security.yaml

   grant-test.tencentcloudcr.com:
   username: xxx
   password: xxx
   grant-test2.tencentcloudcr.com:
   username: xxx
   password: xxx
   ccr.ccs.tencentyun.com:
   username: xxx
   password: xxx

2. Prepare Tencent Cloud secret configuration file secret.yaml

For the one-click migration mode of TCR, the user name and password of the warehouse are not required as access authentication, but the secret information of Tencent Cloud is used.

   ccr:
   secretId: xxx
   secretKey: xxx
   tcr:
   secretId: xxx
   secretKey: xxx

Note: The
file format is shown above, only the secretId and secretKey items are allowed to be modified.

If there is no secret information of ccr, tcr will be used instead. On the contrary, if there is no secret information of tcr, ccr will be used instead.

The secret information is obtained as follows:

contains two information, secretid and secretkey

3. Run the tool

The parameter input here is slightly different from the above two scenarios.

   ./image-transfer --ccrToTcr=true --routines=5 --securityFile=./security.yaml --secretFile=./secret.yaml --tcrName=tcr-test \
   --retry=3 --tcrRegion=ap-guangzhou --ccrRegion=ap-guangzhou --qps=3000

Parameter explanation:

--ccrToTcr=true, means to enable TCR one-key full migration mode.

--secretFile, provides secret.yaml configuration file.

--tcrName=tcr-test, specify the name of the destination tcr warehouse.

--tcrRegion, specify the region where the destination tcr warehouse is located.

--ccrRegion, specify the region where the source ccr warehouse is located.

4. Run results

Image batch migration tool: image-transfer

One-click batch migration will take a long time, because all images of ccr need to be transferred to tcr.

Finally, you can see that 16 jobs have failed. The tool will finally list the source mirror address and destination mirror address of the failed job. For these failed jobs, I went to the warehouse to check and found that the tags of these jobs were invalid. Therefore the transmission fails.

to sum up

This article introduces the image batch migration tool: image-transfer from the aspects of problem analysis, design goals, principle analysis, and best practices. Everyone is welcome to contribute source code, and also welcome to raise issue requirements.