background:
Faced with customers constantly migrating the storage volume of friends and merchants to Alibaba Cloud. The ossimport tool is more and more exposed on the user side, but the reasonable use of the ossimport tool and good migration schema data can help users to migrate efficiently and quickly. However, if you are not familiar with ossimport, and the migration architecture has not been tested, it will reduce our migration efficiency and affect the customer's overall strategic cloud plan.
Evolution of Migration Architecture:
Traditional migration method:
Local localfile -> Migrate to OSS cloud
Third-party storage -> Migrate to OSS cloud
The above traditional migration methods will encounter a functional problem, that is, the interference factor of the public network is unavoidable, especially in the current domestic network environment is complex, it is difficult to ensure that the public network is free from congestion and jitter, even if the physical bandwidth is large. . So there is an improvement in the vpc network environment.
Evolving VPC Environment Migration:
After introducing the concept of VPC, users solved the slow headache caused by the network and brought a series of benefits. The concept of VPC here refers to (transmission through OSS intranet or VPC dedicated line transmission)
- Migrate data sources to the OSS side, no bandwidth traffic restrictions (excluding ECS internal restrictions)
- VPC intranet environment migration, use OSS intranet domain name, no traffic fee.
- Compared with the public network, the delay in the VPC environment is extremely low, and there is basically no packet loss unless there is a line failure
Third-party storage -> data disk copy -> IDC machine -> VPC dedicated line push -> OSS cloud
Third-party storage <- pull ECS -> VPC dedicated line push -> OSS cloud
Third-party storage <- pull ECS -> OSS intranet endpoint push -> OSS cloud
Local file -> OSS intranet endpoint push -> OSS cloud
Local file -> VPC dedicated line push -> OSS cloud
Reasonable migration configuration:
At present, it is recommended that customers migrate according to the VPC environment, which greatly improves the service experience. During the migration process, the OSS SLA does not promise clear indicators of the migration speed, and the migration data is related to multiple factors (machine configuration, data volume, network, thread, current limit, etc.), so the problem is diagnosed according to the actual situation.
First of all, before migrating, users are required to learn about our configuration attribute description, and the function of related files, log storage, exception handling, etc.
https://help.aliyun.com/document_detail/56990.html?spm=a2c4g.11174283.6.1079.sbu1ch
To configure the migration file stand-alone version:
Before migration, it is necessary to clarify the migration volume and the number of files of users. The purpose is to reasonably configure the number of tasks and threads, as well as the working model of ossimport.
Generally, if the migration volume is less than 30TB, it can be migrated in stand-alone mode. The stand-alone version can be migrated in multi-threaded mode, and the workerTaskThreadNum parameter can be adjusted. It should be noted that if it is a high-end, physical machine, the number can be increased. You can refer to the average File size * number of threads, compare whether the memory is sufficient, and also refer to whether the number of CPU cores can withstand this concurrency.