Scene AzCopy use is actually very much, but we can help solve many problems, this time to share two different scenarios, how AzCopy is to help us solve the problem
First look at the first two scenes are what is
1. Need blob download stored on the Azure Global
2. The need to synchronize data between the blob and Azure Global Azure China
Both are actual cases, first look at the first
There is a background vhd file on Azure Global 60T of storage, the file is shared publicly third-party vendors, anyone can download directly, without going through authentication, and we need to do is put this on Global's vhd file migration to China, and then deploy
We tried a variety of methods
1. Direct Download - rate remained at the level kB, very unstable
2. Use major online disk and offline download download tool - simply does not move under
3. Use the tools of science to download indescribable - at the start of the speed is very objective, but after some time speed becomes 0
4. Create a table directly in the Global Azure VM download - MB level in speed position, stability is acceptable but after waiting for some time, interrupted downloads
Several methods are eventually identified as infeasible, eventually solved the problem using AzCopy
First we installed azcopy, and then build a storage container in the Chinese area to store data copies to the
These are all preparatory work
After time we will look at how to copy data, first in the source and target-side are azure blob of, azcopy there will be some restrictions
SAS must append a token to each source URL.
If Azure Active Directory (AD) to provide authorization credentials, the token can be omitted from the SAS target URL.
So, our source must be carried out with SAS certification, and our situation is the direct source is open, so I do not need the SAS
The end goal we need to use to authenticate the Azure AD
Azure AD to authenticate we have said many times, specifically how to operate it, azcopy v10 that allows users to login, so it also shows that we can log in using our azure accounts, and azure users now can be assigned directly read and write access to the blob, relevant role mainly in the following
Storage Blob Data Reader
Storage Blob Data Owner
Storage Blob Data Contributor
Concepts and conventional role somewhat similar to it, we need here is written to the file, so it must not be used reader, we directly assign a permission of the owner
After we started to migrate first to be logged in target-side environment
azcopy login --aad-endpoint https://login.partner.microsoftonline.cn
This visit marked red url in the browser, then enter the code, and the whole process and powershell cli under linux is actually the same
Registration is completed, we try to list the contents of the lower container, you can see has been able to list the contents of the corresponding
azcopy list https://mxy.blob.core.chinacloudapi.cn/test
After we directly use the copy command to copy the vhd file to the container in the Global
azcopy copy https://xxxxx.blob.core.windows.net/release/xxxxx.vhd https://.blob.core.chinacloudapi.cn/sms/sms-connector.vhd
The whole process was completed half an hour, the average speed can reach 400-500Mbps
The reason why the rate can be achieved so quickly, because AzCopy use server-to-server api, therefore replicate data directly between storage servers. The copy operation does not use the computer's network bandwidth.
Let's look at the second scenario:
Synchronize data between the blob, such as synchronization and is likely to be more than a one-time synchronization, it is likely to run on a regular basis after, the source and target-side we do authentication with SAS
About SAS, limited space here is not to go into details, we directly generate a level of SAS storage
After using azcopy sync command to synchronize
azcopy sync 'https://mxxxc.blob.core.windows.net/$web?sv=2019-02-02&ss=bfqt&srt=sco&sp=rwdlacuxxxxxx&st=2020-02-10T03:42:25Z&spr=https,http&sigxxxxxxc%3D' 'https://xxx.blob.core.chinacloudapi.cn/global?sv=2019-02-02&ss=bfqt&srt=sco&xxxxx5:59Z&st=2020-02-10T04:35:59Z&spr=https,http&sigxxxxxx3D' --recursive --s2s-preserve-access-tier=false
Can be seen in the target-side data has come
Then we upload a new file at the source
Run the command again, only this time you can see a synchronized file
In this way, we will soon be able to gloabl data synchronized to the China region, and security can be guaranteed
These are used in two cases azcopy