The core concept and purchase application of Alibaba Cloud OSS object storage

1. Basic introduction to OSS object storage

Official documentation of OSS object storage: https://help.aliyun.com/product/31815.html

1.1. OSS object storage concept

The full name of OSS Object Storage Service (Object Storage Service, OSS), is a massive, safe, low-cost, high-reliability cloud storage service, suitable for storing any type of file.

Elastic expansion of capacity and processing power, multiple storage types to choose from, and comprehensive optimization of storage costs.

OSS object storage provides a RESETful API interface. All upload and download files are completed through the API interface. As long as there is a network, you can access the files in OSS storage at any time and place.

When uploading a large file, OSS object storage will divide the file into pieces, and after all the pieces are uploaded to OSS, all the pieces will be aggregated into one file.

1) OSS component concept

• Storage type (Storage Class)
  • OSS provides four types of data storage, distributed standard, infrequent access, archive, and cold archive.
  • The standard storage type provides high-durability, high-availability, and high-performance object storage services, which can support frequent access to data, and standard storage is generally selected.
  • The infrequently accessed storage type is suitable for long-term storage of infrequently accessed data, and the price is relatively low.
  • The archival storage type is suitable for data that needs to be preserved for a long time.
  • The cold archive storage type is suitable for data that needs to be stored for a long time.
  • Which type of storage will we choose to use when we purchase the OSS resource pack
• Storage space (Bucket)
  • The storage space is equivalent to resource isolation, which is used to store different application data in different storage spaces, one bucket for each application.
  • A storage space is a container for storing objects. Each object must belong to a certain storage space. The storage space has various configuration attributes, including region, access rights, storage type, etc., which can be used for different applications according to our needs. Programs create different storage spaces.
• object
  • An object is the basic unit for storing data in OSS. It can be understood that a file stored in OSS is an object.
  • An object consists of meta information (Object Meta), user data (Data) and file name (Key).
  • Objects are identified by a unique Key inside the storage space. Object meta information is a set of key-value pairs that represent some attributes of the object, such as last modification time, size, and other information.
• ObjectKey
• Region
  • The region indicates the physical location of the OSS, and we can run the OSS in the area closest to the user
• Access Domain Name (Endpoint)
  • OSS has an external network address and an internal network address. When we want to read data in OSS from ECS in other regions, we need to connect through the external network address of OSS. The external network address is a domain name, and the OSS in each region Each has its own domain name.
  • The domain names required to access the same region through the intranet and the extranet are also different
• Access Key (AccessKey)
  • AccessKey, referred to as AK, refers to the AccessKey ID and AccessKey Secret used in access authentication.
  • AccessKey, referred to as AK, refers to the AccessKey ID and AccessKey Secret used in access authentication.
  • OSS uses Key+Secret symmetric encryption to verify the identity of the requester, and resources in OSS can only be accessed after successful verification.
  • The AccessKey ID is used to identify the user; the AccessKey Secret is the key used by the user to encrypt the signature string and OSS to verify the signature string, and must be kept secret.

2) Important features of OSS

• version control
  • Data protection features at the storage space level.
  • After version control is enabled, data overwriting and deletion will be recorded in the form of versions. When we accidentally delete a file, we can restore the stored object to the version at any time in the bucket.
• Bucket Policy
  • It is mainly used to authorize the access permissions of different users for the storage space.
  • For example, you need to authorize an account with read-only, read-write, and full control permissions for the entire storage space.
• Replication across regions
  • Operations such as creating, updating, and deleting object resources can be copied across OSS data centers in different regions from the source storage space to the target storage space in different regions, which can be used as a data disaster recovery solution.
• data encryption
  • When uploading a file, OSS encrypts the received file, and then persists the encrypted file; when downloading a file, OSS automatically decrypts the encrypted file and returns it to the user, and declares the file in the returned HTTP request header Server-side encryption is done.

1.2. Differences between NAS and OSS storage

NAS is a file storage system, which is a storage service for ECS instances. Similar to traditional NFS services, you need to mount NAS to an ECS server to store files. NAS must be in the same region as ECS, otherwise ECS is Unable to connect to NAS, that is to say, if we want to migrate ECS to other regions, NAS must also be migrated, otherwise it will not be available.

OSS is an object storage that provides a RESETful API interface. Each file stored in OSS has a separate access address. We only need to configure the address of OSS and the Bucket used in the program to connect to OSS. Even in the future It is necessary to migrate ECS to another region, and OSS does not need to do any operations, as long as there is a network environment, data can be obtained from OSS.

In addition, if our website has a CDN accelerator, the program images we cache in OSS can also directly request OSS from the CDN without going through the ECS instance.

NAS is mainly used for server file storage and container storage. After the user request reaches the WEB program, the program will find the data in the NAS in a certain path mounted on the server.

OSS is mainly used for cloud data processing, image audio and video file storage, and WEB program dynamic and static resource file storage. OSS is quite a cloud storage. You only need to request the domain name of OSS to get the desired file.

1.3. Application Scenarios of OSS

1) Massive storage for applications such as pictures, audio and video

A large number of files (pictures, files, audio and video) generated by WEB applications can be stored in OSS. The storage provided by OSS is massive. OSS supports two methods: streaming writing and file writing.

As shown in the figure below: the network camera is a program running on the ECS server. The video recorded by the user through the network camera is directly stored in the OSS object storage through the program, and then the video is transcoded through the media program that comes with OSS. When the user needs to play When uploading a video, if the website has a CDN accelerator, the request will first reach the CDN accelerator. If it is cached in the CDN, the video file will be returned to the user directly. If there is no cache in the CDN, the CDN will directly read the video file from OSS. , and then play without going through the ECS server.

The external network address of the OSS connected to the CDN to read the files in the OSS, and the external network address will be different in different regions where the OSS is located.

OSS supports direct writing and reading on the client side.

2) Separation of static and dynamic resources for web pages or mobile applications

Most WEB applications or mobile apps will separate dynamic and static resources. Some companies directly deploy static resources in a server and provide specific domain names for access.

We can also directly store static resources in OSS. When requesting dynamic data, pass through SLB->ECS, and then send the data back to the user. When requesting static data, the request first reaches the CDN accelerator. If the CDN accelerator If there is a static resource cache, then it will be sent back to the user directly. If there is no static resource cache in the CDN, the CND will directly read the static resource from the OSS, and then send it back to the user. There is no need to read the static resource from the SLB and ECS , which reduces the traffic of ECS to some extent.

3) Cloud data processing

After data files are stored in OSS object storage, such as pictures and audio and video, the data can be directly processed through other programs, such as adjusting the picture definition, transcoding and compressing audio and video files, and so on.

OSS is equivalent to a cloud, and other programs obtain data from this cloud.

1.4. OSS Term Correspondence Table

English	Chinese
Bucket	storage
Object	object or file
Endpoint	OSS access domain name
Region	Region or Data Center
AccessKey	The collective name of AccessKeyId and AccessKeySecret, access key
Put Object	simple upload
Post Object	form upload
Multipart Upload	Multipart upload
Append Object	additional upload
Get Object	simple download
Callback	call back
Object Meta	File meta information. Used to describe file information, such as length, type, etc.
Data	file data
Key	file name
ACL (Access Control List)	storage space or file permissions

2. Purchase OSS storage resource pack

Before using OSS, you first need to purchase a storage resource package to provide storage space. If you create a Bucket directly without purchasing a storage resource package, the amount of data in the storage space will be billed on a per-use basis. After purchasing a storage package, the size of the data space will be based on Resource packs are paid for.

The space for storing resource packages in OSS can be mirrored and expanded as needed.

Search for OSS in the console and enter the OSS console.

1) In resource pack management, click Purchase resource pack.

2) Select the standard LRS storage package for the type of resource package, select Common in Mainland China for the region, select 40GB for the storage package specification, select the standard storage package (Mainland China) for the package, and select half a year for the purchase period.

3) Confirm the order information and click to pay.

4) The purchase of the resource pack is complete

3. KodCloud cloud disk access OSS object storage

The general implementation steps of OSS docking application system:

• 1. Create Bucket storage space
• 2. Create a sub-user and grant OSS management rights, and obtain the sub-user's AccessKey and AccessSecret
• 3. Authorize the Bucket storage space to sub-users for management
• 4. Configure the application system to access OSS

KodCloud needs to fill in the following information when accessing OSS, and we need to take this information to create OSS storage.

3.1. Create Bucket storage space

1) Create a Bucket

Click Bucket List -> Create Bucket.

2) Configure Buckets

Bucket name is kodcloud-storage—>region is China North 2—>storage type is standard storage—>disable redundant storage in the same city—>disable version control.

The read and write permissions are private—>Data encryption select None—>Disable real-time directory query—>Disable scheduled backup—>Finally click OK to complete the creation of the bucket.

The most important thing is the read and write permissions. You must choose private, otherwise it is not safe for the data. If the permission is private, the key for verifying identity will be included in the file url. Public read means that everyone can read the files in the bucket. , public reading and writing are the least secure, and everyone can write data to the bucket.

3) View the created Bucket

After the Bucket is created, it will automatically jump to the management page of the current Bucket. In the overview, we can get the domain names for accessing OSS, including intranet domain names and external network domain names, and different buckets also have separate access domain names.

3.2. Create a sub-user to manage Bucket

The bucket storage space has been created. We cannot allow the program to connect through the administrator account of Alibaba Cloud. Therefore, we need to create a sub-user to manage the bucket, assign some permissions to the user, and then authorize the bucket to this user. , we can access the Bucket through the user's Key and Secret, and read or write data in the Bucket.

1) Go to the RAM console

Click Rights Management—>Access Control RAM—>Go to the console.

2) Create a user

Click User in Identity Management -> Create User.

Fill in the user name—>check Enable AccessKey ID and AccessKey Secret—>click OK to complete the user creation.

3.3. Get the user's AccessKey

After the user is created, the user AccessKey will be obtained, and the AccessKey must be copied and saved, otherwise, the Key information cannot be obtained again after refreshing the page.

用户登录名称 [email protected]
AccessKey ID：LTAI5tRMCtvCFthMNXKsKLqE
AccessKey Secret：wWrCbFTCuxxxx6RooH8RbXkbfpni

3.3. Set permissions for users

1) Click the user list again, find the kodcloud-oss user we created, and click Add Permission.

2) Add an AliyunOSSFullAccess permission for the user, which means that the user has administrator-level permissions for OSS, and then click OK.

3) On the user details page, you can see the permissions we added.

3.4. Authorize the Bucket to the created user management

1) Click Permission Management—>Bucket Authorization Policy—>Settings.

2) Click Add Authorization —> Authorize Resources to select the entire Bucket —> Authorize User to select the sub-account just created —> Check Full Control for Simple Settings —> Click OK to complete the authorization.

3) Authorization succeeded.

3.5. Configure the background management of KodCloud cloud disk to access OSS

Enter the background management of the Kodclou cloud disk—>find storage management—>click Add—>storage type is Alibaba Cloud OSS—>name is filled with Alibaba Cloud OSS storage—>space size is 40GB—>Access Key ID fills in the sub-user’s Key ID—>Access Key Secret Fill in the Key Secret of the sub-user—>Bucket fill in the kodcloud-storage storage space—>Regional nodes fill in the domain name of the OSS intranet access—>The storage directory is /kodbox—>Click to save.

The storage is added.

3.6. Upload files to OSS in KodCloud

Enter the Kodclou cloud disk system —> find Alibaba Cloud OSS storage —> click to upload files.

3.7. Check whether there are files generated in OSS

Click the file management in Bucket, you can see that a kodbox directory has been created.

After entering the directory, you can see the files uploaded by the user.

3.8. View the URL address of the file in OSS

Each file in OSS has an independent access address. Our Bucket permissions are private, and there will be Key authentication information in the URL. You must pass the program authentication to access the contents of the file.

Click on any file to view the URL address of the file.

4. What should I do if I forget the Key of the sub-user?

After the sub-user is created, the Accesskey will disappear after refreshing. What should I do if I forget it later? There is also a solution.

Click on the sub-user —> there is a Create AccessKey below —> a new AccessKey will pop up.

After that, you can delete the forgotten key.