Use OSS to quickly build a personal network disk
1. Basic overview
1. Main storage types
Whether on-premises or in a cloud environment, there are three main types of storage: block storage, file storage, and object storage.
1.1 Block Storage
Block storage is a data storage method that stores data in fixed-size blocks .
Block storage divides data into multiple blocks, and the size of each block is usually fixed, such as 4KB, 8KB, 16KB, etc.
The data structure of block storage is relatively fixed and cannot be adjusted freely.
Block storage is usually used in file systems, relational databases and other fields, and is often used to store large amounts of data to improve data read and write efficiency.
The advantage of block storage is that it has high storage efficiency and can store and retrieve data quickly .
1.2 File Storage
File storage is a data storage method that stores data as files .
File storage divides data into multiple files, and each file can contain multiple data blocks. File storage is usually used to store a large amount of data, which can be managed and accessed through the file system and other methods.
The advantage of file storage is that it can manage and access data flexibly , and it can support large-scale data storage and management
1.3 Object Storage
Object storage is a new type of cloud storage service that stores data in the form of objects, which can better meet the storage and management needs of a large amount of unstructured data.
Object storage divides data into layers, each with its own namespace and attributes. Each object has a unique identifier through which data can be accessed and managed.
Object storage is cost-effective and is a very suitable storage system for static data. Its flexibility and flatness mean that it can be expanded to store extremely large amounts of data. Objects have enough information for applications to find data quickly and are good at storing unstructured data.
2. Object Storage OSS
2.1 Storage space
The container used by users to store objects (Object). All objects must belong to a certain storage space, and storage types such as standard storage, low-frequency access storage, archive storage, and cold archive storage are provided.
The personal network disk frequently performs file uploading, downloading, sharing and other operations; the personal network disk needs to frequently perform file uploading, downloading, sharing and other operations. We choose to use "standard storage" for this experiment.
Standard storage is suitable for various business scenarios such as social networking and sharing pictures, audio and video applications, large websites, and big data analysis.
2.2 Region
The physical location of the OSS data center.
Alibaba Cloud's region selection can follow the principle of proximity, giving priority to regional nodes that are closer to the user group. In this experiment, "Eastern China 2 (Shanghai)" is the preferred region.
2.3 Objects
An object is the basic unit of OSS data storage, also called an OSS file.
In the personal network disk scenario, Objects are photo files, text files, video files, etc. stored in the OSS Bucket.
2.4 Read and write permissions
Through the Bucket ACL permission list and permission description, this experiment sets the Bucket ACL permission to private, the owner of the Bucket can read and write the files in the Bucket, and other people cannot access the files in the Bucket.
2.5 Access Domain Name (Endpoint)
The access domain name of the OSS external service, the Endpoint format is:
the external network Endpoint is oss-{Region}.aliyuncs.com,
the internal network Endpoint is oss-{Region}-internal.aliyuncs.com.
When accessing OSS resources through the OSS intranet address, no traffic fee is charged. The network disk software deployed on the ECS instance in the personal network disk scenario can call the OSS intranet endpoint.
2.6 Access Key
AK for short refers to the AccessKeyId and AccessKeySecret used in access authentication.
OSS uses AccessKeyId and AccessKeySecret symmetric encryption to verify the identity of the sender of a request, and the AccessKey can be created through the console
The sources of AccessKey are:
1、Bucket的拥有者申请的AccessKey。
2、被Bucket的拥有者通过RAM授权给第三方请求者的AccessKey。
3、被Bucket的拥有者通过STS授权给第三方请求者的AccessKey。
After understanding the concepts of Endpoint and Accesskey, you need to configure the access domain name (Endpoint) and access key (Accesskey) when using the SDK or tools.
2.7 Common functions
(1) Create storage space
According to actual needs, create different types of storage space to store different data;
(2) Upload files
Before uploading files to OSS, a storage space must be created in any region of Alibaba Cloud.
After creating a storage space, you can upload any file to the storage space. There are simple upload, multi-part upload, additional upload, etc.;
(3) List files
After the Object is uploaded successfully, it is sorted alphabetically by default. It can list all Objects in the current Bucket, Objects with a specified prefix, Objects with a specified number, etc.;
(4) Search files
After the object is successfully uploaded, you can search for the file or folder to quickly find the target file;
(5) Share files
After the file is uploaded to the storage space, you can share the URL of the file to the third party for download or preview;
(6) Download files
After the file (Object) is uploaded to the storage space (Bucket), you can download the file to the default path of the browser or a specified local path;
(7) Delete files
You can delete single or multiple files, fragments, etc. at a time. You can delete expired files regularly to save your storage space.
2.8 Resource access method
Alibaba Cloud Object Storage OSS provides multiple ways to access and operate OSS resources: Alibaba Cloud console, tools provided by OSS, and SDK provided by OSS.
3. Personal network disk architecture
This experiment uses ECS to install Cloudreve to provide network disk services, and OSS to provide storage services.
When a user uses a personal network disk, accessing the public network IP address where Cloudreve ECS is deployed can complete file upload, download, delete, share and other services.
3.1 Cloudreve
Cloudreve can quickly build a network disk service for both self-use and public use, and realize a flexible file management experience through the support of multiple storage strategies and virtual file systems.
3.2 ossfs
ossfs allows you to mount the object storage OSS storage space (Bucket) to the local file system in the Linux system, and operate the OSS object (Object) like a local file to realize data sharing.
2. Quickly build a personal website based on OSS
1. Log in to the OSS console
Click Create Resource on the homepage of Alibaba Cloud Platform, and the following content will be generated, AK ID and AK Secret
After creation, enter the following command line interface by default, we select the remote desktop on the right
After entering the remote desktop, click the Chromium Web Browser icon on the desktop, and opening the browser will automatically open the Alibaba Cloud RAM user login page.
In the login interface, enter the user name in the resource just generated
enter password
After completing the RAM user login, click the icon in the upper left corner of the home page of the Alibaba Cloud console in the browser
Enter "OSS" in the search box, click Object Storage OSS in the search results, and enter the OSS management console.
Have entered the OSS management console
2. Create an OSS Bucket
On the OSS management console, select the Bucket list and click Create Bucket to create a Bucket.
When creating a Bucket, fill in the Bucket name, select the correct region, use the default values for other parameters, and click OK to complete the creation of the Bucket.
Here I use the English name of Nine Pointed Star
ninemangstars
Region Find the corresponding region based on the initially generated resources
In the personal network disk scenario, it needs to be in the same region as the ECS instance. Shanghai used here
3. Install Cloudreve
After submitting, click the icon in the navigation bar on the right side of the experiment page to switch to Web Terminal.
3.1 Download the installation package
Run the following command to download the cloudreve installation package.
wget https://labfileapp.oss-cn-hangzhou.aliyuncs.com/cloudreve_3.3.1_linux_amd64.tar.gz
wget is a command-line tool for Linux systems to download files from the web. Wget adds the link of the file to be downloaded to complete the download of the file.
The returned result is as follows, that is, the download of the cloudreve installation package is complete.
3.2 Decompression
After the download is complete, execute the following command to decompress the cloudreve installation package.
tar -zxvf cloudreve_3.3.1_linux_amd64.tar.gz
tar is the Linux system used to decompress and compress commands, in the decompression parameters
z means to decompress the .gz type compressed package
x means to decompress
v means to display decompression detailed information
f means to process the file.
The returned result is as follows, that is, the decompression of the cloudreve installation package is completed.
3.3 Authorization
Run the following command to grant permissions to cloudreve.
chmod +x ./cloudreve
chmod is used to change the access permission of a file or directory, +x means to grant executable permission, and the next step is to run the cloudreve program, so cloudreve must be given executable permission.
Execute the following command to run cloudreve.
./cloudreve
The returned results are as follows, and you can see the initial administrator account, initial administrator password, and listening port.
The initial password is randomly generated, which requires recording the initial administrator account, password and port, which will be used later
初始管理员账号：admin@cloudreve.org
初始管理员密码：7jx2gsnr
开始监听 :5212
3.4 Verify Cloudreve
When accessing the network disk, you need to use the public network IP, so you need to obtain the public network address of the ECS instance.
Through the left navigation bar of the lab, click on the cloud product resource, and obtain it through the elastic IP option under the ECS cloud server.
In the local browser, open and visit http://<ECS public network address>:<cloudreve listening port>
http://47.103.110.136:5212
Go to login page
Enter the initialization administrator account and initialization administrator password in sequence, and click Login.
login successful
As you can see, the background receives the login log
Press Ctrl+C in the terminal to stop cloudreve from running.
4. Install ossfs
ossfs can mount the object storage OSS storage space (Bucket) to the local file system in the Linux system
OSS objects (Object) can be operated like local files to realize data sharing.
In this experiment scenario, the OSS Bucket is mounted to the ECS instance through ossfs and used as the storage of the personal network disk.
4.1 download ossfs
Run the following command to download the ossfs installation package.
wget https://gosspublic.alicdn.com/ossfs/ossfs_1.80.6_centos8.0_x86_64.rpm
After downloading, use ls to check whether the download is successful
4.2 install ossfs
Use yum installation here
yum is a shell front-end package manager in Fedora and RedHat as well as SUSE.
It can automatically download and install RPM packages from a specified server, automatically handle dependencies, and install all dependent software packages at one time without tedious downloading and installation again and again.
In this scenario, we need to install the ossfs software. During the ossfs installation process, yum can automatically download and install the dependent software packages of the ossfs software.
The ECS system of the experiment environment is Centos 8. Since the official Centos8 yum source is offline, it needs to be replaced with the Aliyun yum source.
4.2.1 Delete expired .repo
First delete the expired .repo file in the system
rm -f /etc/yum.repos.d/*
Then download the new CentOS-Base.repo to /etc/yum.repos.d/
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-vault-8.5.2111.repo
Finally run yum makecache to generate the cache
yum clean all && yum makecache
Finally complete the replacement of the yum source
4.3 install ossfs
使用yum命令安装ossfs时
yum install xx.rpm表示为安装xx.rpm软件包
-y参数表示安装过程中自动选择yes
Enter, install
yum install -y ./ossfs_1.80.6_centos8.0_x86_64.rpm
After executing the installation command, the returned result shows Complete! Indicates that ossfs is installed successfully.
4.4 configure ossfs
First, obtain the OSS Bucket, AccessKey ID, and AccessKey Secret information.
The name can be seen on the OSS console, I use Nine Pointed Star English
nine-mangstars
ID and KEY can be found in the resources generated at the beginning
Execute the following commands to save the Bucket name, AccessKey ID and AccessKey Secret information with access rights to the Bucket in the /etc/passwd-ossfs file.
echo nine-mangstars:LTAIEGhXytkJ8tmA:GVOL1mQPdcnJ23dKStnaY6mz3cl8DH > /etc/passwd-ossfs
Run the following command to grant permissions to the passwd-ossfs file.
Note: In the subsequent stage of mounting the OSS Bucket, you need to use the AccessKeyId and AccessKeySecret in passwd-ossfs to verify whether you have the permission of the Bucket in the passwd-ossfs file.
chmod 640 /etc/passwd-ossfs
5. Mount the OSS Bucket
5.1 Create a mount point
Execute the following command to create a folder named oss
mkdir oss
5.2 Manually mount OSS Bucket
Execute the following command
Mount the Bucket to the specified directory /root/oss,
ossfs nine-mangstars /root/oss -o url=oss-cn-shanghai-internal.aliyuncs.com
The endpoint of classic network access (intranet) of ECS of OSS can be viewed in Bucket List >> Bucket Overview >> Access Port (that is, the content after the url)
Check if the mount is successful
df -h
If the return is as follows, it means that the mount is successful
5.3 Set auto-mounting at startup
Execute the following command to create the file ossfs in the /etc/init.d/ directory
Set the ECS startup script to automatically mount the OSS Bucket.
Enter the Vim editor.
vim /etc/init.d/ossfs
Paste this script in editor mode
The Bucket name and url in the last line should be replaced with your own
#! /bin/bash
#
# ossfs Automount Aliyun OSS Bucket in the specified direcotry.
#
# chkconfig: 2345 90 10
# description: Activates/Deactivates ossfs configured to start at boot time.
ossfs nine-mangstars /root/oss -o url=oss-cn-shanghai-internal.aliyuncs.com -oallow_other
paste
After saving: wq to exit
Execute the following command to grant executable permission to the newly created ossfs script.
chmod a+x /etc/init.d/ossfs
Execute the following command to use the ossfs startup script as another service and start it automatically after booting.
chkconfig ossfs on
6. Configure personal network disk
6.1 Running Cloudreve
Execute the following command to run cloudreve.
./cloudreve
6.2 Configure Cloudreve
Configuring cloudreve is mainly to configure storage policies and bind storage policies to user groups.
Storage policies define storage platforms, uploads, and functional restrictions for files. A user group is bound to a storage policy, and users under this user group will share the same storage policy.
First switch to remote login cloudreve. Open a browser, visit
http://47.103.110.136:5212
Description: View the public network address of ECS through the cloud product resource list
http://<ECS public network address>:<cloudreve listening port>
Enter the previously randomly generated account and password
In the upper right corner of the cloudreve main interface, click Management Panel.
In the Determine Site URL Settings dialog box, click Change.
6.2.1 Configure Storage Policy
In the left navigation bar, click Storage Policies to add a storage policy
Select local storage
On the upload path page of adding a native storage policy, change the storage directory to
/root/oss/uploads/{
uid}/{
path}
/root/oss is the mount point of the oss bucket.
Then default to the next step
On the Direct Link Settings page for adding a native storage policy, click Next.
On the Add Native Storage Policy Upload Restrictions page, click Next.
On the completion page of adding a local storage policy, enter the storage policy name OSS, and click Finish.
6.2.2 Configure User Group
In the left navigation bar, click User Groups, Administrator Edit
In the storage policy on the edit admin page, select oss and click save
6.2.3 Upload files
After returning to the homepage, upload any file to the personal network disk
6.2.4 OSS Verification Result
Switch to the command line interface, you can see the file just uploaded
Then go back to remote desktop again
https://oss.console.aliyun.com/
Click Next in the RAM user login box, copy and paste the sub-user password in the upper left corner of the page to the user password input box, and click Login.
In the left navigation bar, click Bucket List. Find the created Bucket and click the Bucket name.
On the file list page, click the corresponding folder name according to the Object path in the cloud product resource list, and the file just uploaded is stored in the folder named 1
Namely CSDN Nine Pointed Star.txt
