[Cloud Computing | AWS Practice] Build a personal cloud storage service based on the Amazon S3 protocol

This article is included in the column [#Cloud Computing Introduction and Practice - AWS], which includes blog posts related to AWS introduction and practice.

This article is synchronized with my personal public account: [Cloud Computing Insights]

For more information about cloud computing technology, please pay attention to: CSDN [#Introduction and Practice of Cloud Computing - AWS] column.

This series has updated blog posts:

• [Cloud Computing | AWS Practice] A complete guide to using Amazon S3 for bucket and object operations in Java applications
• [Cloud Computing | AWS Practice] How to rename files and folders in Amazon S3 in Java
• [Cloud Computing | AWS Practice] List all AWS S3 objects in a bucket using Java
• [Cloud Computing | AWS Practice] Updating existing Amazon S3 objects using Java
• [Cloud Computing | AWS Practice] Build a personal cloud storage service based on the Amazon S3 protocol
• [Cloud Computing | AWS Practice] Check if a specified key exists in a given Amazon S3 bucket using Java

I. Introduction

MinIO is a high-performance object storage system. It is designed as an alternative to cloud-native storage systems. In fact,its API is fully compatible with Amazon S3; MinIO also provides APIs compatible with domestic cloud vendors such as Alibaba Cloud. Due to the length of this article, I will not include it in this blog. , only introduces the cases related to Amazon AWS and how to quickly use MinIO.

2. About MinIO

2.1 What is MinIO

Minio is a new open source project released by GlusterFS co-founder Anand Babu Periasamy. MinIO was designed from the ground up as a fully compatible alternative to the Amazon S3 storage API.

Legend is the most compatible S3 alternative while also offering comparable performance and scalability.

MinIO also offers a variety of deployment options. It can run as a native application on most popular architectures or deployed as a containerized application using Docker or Kubernetes.

In addition, MinIO is open source software. Organizations can use it free of charge under the terms of the AGPLv3 license (License Agreementclick here). Please note that this option does not provide any support other than online documentation and the MinIO user community. For larger enterprises, paid subscriptions with dedicated support are also available.

MinIO's GitHub location:https://github.com/minio/minio

Due to its S3 API compatibility, ability to run in various deployments, and open source nature, MinIO is an excellent tool for development and testing and DevOps scenarios.

2.2 How object storage works

The concept of object storage is similar to the standard Linux file system, but weuse buckets and objects instead of directories and files.

Buckets can be nested into hierarchies like directories, while objects can be thought of as just collections of bytes. These collections can be arbitrary byte arrays or regular files such as images, PDFs, etc.

An example object storage system might look like this:

/
/images/
  imge1.png
  image2.jpg
/videos/
  video1.mp4
/users/
  /bluetata/
    status-report.docx

Just like directories and files, buckets and objects can also have permissions. Thisallows fine-grained access controlto data, especially in large organizations with many users.

3. Install MinIO

3.1 Install MinIO using Docker

As mentioned before, MinIO is available on almost all platforms. There are standalone installers for Windows, Linux, and MacOS. However, for development and testing purposes, the easiest way to get started is with a containerized distribution.

MinIO requires a persistent volume to store configuration and application data. However, for testing purposes, MinIO can be started by simply passing a directory (/data in the example below). This directory is created in the container file system when the container starts. But all data is lost after the container exits. If you plan to persist it, just mount it.

docker run \
  -p 9000:9000 \
  -p 9001:9001 \
  -e "MINIO_ROOT_USER=MINIO_ROOT_USER" \
  -e "MINIO_ROOT_PASSWORD=MINIO_ROOT_PASSWORD" \
  minio/minio server /data --console-address ":9001"

The execution status screenshot is as follows

While containerized deployments are ideal for evaluating MinIO, there are some limitations to be aware of.

Specifically, some advanced features such as versioning, object locking, and bucket replication will not work. These features requireMinIO's distributed deployment, which is not available in a single-server deployment.

Also note that the above command has set the MinIO username and password. If there is no setting here, you can set it up again as described below.

3.2 Test access to MinIO

Can directly access:http://你的ip地址:9001/login to access your MinIO

4. Configuration and operations related to using MinIO

There are many different ways to interact with the MinIO server and manage buckets and objects. Below, we will explain them one by one.

4.1. MinIO client related configuration operations

4.1.1 Deploy and install the MinIO Client executable program

First, you must deploy the MinIO Client executable program in your own system, otherwise all mc commands will not be recognized and the following error will occur:

-bash: mc: command not found

The solution is:

wget https://dl.minio.io/client/mc/release/linux-amd64/mc
chmod +x mc
sudo mv mc /usr/local/bin

4.1.2 MinIO client related configuration operations

MinIO client provides the same commands as Linux file management commands, such as cp and ls, etc., but is designed for local and remote storage systems . It is fully compatible with AWS S3 and its syntax mimics that of AWS client tools.

The first step in using the MinIO client is to configure it to communicate with the cloud storage system. Let's point it to the containerized deployment above:

mc alias set docker_minio http://127.0.0.1:9000 MINIO_ROOT_USER MINIO_ROOT_PASSWORD

This command is to use MinIO Client (mc) to set an alias (alias). Specifically, it creates an alias named docker_minio, connects to the MinIO instance at http://127.0.0.1:9000, and authenticates using the username MINIO_ROOT_USER and password MINIO_ROOT_PASSWORD.
This is a separate configuration for the client. If you have set the user name and password during the installation process as mentioned above, you can access it directly.

We can verify the connection using the admin subcommand:

[root@aliyun-bluetata-hub-01 user1]# mc admin info docker_minio
●  127.0.0.1:9000
   Uptime: 1 hour
   Version: 2023-11-20T22:40:07Z
   Network: 1/1 OK
   Drives: 1/1 OK
   Pool: 1

Pools:
   1st, Erasure sets: 1, Drives per erasure set: 1

25 KiB Used, 1 Bucket, 1 Object
1 drive online, 0 drives offline

The execution screenshot is as follows:

Now we can start performing basic operations like creating buckets and objects. Many MinIO client subcommands mimic familiar Linux commands:

• cp: Copy files or objects between file systems.
• ls: List files or objects in a bucket.
• mb: Create a bucket (similar to mkdir on Linux).
• mv: Move/relocate files or objects from one file system to another.
• rb: Delete a bucket (similar to rmdir on Linux).
• rm: Delete a file or object.

Most subcommands work on local file systems and cloud storage. For example, we can use the following sequence of commands to create a new bucket, copy files into the bucket, move objects between buckets, and then delete the bucket:

mc mb user1
mc cp ~/test.pdf prattm
mc mb user2
mc cp user1/test.pdf user2
mc rb user1
mc ls user2
[2023-11-22 21:39:10 MDT]     491K test.pdf

4.2 MiniIO console related configuration operations

Another way to manage data in a MinIO deployment is to use the web-based management console. With containerized deployment, we first open the address http://127.0.0.1:9001 in a web browser. We log in using default credentialsMINIO_ROOT_USER/ MINIO_ROOT_PASSWORD.

If you are a remote server, then change the IP to the public IP of your remote server.

We can create our first bucket:

After entering the bucket creation page, directly enter the name to be created and click Create Bucket.

We've said it before: not all options (such as version control) are available for our containerized deployments.

Now we can navigate to the Object Browser and click on our new bucket. First, we can create a sub-bucket using the Create New Path button:

Additionally we can upload the file to the bucket as a new object:

After the upload is successful, the location where the background server stores the file:

In general, the MinIO management console functions the same as the command line client. However, it does have some subtle differences.

First, it is not possible for a client to move objects between buckets like a command line client can.

In addition, the command line client has many subcommands that do not exist in the management console. For example, the diff, du, and pipe subcommands all mimic standard Linux commands and have no equivalents in the administrative console.

5. Use Java to operate MinIO objects (native operations and Amazon Java SDK operations)

5.1 Native operation MinIO

The last way to use MinIO that we'll cover is using the Java SDK. First, we include the required dependencies in our application:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.2</version>
</dependency>

The first step in using the Java SDK is to create a client instance:

MinioClient minioClient =
  MinioClient.builder()
    .endpoint("http://127.0.0.1:9000")
    .credentials("minioadmin", "minioadmin")
    .build();

This client can perform all the same actions we saw previously using command line tools and the administrative console. For example, we can create a Bucket:

minioClient.makeBucket(
  MakeBucketArgs
    .builder()
    .bucket("user1")
    .build());

We can then upload the file as an object into that bucket:

minioClient.putObject(PutObjectArgs
  .builder()
  .bucket("user1")
  .object("Resume.pdf")
  .stream(new FileInputStream("/tmp/Resume.pdf")
  .build());

Finally, let's see how to get the object from the bucket:

try (InputStream stream =
  minioClient.getObject(GetObjectArgs
    .builder()
    .bucket("user2")
    .object("Resume.pdf")
    .build())) {
    
    
    // Read the stream
}

5.2 Using Amazon Java API to operate MinIO

Similarly, you need to introduce dependency packages before using them.

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.21.0</version>
</dependency>

The relevant code for using Amazon JDK operation objects is posted directly below:

public class S3Test {
    
    
    public static void main(String[] args) throws Exception{
    
    
        AmazonS3 s3Client= S3Utils.getS3Client("MINIO_ROOT_USER","MINIO_ROOT_PASSWORD","http://172.0.0.1:9000");

        List<Bucket> bucketList=s3Client.listBuckets();
        List<Bucket> myBuckets=bucketList.stream().filter(bucket -> bucket.getName().equals("bucket01")).collect(Collectors.toList());
        if(myBuckets.size() < 1){
    
    
            s3Client.createBucket("bucket01");
        }


        putObject(s3Client);

        S3Object s3Object = s3Client.getObject(
                "bucket01",
                "test2.jpg"
        );

       downLoadObject(s3Object);

        System.out.println(bucketList);

        ListObjectsV2Result result = s3Client.listObjectsV2("bucket01");
        List<S3ObjectSummary> objects = result.getObjectSummaries();
        for (S3ObjectSummary os : objects) {
    
    
            System.out.println("含有的文件：" + os.getKey());
        }


        // 共享文件
        generatePresignedUrl(s3Client,"bucket01", "test2.jpg");

    }



    public static void putObject(AmazonS3 s3Client) throws Exception{
    
    
        File localFile=new File("D:\\bluetata\\test1.txt");
        ObjectMetadata metadata  = new ObjectMetadata();
        metadata.setContentType("text/plain");
        //metadata.setContentLength(5);


        String s3FileFullPath = UUID.randomUUID()+"a.txt";
        PutObjectResult putResult = s3Client.putObject("bucket01", s3FileFullPath, new FileInputStream(localFile), metadata);
        System.out.println(putResult);
    }

    public static void downLoadObject(S3Object s3Object) throws Exception{
    
    
        File targetFile=new File("D:\\bluetata\\test2.jpg");
        OutputStream out=new FileOutputStream(targetFile);



        try(BufferedInputStream bufferedInputStream = new BufferedInputStream(
                s3Object.getObjectContent()
        )){
    
    
            int len  ;
            byte[] buffer = new byte[1024];
            while((len=bufferedInputStream.read(buffer))!=-1){
    
    
                out.write(buffer, 0, len);
            }
            out.flush();

        }catch (IOException e){
    
    
            GlobalException.throwException(e.getMessage());
        }
        out.close();
    }


    public static String generatePresignedUrl(AmazonS3 s3Client,String bucketName, String key){
    
    
        GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(
                bucketName, key);
        Date expirationDate = null;
        try {
    
    
            expirationDate = new SimpleDateFormat("yyyy-MM-dd").parse("2021-09-19");
        } catch (Exception e) {
    
    
            e.printStackTrace();
        }
        //设置过期时间
        urlRequest.setExpiration(expirationDate);
        URL url =s3Client.generatePresignedUrl(urlRequest);
        System.out.println(url);
        return url.getPath();
    }

}

6. Summary at the end of the article

The article explores the importance and capabilities of MinIO, a high-performance object storage system. It has the advantages of compatibility with Amazon S3 API, multiple deployment options, and open source features, making it suitable for various scenarios, from development and testing to DevOps tool applications.

The article introduces the installation, configuration and operation of MinIO in detail, covering the use of Docker installation, command line client operation, console management, and the use of Java native operations and Amazon Java SDK to operate MinIO objects. These methods not only help users understand the basic operation of MinIO, but also demonstrate its compatibility and flexibility with other systems, making it a powerful and versatile storage solution.

[ 本文作者 ]   bluetata
[ 原文链接 ]   https://bluetata.blog.csdn.net/article/details/134566027
[ 最后更新 ]   11/23/2023 2:15
[ 版权声明 ]   如果您在非 CSDN 网站内看到这一行，
说明网络爬虫可能在本人还没有完整发布的时候就抓走了我的文章，
可能导致内容不完整，请去上述的原文链接查看原文。