springcloud: object storage component MinIO (sixteen)

0 Preface

In actual development, we often face file storage requirements such as storing documents and pictures, and in a distributed architecture, files need to be shared among nodes, similar to the requirements of shared folders, created in distributed servers The cost of shared folders is high, and it is not satisfied even when access across computer rooms is required. At this time, we need a third-party component to realize this type of object storage

There are also object storage components such as OSS, OBS, minIO, and hdfs. Today, we will mainly learn about free and open source MinIO components.

1. Introduction to minio

MinIO is an open-source object storage component developed in go language. It can provide high-performance and high-availability data storage capabilities, support distributed deployment, and provide functions such as data encryption, access control, version control, lifecycle management, and event notification. . It also supports advanced features such as multipart upload and multipart download to improve the processing efficiency of large files.

Official document: https://www.minio.org.cn/

2. minio installation

minio supports docker installation and compressed package installation. Here we use docker installation for the convenience of installation. If you are using mac, you can also use the brew tool to install. For details, please refer to the official website documentation: https://github.com/minio/minio

1. Download the image

docker pull minio/minio

2. Create a data mapping directory

mkdir -p /Library/software/dockerdata/minio/data

3. Create a container

Note that the version I installed here isRELEASE.2023-01-02T09-40-09Z

docker run -p 9000:9000 -p 9090:9090 \
 --name minio \
 -e "MINIO_ACCESS_KEY=minioadmin" \
 -e "MINIO_SECRET_KEY=minioadmin" \
 -v /Library/software/dockerdata/minio/data:/data \
 minio/minio server \
 /data --console-address ":9090" -address ":9000"

4. Log in localhost:9090and enter account/password: minioadmin / minioadmin

3. Minio management terminal introduction

The management terminal enters through port 9090 by default: http://ip:9090/

Some of the more commonly used menus include:

• Object Browser: Object management page, all buckets and files in the bucket in minio can be viewed on this page, the bucket bucket here can be simply understood as a folder - Buckets: bucket management page, used to manage bucket-
  related Configuration, such as bucket access permissions, bucket life cycle (files in the bucket are kept for a few days)
  
• Identity: permission management page, you can create users, groups, and set corresponding permissions, etc.
  
• Monitoring: Monitoring page, monitoring and displaying various health, status, and log information of minio
  

4. minio client use

1. Add pom dependencies

<dependency>
            <groupId>io.minio</groupId>
            <artifactId>minio</artifactId>
            <version>8.5.3</version>
</dependency>

<dependency>
            <groupId>com.squareup.okhttp3</groupId>
            <artifactId>okhttp</artifactId>
            <version>4.10.0</version>
</dependency>

2. Add configuration files

minio:
  # minio地址
  endpoint: http://localhost:9000
  # 账户
  username: minioadmin
  # 密码
  password: minioadmin
  defaultBucketName: test

3. Create a configuration class for generatingMinioClient

/**
 * @author benjamin_5
 * @Description minio配置类
 * @date 2023/8/5
 */
@Configuration
public class MinioConfig {
    
    

    @Value("${minio.endpoint}")
    private String endpoint;

    @Value("${minio.username}")
    private String username;

    @Value("${minio.password}")
    private String password;

    @Value("${minio.defaultBucketName}")
    private String defaultBucketName;

    @Bean
    public MinioClient minioClient(){
    
    
        return MinioClient.builder().credentials(username, password).endpoint(endpoint).build();
    }

}

4. Create a return entity class to facilitate the specification of return information

@Data
public class MinioReturn {
    
    

    /**
     * 文件地址
     */
    private String path;

    /**
     * 原始文件名
     */
    private String inputName;

    /**
     * 最终文件名
     */
    private String outPutName;

}

5. Create MinioTemplatea class to write the minio tool class

@Component
public class MinioTemplate {
    
    

    @Autowired
    private MinioClient minioClient;

    private static final String SLASH = "/";

    @Value("${minio.defaultBucketName}")
    private String defaultBucketName;

    @Value("${minio.endpoint}")
    private String endpoint;

    /**
     * 创建桶
     *
     * @param bucketName
     * @throws Exception
     */
    public void makeBucket(String bucketName) throws Exception {
    
    
        BucketExistsArgs args = BucketExistsArgs.builder().bucket(bucketName).build();
        if (!minioClient.bucketExists(args)) {
    
    
            minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
        }
    }

    /**
     * 上传文件
     *
     * @param file
     * @return
     * @throws Exception
     */
    public MinioReturn putFile(MultipartFile file) throws Exception {
    
    
        return putFile(file, file.getOriginalFilename(), defaultBucketName);
    }

    public MinioReturn putFile(MultipartFile file, String fileName, String bucketName) throws Exception {
    
    
        if (bucketName == null || bucketName.length() == 0) {
    
    
            bucketName = defaultBucketName;
        }
        makeBucket(bucketName);
        minioClient.putObject(PutObjectArgs.builder()
                .bucket(bucketName)
                .object(fileName)
                .stream(file.getInputStream(), file.getSize(), -1)
                .contentType(file.getContentType())
                .build());
        return new MinioReturn(fileLink(bucketName, fileName), file.getOriginalFilename(), fileName);
    }

    /**
     * 删除文件
     *
     * @param bucketName
     * @param fileName
     * @throws Exception
     */
    public void removeFile(String bucketName, String fileName) throws Exception {
    
    
        minioClient.removeObject(RemoveObjectArgs.builder()
                .bucket(bucketName == null || bucketName.length() == 0 ? defaultBucketName : bucketName)
                .object(fileName)
                .build());
    }

    @SneakyThrows
    private String fileLink(String bucketName, String fileName) {
    
    
        return endpoint.concat(SLASH).concat(bucketName).concat(SLASH).concat(fileName);
    }

    private String getFileName(String fileName) {
    
    
        return getFileName(null, fileName);
    }

    private String getFileName(String prefix, String fileName) {
    
    
        String fileNamePre = fileName;
        String fileType = "";
        int index = fileName.lastIndexOf(".");
        if (index > 0) {
    
    
            fileNamePre = fileName.substring(0, index);
            fileType = fileName.substring(index);
        }
        String name = UUID.randomUUID().toString().replace("-", "");
        if (!org.springframework.util.StringUtils.isEmpty(fileNamePre)) {
    
    
            name = fileNamePre + "-" + name + fileType;
        }
        if (!StringUtils.isEmpty(prefix)) {
    
    
            name = prefix + "-" + name;
        }
        return name;
    }


}

6. Writing control class for testing

@RestController
@RequestMapping("minio")
@AllArgsConstructor
public class MinioController {
    
    

    private final MinioTemplate minioTemplate;

    @PostMapping("/upload")
    @ResponseBody
    public MinioReturn upload(MultipartFile file) throws Exception {
    
    
        return minioTemplate.putFile(file);
    }

    @PostMapping("/remove")
    @ResponseBody
    public String remove(String fileName, String bucketName) throws Exception{
    
    
        minioTemplate.removeFile(bucketName, fileName);
        return "success";
    }

}

7. Call the upload interface, as shown in the figure below, the call is successful

We checked in minio, the bucket was automatically created, and the file was uploaded successfully

Test again to delete the interface

View the successful deletion in minio

5. Application scenarios

• 1. Static resource storage

For example, on official websites, portals, homepages and other websites, we often need to load some static resources, such as pictures, js files, video files, etc. Some of them can be directly stored in nginx for loading, on the other hand we can also be stored in minio, through minio to visit

The premise of accessing through minio is to remember to set the bucket permission topublic

The bucket created by the above code defaults to yes private. If you want to adjust the bucket permission through the client code, you can use minioClient.setBucketPolicythe method. After setting, you can access it through the returned address, as shown below (if you want to access through the external network, give the corresponding The internal network address port can be mapped to the external network)

There will be a security problem if the permission is public, that is, when you access the bucket path, you will find that all the files under the bucket will be listed, so that you can access all the files just by splicing the file name

To fix this, just set the permissions to customand then "s3:ListBucket"cancel

If you visit again, you will find that the permission is prohibited, and you can view it normally after adding the file name

Of course, if the old version does not support direct setting in the minio management page, you can download s3browser https://s3browser.com/ , after connecting to minio, right click and Edit Bucket Policyset in

• 2. file storage

Sometimes, when it comes to transferring files between components or systems, the efficiency of direct transfer through the interface is too slow and the timeliness is not enough, or the user does not need to check these files immediately after receiving them. At this time, we can temporarily store the files in minio , send a file address to the customer, and the customer can download it after receiving this address.
Since the file is temporarily stored, it is hoped that it can be automatically deleted. There are two ways to achieve automatic deletion. One is to directly delete the file under the specified path by writing a shell script. The files stored in minio on the server are directly placed in the folder Next, there is no special treatment, so you can delete it directly. The second is to use the life cycle management provided by minio, which can be set in Buckets-Lifecycle. At the same time, this can also be set when creating a bucket in the client. Through the minioClient.setBucketLifecycle();method

• 3. Release bandwidth pressure

Sometimes, when it comes to interface file transfer, the direct transfer takes up a lot of bandwidth, resulting in a failure to go up concurrently, and the client may not need to use the transferred file right away, but just needs to know a status or other information immediately, then you can pass Upload the file to minio, and send a file address to the client, so that the file status notification and download are carried out step by step.
A certain classmate here may think, doesn’t downloading still occupy bandwidth? How can I say that it saves bandwidth resources? Naturally, it is more expensive, but the file download service does not have such a high delay requirement, so it is cheaper to pull a separate one with relatively less performance, which naturally reduces bandwidth costs

6. Summary

So far, we have completed the quick start for minio. This article only describes the most commonly used methods of minio, and there are more methods and instructions waiting for you to explore and learn by yourself.

The demo source code in this article can be found at the following address: https://gitee.com/wuhanxue/wu_study/tree/master/demo/minio_demo