Lecture 03: MinIO Distributed File Server

1. What is MinIO

Minio is an open source object storage suite based on Golang. Although it is lightweight, it has good performance.

• Input:MinIO | High Performance, Kubernetes Native Object Storage
• Official document address:MinIO | The MinIO Quickstart Guideopen in new window
• Official website document (Chinese) address:Official Chinese website address open in new window The Chinese document corresponds to the previous version, and there are some in the new version The content has changed, so it's best to read the English documentation.
• JAVA SDK API:minio java sdk api 文档

What is object storage?

Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service that is suitable for storing any type of files. Capacity and processing power are elastically expanded, with multiple storage types to choose from to fully optimize storage costs.

For small and medium-sized enterprises, if they do not choose to store on the cloud, then Minio is a good choice. Although it is small, it has all the essentials.

2. Install MinIO under CentOS7

2.1. Download MinIO

Create the minio folder under usr/local, create the bin and data directories in the minio file, and copy the downloaded minio file to the bin directory.

To create a directory, command:

mkdir /usr/local/minio && cd /usr/local/minio && mkdir bin data

Effect:

Download MinIO, command:

wget https://dl.min.io/server/minio/release/linux-amd64/minio bin
mv minio bin/

Effect:

2.2. Run MinIO

Grant executable permissions, command:

chmod +x bin/minio

To run MinIO, command:

./bin/minio server ./data

2.3. Add minio as a Linux service

cat > /etc/systemd/system/minio.service << EOF
[Unit]
Description=Minio
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/minio/bin/minio

[Service]
WorkingDirectory=/usr/local/minio/
PermissionsStartOnly=true
ExecStart=/usr/local/minio/bin/minio server /usr/local/minio/data
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target
EOF

In this way, you can use systemctl to start and stop minio, command:

systemctl start minio   # 启动
systemctl stop minio    # 停止

running result:

2.4. Testing

After MinIO Server is successfully started, accesshttp://10.211.55.9:9000, you will see an interface similar to the following:

Ps: Close the firewall or open port 9000

systemctl stop firewalld

Enter username/password minioadmin/minioadmin to enter the web management system:

When it is first opened, there is no bucket. You can create a bucket manually or through java code.

The default permissions of the created bucket are private and cannot be accessed by the outside. You can modify the properties of the bucket, click manage, find Access Policy, and change the permissions to public.

3. SpringBoot integrates MinIO

Step 1: Add dependencies in pom.xml

<dependency>
     <groupId>com.squareup.okhttp3</groupId>
     <artifactId>okhttp</artifactId>
     <version>4.8.1</version>
</dependency>
<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.3.9</version>
</dependency>
<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>fastjson</artifactId>
    <version>1.2.70</version>
</dependency>
        

Step 2: application.yml

server:
  port: 8070
spring:
  servlet:
    multipart:
      max-file-size: 200MB  #设置单个文件的大小  因为springboot内置tomact的的文件传输默认为10MB
      max-request-size: 500MB   #设置单次请求的文件总大小
      enabled: true    #千万注意要设置该参数，否则不生效
# minio 文件存储配置信息
minio:
  endpoint: http://10.211.55.9:9000
  accesskey: minioadmin
  secretKey: minioadmin

Step 3: minio configuration classes and configuration properties

Configuration propertiesMinioProp.java

package demo.config;

import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Data
@Component
@ConfigurationProperties(prefix = "minio")
public class MinioProp {
    
    
    private String endpoint;
    private String accesskey;
    private String secretKey;
}

Configuration class MinioConfig.java

package demo.config;

import io.minio.MinioClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableConfigurationProperties(MinioProp.class)
public class MinioConfig {
    
    
    @Autowired
    private MinioProp minioProp;
    @Bean
    public MinioClient minioClient() throws Exception {
    
    
        return MinioClient.builder().endpoint(minioProp.getEndpoint())
                .credentials(minioProp.getAccesskey(), minioProp.getSecretKey()).build();
    }
}

Step 4: File upload tool class

package demo.utils;

import com.alibaba.fastjson.JSONObject;
import demo.config.MinioProp;
import io.minio.BucketExistsArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;

import java.io.InputStream;

@Slf4j
@Component
public class MinioUtils {
    
    

    @Autowired
    private MinioClient client;
    @Autowired
    private MinioProp minioProp;

    /**
     * 创建bucket
     */
    public void createBucket(String bucketName) {
    
    
        BucketExistsArgs bucketExistsArgs = BucketExistsArgs.builder().bucket(bucketName).build();
        MakeBucketArgs makeBucketArgs = MakeBucketArgs.builder().bucket(bucketName).build();
        try {
    
    
            if (client.bucketExists(bucketExistsArgs))
                return;
            client.makeBucket(makeBucketArgs);
        } catch (Exception e) {
    
    
            log.error("创建桶失败：{}", e.getMessage());
            throw new RuntimeException(e);
        }
    }

    /**
     * @param file       文件
     * @param bucketName 存储桶
     * @return
     */
    public JSONObject uploadFile(MultipartFile file, String bucketName) throws Exception {
    
    
        JSONObject res = new JSONObject();
        res.put("code", 0);
        // 判断上传文件是否为空
        if (null == file || 0 == file.getSize()) {
    
    
            res.put("msg", "上传文件不能为空");
            return res;
        }
        // 判断存储桶是否存在
        createBucket(bucketName);
        // 文件名
        String originalFilename = file.getOriginalFilename();
        // 新的文件名 = 存储桶名称_时间戳.后缀名
        String fileName = bucketName + "_" + System.currentTimeMillis() + originalFilename.substring(originalFilename.lastIndexOf("."));
        // 开始上传
        InputStream inputStream = file.getInputStream();
        PutObjectArgs args = PutObjectArgs.builder().bucket(bucketName).object(fileName)
                .stream(inputStream,inputStream.available(),-1).build();
        client.putObject(args);
        res.put("code", 1);
        res.put("msg", minioProp.getEndpoint() + "/" + bucketName + "/" + fileName);
        return res;
    }
}

Step 5: Test the Controller

package demo.controller;

import com.alibaba.fastjson.JSONObject;
import demo.utils.MinioUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

import javax.servlet.http.HttpServletRequest;

@RestController
public class TestController {
    
    
    @Autowired
    private MinioUtils minioUtils;

    @PostMapping("/uploadimg")
    @ResponseBody
    public String uploadimg(@RequestParam(name = "file", required = false) MultipartFile file,
                            HttpServletRequest request) {
    
    
        JSONObject res = null;
        try {
    
    
            res = minioUtils.uploadFile(file, "qin");
        } catch (Exception e) {
    
    
            e.printStackTrace();
            res.put("code", 0);
            res.put("msg", "上传失败");
        }
        return res.toJSONString();
    }
}

Step 6: Front-end page upload.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Title</title>
    <script src="assets/vue.min-v2.5.16.js"></script>
    <script src="assets/axios.min.js"></script>
</head>
<body>
<div id="app">
    <input type="file" @change="Upload" />
</div>
<script>
    new Vue({
      
      
        el: '#app',
        data: {
      
      

        },
        methods: {
      
      
            Upload(event){
      
      
                const flie = event.target.files[0];
                // 在这里进行一系列的校验
                const formData = new FormData();
                formData.append("file",flie);
                axios.post('http://localhost:8070/uploadimg',formData,{
      
      
                    'Content-type' : 'multipart/form-data'
                }).then(res=>{
      
      
                    console.log(res.data);
                },err=>{
      
      
                    console.log(err)
                })
            }
        }
    });
</script>
</body>
</html>

running result:

Access upload.html with the browser and select the file to upload.

Access the console response URL and find that the file has been successfully uploaded.

<body>
    <img
        src="http://10.211.55.9:9000/qin/qin_1680766685084.jpg"
        style="width: 500px;"
    >
</body>

4. Appendix

4.1、JavaSDK

@SpringBootTest
class DemoApplicationTests {
    
    
    @Test
    public void demo() throws Exception {
    
    

        // 使用 MinIO 服务的 URL 和端口，Access key 和 Secret key 创建一个 MinioClient 对象。
        MinioClient minioClient = MinioClient.builder()
                .endpoint("http://127.0.0.1:9000")
                .credentials("minioadmin", "minioadmin")
                .build();

        // 检查存储桶是否已经存在
        boolean isExist = minioClient.bucketExists(BucketExistsArgs.builder().bucket("kongming").build());
        if (isExist) {
    
    
            System.out.println("Bucket already exists.");
        } else {
    
    
            // 创建一个名为 asiatrip 的存储桶，用于存储文件。
            minioClient.makeBucket(MakeBucketArgs.builder().bucket("kongming").build());
        }

        // 使用 putObject 上传一个文件到存储桶中。
        File file = new File("e:/BluceLee/1.jpg");
        InputStream inputStream = new FileInputStream(file);

        PutObjectArgs args = PutObjectArgs.builder()
                .bucket("kongming")
                .object("xiaolong.jpg")
                .contentType("image/jpg")
                .stream(inputStream, inputStream.available(), -1)
                .build();

        minioClient.putObject(args);
        System.out.println("  successfully uploaded as xiaolong.png to `kongming` bucket.");
	}
}

4.2. Complete tool class

package com.woniu.util;

import io.minio.*;
import io.minio.errors.*;
import io.minio.http.Method;
import io.minio.messages.*;
// import net.minidev.json.JSONObject;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.multipart.MultipartFile;

import java.io.*;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class MinioUtils1 {
    
    

    private static final Logger log = LoggerFactory.getLogger(MinioUtils1.class);

    private final String endpoint;
    private final String accessKey;
    private final String secretKey;

    private MinioClient minioClient;

    public MinioUtils1(String endpoint, String accessKey, String secretKey) {
    
    
        this.endpoint = endpoint;
        this.accessKey = accessKey;
        this.secretKey = secretKey;
        this.minioClient = MinioClient.builder().endpoint(endpoint).credentials(accessKey, secretKey).build();
    }

/*
    @PostConstruct
    private MinioClient client() {
    }
*/

    public boolean doesBucketExists(String bucketName) {
    
    
        BucketExistsArgs args = BucketExistsArgs.builder()
                .bucket(bucketName)
                .build();
        try {
    
    
            return minioClient.bucketExists(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }


    /**
     * 创建 bucket
     *
     * @param bucketName 桶名
     */
    public void createBucket(String bucketName) {
    
    
        BucketExistsArgs bucketExistsArgs = BucketExistsArgs.builder().bucket(bucketName).build();
        MakeBucketArgs makeBucketArgs = MakeBucketArgs.builder().bucket(bucketName).build();

        try {
    
    
            if (minioClient.bucketExists(bucketExistsArgs))
                return;

            minioClient.makeBucket(makeBucketArgs);
        } catch (Exception e) {
    
    
            log.error("创建桶失败：{}", e.getMessage());
            throw new RuntimeException(e);
        }
    }


    /**
     * 判断文件是否存在
     *
     * @param bucketName 存储桶
     * @param objectName 对象
     * @return true：存在
     */
    public boolean doesObjectExist(String bucketName, String objectName) {
    
    
        StatObjectArgs args = StatObjectArgs.builder().bucket(bucketName).object(objectName).build();
        try {
    
    
            minioClient.statObject(args);
        } catch (Exception e) {
    
    
            return false;
        }
        return true;
    }


    /**
     * 判断文件夹是否存在
     *
     * @param bucketName 存储桶
     * @param objectName 文件夹名称（去掉/）
     * @return true：存在
     */
    public boolean doesFolderExist(String bucketName, String objectName) {
    
    
        ListObjectsArgs args = ListObjectsArgs.builder()
                .bucket(bucketName)
                .prefix(objectName)
                .recursive(false)
                .build();
        boolean exist = false;
        try {
    
    
            Iterable<Result<Item>> results = minioClient.listObjects(args);
            for (Result<Item> result : results) {
    
    
                Item item = result.get();
                if (!item.isDir())
                    continue;

                if (objectName.equals(item.objectName())) {
    
    
                    exist = true;
                }
            }
        } catch (Exception e) {
    
    
            exist = false;
        }
        return exist;
    }


    /**
     * 通过 MultipartFile ，上传文件
     *
     * @param bucketName 存储桶
     * @param file       文件
     * @param objectName 对象名
     */
    public ObjectWriteResponse putObject(String bucketName, MultipartFile file, String objectName, String contentType) {
    
    
        try {
    
    
            InputStream inputStream = file.getInputStream();
            PutObjectArgs args = PutObjectArgs.builder()
                    .bucket(bucketName)
                    .object(objectName)
                    .contentType(contentType)
                    .stream(inputStream, inputStream.available(), -1)
                    .build();
            return minioClient.putObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 上传本地文件
     *
     * @param bucketName 存储桶
     * @param objectName 对象名称
     * @param fileName   本地文件路径
     */
    public ObjectWriteResponse putObject(String bucketName, String objectName, String fileName) {
    
    
        try {
    
    
            UploadObjectArgs args = UploadObjectArgs.builder()
                    .bucket(bucketName)
                    .object(objectName)
                    .filename(fileName)
                    .build();
            return minioClient.uploadObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 通过流上传文件
     *
     * @param bucketName  存储桶
     * @param objectName  文件对象
     * @param inputStream 文件流
     */
    public ObjectWriteResponse putObjectByStream(String bucketName, String objectName, InputStream inputStream) {
    
    
        try {
    
    
            PutObjectArgs args = PutObjectArgs.builder()
                    .bucket(bucketName)
                    .object(objectName)
                    .stream(inputStream, inputStream.available(), -1)
                    .build();
            return minioClient.putObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 创建文件夹或目录
     *
     * @param bucketName 存储桶
     * @param objectName 目录路径
     */
    public ObjectWriteResponse putDirObject(String bucketName, String objectName) {
    
    
        PutObjectArgs args = PutObjectArgs.builder()
                .bucket(bucketName)
                .object(objectName)
                .stream(new ByteArrayInputStream(new byte[]{
    
    }), 0, -1)
                .build();
        try {
    
    
            return minioClient.putObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 获取全部 bucket
     */
    public List<Bucket> getAllBuckets() throws Exception {
    
    
        return minioClient.listBuckets();
    }

    /**
     * 根据 bucketName 删除信息
     *
     * @param bucketName 桶名
     */
    public void removeBucket(String bucketName) {
    
    
        try {
    
    
            minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            e.printStackTrace();
        }
    }

    /**
     * 获取⽂件外链
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @param expires    过期时间 <=7
     */
    public String getObjectUrl(String bucketName, String objectName, Integer expires) {
    
    
        GetPresignedObjectUrlArgs args = GetPresignedObjectUrlArgs.builder()
                .method(Method.GET)
                .bucket(bucketName)
                .object(objectName)
                .expiry(expires)    // 单位：秒
                .build();

        try {
    
    
            return minioClient.getPresignedObjectUrl(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidResponseException | InvalidKeyException | NoSuchAlgorithmException | IOException | XmlParserException | ServerException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 获取⽂件外链
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @param duration 过期时间
     * @param unit  过期时间的单位
     */
    public String getObjectUrl(String bucketName, String objectName, int duration, TimeUnit unit) {
    
    
        GetPresignedObjectUrlArgs args = GetPresignedObjectUrlArgs.builder()
                .method(Method.GET)
                .bucket(bucketName)
                .object(objectName)
                .expiry(duration, unit)
                .build();

        try {
    
    
            return minioClient.getPresignedObjectUrl(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidResponseException | InvalidKeyException | NoSuchAlgorithmException | IOException | XmlParserException | ServerException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 获取文件
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @return ⼆进制流
     */
    public InputStream getObject(String bucketName, String objectName) throws Exception {
    
    
        GetObjectArgs args = GetObjectArgs.builder()
                .bucket(bucketName)
                .object(objectName)
                .build();
        return minioClient.getObject(args);
    }

    /**
     * 上传文件
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @param stream     ⽂件流
     * @throws Exception https://docs.minio.io/cn/java-client-api-reference.html#putObject
     */
    public void putObject(String bucketName, String objectName, InputStream stream) {
    
    
        putObjectByStream(bucketName, objectName, stream);
    }

    /**
     * 文件流上传文件
     *
     * @param bucketName  bucket名称
     * @param objectName  ⽂件名称
     * @param stream      ⽂件流
     * @param size        ⼤⼩
     * @param contextType 类型
     */
    public void putObject(String bucketName, String objectName, InputStream stream, long size, String contextType) {
    
    
        putObjectByStream(bucketName, objectName, stream);
    }

    /**
     * 获取文件信息
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @throws Exception https://docs.minio.io/cn/java-client-api-reference.html#statObject
     */
    public StatObjectResponse getObjectInfo(String bucketName, String objectName) {
    
    
        StatObjectArgs args = StatObjectArgs.builder()
                .bucket(bucketName)
                .object(objectName)
                .build();
        try {
    
    
            return minioClient.statObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            throw new RuntimeException(e);
        }
    }

    /**
     * 删除文件
     *
     * @param bucketName bucket名称
     * @param objectName ⽂件名称
     * @throws Exception https://docs.minio.io/cn/java-client-apireference.html#removeObject
     */
    public void removeObject(String bucketName, String objectName) {
    
    
        RemoveObjectArgs args = RemoveObjectArgs.builder()
                .bucket(bucketName)
                .object(objectName)
                .build();
        try {
    
    
            minioClient.removeObject(args);
        } catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
    
    
            e.printStackTrace();
        }
    }
}