[SpringBoot Application] SpringBoot integrates MinIO object storage service

Object storage service MinIO

Introduction to MinIO

MinIO is an object storage service based on the Apache License v2.0 open source protocol. It can be used as a cloud storage solution to save massive pictures, videos, and documents. Because it is implemented in Golang, the server can work on Windows, Linux, OS X and FreeBSD. The configuration is simple, basically copying the executable program and running it with a single line command.

MinIO is compatible with the Amazon S3 cloud storage service interface and is very suitable for storing large-capacity unstructured data, such as pictures, videos, log files, backup data, container/virtual machine images, etc., and an object file can be of any size, ranging from several Ranges from kb to a maximum of 5T.

S3 (Simple Storage Service simple storage service)

basic concept

• bucket – a directory analogous to a file system
• Object – A file analogous to a file system
• Keys – Analog file name

Official website documentation: http://docs.minio.org.cn/docs/

MinIO features

Common cloud storage examples include Qiniu Cloud, Alibaba Cloud, etc. The disadvantage is that it costs money

Private storage systems: fastdfs (installation and deployment are super troublesome, you need to install hadoop, and there is no interface), GridFS that comes with mongodb (it also has many disadvantages in use), so compared with MinIO, the advantages are as follows:

• SDK support

  MinIO is an open source project based on the Golang programming language to develop a high-performance distributed storage solution. Based on the lightweight characteristics of Minio, it is supported by SDKs in languages ​​​​such as Java, Python or Go, and has a very complete official website. document.

• Simple installation and deployment

  In a Linux environment, you only need to download a binary file and execute it to complete the installation and configuration of MinIO in a few minutes. The number of configuration options and variants is kept to a minimum, thus reducing the probability of failed configurations to almost zero. MinIO upgrade is completed through a simple command. This command can complete the MinIO upgrade work without interruption, and the upgrade operation can be completed without downtime, greatly reducing the total usage and operation and maintenance costs.

• There is an operation page

  After the MinIO service is installed, you can log in to the system directly through the browser. The user-friendly simple operation interface makes it very convenient to manage the Bucket and the file resources in it.

• Data protection

  Minio uses Minio Erasure Code to prevent hardware failures. Even if more than half of the driver is damaged, it can still be recovered from it.

• high performance

  As a high-performance object storage, it can achieve a read rate of 55GB/s and a write rate of 35GB/s under standard hardware conditions.

• Expandable

  Different MinIO clusters can form a federation and form a global namespace spanning multiple data centers.

• Simple function

  This design principle makes MinIO less error-prone and faster to start.

• Rich API

  Supports file resource sharing links and shared link expiration policies, bucket operations, file list access, and basic functions of file upload and download, etc.

• Proactive notification of file changes

  If the bucket (Bucket) changes, such as uploading objects and deleting objects, you can use the bucket event notification mechanism to monitor and publish them through the following methods: AMQP, MQTT, Elasticsearch, Redis, NATS, MySQL, Kafka, Webhooks, etc.

Use out of the box

Docker installation starts

We can use docker for environment deployment and startup, the latest2023-08-31T15-31-16Z

docker run -p 9000:9000 -p 9090:9090 --name minio -d --restart=always -e "MINIO_ACCESS_KEY=minio" -e "MINIO_SECRET_KEY=minio123" -v /home/data:/data -v /home/config:/root/.minio minio/minio server /data --console-address ":9090" -address ":9000"

• -eSet environment variables MINIO_ROOT_USERand MINIO_ROOT_PASSWORD. These set the root user credentials . Change the sample values ​​used for your container

Management console

Assuming that our server address is 192.168.171.128, we enter: http://192.168.171.128:9000/ in the address bar to enter the login interface.

The Access Key is minioSecret_key minio123, which is created when creating the docker container. After entering the system, you can see the main interface.

Click create a Bucket to create a bucket. The Bucket here can be understood as a directory where files are stored. Multiple buckets can be created independently of each other.

Quick start

Java upload files to minio

pom dependency

<dependencies>
    <dependency>
        <groupId>io.minio</groupId>
        <artifactId>minio</artifactId>
        <version>7.1.0</version>
    </dependency> 
</dependencies>

Create test class and upload files

public class MinIOTest {
    
    


     public static void main(String[] args) {
    
    

        FileInputStream fileInputStream = null;
        try {
    
    

            fileInputStream =  new FileInputStream("D:\\study\\images\\1.png");;

            //1.创建minio链接客户端
            MinioClient minioClient = MinioClient.builder().credentials("minio", "minio123").endpoint("http://192.168.171.128:9000").build();
            //2.上传
            String objectName = UUID.randomUUID().toString();
            PutObjectArgs putObjectArgs = PutObjectArgs.builder()
                    .object(objectName+".png")//文件名
                    .contentType("image/jpg")//文件类型 MediaType.IMAGE_PNG_VALUE代替
                    .bucket("miniotest")//桶名词  与minio创建的名词一致
                    .stream(fileInputStream, fileInputStream.available(), -1) //文件流
                    .build();
            minioClient.putObject(putObjectArgs);

            System.out.println("http://192.168.171.128:9000/miniotest/"+objectName+".png");

        } catch (Exception ex) {
    
    
            ex.printStackTrace();
        }
    }
}

Use UUID as object name:

• Uniqueness to avoid object name conflicts.

• Hide actual file information and improve certain privacy.

• Object names are not limited by the length of the original file name or by special characters.

In actual scenarios, if you pay more attention to file identification and management, you can consider using file names. If you pay more attention to uniqueness and privacy, consider using UUID.
At the same time, you can also combine the two, such as storing the file name as an object property, and then using the UUID as the object name. This can preserve file information and ensure uniqueness.

Visit the client console and the image has been uploaded successfully.

Configure access permissions

Access the uploaded file through the browser. The file format accessed http://minio服务器ip:9090/存储桶/文件名is here http://192.168.171.128:9000/miniotest/dc7a7378-eb4e-4622-afd6-32f3b41e14bf.png. An error will occur when accessing. The reason is that the created bucket space setting is private and other users do not have access rights.

If we need the files we upload to be accessible by anonymous users, then we need to modify the access permissions to Publicpermissions

Browser access successful

Encapsulate MinIO as starter

Create module zy-minio-starter

Import dependencies

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-autoconfigure</artifactId>
    </dependency>
    <dependency>
        <groupId>io.minio</groupId>
        <artifactId>minio</artifactId>
        <version>7.1.0</version>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-configuration-processor</artifactId>
        <optional>true</optional>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
</dependencies>

Configuration class

MinIOConfigProperties

@Data
@ConfigurationProperties(prefix = "minio")  // 文件上传 配置前缀file.oss
public class MinIOConfigProperties implements Serializable {
    
    

    private String accessKey;
    private String secretKey;
    private String bucket;
    private String endpoint;
    private String readPath;
}

MinIOConfig

@Configuration
@EnableConfigurationProperties({
    
    MinIOConfigProperties.class})
//当引入FileStorageService接口时
@ConditionalOnClass(FileStorageService.class)
public class MinIOConfig {
    
    

   @Autowired
   private MinIOConfigProperties minIOConfigProperties;

    @Bean
    public MinioClient buildMinioClient(){
    
    
        return MinioClient
                .builder()
                .credentials(minIOConfigProperties.getAccessKey(), minIOConfigProperties.getSecretKey())
                .endpoint(minIOConfigProperties.getEndpoint())
                .build();
    }
}

Encapsulation operation minIO class

FileStorageService

public interface FileStorageService {
    
    


    /**
     *  上传图片文件
     * @param prefix  文件前缀
     * @param filename  文件名
     * @param inputStream 文件流
     * @return  文件全路径
     */
    public String uploadImgFile(String prefix, String filename,InputStream inputStream);

    /**
     *  上传html文件
     * @param prefix  文件前缀
     * @param filename   文件名
     * @param inputStream  文件流
     * @return  文件全路径
     */
    public String uploadHtmlFile(String prefix, String filename,InputStream inputStream);

    /**
     * 删除文件
     * @param pathUrl  文件全路径
     */
    public void delete(String pathUrl);

    /**
     * 下载文件
     * @param pathUrl  文件全路径
     * @return
     *
     */
    public byte[]  downLoadFile(String pathUrl);

}

MinIOFileStorageService

@Slf4j
@Import(MinIOConfig.class)
public class MinIOFileStorageService implements FileStorageService {
    
    

    @Autowired
    private MinioClient minioClient;

    @Autowired
    private MinIOConfigProperties minIOConfigProperties;

    private final static String separator = "/";

    /**
     * @param dirPath
     * @param filename  yyyy/mm/dd/file.jpg
     * @return
     */
    public String builderFilePath(String dirPath,String filename) {
    
    
        StringBuilder stringBuilder = new StringBuilder(50);
        if(!StringUtils.isEmpty(dirPath)){
    
    
            stringBuilder.append(dirPath).append(separator);
        }
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd");
        String todayStr = sdf.format(new Date());
        stringBuilder.append(todayStr).append(separator);
        stringBuilder.append(filename);
        return stringBuilder.toString();
    }

    /**
     *  上传图片文件
     * @param prefix  文件前缀
     * @param filename  文件名
     * @param inputStream 文件流
     * @return  文件全路径
     */
    @Override
    public String uploadImgFile(String prefix, String filename,InputStream inputStream) {
    
    
        String filePath = builderFilePath(prefix, filename);
        try {
    
    
            PutObjectArgs putObjectArgs = PutObjectArgs.builder()
                    .object(filePath)
                    .contentType("image/jpg")
                    .bucket(minIOConfigProperties.getBucket()).stream(inputStream,inputStream.available(),-1)
                    .build();
            minioClient.putObject(putObjectArgs);
            StringBuilder urlPath = new StringBuilder(minIOConfigProperties.getReadPath());
            urlPath.append(separator+minIOConfigProperties.getBucket());
            urlPath.append(separator);
            urlPath.append(filePath);
            return urlPath.toString();
        }catch (Exception ex){
    
    
            log.error("minio put file error.",ex);
            throw new RuntimeException("上传文件失败");
        }
    }

    /**
     *  上传html文件
     * @param prefix  文件前缀
     * @param filename   文件名
     * @param inputStream  文件流
     * @return  文件全路径
     */
    @Override
    public String uploadHtmlFile(String prefix, String filename,InputStream inputStream) {
    
    
        String filePath = builderFilePath(prefix, filename);
        try {
    
    
            PutObjectArgs putObjectArgs = PutObjectArgs.builder()
                    .object(filePath)
                    .contentType("text/html")
                    .bucket(minIOConfigProperties.getBucket()).stream(inputStream,inputStream.available(),-1)
                    .build();
            minioClient.putObject(putObjectArgs);
            StringBuilder urlPath = new StringBuilder(minIOConfigProperties.getReadPath());
            urlPath.append(separator+minIOConfigProperties.getBucket());
            urlPath.append(separator);
            urlPath.append(filePath);
            return urlPath.toString();
        }catch (Exception ex){
    
    
            log.error("minio put file error.",ex);
            ex.printStackTrace();
            throw new RuntimeException("上传文件失败");
        }
    }

    /**
     * 删除文件
     * @param pathUrl  文件全路径
     */
    @Override
    public void delete(String pathUrl) {
    
    
        String key = pathUrl.replace(minIOConfigProperties.getEndpoint()+"/","");
        int index = key.indexOf(separator);
        String bucket = key.substring(0,index);
        String filePath = key.substring(index+1);
        // 删除Objects
        RemoveObjectArgs removeObjectArgs = RemoveObjectArgs.builder().bucket(bucket).object(filePath).build();
        try {
    
    
            minioClient.removeObject(removeObjectArgs);
        } catch (Exception e) {
    
    
            log.error("minio remove file error.  pathUrl:{}",pathUrl);
            e.printStackTrace();
        }
    }


    /**
     * 下载文件
     * @param pathUrl  文件全路径
     * @return  文件流
     *
     */
    @Override
    public byte[] downLoadFile(String pathUrl)  {
    
    
        String key = pathUrl.replace(minIOConfigProperties.getEndpoint()+"/","");
        int index = key.indexOf(separator);
        String bucket = key.substring(0,index);
        String filePath = key.substring(index+1);
        InputStream inputStream = null;
        try {
    
    
            inputStream = minioClient.getObject(GetObjectArgs.builder().bucket(minIOConfigProperties.getBucket()).object(filePath).build());
        } catch (Exception e) {
    
    
            log.error("minio down file error.  pathUrl:{}",pathUrl);
            e.printStackTrace();
        }

        ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
        byte[] buff = new byte[100];
        int rc = 0;
        while (true) {
    
    
            try {
    
    
                if (!((rc = inputStream.read(buff, 0, 100)) > 0)) break;
            } catch (IOException e) {
    
    
                e.printStackTrace();
            }
            byteArrayOutputStream.write(buff, 0, rc);
        }
        return byteArrayOutputStream.toByteArray();
    }
}

Join automatic configuration externally

Create new in resourcesMETA-INF/spring.factories

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
 cn.zysheep.file.service.impl.MinIOFileStorageService

Use by other microservices

First, import the dependencies of zy-minio-starter

Second, add the configuration required by minio in the microservice

minio:
  accessKey: minio
  secretKey: minio123
  bucket: miniotest
  endpoint: http://192.168.171.128:9000
  readPath: http://192.168.171.128:9000

Third, inject FileStorageService into the corresponding business class. The example is as follows:

@SpringBootTest(classes = MinIOApplication.class)
public class MinioBootTest {
    
    

    @Autowired
    private FileStorageService fileStorageService;

    @Test
    public void testUpdateImgFile() {
    
    
        try {
    
    
            FileInputStream fileInputStream = new FileInputStream("D:\\study\\images\\docker3.png");
            String fileName = UUID.randomUUID().toString();
            String filePath = fileStorageService.uploadImgFile("", fileName+".jpg", fileInputStream);
            System.out.println(filePath);
        } catch (FileNotFoundException e) {
    
    
            e.printStackTrace();
        }
    }
}

Problems encountered during installation

Question one