This article briefly introduces the basic use of MinIO to help everyone get started with MinIO quickly.
Table of contents
1. What is MinIO?
MinIO is an object storage service based on the Apache License v2.0 open source protocol, which can be used as a cloud storage solution to save massive pictures, videos, and documents. Because it is implemented in Golang, the server can work on Windows, Linux, OS X and FreeBSD. The configuration is simple, basically copying the executable program and running it with a single line command.
MinIO is compatible with Amazon S3 (Simple Storage Service) Cloud storage service interface, very suitable for storing large-capacity unstructured data Data, such as pictures, videos, log files, backup data and container/virtual machine images, etc., and an object file can be of any size, ranging from a few kb to a maximum of 5T.
basic concept
-
bucket – a directory analogous to a file system
-
Object – A file analogous to a file system
-
Keys – Analog file name
Official text:http://docs.minio.org.cn/docs/
Features of MinIO
Data protection
Minio uses Minio Erasure Code to prevent hardware failures. Even if more than half of the driver is damaged, it can still be recovered from it.
high performance
As a high-performance object storage, it can achieve a read rate of 55GB/s and a write rate of 35GB/s under standard hardware conditions.
Expandable
Different MinIO clusters can form a federation and form a global namespace spanning multiple data centers.
SDK support
Based on the lightweight characteristics of Minio, it is supported by SDKs in languages such as Java, Python or Go.
There is an operation page
User-friendly and simple operation interface makes it very convenient to manage Bucket and the file resources in it.
Simple function
This design principle makes MinIO less error-prone and faster to start.
Rich API
Supports file resource sharing links and shared link expiration policies, bucket operations, file list access, and basic functions of file upload and download, etc.
Proactive notification of file changes
If the bucket (Bucket) changes, such as uploading objects and deleting objects, you can use the bucket event notification mechanism to monitor and publish them through the following methods: AMQP, MQTT, Elasticsearch, Redis, NATS, MySQL, Kafka, Webhooks, etc.
2. MinIO installation
本文使用Linux+Docker的形式安装MinIO，安装步骤请看：Detailed steps for installing MinIO with Docker_guaica@’s blog-CSDN blog
3. Quick start with integrated SpringBoot
Create project, demo project:MinIO-Demo: MinIo demo, which simply shows the use of minio. (gitee.com)
1. Import pom dependencies
Please import spring dependencies yourself
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>7.1.0</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.28</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>2.yml configuration
spring:
# 配置文件上传大小限制
servlet:
multipart:
max-file-size: 200MB
max-request-size: 200MB
minio:
accessKey: minioadmin
secretKey: minioadmin
#填写你的桶名称
bucket: xxx
endpoint: http://你的IP:9090
readPath: http://你的IP:9090Generally, a service uses one bucket.
3.Write configuration class
MinIOConfigProperties:
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import java.io.Serializable;
@Data
@ConfigurationProperties(prefix = "minio")
public class MinIOConfigProperties implements Serializable {
private String accessKey;
private String secretKey;
private String bucket;
private String endpoint;
private String readPath;
}MinIOConfig :
import io.minio.MinioClient;
import lombok.Data;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.autoconfigure.condition.ConditionalOnClass;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Data
@Configuration
@EnableConfigurationProperties({MinIOConfigProperties.class})
public class MinIOConfig {
@Autowired
private MinIOConfigProperties minIOConfigProperties;
@Bean
public MinioClient buildMinioClient(){
return MinioClient
.builder()
.credentials(minIOConfigProperties.getAccessKey(), minIOConfigProperties.getSecretKey())
.endpoint(minIOConfigProperties.getEndpoint())
.build();
}
}4.MinIO operation tool class
import com.rikka.config.MinIOConfigProperties;
import io.minio.*;
import io.minio.messages.DeleteObject;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.*;
import java.util.stream.Collectors;
import java.util.stream.Stream;
@Component
@EnableConfigurationProperties(MinIOConfigProperties.class)
@Slf4j
public class MinIOUtil {
@Autowired
private MinIOConfigProperties minIOConfigProperties;
@Autowired
private MinioClient minioClient;
private final static String separator = "/";
/**
* 判断文件是否存在
* @param bucketName
* @return
*/
public boolean existBucket(String bucketName, String objectName){
// 若数据库中存在，根据数据库中的文件信息，则继续判断bucket中是否存在
try {
InputStream inputStream = minioClient.getObject(GetObjectArgs
.builder()
.bucket(bucketName)
.object(objectName)
.build());
if (inputStream == null) {
return false;
}
} catch (Exception e) {
return false;
}
return true;
}
/**
* 判断bucket是否存在，不存在则创建, 默认为yml配置的 bucket
*/
public String existBucket(String bucketName){
if (StringUtils.isBlank(bucketName)) {
bucketName = minIOConfigProperties.getBucket();
}
try {
boolean exist = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
if(!exist){
minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
}
}catch (Exception e){
log.error("minio exist Bucket error.",e);
throw new RuntimeException("判断bucket是否存在，不存在则创建,失败");
}
return bucketName;
}
/**
* 删除存储bucket
* @param bucketName 存储bucket名称，必填
* @return Boolean
*/
public Boolean removeBucket(String bucketName) {
try {
minioClient.removeBucket(RemoveBucketArgs.builder()
.bucket(bucketName)
.build());
} catch (Exception e) {
log.error("minio remove Bucket error.",e);
e.printStackTrace();
return false;
}
return true;
}
/**
* 将本地文件上传到minio
* @param filePath 本地文件路径
* @param bucket 桶
* @param objectName 对象名称
*/
private void uploadToMinio(String filePath, String bucket, String objectName) {
String contentType = objectName.split("\\.")[1];
try {
minioClient.uploadObject(UploadObjectArgs
.builder()
.bucket(bucket)
.object(objectName)
.filename(filePath)
.contentType(contentType)
.build());
} catch (Exception e) {
log.error("上传到文件系统出错");
}
}
/**
* 文件上传(根据本地文件)，以日期为格式
* @param bucketName
* @param prefix 前缀
* @param fileName 本地文件全路径 D:/xx/xx.text
* @return
*/
public String upload(String bucketName, String prefix, String fileName) {
//判断并创建Bucket
bucketName = existBucket(bucketName);
String uuid = UUID.randomUUID().toString().replace("-","");
DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd");
String format = dateFormat.format(new Date());
StringBuilder stringBuilder = new StringBuilder();
if (StringUtils.isNotBlank(prefix)) {
stringBuilder.append(prefix).append(separator);
}
String suffix = fileName.split("\\.")[1];
stringBuilder.append(format).append(separator).append(uuid).append(suffix);
try {
// 上传到minio服务器
minioClient.uploadObject(UploadObjectArgs.builder()
.bucket(bucketName)
.filename(fileName)
.object(stringBuilder.toString())
.build());
return minIOConfigProperties.getReadPath() + separator + minIOConfigProperties.getBucket() +
separator +
stringBuilder.toString();
} catch (Exception e) {
log.error("minio put file error.",e);
throw new RuntimeException("上传文件失败");
}
}
/**
* 文件上传(根据流),文件路径以日期为格式
* @param bucketName
* @param prefix 前缀
* @param file
* @return
*/
public String upload(String bucketName, String prefix, MultipartFile file) {
//判断并创建Bucket
bucketName = existBucket(bucketName);
String filePath = builderFilePath(prefix, file);
try {
InputStream inputStream = file.getInputStream();
// 上传到minio服务器
minioClient.putObject(PutObjectArgs.builder()
.bucket(bucketName)
.object(filePath)
.stream(inputStream, inputStream.available(), -1)
.build());
return minIOConfigProperties.getReadPath() + separator + minIOConfigProperties.getBucket() +
separator +
filePath;
} catch (Exception e) {
log.error("minio put file error.",e);
throw new RuntimeException("上传文件失败");
}
}
/**
* 上传图片文件
* @param bucketName bucket名称，为空时默认yml配置
* @param prefix 文件前缀
* @param multipartFile 文件
* @return 文件全路径
*/
public String uploadImgFile(String bucketName, String prefix, MultipartFile multipartFile) {
//判断并创建Bucket
bucketName = existBucket(bucketName);
//构建文件存储路径
String filePath = builderFilePath(prefix, multipartFile);
try {
InputStream inputStream = multipartFile.getInputStream();
PutObjectArgs putObjectArgs = PutObjectArgs.builder()
.object(filePath)
.contentType("image/jpg")
.bucket(bucketName)
.stream(inputStream,inputStream.available(),-1)
.build();
minioClient.putObject(putObjectArgs);
return minIOConfigProperties.getReadPath() + separator + minIOConfigProperties.getBucket() +
separator +
filePath;
}catch (Exception ex){
log.error("minio put image file error.",ex);
throw new RuntimeException("上传文件失败");
}
}
/**
* 下载文件
* @param bucketName 必填
* @param pathUrl 文件全路径
* @return 文件流
*/
public byte[] downLoadFile(String bucketName, String pathUrl) {
String key = pathUrl.replace(minIOConfigProperties.getEndpoint()+"/","");
int index = key.indexOf(separator);
String bucket = key.substring(0,index);
String filePath = key.substring(index+1);
InputStream inputStream = null;
try {
inputStream = minioClient.getObject(GetObjectArgs.builder().bucket(bucketName).object(filePath).build());
} catch (Exception e) {
log.error("minio down file error. pathUrl:{}",pathUrl);
e.printStackTrace();
}
ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buff = new byte[100];
int rc = 0;
while (true) {
try {
if (!((rc = inputStream.read(buff, 0, 100)) > 0)) break;
} catch (IOException e) {
e.printStackTrace();
}
byteArrayOutputStream.write(buff, 0, rc);
}
return byteArrayOutputStream.toByteArray();
}
/**
*
* @param bucketName
* @param folderPath 分片文件路径
* @param chunkTotal
* @param fileName 合并后的文件名
* @return
*/
public boolean mergeChunkFiles(String fileMd5, String bucketName, String folderPath, int chunkTotal, String fileName) {
List<ComposeSource> sources = Stream.iterate(0, i -> i++).limit(chunkTotal).map(i -> ComposeSource
.builder()
.bucket(minIOConfigProperties.getBucket())
.object(folderPath + i).build()).collect(Collectors.toList());
// 合并后文件的名
String extension = fileName.substring(fileName.lastIndexOf("."));
String objectName = getFilePathByMd5(fileMd5, extension);
//指定合并后的objectName等信息
ComposeObjectArgs composeObjectArgs = ComposeObjectArgs.builder()
.bucket(bucketName)
.object(objectName)
.sources(sources)
.build();
//合并文件,单个分片文件大小必须为5m
try {
minioClient.composeObject(composeObjectArgs);
} catch (Exception e) {
e.printStackTrace();
log.error("合并文件出错了，bucket:{},objectName:{},错误信息:{}",minIOConfigProperties.getBucket(), objectName, e.getMessage());
return false;
}
return true;
}
/**
* 删除文件
* @param bucketName 存储bucket名称，必填
* @param pathUrl 文件全路径
*/
public boolean deleteFile(String bucketName, String pathUrl) {
String key = pathUrl.replace(minIOConfigProperties.getEndpoint()+"/","");
int index = key.indexOf(separator);
String bucket = key.substring(0,index);
String filePath = key.substring(index+1);
// 删除Objects
RemoveObjectArgs removeObjectArgs = RemoveObjectArgs.builder().bucket(bucketName).object(filePath).build();
try {
minioClient.removeObject(removeObjectArgs);
} catch (Exception e) {
log.error("minio remove file error. pathUrl:{}",pathUrl);
e.printStackTrace();
return false;
}
return true;
}
/**
* 批量删除文件对象
* @param bucketName 存储bucket名称，必填
* @param objects 对象名称集合
*/
public boolean removeObjects(String bucketName, List<String> objects) {
Map<String,String > resultMap = new HashMap<>();
List<DeleteObject> dos = objects.stream().map(e -> new DeleteObject(e)).collect(Collectors.toList());
try {
minioClient.removeObjects(RemoveObjectsArgs.builder().bucket(bucketName).objects(dos).build());
}catch (Exception e){
log.error("minio remove file error. objects:{}",objects);
return false;
}
return true;
}
/**
* 文件存储路径构建 yyyy/mm/dd/uuid.jpg
* @param prefix
* @param multipartFile
* @return
*/
public String builderFilePath(String prefix, MultipartFile multipartFile) {
String filename = multipartFile.getOriginalFilename();
assert filename != null;
String suffix = filename.split("\\.")[1];
String uuid = UUID.randomUUID().toString().replace("-","");
StringBuilder stringBuilder = new StringBuilder(50);
if(StringUtils.isNotBlank(prefix)){
stringBuilder.append(prefix).append(separator);
}
SimpleDateFormat sdf = new SimpleDateFormat("yyyy/MM/dd");
String todayStr = sdf.format(new Date());
stringBuilder.append(todayStr).append(separator).append(uuid);
stringBuilder.append(".").append(suffix);
return stringBuilder.toString();
}
/**
* 合并后完整视频路径
* @param fileMd5
* @param fileExt
* @return
*/
private String getFilePathByMd5(String fileMd5, String fileExt) {
return fileMd5.substring(0, 1) + "/" + fileMd5.substring(1, 2) + "/" + fileMd5 + "/" + fileMd5 + fileExt;
}
}Then directly inject the tool class and use it.
Summarize
The above is what I am going to talk about today. If you think it is helpful, please pay attention!