Installation and use of MinIO
1. What is MinIO?
-
MinIO is a high-performance, distributed object storage system. It is a software product that can run 100% on standard hardware. Even low-cost machines such as X86 can also run MinIO well.
-
MinIO is different from traditional storage and other object storage in that it designs its software architecture from the beginning for private cloud standards with higher performance requirements. Because MinIO was only designed for object storage from the beginning. So he designed it in a more user-friendly way. It can realize all the functions required for object storage and is more powerful in performance. It will not compromise for more business functions and lose the ease of use and efficiency of MinIO. . The benefit of this result is that it can more easily implement native object storage services with elastic scalability.
-
MinIO excels in traditional object storage use cases such as secondary storage, disaster recovery and archiving. At the same time, it is also unique in storage technology in machine learning, big data, private cloud, hybrid cloud, etc. Of course, support for data analysis, high-performance application loads, and native cloud is not excluded.
-
MinIO is mainly implemented in Golang language, and the http/https communication protocol is used between the client and the storage server.
-
It is compatible with Amazon S3 cloud storage service API
Information about MinIO
Chinese official website: http://www.minio.org.cn/Chinese
documents: http://docs.minio.org.cn/docs/Chinese
download address: http://www.minio.org.cn/download. shtml#/linux
English official website: https://min.io/English
documentation: https://docs.min.io/English
download address: https://min.io/download#/linux
Github address: https:/ /github.com/minio/minio
2. MinIO installation (centos7)
2.1 Download MinIO
The demo is configured by downloading the binary file from the official website. The download address is shown above. Docker installation is easier~
#创建目录
mkdir /usr/local/minio/
cd /usr/local/minio/
#下载并添加权限
wget http://dl.minio.org.cn/server/minio/release/linux-amd64/minio
#赋值权限
chmod +x minio
2.2 Start MinIO
#创建数据目录，数据目录存储需要大点
mkdir -p /home/data/minio
#创建日志目录
mkdir -p /home/data/minio/log
touch /home/data/minio/log/minio.log
#前台启动minio
./minio server /home/data/minio
#后台启动minio
nohup ./minio server /home/data/minio > /home/data/minio/log/minio.log &
# nohup端口自定义启动服务 指定文件存放路径 /home/data/minio 还有设置日志文件路径 /home/data/minio/log/minio.log
nohup ./minio server --address :9000 --console-address :9001 /home/data/minio > /home/data/minio/log/minio.log 2>&1 &
–address :9000 --console-address :9001 is the configuration port. The default minio port is 9000. If port 9000 is occupied, then add this string of configurations. There is no need to write out the ip before the colon of the port number. Of course If your IP changes dynamically instead of being static, you don’t need to write the previous IP. Of course, it is best to use a static IP.
Turn off the firewall.
If it is not turned off, enter the following command:
systemctl stop firewalld
Note: If you do not want to turn off the firewall, you need to open ip port 9000 and set up static control access IP
Access: http://192.168.92.100:9000View the console.
Here we all use the default configuration account password.minioadmin:minioadmin
2.3 Modify configuration
The account and password here are all used by default. If we want to modify them, we can set and modify the environment variables in the environment variables.
-
Open the /etc/profile file
vim /etc/profile -
Add the following information at the end of the file (see the prompts when starting, new versions need to use
MINIO_ROOT_USERandMINIO_ROOT_PASSWORD, old versions need to useMINIO_ACCESS_KEYandMINIO_SECRET_KEY).
After pressing the i key, enter
(1) the new version at the end of the document:export MINIO_ROOT_USER=minioadmin export MINIO_ROOT_PASSWORD=admin123(2)Old version
export MINIO_ACCESS_KEY=minioadmin export MINIO_SECRET_KEY=admin123 -
Save and exit, refresh and reload environment variables
source /etc/profile
2.4 Write startup script and add it to systemctl
In order to facilitate management, we add the command to the script here
-
Create a new one in
/usr/local/minio/the directoryrun.shvim run.sh -
Then save the following content to
run.shand give it execution permissionschmod +x run.sh#!/bin/bash #配置登陆账号密码 export MINIO_ROOT_USER=minioadmin export MINIO_ROOT_PASSWORD=admin123 # nohup启动服务 指定文件存放路径 /home/data/minio 还有设置日志文件路径 /home/data/minio/log/minio.log nohup ./minio server --address :9000 --console-address :9001 /home/data/minio > /home/data/minio/log/minio.log 2>&1 & -
Then start minio
# 启动minio服务 bash run.sh # 查看日志 tail -200f /home/data/minio/log/minio.log
Then there will be log printing information, and then you can see the minio server address and console information address.
Then access the address in the browser
http://192.168.92.100:9000. After entering this address, you will be redirected to the console login address.http://192.168.92.100:9001/login
Add to systemctl startup command
-
Write minio.service file
vim /usr/lib/systemd/system/minio.serviceFill in the following
[Unit] Description=minio Documentation=https://docs.min.io Wants=network-online.target After=network-online.target AssertFileIsExecutable=/usr/local/minio/minio [Service] #User and group User=root Group=root ExecStart=/usr/local/minio/run.sh #Let systemd restart this service always Restart=always #Specifies the maximum file descriptor number that can be opened by this process LimitNOFILE=65536 #Disable timeout logic and wait until process is stopped TimeoutStopSec=infinity SendSIGKILL=no [Install] WantedBy=multi-user.target -
Assign permissions and start
chmod +x /usr/lib/systemd/system/minio.serviceStart, view, and set startup
systemctl daemon-reload systemctl start minio systemctl enable minio systemctl status minioStart error handling
systemctl start minio Assertion failed on job for minio.service. # 是 minio.service 的 AssertFileIsExecutable 路径错误 AssertFileIsExecutable=/usr/local/minio # 改为 AssertFileIsExecutable=/usr/local/minio/minio
The stand-alone version has been installed here!
3. Springboot integrates MinIO
3.1 Project application
- Introduce minio dependencies into the project's pom file
<properties>
<minio.version>7.1.0</minio.version>
<dependencies>
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>${
minio.version}</version>
</dependency>
</dependencies>
application.ymlConfigure minio in file
minio:
endpoint: http://192.168.92.100
port: 9000
accessKey: minioadmin
secretKey: admin123
bucketName: test
secure: false
spring:
#设置文件上传大小限制
servlet:
multipart:
max-file-size: 100MB
max-request-size: 150MB
- Create minio configuration class and tool class
configuration class:
package cn.cvzhanshi.wechatpush.config;
import io.minio.MinioClient;
import io.minio.errors.InvalidPortException;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;
@Configuration
@Component
@ConfigurationProperties(prefix = "minio")
@Getter
@Setter
public class MinioConfig {
private String endpoint;
private int port;
private String accessKey;
private String secretKey;
private Boolean secure;
private String bucketName;
@Bean
public MinioClient getMinioClient() throws InvalidPortException {
MinioClient minioClient = MinioClient.builder().endpoint(endpoint, port, secure)
.credentials(accessKey, secretKey)
.build();
return minioClient;
}
//
// @Bean(name = "multipartResolver")
// public MultipartResolver multipartResolver(){
// CommonsMultipartResolver resolver = new CommonsMultipartResolver();
// resolver.setDefaultEncoding("UTF-8");
// //resolveLazily属性启用是为了推迟文件解析，以在在UploadAction中捕获文件大小异常
// resolver.setResolveLazily(true);
// resolver.setMaxInMemorySize(40960);
// //上传文件大小 50M 50*1024*1024
// resolver.setMaxUploadSize(50*1024*1024);
// return resolver;
// }
}
Tools:
package cn.cvzhanshi.wechatpush.utils;
import io.minio.*;
import io.minio.errors.*;
import io.minio.http.Method;
import io.minio.messages.Bucket;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.ArrayList;
import java.util.List;
/**
* MinIO 客户端工具类
*/
@Component
@Slf4j
public class MinioClientUtils {
@Autowired
private MinioClient minioClient;
private static final int DEFAULT_EXPIRY_TIME = 7 * 24 * 3600;
/**
* 检查存储桶是否存在
*
* @param bucketName 存储桶名称
* @return boolean
*/
public boolean bucketExists(String bucketName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = false;
flag = minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
if (flag) {
return true;
}
return false;
}
/**
* 创建存储桶
*
* @param bucketName 存储桶名称
*/
public boolean makeBucket(String bucketName) throws IOException, InvalidKeyException, InvalidResponseException, RegionConflictException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (!flag) {
minioClient.makeBucket(
MakeBucketArgs.builder()
.bucket(bucketName)
.build());
return true;
} else {
return false;
}
}
/**
* 列出所有存储桶名称
*
* @return List<String>
*/
public List<String> listBucketNames() throws IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
List<Bucket> bucketList = listBuckets();
List<String> bucketListName = new ArrayList<>();
for (Bucket bucket : bucketList) {
bucketListName.add(bucket.name());
}
return bucketListName;
}
/**
* 列出所有存储桶
*
* @return List<Bucket>
*/
public List<Bucket> listBuckets() throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
return minioClient.listBuckets();
}
/**
* 删除存储桶
*
* @param bucketName 存储桶名称
* @return boolean
*/
public boolean removeBucket(String bucketName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
Iterable<Result<Item>> myObjects = listObjects(bucketName);
for (Result<Item> result : myObjects) {
Item item = result.get();
// 有对象文件，则删除失败
if (item.size() > 0) {
return false;
}
}
// 删除存储桶，注意，只有存储桶为空时才能删除成功。
minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
flag = bucketExists(bucketName);
if (!flag) {
return true;
}
}
return false;
}
/**
* 列出存储桶中的所有对象名称
*
* @param bucketName 存储桶名称
* @return List<String>
*/
public List<String> listObjectNames(String bucketName) throws XmlParserException, IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, InvalidBucketNameException, InsufficientDataException, InternalException {
List<String> listObjectNames = new ArrayList<>();
boolean flag = bucketExists(bucketName);
if (flag) {
Iterable<Result<Item>> myObjects = listObjects(bucketName);
for (Result<Item> result : myObjects) {
Item item = result.get();
listObjectNames.add(item.objectName());
}
}
return listObjectNames;
}
/**
* 列出存储桶中的所有对象
*
* @param bucketName 存储桶名称
* @return Iterable<Result<Item>>
*/
public Iterable<Result<Item>> listObjects(String bucketName) throws XmlParserException, IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
if (flag) {
return minioClient.listObjects( ListObjectsArgs.builder().bucket(bucketName).build());
}
return null;
}
/**
* 通过文件上传到对象
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param fileName File name
* @return boolean
*/
public boolean uploadObject(String bucketName, String objectName, String fileName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
minioClient.uploadObject(
UploadObjectArgs.builder()
.bucket(bucketName).object(objectName).filename(fileName).build());
ObjectStat statObject = statObject(bucketName, objectName);
if (statObject != null && statObject.length() > 0) {
return true;
}
}
return false;
}
/**
* 文件上传
*
* @param bucketName 存储捅名称
* @param multipartFile 文件
* @param filename 文件名
*/
public void putObject(String bucketName, MultipartFile multipartFile, String filename) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
PutObjectOptions putObjectOptions = new PutObjectOptions(multipartFile.getSize(), PutObjectOptions.MIN_MULTIPART_SIZE);
putObjectOptions.setContentType(multipartFile.getContentType());
minioClient.putObject(
PutObjectArgs.builder().bucket(bucketName).object(filename).stream(
multipartFile.getInputStream(), multipartFile.getSize(), -1).contentType(multipartFile.getContentType())
.build());
}
/**
* 通过InputStream上传对象
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param inputStream 要上传的流
* @param contentType 上传的文件类型 例如 video/mp4 image/jpg
* @return boolean
*/
public boolean putObject(String bucketName, String objectName, InputStream inputStream,String contentType) throws IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
if (flag) {
minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(
//不清楚文件的大小时，可以传-1，10485760。如果知道大小也可以传入size，partsize。
inputStream, -1, 10485760)
.contentType(contentType)
.build());
ObjectStat statObject = statObject(bucketName, objectName);
if (statObject != null && statObject.length() > 0) {
return true;
}
}
return false;
}
/**
* 以流的形式获取一个文件对象
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @return InputStream
*/
public InputStream getObject(String bucketName, String objectName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
ObjectStat statObject = statObject(bucketName, objectName);
if (statObject != null && statObject.length() > 0) {
InputStream stream = minioClient.getObject( GetObjectArgs.builder()
.bucket(bucketName)
.object(objectName)
.build());
return stream;
}
}
return null;
}
/**
* 以流的形式获取一个文件对象（断点下载）
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param offset 起始字节的位置
* @param length 要读取的长度 (可选，如果无值则代表读到文件结尾)
* @return InputStream
*/
public InputStream getObject(String bucketName, String objectName, long offset, Long length) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
ObjectStat statObject = statObject(bucketName, objectName);
if (statObject != null && statObject.length() > 0) {
InputStream stream = minioClient.getObject( GetObjectArgs.builder()
.bucket(bucketName)
.object(objectName)
.offset(1024L)
.length(4096L)
.build());
return stream;
}
}
return null;
}
/**
* 下载并将文件保存到本地
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param fileName File name
* @return boolean
*/
public boolean downloadObject(String bucketName, String objectName, String fileName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
ObjectStat statObject = statObject(bucketName, objectName);
if (statObject != null && statObject.length() > 0) {
minioClient.downloadObject(DownloadObjectArgs.builder()
.bucket(bucketName)
.object(objectName)
.filename(fileName)
.build());
return true;
}
}
return false;
}
/**
* 删除一个对象
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
*/
public boolean removeObject(String bucketName, String objectName) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
boolean flag = bucketExists(bucketName);
if (flag) {
minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(objectName).build());
return true;
}
return false;
}
/**
* 删除指定桶的多个文件对象,返回删除错误的对象列表，全部删除成功，返回空列表
*
* @param bucketName 存储桶名称
* @param objectNames 含有要删除的多个object名称的迭代器对象
* @return
* eg:
* List<DeleteObject> objects = new LinkedList<>();
* objects.add(new DeleteObject("my-objectname1"));
* objects.add(new DeleteObject("my-objectname2"));
* objects.add(new DeleteObject("my-objectname3"));
*/
public List<String> removeObjects(String bucketName, List<DeleteObject> objectNames) throws IOException, InvalidKeyException, InvalidResponseException, InsufficientDataException, NoSuchAlgorithmException, ServerException, InternalException, XmlParserException, InvalidBucketNameException, ErrorResponseException {
List<String> deleteErrorNames = new ArrayList<>();
boolean flag = bucketExists(bucketName);
if (flag) {
Iterable<Result<DeleteError>> results = minioClient.removeObjects(RemoveObjectsArgs.builder().bucket(bucketName).objects(objectNames).build());
for (Result<DeleteError> result : results) {
DeleteError error = result.get();
deleteErrorNames.add(error.objectName());
}
}
return deleteErrorNames;
}
/**
* 生成一个给HTTP GET请求用的presigned URL。
* 浏览器/移动端的客户端可以用这个URL进行下载，即使其所在的存储桶是私有的。这个presigned URL可以设置一个失效时间，默认值是7天。
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param expires 失效时间（以秒为单位），默认是7天，不得大于七天
* @return
*/
public String getPresignedObjectUrl(String bucketName, String objectName, Integer expires) throws InvalidExpiresRangeException, IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
String url = "";
if (flag) {
if (expires < 1 || expires > DEFAULT_EXPIRY_TIME) {
throw new InvalidExpiresRangeException(expires,
"expires must be in range of 1 to " + DEFAULT_EXPIRY_TIME);
}
try {
url = minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucketName)
.object(objectName)
.expiry(expires)//动态参数
// .expiry(24 * 60 * 60)//用秒来计算一天时间有效期
// .expiry(1, TimeUnit.DAYS)//按天传参
// .expiry(1, TimeUnit.HOURS)//按小时传参数
.build());
} catch (ErrorResponseException | InsufficientDataException | InternalException | InvalidBucketNameException | InvalidExpiresRangeException | InvalidKeyException | InvalidResponseException | IOException | NoSuchAlgorithmException | ServerException | XmlParserException e) {
e.printStackTrace();
}
}
return url;
}
/**
* 生成一个给HTTP PUT请求用的presigned URL。
* 浏览器/移动端的客户端可以用这个URL进行上传，即使其所在的存储桶是私有的。这个presigned URL可以设置一个失效时间，默认值是7天。
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @param expires 失效时间（以秒为单位），默认是7天，不得大于七天
* @return String
*/
public String presignedPutObject(String bucketName, String objectName, Integer expires) throws IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
String url = "";
if (flag) {
if (expires < 1 || expires > DEFAULT_EXPIRY_TIME) {
try {
throw new InvalidExpiresRangeException(expires,
"expires must be in range of 1 to " + DEFAULT_EXPIRY_TIME);
} catch (InvalidExpiresRangeException e) {
e.printStackTrace();
}
}
try {
url = minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder()
.method(Method.PUT)
.bucket(bucketName)
.object(objectName)
.expiry(expires)//动态参数
// .expiry(24 * 60 * 60)//用秒来计算一天时间有效期
// .expiry(1, TimeUnit.DAYS)//按天传参
// .expiry(1, TimeUnit.HOURS)//按小时传参数
.build());
} catch (ErrorResponseException | InsufficientDataException e) {
e.printStackTrace();
} catch (InternalException e) {
log.error("InternalException",e);
} catch (InvalidBucketNameException e) {
log.error("InvalidBucketNameException",e);
} catch (InvalidExpiresRangeException e) {
log.error("InvalidExpiresRangeException",e);
} catch (InvalidKeyException e) {
log.error("InvalidKeyException",e);
} catch (InvalidResponseException e) {
log.error("InvalidResponseException",e);
} catch (IOException e) {
log.error("IOException",e);
} catch (NoSuchAlgorithmException e) {
log.error("NoSuchAlgorithmException",e);
} catch (ServerException e) {
log.error("ServerException",e);
} catch (XmlParserException e) {
log.error("XmlParserException",e);
}
}
return url;
}
/**
* 获取对象的元数据
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @return
*/
public ObjectStat statObject(String bucketName, String objectName) throws IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
if (flag) {
ObjectStat statObject = null;
try {
statObject = minioClient.statObject(StatObjectArgs.builder().bucket(bucketName).object(objectName).build());
} catch (ErrorResponseException e) {
log.error("ErrorResponseException",e);
} catch (InsufficientDataException e) {
log.error("ErrorResponseException",e);
e.printStackTrace();
} catch (InternalException e) {
log.error("InternalException",e);
} catch (InvalidBucketNameException e) {
log.error("InvalidBucketNameException",e);
} catch (InvalidKeyException e) {
log.error("InvalidKeyException",e);
} catch (InvalidResponseException e) {
log.error("InvalidResponseException",e);
} catch (IOException e) {
log.error("IOException",e);
} catch (NoSuchAlgorithmException e) {
log.error("NoSuchAlgorithmException",e);
} catch (ServerException e) {
log.error("ServerException",e);
} catch (XmlParserException e) {
log.error("XmlParserException",e);
}
return statObject;
}
return null;
}
/**
* 文件访问路径
*
* @param bucketName 存储桶名称
* @param objectName 存储桶里的对象名称
* @return String
*/
public String getObjectUrl(String bucketName, String objectName) throws IOException, InvalidResponseException, InvalidKeyException, NoSuchAlgorithmException, ServerException, ErrorResponseException, XmlParserException, InvalidBucketNameException, InsufficientDataException, InternalException {
boolean flag = bucketExists(bucketName);
String url = "";
if (flag) {
try {
url = minioClient.getObjectUrl(bucketName, objectName);
} catch (ErrorResponseException e) {
log.error("XmlParserException",e);
} catch (InsufficientDataException e) {
log.error("InsufficientDataException",e);
} catch (InternalException e) {
log.error("InternalException",e);
} catch (InvalidBucketNameException e) {
log.error("InvalidBucketNameException",e);
} catch (InvalidKeyException e) {
log.error("InvalidKeyException",e);
} catch (InvalidResponseException e) {
log.error("InvalidResponseException",e);
} catch (IOException e) {
log.error("IOException",e);
} catch (NoSuchAlgorithmException e) {
log.error("NoSuchAlgorithmException",e);
} catch (ServerException e) {
log.error("ServerException",e);
} catch (XmlParserException e) {
log.error("XmlParserException",e);
}
}
return url;
}
public void downloadFile(String bucketName, String fileName, String originalName, HttpServletResponse response) {
try {
InputStream file = minioClient.getObject(GetObjectArgs.builder()
.bucket(bucketName)
.object(fileName)
.build());
String filename = new String(fileName.getBytes("ISO8859-1"), StandardCharsets.UTF_8);
if (StringUtils.isNotEmpty(originalName)) {
fileName = originalName;
}
response.setHeader("Content-Disposition", "attachment;filename=" + filename);
ServletOutputStream servletOutputStream = response.getOutputStream();
int len;
byte[] buffer = new byte[1024];
while ((len = file.read(buffer)) > 0) {
servletOutputStream.write(buffer, 0, len);
}
servletOutputStream.flush();
file.close();
servletOutputStream.close();
} catch (ErrorResponseException e) {
log.error("ErrorResponseException",e);
} catch (Exception e) {
log.error("Exception",e);
}
}
}
-
Create a data table to save information about files uploaded to minio
CREATE TABLE `minio_file` ( `id` bigint(20) NOT NULL COMMENT '文件id', `original_file_name` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '原始文件名称', `file_ext_name` varchar(15) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '文件拓展名', `file_size` bigint(20) DEFAULT NULL COMMENT '文件大小(单位：字节）', `file_name` varchar(35) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '存入minio时的文件名称', `mime` varchar(50) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '文件的content-type', `file_url` varchar(255) COLLATE utf8mb4_bin DEFAULT NULL COMMENT '文件路径', `is_delete` tinyint(1) DEFAULT NULL COMMENT '是否删除 0 否 1 是', `create_by` varchar(25) COLLATE utf8mb4_bin DEFAULT NULL, `create_time` datetime DEFAULT NULL, `update_by` varchar(25) COLLATE utf8mb4_bin DEFAULT NULL, `update_time` datetime DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; -
Create minio upload interface
package cn.cvzhanshi.wechatpush.controller; import cn.cvzhanshi.wechatpush.config.MinioConfig; import cn.cvzhanshi.wechatpush.utils.MinioClientUtils; import cn.hutool.core.io.FileUtil; import io.swagger.annotations.Api; import io.swagger.annotations.ApiImplicitParam; import io.swagger.annotations.ApiImplicitParams; import io.swagger.annotations.ApiOperation; import lombok.AllArgsConstructor; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.RandomStringUtils; import org.apache.commons.lang3.math.NumberUtils; import org.springframework.stereotype.Controller; import org.springframework.util.CollectionUtils; import org.springframework.web.bind.annotation.*; import org.springframework.web.multipart.MultipartFile; import java.io.FileInputStream; import java.time.Instant; import java.util.ArrayList; import java.util.List; @RestController @RequestMapping("/fileHandle") @Slf4j @AllArgsConstructor @Api(tags = "文件处理模块") public class FileHandleController { private MinioClientUtils minioClientUtils; private MinioConfig minioConfig; @ApiOperation(value = "上传文件,支持批量上传") @ApiImplicitParams({ @ApiImplicitParam(name = "files", value = "文件流对象,接收数组格式", paramType = "query",required = true, dataType = "MultipartFile", allowMultiple = true)} ) @PostMapping("/uploadFile") public ApiResult uploadFile(@RequestParam(value = "files", required = true) MultipartFile[] files) { log.info(files.toString()); /* if (CollectionUtils.isEmpty(files)){ return ApiResult.error("未选择文件！"); }*/ List<String> MinioResponseDTOList = new ArrayList<>(); for (MultipartFile file : files) { String originalFilename = file.getOriginalFilename(); // 获取文件拓展名 String extName = FileUtil.extName(originalFilename); log.info("文件拓展名:" + extName); // 生成新的文件名，存入到minio long millSeconds = Instant.now().toEpochMilli(); String minioFileName = millSeconds + RandomStringUtils.randomNumeric(12) + "." + extName; String contentType = file.getContentType(); log.info("文件mime:{}", contentType); // 返回文件大小,单位字节 long size = file.getSize(); log.info("文件大小：" + size); try { String bucketName = minioConfig.getBucketName(); minioClientUtils.putObject(bucketName, file, minioFileName); String fileUrl = minioClientUtils.getObjectUrl(bucketName, minioFileName); /* MinioFile minioFile = new MinioFile(); minioFile.setOriginalFileName(originalFilename); minioFile.setFileExtName(extName); minioFile.setFileName(minioFileName); minioFile.setFileSize(size); minioFile.setMime(contentType); minioFile.setIsDelete(NumberUtils.INTEGER_ZERO); minioFile.setFileUrl(fileUrl); boolean insert = minioFile.insert(); if (insert) { MinioResponseDTO minioResponseDTO = new MinioResponseDTO(); minioResponseDTO.setFileId(minioFile.getId()); minioResponseDTO.setOriginalFileName(originalFilename); minioResponseDTO.setFileUrl(fileUrl); MinioResponseDTOList.add(minioResponseDTO); }*/ MinioResponseDTOList.add(fileUrl); } catch (Exception e) { log.error("上传文件出错:{}", e); return ApiResult.error("上传文件出错"); } } return ApiResult.success(MinioResponseDTOList); } /** * 仅仅用于测试，是否可以正常上传文件 * * @return * @throws Exception */ @GetMapping("/test") @ApiOperation(value = "测试minio文件上传") public ApiResult testPutObject() throws Exception { FileInputStream fileInputStream = new FileInputStream("C:\\Users\\MSI\\Desktop\\新建文本文档.txt"); boolean bs = minioClientUtils.putObject("fsp-dev", "新建文本文档.txt", fileInputStream, "image/jpg"); log.info("上传成功?" + bs); return ApiResult.success("上传成功"); } } -
test interface
4. Image compression and upload in Java
4.1 Background
Nowadays, everyone uses smartphones to take photos, and the photos taken are as small as 2-3 MB and as large as more than ten MB, so the picture display is slow. After thinking again and again, I decided to compress the image and upload it to the image server to solve the problem of slow image display.
4.2 Development preparation
-
Introduce maven dependencies
<!-- 图片压缩 --> <dependency> <groupId>net.coobird</groupId> <artifactId>thumbnailator</artifactId> <version>0.4.8</version> </dependency>This time we chose to use thumbnailator as the compression tool
-
thumbnailator introduction
Thumbnailator is a Java class library used to generate image thumbnails. It can generate image thumbnails through very simple code. It can also directly generate thumbnails for an entire directory of images. It supports image scaling, area cropping, watermarking, rotation, and retention
. Proportion -
Compression preparation
to determine whether it is a picture method/** * 判断文件是否为图片 */ public boolean isPicture(String imgName) { boolean flag = false; if (StringUtils.isBlank(imgName)) { return false; } String[] arr = { "bmp", "dib", "gif", "jfif", "jpe", "jpeg", "jpg", "png", "tif", "tiff", "ico"}; for (String item : arr) { if (item.equals(imgName)) { flag = true; break; } } return flag; }
4.3 Compressed upload
/**
* 上传文件
*
* @param file 文件
* @return
*/
public JSONObject uploadFile(MultipartFile file) throws Exception {
JSONObject res = new JSONObject();
res.put("code", 500);
// 判断上传文件是否为空
if (null == file || 0 == file.getSize()) {
res.put("msg", "上传文件不能为空");
return res;
}
// 判断存储桶是否存在
if (!client.bucketExists("test")) {
client.makeBucket("test");
}
// 拿到文件后缀名，例如：png
String suffix = file.getOriginalFilename().substring(file.getOriginalFilename().lastIndexOf(".") + 1);
// UUID 作为文件名
String uuid = String.valueOf(UUID.randomUUID());
// 新的文件名
String fileName = DateUtils.getYyyymmdd() + "/" + uuid + "." + suffix;
/**
* 判断是否是图片
* 判断是否超过了 100K
*/
if (isPicture(suffix) && (1024 * 1024 * 0.1) <= file.getSize()) {
// 在项目根目录下的 upload 目录中生成临时文件
File newFile = new File(ClassUtils.getDefaultClassLoader().getResource("upload").getPath() + uuid + "." + suffix);
// 小于 1M 的
if ((1024 * 1024 * 0.1) <= file.getSize() && file.getSize() <= (1024 * 1024)) {
Thumbnails.of(file.getInputStream()).scale(1f).outputQuality(0.3f).toFile(newFile);
}
// 1 - 2M 的
else if ((1024 * 1024) < file.getSize() && file.getSize() <= (1024 * 1024 * 2)) {
Thumbnails.of(file.getInputStream()).scale(1f).outputQuality(0.2f).toFile(newFile);
}
// 2M 以上的
else if ((1024 * 1024 * 2) < file.getSize()) {
Thumbnails.of(file.getInputStream()).scale(1f).outputQuality(0.1f).toFile(newFile);
}
// 获取输入流
FileInputStream input = new FileInputStream(newFile);
// 转为 MultipartFile
MultipartFile multipartFile = new MockMultipartFile("file", newFile.getName(), "text/plain", input);
// 开始上传
client.putObject("test", fileName, multipartFile.getInputStream(), file.getContentType());
// 删除临时文件
newFile.delete();
// 返回状态以及图片路径
res.put("code", 200);
res.put("msg", "上传成功");
res.put("url", minioProp.getEndpoint() + "/" + "test" + "/" + fileName);
}
// 不需要压缩，直接上传
else {
// 开始上传
client.putObject("test", fileName, file.getInputStream(), file.getContentType());
// 返回状态以及图片路径
res.put("code", 200);
res.put("msg", "上传成功");
res.put("url", minioProp.getEndpoint() + "/" + "test" + "/" + fileName);
}
return res;
}
- Here we judge that when the file is a picture and its size exceeds (1024 * 1024 * 0.1), which is about 100K, compression will be performed
- We first created a temporary file newFile in the upload directory under the root directory
- Thumbnails.of(file.getInputStream()).scale(1f).outputQuality(0.3f).toFile(newFile); Output the compressed file to a temporary file
- Then convert FileInputStream to MultipartFile for upload
- Finally delete the temporary file newFile.delete();
- Complete image compression upload
Problems encountered:
Thumbnails.scale effect will cause the image size to become larger
There should be a bug in Thumbnails, but it has not been updated. Therefore, according to the results of many tests: converting jpg to jpg has the best effect. So when the picture is png, first change it to jpg format and then compress it.
public static String imgConvert(String tempDirPath, String fileName, String fileExt) throws IOException {
String srcPath = tempDirPath + fileName; //原始图片路径
if("png".equals(fileExt)) {
//生成新图片名称
SimpleDateFormat df = new SimpleDateFormat("yyyyMMddHHmmss");
String fileString = df.format(new Date()) + "_" + new Random().nextInt(1000) + ".jpg";
//新图片全路径
String newJpg = tempDirPath + fileString;
// 1、先转换成jpg
Thumbnails.of(srcPath).scale(1f).toFile(newJpg);
//2.jpg图片压缩
Thumbnails.of(newJpg).scale(1f).outputQuality(0.25d).toFile(newJpg);
//压缩成功后,删除png图片
File f = new File(srcPath);
f.delete();
return fileString;
} else {
Thumbnails.of(srcPath).scale(1f).outputQuality(0.25d).toFile(srcPath);
}
return null;
}