Summary of the use of distributed file storage system Minio

Summary of the use of distributed file storage system Minio

1. Distributed file system application:

1.1. Minlo introduction:

Minlo is an object storage service based on the Apache License v2.0 open source protocol. It is compatible with the Amazon S3 cloud storage service interface, and is very suitable for storing large-capacity unstructured data, such as pictures, videos, log files, backup data, and container/virtual machine images, etc., and an object file can be of any size, from several kb to a maximum of 5T.

MinIo is a very lightweight service that can be easily integrated with other applications, like NodeJS, Redis or MySQL.

Official website: https://min.io/ http://www.minio.org.cn/
Object Storage Service (OSS) is a massive, secure, low-cost, and highly reliable cloud storage service, suitable for Store any type of file. Elastic expansion of capacity and processing power, multiple storage types to choose from, and comprehensive optimization of storage costs.
For small and medium-sized enterprises, if they do not choose to store on the cloud, then Minio is a good choice. Although the sparrow is small, it has all the internal organs. Of course, in addition to being directly used as object storage, Minio can also be used as the gateway layer of object storage services on the cloud, seamlessly connecting to Amazon S3 and Microsoft Azure.

In China: Alibaba, Tencent, Baidu, China Unicom, Huawei, China Mobile and more than 9,000 companies are also using MinlO products.

Advantages of Minio

Easy to deploy: a single binary is everything and supports various platforms.
minio supports massive storage, can be expanded by zone (the original zone will not be affected in any way), and supports a single object up to 5TB;
compatible with Amazon S3 interface, fully considering the needs and experience of developers;
Turing Academy
has low redundancy and high tolerance for disk damage, The standard and highest data redundancy factor is 2 (that is, to store a 1M data object, the actual occupied
disk space is 2M). However, data can still be read when any n/2 disks are damaged (n is the
number of disks in an Erasure Coding Set). And this kind of damage recovery is based on a single object, not based on the entire storage volume.
Excellent read and write performance

1.2 The basic concept of MinIO

**Object: **Basic objects stored in Minio, such as files, byte streams, Anything...
**Bucket: **Logical space used to store Objects. The data between each bucket is isolated from each other (playing an isolation role). For the client, it is equivalent to a top-level folder for storing files.
**Drive:** is the disk for storing data, (after the file is uploaded, minIo will create a Bucket 'directory' with the corresponding name on the disk of the server where it is located, and store the file under the corresponding directory) when MinIO starts , which is passed in as a parameter. All object data in Minio will be stored in Drive.
**Set:** is a set of Drives. Distributed deployment automatically divides one or more Sets according to the cluster size, and the Drives in each Set are
distributed in different locations. An object is stored on a Set. (For example: {1...64} is divided into 4 sets each of size 16.)
. An object is stored on a Set
. A cluster is divided into multiple Sets
. The number of Drives included in a Set is fixed, and is automatically calculated by the system based on the cluster size by default
. Drives in a SET are distributed on different nodes as much as possible

1.3 Erasure code EC (Erasure Code)

MinIO uses an erasure code mechanism to ensure high reliability, and highwayhash to deal with data corruption (Bit Rot Protection). Regarding the erasure code, simply speaking, it is possible to restore the lost data through mathematical calculations. It can add n copies of original data to m copies of data, and can restore any n copies of data in n+m copies to Raw data. That is, if any data less than or equal to m is invalid, it can still be restored through the remaining data.

纠删码EC:
   模式生效最少需要四块盘。

纠删码模式:
比方说有六块盘，其中4块盘存放数据，其中另外两块盘存放经过纠删码机制的一个算法计算形成的一个文件。如果丢失了一份文件：可能是数据文件，也可以是经过纠删码之后的文件都可以通过其他纠删码还原。

而判断文件是否损坏:
  可以对上的文件进行哈希计算得到一个Hash码，而Hash码是唯一的，如果下次再对这个文件进行哈希计算，得到的Hash码不同，则可以证明该文件损坏。

1.4 Storage form

When the file object is uploaded to MinIO, it will be stored in the corresponding data storage disk, with the Bucket name as the directory, the file name as the next-level
directory , and the file name is part.1 and xl.meta (the old version, the latest version is shown below) , the former is the encoded data block and the inspection block, and the latter is the metadata
file.

1.5. Storage scheme

2. Minio environment construction

Official document: https://docs.min.io/docs/
Chinese document: http://docs.minio.org.cn/docs/ (not updated in time, easy to be cheated)
minio supports multiple server startup modes:

2.1 Stand-alone deployment

The standalone mode of the minio server, that is, the disks to be managed are all local to the host. This startup mode is generally only used for
verification and learning in the experimental environment and test environment. In the standalone mode, it can also be divided into non-erasure code mode and erasure
code mode.

non-erasure code mode

In this startup mode, for each piece of object data, minio directly stores the data under data without creating a copy or enabling the
erasure code mechanism. Therefore, in this mode, both the service instance and the disk are "single points", without any high availability guarantee, and disk damage
means data loss.

erasure code mode

This mode passes in multiple local disk parameters for the minio server instance. Once encountering more than one disk parameter, minio server will automatically
enable erasure code mode. The erasure code has requirements for the number of disks. If the requirements are not met, the instance startup will fail
.
After the erasure code is enabled, at least 4 endpoints (in standalone mode, that is, directories on the local disk) are required to be passed to the minio server .

Based on centos7

operating system	CPU architecture	address
GNU/Linux	64-bit Intel	http://dl.minio.org.cn/server/minio/release/linux-amd64/minio

wget -q http://dl.minio.org.cn/server/minio/release/linux-amd64/minio
chmod +x minio
#启动minio server服务，指定数据存储目录/mnt/data
./minio server /mnt/data

The default username and password minioadmin:minioadmin, to modify the default username and password can be used:

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678

The default configuration directory is ${HOME}/.minio, you can customize the configuration directory through the --config-dir command:

./minio server --config-dir /mnt/config /mnt/data

The console listening port is dynamically generated, and a static port can be specified by --console-address ":port"

./minio server --console-address ":50000" /mnt/data

Access minio console: http://192.168.3.14:50000/dashboard

based on docker

There is a problem: The browser cannot access the minio console because the console port is not exposed to the outside world

docker run -p 9000:9000 --name minio \
-v /mnt/data:/data \
-v /mnt/config:/root/.minio \
minio/minio server /data

There is a problem: The browser cannot access the minio console because the console port is not exposed to the outside world

Expose the port of the minio console externally, and specify the console port as a static port through --console-address ":50000"

docker run -p 9000:9000 -p 50000:50000 --name minio \
-v /mnt/data:/data \
-v /mnt/config:/root/.minio \
minio/minio server --console-address ":50000" /data

Access the minio console: http://192.168.3.14:50000/

MinIO custom username and password

docker run -d -p 9000:9000 -p 50000:50000 --name minio \
-e "MINIO_ROOT_USER=admin" \
-e "MINIO_ROOT_PASSWORD=12345678" \
-v /mnt/data:/data \
-v /mnt/config:/root/.minio \
minio/minio server --console-address ":50000" /data

2.2, minio erasure code mode

	Minio使用纠删码erasure code 和校验和checksum 来保护数据免受硬件故障和无声数据损坏。 即便
您丢失一半数量（N/2）的硬盘，您仍然可以恢复数据。

	纠删码是一种恢复丢失和损坏数据的数学算法， Minio采用Reed-Solomon code将对象拆分成N/2数据和N/2 奇偶校验块。 这就意味着如果是12块盘，一个对象会被分成6个数据块、6个奇偶校验块，你可以丢失任意6块盘（不管其是存放的数据块还是奇偶校验块），你仍可以从剩下的盘中的数据进行恢复。

Use the Minio Docker image to start the Minio service on 8 disks:

docker run -d -p 9000:9000 -p 50000:50000 --name minio \
-v /mnt/data1:/data1 \
-v /mnt/data2:/data2 \
-v /mnt/data3:/data3 \
-v /mnt/data4:/data4 \
-v /mnt/data5:/data5 \
-v /mnt/data6:/data6 \
-v /mnt/data7:/data7 \
-v /mnt/data8:/data8 \
minio/minio server /data{
    
    1...8} --console-address ":50000"

3 Distributed cluster deployment

Distributed Minio allows you to combine multiple hard disks (even on different machines) into an object storage service. Since hard disks are distributed on different nodes, distributed Minio avoids a single point of failure.

Common Methods for Distributed Storage Reliability

The key point of distributed storage is the reliability of data, that is, to ensure the integrity of data without loss or damage. Only under the premise of achieving reliability can there be a basis for pursuing consistency, high availability, and high performance. In the field of storage, there are generally two types of methods to ensure data reliability, one is the redundancy method, and the other is the verification method.

redundancy

The redundancy method is the most simple and direct, that is, to make a copy backup of the stored data. When the data is lost or damaged, the backup content can be used for recovery, and the number of copies determines the reliability of the data. There will be cost considerations in this. The more replica data, the more reliable the data, but the more equipment is required, the higher the cost. Reliability is allowing one piece of data to be lost. At present, many distributed systems are implemented in this way, such as Hadoop file system (3 copies), Redis cluster, MySQL master-backup mode, etc.

check

The verification method is to verify and restore the lost or damaged data through the mathematical calculation of the verification code. Note that there are two functions here, one check, by calculating the checksum (checksum) of the data, you can check whether the data is complete, whether there is damage or change, which is often used in data transmission and storage, such as the TCP protocol ; The second is recovery and restoration. By combining the data with check codes and mathematical calculations, the lost or damaged data can be restored, which can reduce redundancy on the premise of ensuring data reliability, such as RAID technology in stand-alone hard disk storage, erasure codes (Erasure Code) technology, etc. MinIO uses erasure coding technology.

Advantages of Distributed Minio

data protection

Distributed Minio uses erasure codes to prevent multiple node downtime and bit decay bit rot .

Distributed Minio requires at least 4 hard disks, and the use of distributed Minio automatically introduces the erasure code function.

high availability

There is a single point of failure in the stand-alone Minio service. On the contrary, if it is a distributed Minio with N hard disks, as long as there are N/2 hard disks online, your data is safe. But you need at least N/2+1 hard drives to create new objects.

For example, a 16-node Minio cluster with 16 hard disks per node, even if 8 servers go down, the cluster is still readable, but you need 9 servers to write data.

consistency

In distributed and stand-alone mode, Minio strictly abides by the read-after-write consistency model for all read and write operations.

Running Distributed Minio

To start a distributed Minio instance, you only need to pass the hard disk location as a parameter to the minio server command, and then you need to run the same command on all other nodes.

• All nodes in distributed Minio need to have the same access key and secret key so that these nodes can establish connections. In order to achieve this, you need to export the access key and secret key as environment variables before executing the minio server command. The new version uses MINIO_ROOT_USER&MINIO_ROOT_PASSWORD.

• The disks used by Distributed Minio must be clean with no data in them.

• The IP in the example below is for example reference only, you need to change it to your actual IP and folder path.

• The node time difference in Distributed Minio cannot exceed 3 seconds, you can use NTP to ensure consistent time.

• Running distributed Minio under Windows is in the experimental stage, please take it easy.

8 nodes, 1 disk per node

To start a distributed Minio instance with 8 nodes and 1 disk per node, you need to run the following command on all 8 nodes:

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678
minio server http://192.168.1.11/export1 
http://192.168.1.12/export2 \
http://192.168.1.13/export3 
http://192.168.1.14/export4 \
http://192.168.1.15/export5 
http://192.168.1.16/export6 \
http://192.168.1.17/export7 
http://192.168.1.18/export8

4 nodes, 4 disks per node

To start a distributed Minio instance, 4 nodes, 4 disks per node, you need to run the following command on all 4 nodes

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678
minio server http://192.168.1.11/export1 http://192.168.1.11/export2 \
http://192.168.1.11/export3 http://192.168.1.11/export4 \
http://192.168.1.12/export1 http://192.168.1.12/export2 \
http://192.168.1.12/export3 http://192.168.1.12/export4 \
http://192.168.1.13/export1 http://192.168.1.13/export2 \
http://192.168.1.13/export3 http://192.168.1.13/export4 \
http://192.168.1.14/export1 http://192.168.1.14/export2 \
http://192.168.1.14/export3 http://192.168.1.14/export4

Start by script

View logs and processes after executing scripts

Test upload:

Storage structure:

Deploy MinIO with Docker Compose

Notice:

Minio在Linux系统中通过多个部署时启动的时候，一定要指定文件存储的哪个磁盘？

在初次搭建MinIO的时候，可是化界面console的端口并不是固定的，而是动态变化的，每次打开都是不同的。可以通过命令进行设置固定的端口号。

​

https://docs.min.io/docs/deploy-minio-on-docker-compose.html
To deploy distributed MinIO on Docker Compose, please download docker-compose.yaml and nginx.conf to your current
job Table of contents.

docker-compose pull
docker-compose up

Extend an existing distributed cluster

For example, we start the MinIO cluster through the zone, the command line is as follows:

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678
minio server http://host{
    
    1...32}/export{
    
    1...32}

MinIO supports commands to specify new clusters to expand existing clusters (erasure code mode), the command line is as follows:

export MINIO_ROOT_USER=admin
export MINIO_ROOT_PASSWORD=12345678
minio server http://host{
    
    1...32}/export{
    
    1...32}
http://host{
    
    33...64}/export{
    
    1...32}

Now the entire cluster has expanded to 1024 disks, and the total number of disks has become 2048. New object upload requests will be automatically allocated to the least used cluster
. Through the above expansion strategy, you can expand your cluster on demand. Restarting the cluster after reconfiguration will take effect immediately in the cluster
and will not affect the existing cluster. In the above command, we can regard the original cluster as one zone, and the new cluster as another zone, and new objects
are placed in the zone according to the proportion of the available space in each zone. Within each zone, the location is determined based on a deterministic hashing algorithm.

Explanation: Each region you add must have the same disk count (erasure code set) size as the original region in order to maintain the same
data redundancy SLA. For example, the first zone has 8 disks, you can expand the cluster to zones with 16, 32 or 1024 disks
, you just need to ensure that the deployment SLA is a multiple of the original zone.

Implement loadbalancer based on nginx

upstream minio {
    
    
    server 192.168.3.14:9001;
    server 192.168.3.14:9002;
    server 192.168.3.14:9003;
    server 192.168.3.14:9004;
}
    upstream console {
    
    
        ip_hash;
        server 192.168.3.14:50001;
        server 192.168.3.14:50002;
        server 192.168.3.14:50003;
        server 192.168.3.14:50004;
    }
server {
    
    
    listen 9000;
    listen [::]:9000;
    server_name localhost;
    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific
    value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;
		location / {
    
    
            proxy_set_header Host $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_connect_timeout 300;
            # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
            proxy_http_version 1.1;
            proxy_set_header Connection "";
            chunked_transfer_encoding off;
            proxy_pass http://minio;
		}
}
server {
    
    
    listen 50000;
    listen [::]:50000;
    server_name localhost;
    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific
    value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;
        location / {
    
    
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;
        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;
        proxy_pass http://console;
 		 }
}

Test: http://192.168.3.14:50001/dashboard

3.3 Use of Minio client

MinIO Client (mc) provides an alternative to UNIX commands like ls, cat, cp, mirror, diff, find, etc. It supports file systems and Amazon S3-compatible cloud storage services (AWS Signature v2 and v4).

ls 列出文件和文件夹。
mb 创建一个存储桶或一个文件夹。
cat 显示文件和对象内容。
pipe 将一个STDIN重定向到一个对象或者文件或者STDOUT。
share 生成用于共享的URL。
cp 拷贝文件和对象。
mirror 给存储桶和文件夹做镜像。
find 基于参数查找文件。
diff 对两个文件夹或者存储桶比较差异。
rm 删除文件和对象。
events 管理对象通知。
watch 监视文件和对象的事件。
policy 管理访问策略。
config 管理mc配置文件。
update 检查软件更新。
version 输出版本信息。

Deploy client mc

platform	CPU architecture	URL
GNU/Linux	64-bit Intel	http://dl.minio.org.cn/client/mc/release/linux-amd64/mc

wget http://dl.minio.org.cn/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help
mv mc /usr/local/sbin/

platform	CPU architecture	URL
Microsoft Windows	64-bit Intel	http://dl.minio.org.cn/client/mc/release/windows-amd64/mc.exe

configure mc

mc stores all configuration information in ~/.mc/config.json file

# 查询mc host配置
mc config host ls
# 添加minio服务
mc config host add minio-server http://192.168.3.14:9000 admin 12345678
# 删除host
mc config host remove minio-server

mc command use

ls - list buckets and objects	mb - create bucket	cat - merge objects
cp - copy object	rm - delete an object	pipe - Pipe to an object
share - to share	mirror - bucket mirroring	find - find files and objects
diff - compare bucket differences	policy - set the access policy for the bucket or prefix
config - Manage configuration files	watch - event listener	events - Manage bucket events
update - Manage software updates	version - display version information

Upload Download

# 查询minio服务上的所有buckets(文件和文件夹)
mc ls minio-server
# 下载文件
mc cp minio-server/tulingmall/fox/fox.jpg /tmp/
#删除文件
mc rm minio-server/tulingmall/fox/fox.jpg
#上传文件
mc cp zookeeper.out minio-server/tulingmall/

Bucket management

# 创建bucket
mc mb minio-server/bucket01
# 删除bucket
mc rb minio-server/bucket02
# bucket不为空，可以强制删除 慎用
mc rb --force minio-server/bucket01

#查询bucket03磁盘使用情况
mc du minio-server/bucket03

mc admin use

MinIO Client (mc) provides an "admin" subcommand to perform administrative tasks on your MinIO deployment.

service 服务重启并停止所有MinIO服务器
update 更新更新所有MinIO服务器
info 信息显示MinIO服务器信息
user 用户管理用户
group 小组管理小组
policy MinIO服务器中定义的策略管理策略
config 配置管理MinIO服务器配置
heal 修复MinIO服务器上的磁盘，存储桶和对象
profile 概要文件生成概要文件数据以进行调试
top 顶部提供MinIO的顶部统计信息
trace 跟踪显示MinIO服务器的http跟踪
console 控制台显示MinIO服务器的控制台日志
prometheus Prometheus管理Prometheus配置
kms kms执行KMS管理操作

User Management

mc admin user --help
#新建用户
mc admin user add minio-server fox
mc admin user add minio-server fox02 12345678
#查看用户
mc admin user list minio-server
#禁用用户
mc admin user disable minio-server fox02
#启用用户
mc admin user disable minio-server fox02
#查看用户信息
mc admin user info minio-server fox
#删除用户
mc admin user remove minio-server fox02

policy management

policy command to add, delete, list policies, get information about policies and set policies for users on the MinIO server.

mc admin policy --help
#列出MinIO上的所有固定策略
mc admin policy list minio-server
# 查看plicy信息
mc admin policy info minio-server readwrite

add new policy

Write policy file: /root/tulingmall.json

{
    
    
    "Version": "2012-10-17",
    "Statement": [
    {
    
    
    "Effect": "Allow",
    "Action": [
    "s3:GetBucketLocation",
    "s3:GetObject"
    ],
        "Resource": [
        "arn:aws:s3:::tulingmall"
        ]
},{
    
    
    "Effect": "Allow",
    "Action": [
    "s3:*"
        ],
    "Resource": [
    "arn:aws:s3:::tulingmall/*"
    		]
		}
	]
}


"Action": [
    "s3:GetBucketLocation",
    "s3:ListBucket",
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject"
]

Add tulingmall.json to policy database

# 添加新的策略
mc admin policy add minio-server tulingmall-admin /root/tulingmall.json
mc admin policy list minio-server

mc admin user add minio-server fox03 12345678
# 设置用户的访问策略
mc admin policy set minio-server tulingmall-admin user=fox03

Test: fox03/12345678 logs in to the minio console http://192.168.3.14:50000/, only the bucket of tulingmall can be operated

4、Mini Java Client使用

MinIO Java Client SDK provides a simple API to access any Amazon S3-compatible object storage service.
Official demo: https://github.com/minio/minio-java
Official documentation: https://docs.min.io/docs/java-client-api-reference.html

Introduce dependencies

<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.3.0</version>
</dependency>
<dependency>
<groupId>me.tongfei</groupId>
<artifactId>progressbar</artifactId>
<version>0.5.3</version>
</dependency>
<dependency>
<groupId>com.squareup.okhttp3</groupId>
<artifactId>okhttp</artifactId>
<version>4.8.1</version>
</dependency>

4.1 File upload

public class FileUploader {
    
    
public static void main(String[] args)
	throws IOException, NoSuchAlgorithmException, InvalidKeyException {
    
    
        try {
    
    
        // Create a minioClient with the MinIO server playground, its access key
        and secret key.
        MinioClient minioClient =
        MinioClient.builder()
        .endpoint("http://192.168.3.14:9000")
        .credentials("admin", "12345678")
        .build();
        // 创建bucket
        String bucketName = "tulingmall";
        boolean exists =
        minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
            if (!exists) {
    
    
                // 不存在，创建bucket
                 minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
                }
			// 上传文件
            minioClient.uploadObject(
            UploadObjectArgs.builder()
            .bucket(bucketName)
            .object("tuling-mall-master.zip")
            .filename("F:\\mall\\tuling-mall-master.zip")
            .build());
            System.out.println("上传文件成功");
            } catch (MinioException e) {
    
    
            System.out.println("Error occurred: " + e);
            System.out.println("HTTP trace: " + e.httpTrace());
			}
		}
}	

4.2 File download

public class DownLoadDemo {
    
    
public static void main(String[] args) {
    
    
    // Create a minioClient with the MinIO server playground, its access key
    and secret key.
    MinioClient minioClient =
    MinioClient.builder()
    .endpoint("http://192.168.3.14:9000")
    .credentials("admin", "12345678")
    .build();
	// Download object given the bucket, object name and output file name
    try {
    
    
    minioClient.downloadObject(
    DownloadObjectArgs.builder()
    .bucket("tulingmall")
    .object("fox/fox.jpg")
    .filename("fox.jpg")
    .build());
    } catch (Exception e) {
    
    
    e.printStackTrace();
		}
	}
}

4.3 Spring boot integrates minio

Build the MinioClient object and hand it over to spring management

// 获取yml文件中属性值
@Data
@Component
@ConfigurationProperties(prefix = "minio")
public class MinioProperties {
    
    
    private String endpoint;
    private String accessKey;
    private String secretKey;
 }

//minio 配置属性
minio:
endpoint: http://192.168.3.14:9000
accesskey: admin
secretKey: 12345678

// minio 配置类
@Configuration
public class MinioConfig {
    
    
@Autowired
private MinioProperties minioProperties;
@Bean
public MinioClient minioClient(){
    
    
    MinioClient minioClient =
    MinioClient.builder()
    .endpoint(minioProperties.getEndpoint())
    .credentials(minioProperties.getAccessKey(),
    minioProperties.getSecretKey())
    .build();
	return minioClient;
	}
}

Implement file upload, download, delete operations

package cn.js.Demo04;

import sun.misc.IOUtils;

import java.io.IOException;
import java.text.DecimalFormat;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

@RestController
@Slf4j
public class MinioController {
    
    

    @Autowired
    private MinioClient minioClient;

    @Value("${minio.bucketName}")
    private String bucketName;

    @GetMapping("/list")
    public List<Object> list() throws Exception {
    
    
        //获取bucket列表
        Iterable<Result<Item>> myObjects = minioClient.listObjects(
                ListObjectsArgs.builder().bucket(bucketName).build());
        Iterator<Result<Item>> iterator = myObjects.iterator();
        List<Object> items = new ArrayList<>();
        String format = "{'fileName':'%s','fileSize':'%s'}";
        while (iterator.hasNext()) {
    
    
            Item item = iterator.next().get();
            items.add(JSON.parse(String.format(format, item.objectName(),
                    formatFileSize(item.size()))));
        }
        return items;
    }

    @PostMapping("/upload")
    public Res upload(@RequestParam(name = "file", required = false)
                              MultipartFile[] file) {
    
    
        if (file == null || file.length == 0) {
    
    
            return Res.error("上传文件不能为空");
        }
        List<String> orgfileNameList = new ArrayList<>(file.length);
        for (MultipartFile multipartFile : file) {
    
    
            String orgfileName = multipartFile.getOriginalFilename();
            orgfileNameList.add(orgfileName);
            try {
    
    
//文件上传
                InputStream in = multipartFile.getInputStream();
                minioClient.putObject(
                        PutObjectArgs.builder().bucket(bucketName).object(orgfileName).stream(
                                in, multipartFile.getSize(), -1)
                                .contentType(multipartFile.getContentType())
                                .build());
                in.close();
            } catch (Exception e) {
    
    
                log.error(e.getMessage());
                return Res.error("上传失败");
            }
        }
        Map<String, Object> data = new HashMap<String, Object>();
        data.put("bucketName", bucketName);
        data.put("fileName", orgfileNameList);
        return Res.ok("上传成功", data);
    }

    @RequestMapping("/download/{fileName}")
    public void download(HttpServletResponse response, @PathVariable("fileName")
            String fileName) {
    
    
        InputStream in = null;
        try {
    
    
// 获取对象信息
            StatObjectResponse stat = minioClient.statObject(
                    StatObjectArgs.builder().bucket(bucketName).object(fileName).build());
            response.setContentType(stat.contentType());
            response.setHeader("Content-Disposition", "attachment;filename=" +
                    URLEncoder.encode(fileName, "UTF-8"));
//文件下载
            in = minioClient.getObject(
                    GetObjectArgs.builder()
                            .bucket(bucketName)
                            .object(fileName)
                            .build());
            IOUtils.copy(in, response.getOutputStream());
        } catch (Exception e) {
    
    
            log.error(e.getMessage());
        } finally {
    
    
            if (in != null) {
    
    
                try {
    
    
                    in.close();
                } catch (IOException e) {
    
    
                    log.error(e.getMessage());
                }
            }
        }
    }

    @DeleteMapping("/delete/{fileName}")
    public Res delete(@PathVariable("fileName") String fileName) {
    
    
        try {
    
    
            minioClient.removeObject(
                    RemoveObjectArgs.builder().bucket(bucketName).object(fileName).build());
        } catch (Exception e) {
    
    
            log.error(e.getMessage());
            return Res.error("删除失败");
        }
        return Res.ok("删除成功", null);
    }

    private static String formatFileSize(long fileS) {
    
    
        DecimalFormat df = new DecimalFormat("#.00");
        String fileSizeString = "";
        String wrongSize = "0B";
        if (fileS == 0) {
    
    
            return wrongSize;
        }
        if (fileS < 1024) {
    
    
            fileSizeString = df.format((double) fileS) + " B";
        } else if (fileS < 1048576) {
    
    
            fileSizeString = df.format((double) fileS / 1024) + " KB";
        } else if (fileS < 1073741824) {
    
    
            fileSizeString = df.format((double) fileS / 1048576) + " MB";
        } else {
    
    
            fileSizeString = df.format((double) fileS / 1073741824) + " GB";
        }
        return fileSizeString;
    }
}