Introduction, deployment steps and code examples of MinIO, a high-performance, scalable, distributed object storage system

Detailed introduction

MinIO is a popular open source object storage system designed to be compatible with Amazon S3 API and mainly used in private cloud and edge computing scenarios. It provides high performance, high availability, and easy-to-manage object storage services. The following is a detailed introduction and advantages and disadvantages of MinIO:

Architecture and features:
- Open source and cross-platform : MinIO is open source and free (Apache v2.0 license), and supports deployment on multiple platforms such as Linux, Windows, macOS, etc.
- Distributed storage : MinIO can be deployed in a distributed manner, achieving data redundancy through erasure coding (Erasure Coding) technology. Data can be recovered even if some hard disks fail. It is usually configured to allow the loss of N/2 number of disks while maintaining data availability. recover.
- Amazon S3 Compatibility : Fully compatible with Amazon S3’s v2 and v4 APIs, allowing S3-based applications and services to seamlessly switch to MinIO.
- High performance : Designed for large-scale workloads, able to handle extremely high concurrent read and write requests, and perform well on modern hardware.
- Security : Provides a variety of security measures, including server-side encryption (SSE), client-side encryption, authentication (using JWT or custom authentication mechanisms), and access policy controls.
Advantage:
- Cost-effectiveness : Compared to commercial object storage solutions, MinIO requires no additional licensing fees, reducing storage costs.
- Simple and easy to use : The installation and configuration are relatively simple and the user interface is friendly. It can be managed and operated through command line tools, Web UI or SDK.
- Scalability : Excellent horizontal scalability, nodes can be added as needed to increase capacity and performance.
- Data protection : Use erasure coding technology to ensure data reliability, and support verification and checksum to ensure data integrity.
Advantages and disadvantages in front-end usage scenarios:
- advantage :
  - For projects built through Webpack engineering, since the running environment includes Node.js, files can be directly uploaded to the MinIO server through the API.
  - Compatible with AWS SDK, it is convenient for developers to call existing S3 interface code for development.
- shortcoming :
  - It is not suitable for projects built with pure browser module loaders such as Vite, because Node.js native modules cannot be used.
  - The front-end direct upload lacks intuitive feedback on the upload progress, resulting in a poor user experience.
  - If configuration information such as ports, account passwords, etc. are hard-coded on the front end, there will be security risks and it will be difficult to maintain and expand.
Overall pros and cons:
- advantage :
  - High performance, high reliability and easy scalability
  - Good compatibility and low migration costs for organizations already using the AWS S3 ecosystem
  - The open source community is active and constantly updated and improved.
- shortcoming :
  - May lack some advanced features and professional support relative to larger enterprise-class storage solutions
  - In some specific application scenarios (such as the security and experience issues when directly connecting the front end mentioned above), additional development work is required to improve the solution.
  - For teams unfamiliar with the S3 API, there may be a learning curve.

MinIO is a powerful, ideal storage solution for cloud-native applications, especially for organizations looking for low-cost, high-availability and flexible scalability private cloud storage needs. However, in actual use, attention should be paid to weighing its advantages and disadvantages, and appropriate strategies and technical means should be adopted according to specific business scenarios to overcome potential challenges.

官网：MinIO | High Performance, Kubernetes Native Object Storage

MinIO deployment steps

There are some differences in how MinIO is deployed on different systems and environments, but the basic concept is similar, that is, running an S3 API-compatible object storage service through containerization or direct installation. The following will briefly introduce how to deploy the server side of MinIO on Kubernetes, Docker, CentOS, Windows, macOS and other systems, and provide some basic guidance on client connection.

Kubernetes (k8s) deploy MinIO Server

Create a MinIO StatefulSet or Deployment:

Yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minio-deployment
spec:
  replicas: 4 # 根据需求设置副本数量以实现高可用性
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      labels:
        app: minio
    spec:
      containers:
      - name: minio
        image: minio/minio:latest
        args: ["server", "/data"]
        ports:
        - containerPort: 9000
        env:
        - name: MINIO_ACCESS_KEY
          value: "your-access-key"
        - name: MINIO_SECRET_KEY
          value: "your-secret-key"
        volumeMounts:
        - mountPath: /data
          name: minio-data
      volumes:
      - name: minio-data
        persistentVolumeClaim:
          claimName: minio-pvc # 需要预先创建并绑定到实际PV

Use kubectl applythe command to apply a YAML configuration file to create a StatefulSet or Deployment resource. The configuration file usually specifies the image, persistent volume (PV) and persistent volume claim (PVC), port mapping, and required environment variables.
Configure Service resources: Create a Service resource to expose the MinIO service and provide external access through ClusterIP, NodePort or LoadBalancer.
Initialize the MinIO cluster (if necessary): If you plan to deploy distributed MinIO, you need to ensure that the startup parameters and environment variables are configured correctly so that each Pod can discover each other and form a cluster.

Docker deploys MinIO Server

Pull the image :
```
docker pull minio/minio
```

Running the container : Single node instance:

docker run -p 9000:9000 -p 9001:9001 --name minio \
-e "MINIO_ACCESS_KEY=youraccesskey" \
-e "MINIO_SECRET_KEY=yoursecretkey" \
-v /path/to/data:/data \          #新版目录已经更改为/mnt/data
minio/minio server /data

In distributed mode, additional environment variables are required to define the cluster mode and other cluster members.

docker-compose.yaml

version: '3.9'  # 使用最新的Docker Compose版本以利用新特性

services:
  minio:
    image: minio/minio:RELEASE.2024-01-05T22-17-24Z.fips  # 使用最新稳定版镜像（或指定特定版本）
    container_name: minio
    ports:
      - "19000:9000"  # 根据实际需求决定是否暴露9001端口，用于HTTPS访问时需要
      - "19001:9001"  # 根据实际需求决定是否暴露9001端口，用于HTTPS访问时需要
    environment:
      MINIO_ROOT_USER: youradminaccount
      MINIO_ROOT_PASSWORD: youradminpassword
      #MINIO_BROWSER: off  # （可选）关闭Web浏览器界面，如果只通过API访问
      #MINIO_OPTS: server --address ":9000"  # （可选）自定义启动参数，比如启用多节点集群模式等
    volumes:
      - /your pathto/minio_data:/data  # 确保宿主机目录存在并有合适的权限
    command: server /data --console-address ":9001"  --address ":9000" # 指定控制台监听的静态端口为9001

subman

podman run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"

podman run --name my-mc --hostname my-mc -it --entrypoint /bin/bash --rm minio/mc
[root@my-mc /]# mc alias set myminio/ https://my-minio-service MY-USER MY-PASSWORD
[root@my-mc /]# mc ls myminio/mybucket

Install MinIO Server directly on CentOS

Install dependencies :

sudo yum install -y epel-release
sudo yum install -y mc httpd-tools

Download and decompress the MinIO binary package : Download the latest version of the MinIO binary package, decompress and move it to /usr/local/bin a directory or a directory you specify. When customizing the directory, it is recommended to add the directory to the PATH of the system environment.
Running MinIO : Similar to data volume mounting in Docker, you need to provide a local directory as the storage path:
```
mkdir -p /mnt/minio/data
nohup minio server /mnt/minio/data &
```

Deploying MinIO Server on Windows

Precautions:

Step 1: Download MinIO Server

Visit MinIO official download page to obtain the executable file for Windows operating system: Download address: MinIO | Code and downloads to create high performance object storage
Step 2: Unzip and install
After the download is complete, unzip the file to the location where you wish to store the MinIO service program, for example C:\develop\minio.
Step 3: Create data storage directory

In order to store data persistently, a directory needs to be created on the local hard disk to save uploaded files. For example, D:\minio-data creating a storage directory.

Step 4: Start MinIO Server

Open a Command Prompt (CMD) window, run as administrator, and change to the directory where the MinIO executable is located:
```
cd C:\develop\minio
```
Then start the MinIO Server using the following command, replacing <MINIO_ACCESS_KEY>, <MINIO_SECRET_KEY> with your custom access key and private key, and specifying the directory where the data will be stored:
```
.\minio.exe server D:\minio-data --address ":9000" --access-key <MINIO_ACCESS_KEY> --secret-key <MINIO_SECRET_KEY>
```
Step 5: Environment configuration (optional)

If you need to set a domain name or other advanced options, you can use environment variables to configure them. For example, set the MINIO_DOMAIN environment variable:
```
set MINIO_DOMAIN=my-minio-server.local
```
Then start the MinIO service by including environment variables.

Step 6: Verification and Access
MinIO Server listens to port 9000 by default, so you can access the web management interface through a browser http://localhost:9000 .
Log in using the access key and private key you set previously (the initial defaults are minioadmin and minioadmin).
Please ensure that the firewall or security group rules allow access to port 9000. In a production environment, be sure to change the default access keys and private keys to ensure system security. If you need to deploy MinIO in distributed mode, please refer to the official documentation for specific steps on cluster deployment. For multi-node deployments, additional configuration parameters are required to define information such as cluster membership and erasure coding.

powershell way:

## 安装minio，先直接拉取exe
Invoke-WebRequest -Uri "https://dl.min.io/server/minio/release/windows-amd64/minio.exe" -OutFile "C:\minio.exe"
setx MINIO_ROOT_USER admin
setx MINIO_ROOT_PASSWORD password
C:\minio.exe server F:\Data --console-address ":9001"


## client
Invoke-WebRequest -Uri "https://dl.minio.io/client/mc/release/windows-amd64/mc.exe" -OutFile "C:\mc.exe"
C:\mc.exe alias set myminio/ http://MINIO-SERVER MYUSER MYPASSWORD

Deploying MinIO Server on macOS

Install Homebrew (if not already installed):

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Install MinIO using Homebrew :
```
brew install minio/stable/minio
```
Running MinIO : Run the MinIO server in a terminal pointing to a local directory:
```
mkdir -p ~/minio-storage
minio server ~/minio-storage
```

Client connection and use:

For client connections, regardless of the environment where the MinIO server is deployed, you can use the following steps to connect:

Set environment variables (optional, to simplify command line operations):

export MINIO_ENDPOINT=http://localhost:9000
export MINIO_ACCESS_KEY=youraccesskey
export MINIO_SECRET_KEY=yoursecretkey

Using the MinIO client tool mc : Install mc ( minio/mc) and then connect to the MinIO server using the following command:
```
mc alias set myminio $MINIO_ENDPOINT $MINIO_ACCESS_KEY $MINIO_SECRET_KEY
```
Verify connection : Run mc ls myminio to view the bucket list, if everything is OK you should be able to see the bucket information on the server.

Please note that in a production environment, be sure to replace "youraccesskey" and "yoursecretkey" in the above example with your actual access key and private key, and adjust the network configuration and persistence storage settings according to the actual situation. At the same time, be sure to follow best security practices, such as restricting public network access, enabling SSL encryption, etc.

Browser access:

Access and manage MinIO servers

After starting the MinIO server, you can access and manage the MinIO server through a browser or through the S3 API.

python sample code:

The following is a sample code written in Python that shows how to interact with the MinIO server using MinIO's Python SDK.

import boto3

# 创建MinIO客户端
s3 = boto3.client('s3',
                  endpoint_url='http://localhost:9000',
                  aws_access_key_id='YOUR_ACCESS_KEY',
                  aws_secret_access_key='YOUR_SECRET_KEY')

# 列出所有桶
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

# 创建一个桶
s3.create_bucket(Bucket='mybucket')

# 上传文件
s3.upload_file('myfile.txt', 'mybucket', 'myfile.txt')

# 下载文件
s3.download_file('mybucket', 'myfile.txt', 'downloaded_file.txt')

# 删除文件
s3.delete_object(Bucket='mybucket', Key='myfile.txt')

# 删除桶
s3.delete_bucket(Bucket='mybucket')

The above code uses the boto3 library to interact with the MinIO server. Note that endpoint_urlthe parameters should point to the correct MinIO server address, aws_access_key_idand aws_secret_access_keythe parameters should be set to the correct access keys.

nodejs example:

Install MinIO Node.js SDK

First, install the MinIO SDK in your Node.js project:

npm install minio

Sample code

1. Initialize MinIO client

const Minio = require('minio');

// 创建一个MinIO客户端实例
var minioClient = new Minio.Client({
    endPoint: 'your-minio-server-endpoint',
    port: 9000, // 默认端口为9000，如果自定义了端口，请修改
    useSSL: false, // 如果是https连接，则设置为true
    accessKey: 'your-access-key',
    secretKey: 'your-secret-key'
});

// 检查连接
minioClient.ping((err) => {
  if (err) throw err;
  console.log('Connected to MinIO server successfully.');
});

2. Create a bucket

minioClient.makeBucket('my-bucket', 'us-west-1', function(err) {
  if (err) return console.log(err);
  console.log('Bucket created successfully');
});

3. Upload a file to the bucket

// 读取本地文件
const fs = require('fs');
fs.readFile('local-file.txt', (err, data) => {
  if (err) throw err;

  // 上传文件到MinIO
  minioClient.putObject('my-bucket', 'remote-file.txt', data, data.length, 'application/text', function(err, etag) {
    if (err) return console.log(err);
    console.log("File uploaded successfully.");
  });
});

4. List all objects in the bucket

minioClient.listObjectsV2('my-bucket', '', true, function(err, objects) {
  if (err) return console.log(err);
  
  for (let obj of objects) {
    console.log(obj.name);
  }
});

your-minio-server-endpointPlease make sure to replace , your-access-keyand in the above code your-secret-keywith your actual MinIO server address, access key, and private key. Also, adjust the bucket name, local file path, and remote object name as needed.