Building a Private Object Storage Service with Minio_Best Practices of Open Source PaaS Rainbond

Overview

Minio is built on a cloud-native foundation; features such as distributed and shared storage; an object storage service designed to scale sustainably in a multi-tenant environment. It is most suitable for storing unstructured data such as: photos, videos, log files, containers/VMs/images, etc. The size of a single storage object can be up to 5TB.

Implementation Architecture

single node

• Depending on whether the storage is remote, you can directly use FS or NFS to directly operate the objects in the storage
• Call the S3 interface and use FS or NFS to operate the Object through Minio

multi-node

Multi-node Minio will distinguish different tenants according to different Access_key and Secret_Key, and each tenant can operate the corresponding Server to obtain Object. Minio Servers can be isolated from each other through different process models , containers or virtual machines.

distributed

Distributed Minio can be extended to as many servers as needed in a shared-nothing architecture, and all nodes need to use the same Access_key and Secret_key to log in. Distributed Minio uses a web load balancer or DNS round-robin to achieve load balancing among servers.

Features

• Amazon S3 Compatible

Minio uses Amazon S3 v2/v4 API. Minio server can be accessed using Minio SDK, Minio Client, AWS SDK and AWS CLI.

• data protection

Minio uses Minio Erasure Code to prevent hardware failure. Maybe more than half of the driver will be damaged, but still recover from it.

• highly available

Minio servers can tolerate up to (N/2)-1 node failures in a distributed setup. Also, you can configure the Minio server to store data between Minio and any Amazon S3 compatible server.

• Lambda computing

The Minio server triggers Lambda functions through its AWS SNS/SQS compatible event notification service. Supported targets are message queues such as Kafka, NATS, AMQP, MQTT, Webhooks, and databases such as Elasticsearch, Redis, Postgres, and MySQL.

• Encrypted and Tamper Resistant

Minio provides confidentiality, integrity, and authenticity guarantees for encrypted data with minimal performance overhead. Server-side and client-side encryption is supported using AES-256-GCM , ChaCha20-Poly1305 and AES-CBC . Encrypted objects are tamper-proof using AEAD server-side encryption.

• Can be connected to back-end storage

In addition to Minio's own file system, DAS, JBODs, NAS, Google Cloud Storage, and Azure Blob Storage are also supported.

• sdk support

Based on the lightweight characteristics of Minio, it is supported by SDKs in languages ​​such as Java, Python or Go.

For example: Java classes in the case of using Maven to manage the Jar, pom.xmlspecify Minio in:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>3.0.12</version>
</dependency>

use

Minio provides: Server side with graphical interface; Client side with command line operation. The following introduces the use of Minio Server and Minio Client.

Minio Server

• Use the Access Key and Secret Key to log in to Minio. After successful login, enter the following interface

Access Key and Secret Key can be randomly generated by Minio Server, or can be customized by variables. The following self-service deployment describes how to define and obtain the Access Key and Secret Key.

• Click bucketthe corresponding part of module 1 to create a new bucket, which can be seen in module 2 bucket.

• Choose one in module 2 bucket. Click the corresponding part of module 1 uploadto upload the file to the selected one bucket. bucketYou can see the uploaded files in the corresponding white part of the page .

• Move the mouse to any module 2 bucket, and click on the corresponding bucketright side to operate policyand deleteoperate. The request permissions that policycan be set .Prefix

• Click Module 3 to view and set the basic information of the Object:

  • View shared addressesShareable Link

  • Set the expiration time, the maximum storage time is 7 days

  • The Object pops up at the top of the dialog and the remaining expiration time

Minio Client

download binary

wget https://dl.minio.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help

Basic Operation Commands

• Configure an existing Minio Server

./mc config host add <custom_name> <Minio_Server_address> <access_key> <secret_keyt> S3v4

example:

./mc config host add test http://9000.gr8be71d.grapps.ali-hz.goodrain.net access_key secret_key S3v4

• Create bucket

./mc mb <custom_name>/[bucket_name]/[object_name]

example:

./mc mb test/data

• View the bucket and object of Minio Server

./mc ls <custom_name>/[bucket_name]/[object_name]

example:

./mc ls test/data

• Upload/Download Object

# cp到Minio Server(上传)
./mc cp <object> <custom_name>/[bucket_name]

# cp到本地(下载)
./mc cp <custom_name>/[bucket_name]/[object_name] <local_path>

example:

./mc cp README.md test/data

• Delete Object or Bucket

./mc rm <custom_name>/[bucket_name]/[object_name]

example:

# 删除bucket，因为data下存在名为README.md的object,故需追加--force参数来强制删除bucket
./mc rm test/data --force

* Shared access

mcProvides the share method, which can temporarily upload or download objects through the URL generated by authorization.

• download

Specify an Object in Minio Server to generate a URL for temporarily downloading the Object.

./mc share download [—expire [h|m|s]] <custom_name>/[bucket_name]/[object_name]

example:

# --expire 168h代表生成的URL有效时间仅168小时
./mc share download --expire 168h test/data/README.md              

• upload

Specify the path after uploading a file to Minio Server, and generate a command for temporary upload.

./mc share upload [—expire [h|m|s]] <custom_name>/[bucket_name]/[object_name]

example:

./mc share upload test/data/README.md

Generate a command similar to the following:

curl http://9000.gr17b6e1.grapps.ali-hz.goodrain.net/data/ \
-F x-amz-credential=access_key/20180425/us-east-1/s3/aws4_request \
-F x-amz-date=20180425T031310Z \
-F x-amz-signature=68ac9f102afd6a87526ecb9ce6025dee4f85b25cf054f5a7668a73ae0ef9f4dc \
-F bucket=data \
-F policy=eyJleHBpcmF0aW9uIjoiMjAxOC0wNS0wMlQwMzoxMzowOS45MTlaIiwiY29uZGl0aW9ucyI6W1siZXEiLCIkYnVja2V0IiwiZGF0YSJdLFsiZXEiLCIka2V5IiwiaW5zdGFsbC5zaCJdLFsiZXEiLCIkeC1hbXotZGF0ZSIsIjIwMTgwNDI1VDAzMTMxMFoiXSxbImVxIiwiJHgtYW16LWFsZ29yaXRobSIsIkFXUzQtSE1BQy1TSEEyNTYiXSxbImVxIiwiJHgtYW16LWNyZWRlbnRpYWwiLCJhY2Nlc3Nfa2V5LzIwMTgwNDI1L3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QiXV19 \
-F x-amz-algorithm=AWS4-HMAC-SHA256 \
-F key=README.md \
# <FILE> 为需要上传的Object
-F file=@<FILE>

• list

View all RULs created for download or upload

./mc share list [download/upload]

deploy

From cloud city

You can deploy Minio application from Cloud Market with one click (click to get it) .

From the Cloud Gang

You can also use the docker runcommand to deploy on the cloud by yourself:

• Enter the cloud help - create an application interface and select to create an application from a Docker image

• edit docker runcommand

  docker run -p 9000:9000 \
    -e MINIO_ACCESS_KEY=<Custom Access Key> \
    -e MINIO_SECRET_KEY=<Custom Secret Key> \
    -v /mnt/data:/data \
    -v /mnt/config:/root/.minio \
    minio/minio:RELEASE.2018-04-19T22-54-58Z \
    server /data

When accessing Minio object storage, the Access Key and Secret Key required for verification can be customized and generated according to the incoming variables MINIO_ACCESS_KEYand variables .MINIO_SECRET_KEY

• MINIO_ACCESS_KEYWe have initialized the variables and variables of the Minio application in the cloud market MINIO_SECRET_KEY. You can obtain the corresponding values ​​of the variables in the application console - application dependency information to log in to the Minio application.

Distributed Minio

Using distributed Minio allows you to combine multiple drives (even on different computers) into a single object storage server. Since drivers can be distributed on multiple nodes, distributed Minio can withstand multiple points of failure and ensure data security.

The current Minio application version of Yunshi does not support distributed Minio for the time being, and distributed Minio will be updated in the future.

Official website: https://www.minio.io

Documentation address: https://docs.minio.io/

Other Minio support reference: https://www.minio.io/dcos.html

Learn more about Rainbond v3.5.1

Rainbond is an application-centric open source PaaS that deeply integrates Kubernetes-based container management, Service Mesh microservice architecture best practices, multi-type CI/CD application construction and delivery, and multi-data center resource management and other technologies to provide users with cloud-native application full life cycle solutions, build an ecosystem of interconnection between applications and infrastructure, applications and applications, and infrastructure and infrastructure, to meet the needs of agile development and high efficiency to support rapid business development. Operations and lean management requirements.

• Website: https://www.rainbond.com
• Try Rainbond Public Cloud: https://www.goodrain.com
• Github: https://github.com/goodrain/rainbond
• Code Cloud: https://gitee.com/rainbond/Rainbond
• WeChat group: Add WeChat "qingguo-wei" and accept the invitation to join the group