Minio-based object storage usage guide

Official documentation

what is

• MinIO is the world's leading object storage pioneer, with millions of users around the world. On standard hardware, read/write speeds are up to 183 GB/sec and 171 GB/sec.
• Object storage can serve as the main storage layer to handle various complex workloads such as Spark, Presto, TensorFlow, H2O.ai and become a replacement for Hadoop HDFS.
• MinIO is used as the primary storage for cloud-native applications that require higher throughput and lower latency than traditional object storage. These are the performance indicators that MinIO can achieve.
• Simple and scalable
• Compatible with AWS S3 to achieve seamless connection

Install

• For other installation methods, please refer to the official documentation.
• docker install minio

docker run -p 9000:9000 --name minio1 \
  -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
  -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
  -v /mnt/data:/data \
  -v /mnt/config:/root/.minio \
  minio/minio server /data

Object storage principles

• MinIO's encoding method encodes an object into several data blocks and check blocks. We call it Erasure Code for short. This is the type of encoding. This type of encoding also requires algorithms to implement. Minio uses Reed- Solomon's algorithm
• Even if you lose half (N/2) of your hard drives, you can still recover your data.
• Minio uses Reed-Solomon code by default to split the data into N/2 data blocks and N/2 parity blocks.
• With 16 disks, an object will be divided into 8 data blocks and 8 parity blocks. You can lose any 8 disks (regardless of whether they store data blocks or parity blocks), and you can still get data from the remaining disks. Recover data from
  
• explain
  • B is a matrix of nx (m + n)
  • D is a real data block that we can prove
  • Figure 1
    • When storing, store it as n+m, that is, the amount of redundancy is m
  • Figure II
    • At this time, the data D1, D4, and C2 are lost.
  • Figure 3
    • According to matrix operations, it can be seen that the original data matrix D can be obtained by the invertible matrix of Survivors matrix *B`
• above all
  • The maximum tolerance for data loss is m
  • The amount of data redundancy is m
  • In general, m = n, so it is 2 times redundant.

Bucket notification guide

• If changes occur in the bucket, such as uploading objects or deleting objects, you can use the bucket event notification mechanism to monitor and publish them in the following ways
• AMQP，MQTT，Elasticsearch，Redis，NATS，PostgreSQL，MySQL，Apache Kafka，Webhooks
• Configure an es notification case
  • The default path of the MinIO Server configuration file is ~/.minio/config.json
  • The ES configuration information is under the elasticsearch node under the notify node.

  "elasticsearch": {
        
        
      "1": {
        
        
          "enable": true,
          "format": "namespace",
          "url": "http://127.0.0.1:9200",
          "index": "minio_events"
      }
  },
  

  • Restart the server
  • Add notification event

  mc mb myminio/images
  mc event add  myminio/images arn:minio:sqs:us-east-1:1:elasticsearch --suffix .jpg
  

  • verify

  $ mc cp myphoto.jpg myminio/images # 上传一张图片到存储桶
  $ curl  "http://localhost:9200/minio_events/_search?pretty=true" # 访问es以查看事件通知
  
  

Client installation

• Please refer to the detailed documentation
• Install on mac

brew install minio/stable/mc
mc --help

sdk guide

• Please refer to the detailed documentation

# 引入MinIO包。
from minio import Minio
from minio.error import (ResponseError, BucketAlreadyOwnedByYou,
                         BucketAlreadyExists)

# 使用endpoint、access key和secret key来初始化minioClient对象。
minioClient = Minio('play.min.io',
                    access_key='Q3AM3UQ867SPQQA43P2F',
                    secret_key='zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG',
                    secure=True)

# 调用make_bucket来创建一个存储桶。
try:
       minioClient.make_bucket("maylogs", location="us-east-1")
except BucketAlreadyOwnedByYou as err:
       pass
except BucketAlreadyExists as err:
       pass
except ResponseError as err:
       raise
else:
        try:
               minioClient.fput_object('maylogs', 'pumaserver_debug.log', '/tmp/pumaserver_debug.log')
        except ResponseError as err:
               print(err)