Minio environment to build detailed

MinIO build

MinIO is based on Apache License v2.0 open source license object storage service. It is compatible with Amazon S3 cloud storage service interface is very suitable for a large-capacity storage unstructured data, such as pictures, videos, log files, and backup data container / virtual machine image and the like, and an object file can be any size, from a few 5T kb to the maximum range.

MinIO is a very lightweight service that makes it simple and combined with other applications, similar NodeJS, Redis or MySQL.

The official definition of Minio is a high performance object storage.

Build high performance data infrastructure
for machine learning, analytics and
application data workloads with MinIO 。

Minio's official website is https://min.io ,
Minio official document is https://docs.min.io/cn/ ,
GitHub repository address is: https://github.com/minio/minio , there are 18.8k Stars (2019.11.9)

Minio's benchmark description:

https://min.io/resources/docs/MinIO-vs-HDFS-MapReduce-performance-comparison.pdf
https://min.io/resources/docs/MinIO-throughput-benchmarks-on-NVMe-SSD.pdf
https://min.io/resources/docs/Performance-comparison-Starburst-Presto-SQL.pdf
https://min.io/resources/docs/MinIO-throughput-benchmarks-on-HDD.pdf
https://min.io/resources/docs/Performance-comparison-Apache-Spark.pdf

Quick Start MinIO a single node

MinIO divided into server and client, the server provides an object store, also offers a Web-based management page. The client is used to manage, view a server-side tool. Server binary file is the minioclient binary files are mc.

Use binary start

First to download the server binaries, and secondly to prepare for a storage disk or directory,

After the download is good given execute permission, you can use the following command to the Quick Start.

chmod +x minio
./minio server /erdong/data

After start by visiting http: 9000: // localhost to access your object store.

Use Docker start

MinIO a persistent volumes required to store configuration and application data. However, if just to test it, you can simply pass a directory (as in the example below /erdong/data) to start MinIO. This directory will be created in the container when the container file system startup, but all data will be lost when the container exit.

To create MinIO container has persistent storage, you need to be persistent local directory from the host operating system is mapped to the virtual configuration ~ / .minio and export / data directory. To do this, run the following command

docker run -p 9000:9000 --name minio \
  -e "MINIO_ACCESS_KEY=minioadmin" \
  -e "MINIO_SECRET_KEY=minioadminpassword" \
  -v /erdong/minio/data:/data \
  -v /erdong/minio/config:/root/.minio \
  minio/minio server /data

After starting, you can visit http: // localhost: 9000 to access your object store.

The above command set in the following parameters:

MINIO_ACCESS_KEY administrator access keys
MINIO_SECRET_KEY administrator of keys
/ erdong / Minio / the Data: / the Data mount persistent data directory
/erdong/minio/config:/root/.minio mount configuration files
Start Server server mode, in addition to this mode there is a Gateway mode.
/ Data designating data directory

If you need to specify the following parameters port

--address ":9000"

Use TLS secure access MinIO Service

Next we look at how to configure MinIO service on linux using TLS.

Use TLS premise is to download a good MinIO Server.

If you already have a private key and a public key certificate, you need to copy them to MinIO the $HOME/.minio/certsfolder name must be the private key private.key, the name of the public key certificate must be public.crt. MinIO supports Linux only PEM format keys and certificates support only keys and certificates in PEM format on Windows, it does not currently support PFX certificate.

If this certificate is issued by a certificate authority, public.crtit should be the server's certificate.

Linux can use the following tool to generate a certificate

Let‘s Encrypt
generate_cert.go
OpenSSL

Windows can use the following tool to generate a certificate

GnuTLS

Minio can also be configured to connect to other services, whether Minio node or as NATs, Redis these. If these services are not registered with a certificate in the known certificate authority, you can trust these services let Minio CA, how to do it, these certificates will be put under Minio configuration path (~ / .minio / certs / CAs / Linux or C : \ Users <Username> .minio \ certs \ CAs Windows).

How to use OpenSSL to generate a self-signed certificate

Use the following command to generate a private key, the private key is generated in the directory executing commands

openssl genrsa -out private.key 2048

Generate a self-signed certificate, the certificate will be generated in the directory executing commands

openssl req -new -x509 -days 3650 -key private.key -out public.crt -subj "/C=US/ST=state/L=location/O=organization/CN=domain"

Where the content can be modified according to the actual situation, such as /Ca country, China is CN, /STis a state or province, /La municipal or district /CNis the domain name.

Good keys and certificate generation later, in accordance with the requirements placed on top of the corresponding directory.

Q&A

Thanos when connecting MinIO encountered some problems, there are many error log, listed below for reference.

1. Tip colon too

Log given as follows:

level=info ts=2019-11-08T02:43:40.981858805Z caller=main.go:170 msg="Tracing will be disabled"
level=info ts=2019-11-08T02:43:40.982326667Z caller=factory.go:39 msg="loading bucket configuration"
level=error ts=2019-11-08T02:43:40.982682262Z caller=main.go:200 err="store command failed: create bucket client: create S3 client: initialize s3 client: address http://127.0.0.1:9000: too many colons in address"

Log describe literal translation is too much colon, because in the configuration file, fill the endpoint of the S3 protocol MinIO provided when more than filled http://, leading to prompt the error. Intended use of the intranet, HTTP protocol can, and you do not need to turn on HTTPS, but Thanos default when connecting S3 storage is to use HTTPS can not be used in this way by HTTP.

Wrong configuration is as follows:

type: S3
config:
  bucket: "disk1"
  endpoint: "http://127.0.0.1:9000"

The correct configuration is as follows:

type: S3
config:
  bucket: "disk1"
  endpoint: "127.0.0.1:9000"

2. Tip the other party is not using HTTPS

Logs being given as follows

level=info ts=2019-11-08T02:42:04.366000876Z caller=main.go:170 msg="Tracing will be disabled"
level=info ts=2019-11-08T02:42:04.366417674Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-11-08T02:42:04.384413714Z caller=cache.go:172 msg="created index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=math.MaxInt64
level=error ts=2019-11-08T02:42:04.385632149Z caller=main.go:200 err="store command failed: bucket store initial sync: sync block: iter: Get https://127.0.0.1:9000/prometheus-store/?delimiter=%2F&max-keys=1000&prefix=: http: server gave HTTP response to HTTPS client"

This is object storage Thanos connection provides only HTTP, did not provide HTTPS, this time need to make object storage Add the certificate to enable HTTPS.

3. can not authenticate a domain name

Logs given as follows:

level=info ts=2019-11-08T03:06:57.90508837Z caller=main.go:170 msg="Tracing will be disabled"
level=info ts=2019-11-08T03:06:57.905445182Z caller=factory.go:39 msg="loading bucket configuration"
level=info ts=2019-11-08T03:06:57.923283984Z caller=cache.go:172 msg="created index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=math.MaxInt64
level=error ts=2019-11-08T03:06:57.927125234Z caller=main.go:200 err="store command failed: bucket store initial sync: sync block: iter: Get https://10.23.80.18:9000/prometheus-store/?delimiter=%2F&max-keys=1000&prefix=: x509: cannot validate certificate for minio-erdong.site because it doesn't contain any IP SANs"

Because Thanos HTTP configuration connection Minio's, there is an insecure_skip_verifyoption, which defaults to false, the need for the certificate domain name for authentication, the use of a self-signed certificate, did not do certification authority of the CA, so during the connection process will prompt insecurity, change the value to true, skip this validation on it.

  http_config:
    idle_conn_timeout: 90s
    response_header_timeout: 2m
    insecure_skip_verify: true