table of Contents
1. Overview of Distributed Minio
1.What are the benefits of distributed Minio?
2. Synchronize the cluster time
2. Minio distributed cluster construction
2. Modify the host name and hosts
3. Modify the maximum number of files in the system
1. Overview of Distributed Minio
Distributed Minio allows you to combine multiple hard drives (even on different machines) into an object storage service. Since hard drives are distributed on different nodes, distributed Minio avoids a single point of failure.
1.What are the benefits of distributed Minio?
In the field of big data, the usual design concepts are centerless and distributed. Minio distributed mode can help you build a highly available object storage service. You can use these storage devices regardless of their actual physical location.
Data protection
Distributed Minio uses erasure code (erasure code) to prevent multiple node downtime and bit rot.
Distributed Minio requires at least 4 nodes. Using distributed Minio automatically introduces erasure coding.
High availability
There is a single point of failure for a stand-alone Minio service. On the contrary, if it is an N-node distributed Minio, as long as there are N/2 nodes online, your data is safe. But you need at least N/2+1 Quorum nodes to create new objects.
For example, an 8-node Minio cluster, each node has a disk, even if 4 nodes are down, the cluster is still readable, but you need 5 nodes to write data.
limit
Distributed Minio single tenants have a minimum of 4 disks and a maximum of 16 disks (limited by erasure codes). This restriction ensures the simplicity of Minio while still having scalability. If you need to build a multi-tenant environment, you can easily use the orchestration tool (Kubernetes) to manage multiple Minio instances.
Note that you can combine different nodes and several disks per node as long as you comply with the restrictions of distributed Minio. For example, you can use 2 nodes with 4 disks per node, or you can use 4 nodes with two disks per node, and so on.
consistency
In Minio distributed and stand-alone mode, all read and write operations strictly follow the read-after-write consistency model.
2. Synchronize the cluster time
The time of all machines in the cluster cannot exceed 3 seconds. First, you need to install the ntp time server, which is very simple
https://blog.csdn.net/ct_666/article/details/112598442
2. Minio distributed cluster construction
Check the disk space and select the appropriate installation directory
df -hAt least 4 nodes in the production environment
| node | IP | Data directory |
| minio1 | 192.168.44.10 | / home / minio / data |
| minio2 | 192.168.44.11 | / home / minio / data |
| minio3 | 192.168.44.12 | / home / minio / data |
| minio4 | 192.168.44.13 | / home / minio / data |
1. Get Minio
https://dl.min.io/server/minio/release/linux-amd64/minio
2. Modify the host name and hosts
# 在对应的主机执行对应的语句
hostnamectl set-hostname minio1
hostnamectl set-hostname minio2
hostnamectl set-hostname minio3
hostnamectl set-hostname minio4# 在所有主机执行
cat >> /etc/hosts <<EOF
192.168.44.10 minio1
192.168.44.11 minio2
192.168.44.12 minio3
192.168.44.13 minio4
EOF3. Modify the maximum number of files in the system
# 所有主机执行
echo "* soft nofile 65535" >> /etc/security/limits.conf
echo "* hard nofile 65535" >> /etc/security/limits.conf4. Catalog creation
Startup script and binary file directory run
data storage directory data
configuration file directory /etc/minio
# 所有主机执行
mkdir -p /home/minio/{run,data} && mkdir -p /etc/minio5. Cluster startup file
vim /home/minio/run/run.shMINIO_ACCESS_KEY: User name, the minimum length is 5 characters
MINIO_SECRET_KEY: Password, the password cannot be set too simple, otherwise minio will fail to start, the minimum length is 8 characters
–config-dir: Specify the cluster configuration file directory
# 所有主机添加
#!/bin/bash
export MINIO_ACCESS_KEY=minio
export MINIO_SECRET_KEY=admin#2020
/home/minio/run/minio server --config-dir /etc/minio \
http://192.168.44.10/home/minio/data \
http://192.168.44.11/home/minio/data \
http://192.168.44.12/home/minio/data \
http://192.168.44.13/home/minio/data \6 、 minio.service
WorkingDirectory: Binary file directory
ExecStart: Specify cluster startup script
# 所有主机执行
cat > /usr/lib/systemd/system/minio.service <<EOF
[Unit]
Description=Minio service
Documentation=https://docs.minio.io/
[Service]
WorkingDirectory=/home/minio/run/
ExecStart=/home/minio/run/run.sh
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF7. Upload binary files
Upload the downloaded minio binary file to the /home/minio/run directory
8. Permission modification
Add permissions to all involved files or directories
service file
binary file
cluster startup script
# 所有主机执行
chmod +x /usr/lib/systemd/system/minio.service && chmod +x /home/minio/run/minio && chmod +x /home/minio/run/run.sh9. Start the cluster
# 所有主机执行
systemctl daemon-reload
systemctl start minio && systemctl enable minio
systemctl disable firewalld && systemctl stop firewalld
# 当4台服务器都部署完毕后，查看状态如下：
[root@minio1 conf]# systemctl status minio
● minio.service - Minio service
Loaded: loaded (/usr/lib/systemd/system/minio.service; enabled; vendor preset: disabled)
Active: active (running) since 三 2021-01-13 16:07:08 CST; 9h ago
Docs: https://docs.minio.io/
Main PID: 6196 (run.sh)
CGroup: /system.slice/minio.service
├─6196 /bin/bash /home/minio/run/run.sh
└─6221 /home/minio/run/minio server --config-dir /etc/minio http://192.168.44.10/home/minio/data http://192.168.44.11/home/minio/data http://192.168.44.12/home/minio/data http://192.168.44.13/hom...
1月 13 18:19:02 minio1 run.sh[6196]: Endpoint: http://192.168.44.10:9000 http://127.0.0.1:9000
1月 13 18:19:02 minio1 run.sh[6196]: Browser Access:
1月 13 18:19:02 minio1 run.sh[6196]: http://192.168.44.10:9000 http://127.0.0.1:9000
1月 13 18:19:02 minio1 run.sh[6196]: Object API (Amazon S3 compatible):
1月 13 18:19:02 minio1 run.sh[6196]: Go: https://docs.min.io/docs/golang-client-quickstart-guide
1月 13 18:19:02 minio1 run.sh[6196]: Java: https://docs.min.io/docs/java-client-quickstart-guide
1月 13 18:19:02 minio1 run.sh[6196]: Python: https://docs.min.io/docs/python-client-quickstart-guide
1月 13 18:19:02 minio1 run.sh[6196]: JavaScript: https://docs.min.io/docs/javascript-client-quickstart-guide
1月 13 18:19:02 minio1 run.sh[6196]: .NET: https://docs.min.io/docs/dotnet-client-quickstart-guide
1月 13 18:19:02 minio1 run.sh[6196]: Waiting for all MinIO IAM sub-system to be initialized.. lock acquired10. Agent cluster
The production environment needs to use Nginx to proxy the cluster address and unified entrance to the outside world. Because nginx is deployed on 192.168.44.10, the port 9000 can no longer be used. The used port 9001 adopts the load balancing polling mechanism by default, and forwards 4 addresses evenly.
# 配置nginx将集群代理端口映射到9001
upstream minio{
server 192.168.44.10:9000;
server 192.168.44.11:9000;
server 192.168.44.12:9000;
server 192.168.44.13:9000;
}
server {
listen 9001;
server_name minio;
location / {
proxy_pass http://minio;
proxy_set_header Host $http_host;
client_max_body_size 1000m;
}
}11. Test
The browser accesses the minio cluster proxy address + 9001 port, the user name and password are the ones we set in the start file run.sh above
<IP_ADDRESS>:9001
This deployment method can have fault tolerance for one machine.
When downtime=1, readable and writable, when downtime=2, readable and not writeable, when downtime>2, the cluster is unavailable