基于Docker的Minio分布式集群实践

目录

1. 说明

2. 配置表

3. 步骤

3.1 放行服务端口

3.2 docker-compose 编排

4. 入口反向代理与负载均衡配置

4.1 api入口

4.2 管理入口

5. 用例

6. 参考

---

1. 说明

以多节点的Docker容器方式实现minio存储集群，并配以nginx反向代理及负载均衡作为访问入口。

2. 配置表

服务器 (2 nodes)	ip	备注
center01.dev.sb	172.16.20.20	Minio 入口（http://minio01.dev.sb） 管理入口 （http://minio01-console.dev.sb） 硬件配置：8核16G2T 软件配置：ubuntu22.04 + 宝塔面板 + nginx
node1	172.16.20.10	...

3. 步骤

3.1 放行服务端口

- 9000，9001

3.2 docker-compose 编排

services:minio:image: minio/minio:latestcontainer_name: minio-node1hostname: minio{x}expose:- "9000"- "9001"environment:- MINIO_ROOT_USER=admin- MINIO_ROOT_PASSWORD=you knowvolumes:- /data0/Server/Db/minio/data0:/data0- /data0/Server/Db/minio/data1:/data1command: server --console-address ':9001' --address ':9000' http://minio{0...1}/data{0...1}privileged: truehealthcheck:test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]interval: 30stimeout: 20sretries: 3extra_hosts:- minio0:172.16.20.10- minio1:172.16.20.20restart: alwaysnetwork_mode: host

4. 入口反向代理与负载均衡配置

4.1 api入口

server {listen 80;listen  [::]:80;server_name minio01.dev.sb;# Allow special characters in headersignore_invalid_headers off;# Allow any size file to be uploaded.# Set to a value such as 1000m; to restrict file size to a specific valueclient_max_body_size 0;# Disable bufferingproxy_buffering off;proxy_request_buffering off;location / {proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto $scheme;proxy_set_header Host $http_host;proxy_cache_convert_head off;proxy_cache_key "$scheme$proxy_host$uri$is_args$args|$request_body";proxy_read_timeout 300s;proxy_connect_timeout 75s;proxy_http_version 1.1;proxy_set_header Connection "";chunked_transfer_encoding on;proxy_redirect off;proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance}
}upstream minio_s3 {server 172.16.20.10:9000 weight=5;server 172.16.20.20:9000 weight=1;
}

4.2 管理入口

server {listen 80;listen  [::]:80;server_name minio01-console.dev.sb;# Allow special characters in headersignore_invalid_headers off;# Allow any size file to be uploaded.# Set to a value such as 1000m; to restrict file size to a specific valueclient_max_body_size 0;# Disable bufferingproxy_buffering off;proxy_request_buffering off;location / {rewrite ^/minio/ui/(.*) /$1 break;proxy_set_header Host $http_host;proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto $scheme;proxy_set_header X-NginX-Proxy true;# This is necessary to pass the correct IP to be hashedreal_ip_header X-Real-IP;proxy_connect_timeout 300;# To support websockets in MinIO versions released after January 2023proxy_http_version 1.1;proxy_set_header Upgrade $http_upgrade;proxy_set_header Connection "upgrade";# Some environments may encounter CORS errors (Kubernetes + Nginx Ingress)# Uncomment the following line to set the Origin request to an empty string# proxy_set_header Origin '';chunked_transfer_encoding off;proxy_pass http://minio_console; # This uses the upstream directive definition to load balance}
}upstream minio_console {server 172.16.20.10:9001;server 172.16.20.20:9001;
}

5. 用例

from minio import Minio
from minio.error import S3Errordef main():# 创建 MinIO 客户端实例client = Minio(# "172.16.20.10:9000",  # MinIO 服务器地址和端口"minio01.dev.sb",  # MinIO 服务器地址和端口access_key="my key",  # 替换为你的 MinIO 访问密钥secret_key="you know it",  # 替换为你的 MinIO 秘密密钥secure=False,  # 使用 False 如果是 HTTP 而非 HTTPS)# 创建一个名为 "mybucket" 的存储桶bucket_name = "mybucket"if not client.bucket_exists(bucket_name):client.make_bucket(bucket_name)else:print(f"Bucket '{bucket_name}' already exists")# 上传文件file_path = "D:/Temp/demo/videos/acfa586c0b3248ec95df81267a767258.mp4"object_name = "acfa586c0b3248ec95df81267a767258.png"client.fput_object(bucket_name, object_name, file_path)print(f"'{file_path}' is successfully uploaded as object '{object_name}' to bucket '{bucket_name}'.")# 下载文件client.fget_object(bucket_name, object_name, "D:/Temp/tmp10/a2.mp4")print(f"'{object_name}' is successfully downloaded.")# 列出存储桶中的对象objects = client.list_objects(bucket_name)for obj in objects:print(obj.object_name)# 删除文件# client.remove_object(bucket_name, object_name)# print(f"'{object_name}' is successfully deleted.")if __name__ == "__main__":try:main()except S3Error as exc:print("error occurred.", exc)

 如图：

6. 参考

- Stat: Access Denied for Minio S3 Bucket behind Nginx Reverse Proxy with https enabled · Issue #4860 · restic/restic · GitHub

- Deploy MinIO: Multi-Node Multi-Drive — MinIO Object Storage for Linux