k8s 1.28.2 集群部署 docker registry 接入 MinIO 存储

docker registry dockerfile

docker registry 配置文件

S3 storage driver

registry:2.8.3 Image hierarchy

docker registry 部署

生成 htpasswd 文件

<username> <password> 改成自己想配置的，如果密码有特殊字符，要用单引号包起来

docker run --rm \docker.m.daocloud.io/httpd:latest \htpasswd -Bbn <username> <password> > htpasswd

生成 secret 文件

kubectl create secret generic docker-registry-auth \-n registry \--from-file=htpasswd

生成 registry 配置文件

因为涉及到 MinIO 的 accesskey 和 secretkey，这里采用 secret 的方式来生成配置文件

---
apiVersion: v1
kind: Secret
metadata:name: docker-registry-cmnamespace: registry
stringData:config.yml: |-version: 0.1log:level: infofields:service: registrystorage:delete:enabled: truecache:blobdescriptor: inmemorys3:accesskey: wJpkHB8rznvZBRLfKmBzsecretkey: ZHIyklv5tktYvGR0iFqBiL9NKh7JKbhyDR9SNAYpregion: defaultregionendpoint: http://minio.api.devops.icuforcepathstyle: trueaccelerate: falsebucket: docker-registryencrypt: falsesecure: falsev4auth: truechunksize: 5242880multipartcopymaxconcurrency: 10http:addr: :5000debug:addr: :5001prometheus:enabled: truepath: /metricsheaders:X-Content-Type-Options: [nosniff]health:storagedriver:enabled: trueinterval: 10sthreshold: 3auth:htpasswd:realm: basic-realmpath: /auth/htpasswd
type: Opaque

创建 service

---
apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/name: docker-registryname: docker-registry-svcnamespace: registry
spec:ports:- name: httpport: 5000targetPort: http- name: http-metricsport: 5001targetPort: http-metricsselector:app.kubernetes.io/name: docker-registrytype: ClusterIP

创建 statefulset

---
apiVersion: apps/v1
kind: StatefulSet
metadata:labels:app.kubernetes.io/name: docker-registryname: docker-registrynamespace: registry
spec:replicas: 1selector:matchLabels:app.kubernetes.io/name: docker-registryserviceName: docker-registry-svctemplate:metadata:labels:app.kubernetes.io/name: docker-registryspec:affinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- podAffinityTerm:labelSelector:matchLabels:app.kubernetes.io/name: docker-registrytopologyKey: kubernetes.io/hostnameweight: 1containers:- image: docker.m.daocloud.io/registry:2.8.3livenessProbe:failureThreshold: 60initialDelaySeconds: 5periodSeconds: 10successThreshold: 1tcpSocket:port: httptimeoutSeconds: 1name: docker-registryports:- containerPort: 5000name: http- containerPort: 5001name: http-metricsreadinessProbe:failureThreshold: 60initialDelaySeconds: 5periodSeconds: 10successThreshold: 1tcpSocket:port: httptimeoutSeconds: 1resources:limits:cpu: 2000mmemory: 2.5Girequests:cpu: 100mmemory: 100MistartupProbe:failureThreshold: 60initialDelaySeconds: 5periodSeconds: 10successThreshold: 1tcpSocket:port: httptimeoutSeconds: 1volumeMounts:- mountPath: /etc/docker/registryname: config- mountPath: /authname: authterminationGracePeriodSeconds: 30volumes:- name: configsecret:secretName: docker-registry-cm- name: authsecret:secretName: docker-registry-auth

创建 ingress

没有 ingress 可以开 nodeport 来实现

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/proxy-body-size: 5Gname: docker-registrynamespace: registry
spec:ingressClassName: nginxrules:- host: registry.devops.icuhttp:paths:- backend:service:name: docker-registry-svcport:number: 5000path: /pathType: Prefix

验证 docker registry

/etc/docker/daemon.json 增加 registry 地址

"insecure-registries": ["ip:端口"]
# 或者
"insecure-registries": ["域名"]

登录 docker registry

docker login http://registry.devops.icu

修改 tag

docker tag docker.m.daocloud.io/registry:2.8.3 registry.devops.icu/registry:2.8.3

上传镜像

docker push registry.devops.icu/registry:2.8.3

docker registry 监控

grafana id：9621

prometheus 配置文件参考

    - job_name: docker-registrykubernetes_sd_configs:- role: endpointsrelabel_configs:- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name]regex: registry;docker-registry-svcaction: keep- source_labels: [__meta_kubernetes_pod_ip]regex: (.+)target_label: __address__replacement: ${1}:5001- source_labels: [__meta_kubernetes_endpoints_name]action: replacetarget_label: endpoint- source_labels: [__meta_kubernetes_pod_name]action: replacetarget_label: pod- source_labels: [__meta_kubernetes_service_name]action: replacetarget_label: service- source_labels: [__meta_kubernetes_namespace]action: replacetarget_label: namespace

docker registry ui

Github 项目地址：Joxit/docker-registry-ui-2.5.7

相关的变量和参数详见：available-options

---
apiVersion: v1
kind: Service
metadata:labels:app.kubernetes.io/name: docker-registry-uiname: docker-registry-ui-svcnamespace: registry
spec:ports:- name: httpport: 8080protocol: TCPtargetPort: 8080selector:app.kubernetes.io/name: docker-registry-uitype: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app.kubernetes.io/name: docker-registry-uiname: docker-registry-uinamespace: registry
spec:replicas: 1selector:matchLabels:app.kubernetes.io/name: docker-registry-uitemplate:metadata:labels:app.kubernetes.io/name: docker-registry-uispec:containers:- env:- name: SINGLE_REGISTRYvalue: "true"- name: SHOW_CATALOG_NB_TAGSvalue: "true"- name: REGISTRY_SECUREDvalue: "true"- name: NGINX_PROXY_PASS_URLvalue: http://docker-registry-svc.registry.svc.cluster.local:5000- name: NGINX_PROXY_HEADER_Authorizationvalue: $http_authorizationimage: joxit/docker-registry-ui:2.5.7imagePullPolicy: IfNotPresentname: docker-registry-uisecurityContext:fsGroup: 101runAsUser: 101
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: docker-registry-uinamespace: registry
spec:ingressClassName: nginxrules:- host: registry.ui.devops.icuhttp:paths:- backend:service:name: docker-registry-ui-svcport:number: 8080path: /pathType: Prefix