K8S下redis哨兵集群使用secret隐藏configmap内明文密码方案详解

#作者：朱雷

一、背景环境及方案说明

Redis 哨兵configmap里包含明文密码，需要处理不在configmap内显示明文密码。

1.1、环境说明

方案基于Redis-5.0.14 哨兵版本（5.x，6.x版本兼容）
方案基于redis-sentinel-exporter-5.0.8版本
方案基于容器环境变量
使用两种方案的其中任意一种均可实现

1.2、方案一：使用配置文件设置密码

参照 2.2.1 和2.2.2 修改configmap
参照 2.3.1 和2.3.2 修改deployment

1.3、方案二：使用args 的命令行传参设置密码

参照 2.2.3.1 和2.2.3.2 修改configmap
参照 2.3.3.1 和2.3.3.2 修改deployment

二、redis secret configmap deployment参考

2.1 创建secret-redis.yaml参考

${}内为redis密码的base64的编码，若认证和同步密码不一致分别定义即可
apiVersion: v1
data:password: ${aGFyYm9yMjM0NSM=}
kind: Secret
metadata:name: redis-auth-secretnamespace: paas-middleware

2.2 修改configmap配置参考

2.2.1 哨兵节点修改（每个节点都修改）

apiVersion: v1
data:redis-docker-entrypoint.sh: |#!/bin/bashif [ ! -f "/redis-conf/redis.conf" ]; thencp /etc/redis/redis.conf /redis-conf/redis.conf**echo -e "sentinel auth-pass mymaster ${REDIS_PASSWORD}" >> /redis-conf/redis.conffi**redis-sentinel /redis-conf/redis.conf $@redis.conf: |port 26379protected-mode nodaemonize nosentinel monitor mymaster 169.169.164.253 6379 2sentinel down-after-milliseconds mymaster 15000sentinel failover-timeout mymaster 60000sentinel deny-scripts-reconfig yessentinel parallel-syncs mymaster 2sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:labels:app: redis-base-1type: redisname: redis-base-1-sentinel-1namespace: paas-middleware每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

2.2.2 主从节点配置修改

apiVersion: v1
data:redis-docker-entrypoint.sh: |#!/bin/bashif [ ! -f "/redis-conf/redis.conf" ]; thencp /etc/redis/redis.conf /redis-conf/redis.conf**echo -e "masterauth ${REDIS_MASTER_PASSWORD}" >> /redis-conf/redis.confecho -e "requirepass ${REDIS_PASSWORD}" >> /redis-conf/redis.conf**firedis-server /redis-conf/redis.conf $@redis.conf: |bind 0.0.0.0 ::port 6379daemonize noprotected-mode notimeout 300tcp-keepalive 300replica-read-only yesreplica-serve-stale-data yesmaxclients 20000maxmemory 0maxmemory-policy noevictionmasterauth somepassword  # 删除此行配置requirepass somepassword  # 删除此行配置rename-command FLUSHALL ""dir "/data/"pidfile "/data/redis.pid"logfile "/data/redis.log"
kind: ConfigMap
metadata:labels:app: redis-base-1type: redisname: redis-base-1-masternamespace: paas-middleware所有主从configmap配置文件都修改， 有****不带删除线的为新增行

2.2.3 使用命令行参数指定密码（本小节与上面两小节互斥）

1. 以下为哨兵节点configmap 修改

apiVersion: v1
data:redis-docker-entrypoint.sh: |#!/bin/bashif [ ! -f "/redis-conf/redis.conf" ]; thencp /etc/redis/redis.conf /redis-conf/redis.conffiredis-sentinel /redis-conf/redis.conf $@redis.conf: |port 26379protected-mode nodaemonize nosentinel monitor mymaster 169.169.164.253 6379 2sentinel down-after-milliseconds mymaster 15000sentinel failover-timeout mymaster 60000sentinel deny-scripts-reconfig yessentinel parallel-syncs mymaster 2sentinel auth-pass mymaster somepassword   # 删除这行配置
kind: ConfigMap
metadata:labels:app: redis-base-1type: redisname: redis-base-1-sentinel-1namespace: paas-middleware每个哨兵的configmap 都修改下， 有****不带删除线的为新增行

2. 以下为主从节点configmap 修改

下面为主从节点实例configmap修改，有****不带删除线为新增行
apiVersion: v1
data:
redis-docker-entrypoint.sh: |#!/bin/bashif [ ! -f "/redis-conf/redis.conf" ]; thencp /etc/redis/redis.conf /redis-conf/redis.conffiredis-server /redis-conf/redis.conf $@
redis.conf: |bind 0.0.0.0 ::port 6379daemonize noprotected-mode notimeout 300tcp-keepalive 300replica-read-only yesreplica-serve-stale-data yesmaxclients 20000maxmemory 0maxmemory-policy noevictionmasterauth somepassword  # 删除此行配置requirepass somepassword  # 删除此行配置rename-command FLUSHALL ""dir "/data/"pidfile "/data/redis.pid"logfile "/data/redis.log"
kind: ConfigMap
metadata:
labels:app: redis-base-1type: redis
name: redis-base-1-master
namespace: paas-middleware

2.3 修改deployment配置参考

2.3.1 master&slaves节点的deployment的yaml修改

所有主从节点配置文件都修改，  有**xxx**为新增行
apiVersion: apps/v1
kind: Deployment
metadata:annotations:
deployment.kubernetes.io/revision: "1"labels:app: redis-base-1type: redisname: redis-base-1-masternamespace: paas-middleware
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: redis-base-1name: redis-base-1-masterservicename: redis-base-1type: rediswithexporter: "yes"strategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:annotations:prometheus.io/port: "9121"prometheus.io/scrape: "true"labels:app: redis-base-1name: redis-base-1-masterservicename: redis-base-1type: rediswithexporter: "yes"spec:containers:- args:- --replica-announce-ip- 169.169.164.253- --replica-announce-port- "6379"command:- /etc/redis/redis-docker-entrypoint.shimage: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latestimagePullPolicy: Alwaysname: redis**env:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password- name: REDIS_MASTER_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password**ports:- containerPort: 6379name: clientprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/redis/name: config- mountPath: /dataname: data- mountPath: /redis-confname: actual-config- args:- --redis.addr- redis://localhost:6379- --redis.password- somepassword  #密码替换成 $(REDIS_PASSWORD) 变量- **$(REDIS_PASSWORD)**- --web.listen-address- 0.0.0.0:9121image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latestimagePullPolicy: Alwaysname: redis-exporter**env:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password**ports:- containerPort: 9121name: redis-exporterprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilednsPolicy: ClusterFirstnodeSelector:kubernetes.io/hostname: 10.179.75.111restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30volumes:- configMap:defaultMode: 509name: redis-base-1-mastername: config- hostPath:path: /data/redis/redis-base-1-master/datatype: ""name: data- hostPath:path: /data/redis/redis-base-1-master/redis-conftype: ""name: actual-config

2.3.2 哨兵节点的deployment的yaml修改

所有哨兵节点配置文件都修改， 有****为新增行
apiVersion: apps/v1
kind: Deployment
metadata:annotations:deployment.kubernetes.io/revision: "1"labels:app: redis-base-1type: redisname: redis-base-1-sentinel-1namespace: paas-middleware
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: redis-base-1name: redis-base-1-sentinel-1role: sentineltype: rediswithexporter: "no"strategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app: redis-base-1name: redis-base-1-sentinel-1role: sentineltype: rediswithexporter: "no"spec:containers:- args:- --sentinel- announce-ip- 169.169.196.242- --replica-announce-port- "26379"command:- /etc/redis/redis-docker-entrypoint.shimage: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latestimagePullPolicy: Alwaysname: redis**env:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password- name: REDIS_MASTER_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password**ports:- containerPort: 26379name: clientprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/redis/name: config- mountPath: /dataname: data- mountPath: /redis-confname: actual-configdnsPolicy: ClusterFirstnodeSelector:kubernetes.io/hostname: 10.179.75.111restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30volumes:- configMap:defaultMode: 509name: redis-base-1-sentinel-1name: config- hostPath:path: /data/redis/redis-base-1-sentinel-1/datatype: ""name: data- hostPath:path: /data/redis/redis-base-1-sentinel-1/redis-conftype: ""name: actual-config
status:availableReplicas: 1conditions:- lastTransitionTime: "2023-11-09T03:25:41Z"lastUpdateTime: "2023-11-09T03:25:43Z"message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.reason: NewReplicaSetAvailablestatus: "True"type: Progressing- lastTransitionTime: "2024-07-21T16:48:34Z"lastUpdateTime: "2024-07-21T16:48:34Z"message: Deployment has minimum availability.reason: MinimumReplicasAvailablestatus: "True"type: AvailableobservedGeneration: 3readyReplicas: 1replicas: 1updatedReplicas: 1

2.3.3 使用命令行参数指定密码（本小节与上面两小节互斥）

1. 哨兵节点deployment 修改
   所有哨兵节点配置文件都修改， 有****不带删除线的为新增行

apiVersion: apps/v1
kind: Deployment
metadata:annotations:deployment.kubernetes.io/revision: "1"labels:app: redis-base-1type: redisname: redis-base-1-sentinel-1namespace: paas-middleware
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: redis-base-1name: redis-base-1-sentinel-1role: sentineltype: rediswithexporter: "no"strategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:creationTimestamp: nulllabels:app: redis-base-1name: redis-base-1-sentinel-1role: sentineltype: rediswithexporter: "no"spec:containers:- args:- --sentinel- announce-ip- 169.169.196.242- --replica-announce-port- "26379"- --sentinel- auth-pass- mymaster- $(REDIS_PASSWORD)command:- /etc/redis/redis-docker-entrypoint.shimage: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latestimagePullPolicy: Alwaysname: redisenv:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password- name: REDIS_MASTER_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: passwordports:- containerPort: 26379name: clientprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/redis/name: config- mountPath: /dataname: data- mountPath: /redis-confname: actual-configdnsPolicy: ClusterFirstnodeSelector:kubernetes.io/hostname: 10.179.75.111restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30volumes:- configMap:defaultMode: 509name: redis-base-1-sentinel-1name: config- hostPath:path: /data/redis/redis-base-1-sentinel-1/datatype: ""name: data- hostPath:path: /data/redis/redis-base-1-sentinel-1/redis-conftype: ""name: actual-config
status:availableReplicas: 1conditions:- lastTransitionTime: "2023-11-09T03:25:41Z"lastUpdateTime: "2023-11-09T03:25:43Z"message: ReplicaSet "redis-base-1-sentinel-1-668c76f9bc" has successfully progressed.reason: NewReplicaSetAvailablestatus: "True"type: Progressing- lastTransitionTime: "2024-07-21T16:48:34Z"lastUpdateTime: "2024-07-21T16:48:34Z"message: Deployment has minimum availability.reason: MinimumReplicasAvailablestatus: "True"type: AvailableobservedGeneration: 3readyReplicas: 1replicas: 1updatedReplicas: 1

2、以下为主从实例deployment 配置修改
所有主从节点配置文件都修改， 有****不带删除线的为新增行

apiVersion: apps/v1
kind: Deployment
metadata:annotations:
deployment.kubernetes.io/revision: "1"labels:app: redis-base-1type: redisname: redis-base-1-masternamespace: paas-middleware
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: redis-base-1name: redis-base-1-masterservicename: redis-base-1type: rediswithexporter: "yes"strategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:annotations:prometheus.io/port: "9121"prometheus.io/scrape: "true"labels:app: redis-base-1name: redis-base-1-masterservicename: redis-base-1type: rediswithexporter: "yes"spec:containers:- args:- --replica-announce-ip- 169.169.164.253- --replica-announce-port- "6379"- --**requirepass- $(REDIS_PASSWORD)- -- masterauth- $(REDIS_MASTER_PASSWORD)**command:- /etc/redis/redis-docker-entrypoint.shimage: harbor.somedomain/paas_middleware/redis-sentinel-main-5.0.8:latestimagePullPolicy: Alwaysname: redis**env:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password- name: REDIS_MASTER_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password**ports:- containerPort: 6379name: clientprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /etc/redis/name: config- mountPath: /dataname: data- mountPath: /redis-confname: actual-config- args:- --redis.addr- redis://localhost:6379- --redis.password- somepassword  #密码替换成 $(REDIS_PASSWORD) 变量- **$(REDIS_PASSWORD)**- --web.listen-address- 0.0.0.0:9121image: harbor.somedomain/paas_middleware/redis-sentinel-exporter-5.0.8:latestimagePullPolicy: Alwaysname: redis-exporter**env:- name: REDIS_PASSWORDvalueFrom:secretKeyRef:name: redis-auth-secretkey: password**ports:- containerPort: 9121name: redis-exporterprotocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilednsPolicy: ClusterFirstnodeSelector:kubernetes.io/hostname: 10.179.75.111restartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30volumes:- configMap:defaultMode: 509name: redis-base-1-mastername: config- hostPath:path: /data/redis/redis-base-1-master/datatype: ""name: data- hostPath:path: /data/redis/redis-base-1-master/redis-conftype: ""name: actual-config

2.3.4 镜像环境变量参考

https://hub.docker.com/r/bitnami/redis#configuration
https://github.com/oliver006/redis_exporter#flags

三、测试是否生效

Master节点

Slave节点

哨兵节点

测试redis-sentinel-exporter 指标抓取

四、 注意事项

1. 所有节点configmap和deployment yaml 配置文件都按照上面修改别遗漏
2. 修改完先在测试环境验证没有问题，再连接到连接哨兵集群进行读写测试