nacos在k8s上的集群安装实践

概述

   本文主要对 nacos 在k8s上的集群安装 进行说明与实践。主要版本信息，k8s: 1.27.x，nacos: 2.0.3。运行环境为 centos 7.x。

实践

在线安装参考: nacos官网安装指南
离线安装: 需要参考 在线安装，整理其中的镜像，下载后上传私服(注意镜像地址及nfs相关修改，其它默认就好)。
懒人快递: nfs-subdir-external-provisioner-4.0.2， mysql离线镜像安装包，版本为5.7.26 ，
nacos在k8s上集群化安装-nacos-2.0.3-k8s-1.27.x-yaml配置文件 , nacos-peer-finder-plugin-nacos集群安装所需要的镜像

nacos-server离线镜像安装包:链接：https://pan.baidu.com/s/1uQ07yzIkXyQcu2wUPCHDqQ?pwd=4utl
提取码：4utl
–来自百度网盘超级会员V2的分享

nfs

安装

   centos 7.x 上安装 nfs

repotrack -p nfs nfs-utils
repotrack -p nfs rpcbind
zip -r nfs.zip ./nfs
rpm -ivh --replacefiles --replacepkgs --force --nodeps *.rpm
# 安装nfs和rpc的软件包
yum install -y nfs-utils rpcbind
# 开启nfs、rpcbind服务
systemctl start nfs
systemctl start rpcbind
# 开机自启
systemctl enable nfs
systemctl enable rpcbind

使用

[root@hadoop02 ~]# mkdir -p /data/nfs
[root@hadoop02 data]# chmod  777 ./nfs
ls  /etc/exports
cat  /etc/exports
vi /etc/exports
/data/nfs  *(rw,sync,no_root_squash)
systemctl restart nfs
systemctl restart rpcbind
showmount -e 10.xx.xx.143

k8s持久化

   nacos集群安装，使用到了两种类型的持久化策略，一种直接使用卷挂载 nfs，这种在mysql中使用，另一种使用 pvc，应用场景在 nacos 上。

mysql-nfs.yaml

apiVersion: v1
kind: ReplicationController
metadata:namespace: defaultname: mysqllabels:name: mysql
spec:replicas: 1selector:name: mysqltemplate:metadata:labels:name: mysqlspec:containers:- name: mysqlimage: harbor.easzlab.io.local:8443/library/nacos/nacos-mysql:5.7ports:- containerPort: 3306volumeMounts:- name: mysql-datamountPath: /var/lib/mysqlenv:- name: MYSQL_ROOT_PASSWORDvalue: "root"- name: MYSQL_DATABASEvalue: "nacos_devtest"- name: MYSQL_USERvalue: "nacos"- name: MYSQL_PASSWORDvalue: "nacos"volumes:- name: mysql-datanfs:server: 10.xx.xx.143path: /data/nfs/nacos/mysql
---
apiVersion: v1
kind: Service
metadata:name: mysqllabels:name: mysql
spec:ports:- port: 3306targetPort: 3306selector:name: mysql

nacos-pvc-nfs.yaml
部分代码如下

apiVersion: v1
kind: ConfigMap
metadata:namespace: testname: nacos-cm
data:mysql.host: "mysql"mysql.db.name: "nacos_devtest"mysql.port: "3306"mysql.user: "nacos"mysql.password: "nacos"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:namespace: testname: nacos
spec:podManagementPolicy: ParallelserviceName: nacos-headlessreplicas: 3template:metadata:labels:app: nacosannotations:pod.alpha.kubernetes.io/initialized: "true"spec:affinity:podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:matchExpressions:- key: "app"operator: Invalues:- nacostopologyKey: "kubernetes.io/hostname"serviceAccountName: nfs-client-provisionerinitContainers:- name: peer-finder-plugin-installimage: harbor.easzlab.io.local:8443/library/nacos/nacos-peer-finder-plugin:1.1imagePullPolicy: AlwaysvolumeMounts:- mountPath: /home/nacos/plugins/peer-findername: datasubPath: peer-findercontainers:- name: nacosimagePullPolicy: Alwaysimage: harbor.easzlab.io.local:8443/library/nacos/nacos-server:latestresources:requests:memory: "2Gi"cpu: "500m"ports:- containerPort: 8848name: client-port- containerPort: 9848name: client-rpc- containerPort: 9849name: raft-rpc- containerPort: 7848name: old-raft-rpcenv:- name: NACOS_REPLICASvalue: "3"- name: SERVICE_NAMEvalue: "nacos-headless"- name: DOMAIN_NAMEvalue: "cluster.local"- name: POD_NAMESPACEvalueFrom:fieldRef:apiVersion: v1fieldPath: metadata.namespace- name: MYSQL_SERVICE_HOSTvalueFrom:configMapKeyRef:name: nacos-cmkey: mysql.host- name: MYSQL_SERVICE_DB_NAMEvalueFrom:configMapKeyRef:name: nacos-cmkey: mysql.db.name- name: MYSQL_SERVICE_PORTvalueFrom:configMapKeyRef:name: nacos-cmkey: mysql.port- name: MYSQL_SERVICE_USERvalueFrom:configMapKeyRef:name: nacos-cmkey: mysql.user- name: MYSQL_SERVICE_PASSWORDvalueFrom:configMapKeyRef:name: nacos-cmkey: mysql.password- name: SPRING_DATASOURCE_PLATFORMvalue: "mysql"- name: NACOS_SERVER_PORTvalue: "8848"- name: NACOS_APPLICATION_PORTvalue: "8848"- name: PREFER_HOST_MODEvalue: "hostname"volumeMounts:- name: datamountPath: /home/nacos/plugins/peer-findersubPath: peer-finder- name: datamountPath: /home/nacos/datasubPath: data- name: datamountPath: /home/nacos/logssubPath: logsvolumeClaimTemplates:- metadata:name: dataannotations:volume.beta.kubernetes.io/storage-class: "managed-nfs-storage"spec:accessModes: [ "ReadWriteMany" ]resources:requests:storage: 2Giselector:matchLabels:app: nacos

nacos安装

创建角色

[root@hadoop01 nacos-k8s]# kubectl create -f deploy/nfs/rbac.yaml
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
[root@hadoop01 nacos-k8s]# kubectl create -f deploy/nfs/deployment.yaml
serviceaccount/nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
[root@hadoop01 nacos-k8s]# kubectl create -f deploy/nfs/class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
[root@hadoop01 nacos-k8s]# kubectl get pod -l app=nfs-client-provisioner
NAME                                      READY   STATUS    RESTARTS   AGE
nfs-client-provisioner-7ddb7b97b6-n4vh2   1/1     Running   0          24s

部署数据库

[root@hadoop01 nacos-k8s]# kubectl create -f deploy/mysql/mysql-nfs.yaml
replicationcontroller/mysql created
service/mysql created
[root@hadoop01 nacos-k8s]# kubectl get pod
NAME                                      READY   STATUS    RESTARTS   AGE
mysql-dl99b                               1/1     Running   0          15s
nfs-client-provisioner-7ddb7b97b6-n4vh2   1/1     Running   0          2m23skubectl exec -it mysql-dl99b  CONTAINER_ID mysql -u nacos
kubectl exec -it mysql-dl99b sh
# mysql -h localhost -u nacos -p
Enter password: 

执行数据库初始化语句

不用执行，已经好了

mysql> show tables;
+-------------------------+
| Tables_in_nacos_devtest |
+-------------------------+
| config_info             |
| config_info_aggr        |
| config_info_beta        |
| config_info_tag         |
| config_tags_relation    |
| group_capacity          |
| his_config_info         |
| permissions             |
| roles                   |
| tenant_capacity         |
| tenant_info             |
| users                   |
+-------------------------+
12 rows in set (0.00 sec)mysql> select * from uses;
ERROR 1146 (42S02): Table 'nacos_devtest.uses' doesn't exist
mysql> select * from users;
+----------+--------------------------------------------------------------+---------+
| username | password                                                     | enabled |
+----------+--------------------------------------------------------------+---------+
| nacos    | $2a$10$EuWPZHzz32dJN7jexM34MOeYirDdFAZm2kuWj7VEOJhhZkDrxfvUu |       1 |
+----------+--------------------------------------------------------------+---------+
1 row in set (0.00 sec)mysql> 

部署nacos

kubectl create -f deploy/nacos/nacos-pvc-nfs.yaml

ingress

ingress-nacos.yaml
如有疑问，请参考 k8s中ingress-nginx离线安装实践

#ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-nacosnamespace: defaultannotations:nginx.ingress.kubernetes.io/rewrite-target: /
#    kubernetes.io/ingress.class: nginx
spec:ingressClassName: nginxrules:- host: "nacos.fun.com"http:paths:- pathType: Prefixpath: /backend:service:name: nacos-headlessport:number: 8848 

效果展示

访问地址：http://nacos.fun.com:32570/nacos/index.html
直接使用 http://nacos.fun.com 是无法访问的。

问题修复

解决需要端口进行访问：http://nacos.fun.com:32570/nacos/index.html

#ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-nacosnamespace: defaultannotations:
#    nginx.ingress.kubernetes.io/rewrite-target: /
#    kubernetes.io/ingress.class: nginx
spec:ingressClassName: nginxrules:- host: "nacos.fun.com"http:paths:- pathType: Prefixpath: /backend:service:name: nacos-headlessport:number: 8848

红线部分，解决了带端口访问问题

结束

nacos在k8s上的集群安装实践