k8s 集群搭建的一些坑

k8s集群部署的时候会遇到很多的坑，即使看网上的文档也可能遇到各种的坑。

安装准备

1、虚拟机两台（ip按自己的网络环境相应配置）(master/node)

192.168.100.215 k8s-master
192.168.100.216 k8s-node1
2、关闭防火墙(master/node)

systemctl stop firewalld
systemctl disable firewalld
3、关闭selinux(master/node)
setenforce 0 # 临时关闭
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/selinux/config # 永久关闭
4、关闭swap(master/node)
swapoff -a # 临时关闭；关闭swap主要是为了性能考虑
free # 可以通过这个命令查看swap是否关闭了
sed -ri ‘s/.swap./#&/’ /etc/fstab # 永久关闭
5、添加主机名与IP对应的关系(master/node)
$ vim /etc/hosts
#添加如下内容：
192.168.100.215 k8s-master
192.168.100.216 k8s-node1
#保存退出
6、修改主机名(master/node)
#k8s-master
[root@localhost ~] hostname
localhost.localdomain
[root@localhost ~] hostname k8s-master ##临时生效
[root@localhost ~] hostnamectl set-hostname k8s-master ##重启后永久生效
#k8s-node1
[root@localhost ~] hostname
localhost.localdomain
[root@localhost ~] hostname k8s-node1 ##临时生效
[root@localhost ~] hostnamectl set-hostname k8s-node1 ##重启后永久生效
7、桥接设置(master/node)
$ cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sysctl --system
p.s.

以上几步最好照着都执行一下，以免后面报一大堆错。

安装docker

这一步需要注意docker版本与k8s版本对照关系，否则执行master初始化的时候会报错。

部署Kubernetes (master)

kubeadm init
kubeadm init
–apiserver-advertise-address=192.168.100.215
–image-repository registry.aliyuncs.com/google_containers
–kubernetes-version v1.18.0
–service-cidr=10.1.0.0/16
–pod-network-cidr=10.244.0.0/16

执行命令会出现可能一些告警，需要处理一下

cgroupfs切换为systemd

[WARNING IsDockerSystemdCheck]: detected “cgroupfs” as the Docker cgroup driver. The recommended driver is “systemd”. Please follow the guide at https://kubernetes.io/docs/setup/cri/

解决方案：
新建daemon.json切换为systemd

mkdir /etc/docker/
cat > /etc/docker/daemon.json << EOF
{
“registry-mirrors”: [“https://gqs7xcfd.mirror.aliyuncs.com”,“https://hub-mirror.c.163.com”],
“exec-opts”: [“native.cgroupdriver=systemd”],
“log-driver”: “json-file”,
“log-opts”: {
“max-size”: “100m”
},
“storage-driver”: “overlay2”
}
EOF

Docker 与k8s版本不兼容

[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 19.03

解决方案：
删除旧版本的docker
sudo yum remove docker-ce
sudo rm -rf /var/lib/docker

安装兼容版本的docker
yum -y install docker-ce-19.03.9-3.el7 docker-ce-cli-19.03.9-3.el7
systemctl enable docker
systemctl start docker

在两台Node节点上执行join命令

kubeadm join 192.168.200.128:6443 --token 29f2jw.4lczl18ymfnzcax7
–discovery-token-ca-cert-hash sha256:57e05025b3a40415cd9297e089f0439f265a6e5df46ff1b431e017f326b313

cannot stat ‘/etc/kubernetes/admin.conf’

从master节点拷贝过来。

# master节点执行
[root@master ~]# scp /etc/kubernetes/admin.conf root@192.168.247.137:/etc/kubernetes/admin.conf
The authenticity of host '192.168.247.137 (192.168.247.137)' can't be established.
ECDSA key fingerprint is SHA256:oj2gG+nDvkzbS+IggsOxKVwMTVP2zxClBoNEPXmHvzk.
ECDSA key fingerprint is MD5:fa:f6:9a:89:1a:4a:54:63:fe:3d:70:88:c4:a4:4d:ea.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.247.137' (ECDSA) to the list of known hosts.
root@192.168.247.137's password: 
admin.conf                                                                    100% 5451     3.5MB/s   00:00    # node节点执行
[root@slave2 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@slave2 ~]# source ~/.bash_profile
[root@slave2 ~]# chown $(id -u):$(id -g) $HOME/.kube/config# 验证
[root@slave2 ~]# kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6867cdf567-vkf99   1/1     Running   0          6m38s

node节点 kubectl get nodes 报错

error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

[root@localhost ~]# kubectl get nodes
error: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

解决方案

[root@localhost ~]# find / -name  admin.conf
/etc/kubernetes/admin.conf
[root@localhost ~]# mkdir -p $HOME/.kube
[root@localhost ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@localhost ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

删除pod service deployment

删除service
kubectl delete deployment grafana -n kube-system
kubectl delete service grafana -n kube-system
kubectl delete serviceaccount grafana -n kube-system
kubectl delete serviceaccount grafana -n kube-system --force --grace-period=0查看
kubectl get pods,deployments,services,serviceaccounts -n kube-system | grep grafana删除 deployment
kubectl delete deployment monitoring-grafana -n kube-system
kubectl delete pod monitoring-grafana-b46f4c778-z74wg -n kube-system