安装kubernetes

master1	10.10.10.10	docker、kubelet、kubeadm、kubectl
master2	10.10.10.11	docker、kubelet、kubeadm、kubectl
node1	10.10.10.12	docker、kubelet、kubeadm、kubectl
node2	10.10.10.13	docker、kubelet、kubeadm、kubectl

1.关闭防火墙（所有节点执行）

systemctl stop firewalld
sed -i 's/enforcing/disabled/' /etc/selinux/config

2.关闭swap分区（所有节点执行）

sed -ri 's/.*swap.*/#&/' /etc/fstab

3.添加节点名称（所有节点执行）

cat >> /etc/hosts << EOF
10.10.10.10 master1
10.10.10.11 master2
10.10.10.12 node1
10.10.10.13 node2
EOF

4.在所有节点安装docker（所有节点执行）

yum -y install docker

5.所有节点都配置阿里云加速（所有节点执行）

sudo mkdir -p /etc/docker
sudo cat /etc/docker/daemon.json <<-'EOF'
{"registry-mirrors": ["https://xxxxxxxx.mirror.aliyuncs.com"]}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

6.master和node节点都安装kubadm，kubctl，kublet软件

6.1 master节点和node节点都配置

vim /etc/yum.repos.d/kubernetes.repo

[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

6.2 master节点和node节点都安装

yum install -y kubelet kubeadm kubectl

6.3 master节点和node节点都设置开机启动

systemctl enable  kubelet
systemctl start kubelet

7.在master节点用kubeadm跑初始化

kubeadm init --image-repository=registry.aliyuncs.com/google_containers

7.1报错1：

[root@localhost ~]# kubeadm init --image-repository=registry.aliyuncs.com/google_containers
[init] Using Kubernetes version: v1.26.2
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR CRI]: container runtime is not running: output: E0317 15:44:38.904235   46657 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
time="2023-03-17T15:44:38+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
, error: exit status 1

解决：
[root@localhost ~]# yum -y install containerd jq	
[root@localhost ~]# containerd config default > /etc/containerd/config.toml
[root@localhost ~]# systemctl enable --now containerd
[root@localhost ~]# vim /etc/containerd/config.toml	
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
[root@localhost ~]# systemctl restart containerd.service

7.2报错2：

[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

解决：
[root@localhost ~]# echo "1" >/proc/sys/net/bridge/bridge-nf-call-iptables

正常初始化输出内容：
[root@localhost ~]# kubeadm init --image-repository=registry.aliyuncs.com/google_containers
[init] Using Kubernetes version: v1.26.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost.localdomain] and IPs [10.96.0.1 10.100.208.21]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost localhost.localdomain] and IPs [10.100.208.21 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost localhost.localdomain] and IPs [10.100.208.21 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 7.002772 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node localhost.localdomain as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node localhost.localdomain as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: iarelq.4xv3k2uinhpy43gl
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxyYour Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/Then you can join any number of worker nodes by running the following on each as root:kubeadm join 10.100.208.21:6443 --token iarelq.4xv3k2uinhpy43gl \--discovery-token-ca-cert-hash sha256:31fa6d668197e7652f69b422d5af53888ba5f667fdb87a3eea9fd4c04b34f352 

继续在master节点上执行：

[root@localhost ~]# mkdir -p $HOME/.kube
[root@localhost ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@localhost ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

注意：保存最后产生的token值
如果忘记，重新获取：kubeadm token create --print-join-command

8.将master2节点加入到master1集群

在master1上执行：
[root@master1 ~]# kubeadm reset  #还原由 kubeadm init 或 kubeadm join 所做的更改
[root@master1 ~]# kubeadm init phase upload-certs --upload-certs
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
8ac7964d0808f102629488bb6fc2fbc31db76836245728f03d9303fa1fd7b0bb
[root@master1 ~]# kubeadm token create --print-join-command
kubeadm join 10.100.208.21:6443 --token ze93cl.wmlal7givxmbsre1 --discovery-token-ca-cert-hash sha256:31fa6d668197e7652f69b422d5af53888ba5f667fdb87a3eea9fd4c04b34f352 进行命令拼接：
kubeadm join 10.100.208.21:6443 --token iarelq.4xv3k2uinhpy43gl \--discovery-token-ca-cert-hash sha256:31fa6d668197e7652f69b422d5af53888ba5f667fdb87a3eea9fd4c04b34f352 \--control-plane --certificate-key 8ac7964d0808f102629488bb6fc2fbc31db76836245728f03d9303fa1fd7b0bb

 在master2上执行加入集群启动不起来：

[root@master2 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node AgentLoaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)Drop-In: /usr/lib/systemd/system/kubelet.service.d└─10-kubeadm.confActive: activating (auto-restart) (Result: exit-code) since 一 2023-03-20 17:20:38 CST; 4s agoDocs: https://kubernetes.io/docs/Process: 92264 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)Main PID: 92264 (code=exited, status=1/FAILURE)3月 20 17:20:38 master2 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
3月 20 17:20:38 master2 systemd[1]: Unit kubelet.service entered failed state.
3月 20 17:20:38 master2 systemd[1]: kubelet.service failed.

[root@master2 ~]# journalctl -xefu kubelet
3月 20 17:13:35 master2 systemd[1]: kubelet.service holdoff time over, scheduling restart.
3月 20 17:13:35 master2 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
3月 20 17:13:35 master2 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
3月 20 17:13:35 master2 kubelet[91401]: E0320 17:13:35.739677   91401 run.go:74] "command failed" err="failed to validate kubelet flags: the container runtime endpoint address was not specified or empty, use --container-runtime-endpoint to set"
3月 20 17:13:35 master2 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
3月 20 17:13:35 master2 systemd[1]: Unit kubelet.service entered failed state.
3月 20 17:13:35 master2 systemd[1]: kubelet.service failed.

9.在node节点上执行：

kubeadm join 10.100.208.21:6443 --token iarelq.4xv3k2uinhpy43gl \--discovery-token-ca-cert-hash sha256:31fa6d668197e7652f69b422d5af53888ba5f667fdb87a3eea9fd4c04b34f352

正常输出内容：
[root@localhost ~]# kubeadm join 10.100.208.21:6443 --token 3cjcra.akfpvh70hmhb3u7v \
> --discovery-token-ca-cert-hash sha256:faa3ff1629387afc41774da2e5a15a7bbf314ecbb05ca190b50a56dbed2f1dc3
[preflight] Running pre-flight checks
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.Run 'kubectl get nodes' on the control-plane to see this node join the cluster.	

验证：在master上执行：kubectl get nodes

[root@master1 ~]# kubectl get nodes -owide 
NAME      STATUS     ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION                CONTAINER-RUNTIME
master1   NotReady   control-plane   5h3m    v1.26.3   10.10.10.10   <none>        CentOS Linux 7 (Core)   3.10.0-1160.76.1.el7.x86_64   containerd://1.6.18
node1     NotReady   <none>          4h56m   v1.26.3   10.10.10.12   <none>        CentOS Linux 7 (Core)   6.2.6-1.el7.elrepo.x86_64     containerd://1.6.18