使用KubeKey 快速交付k8s v1.28.8集群

服务器配置

主机名	IP	CPU	内存	系统盘	数据盘	用途
vm-16-11-ubuntu	192.168.9.131	128	256Gi	500GB	1TB	k8s-master
vm-16-16-ubuntu	192.168.9.132	128	256Gi	500GB	1TB	k8s-master
vm-16-7-ubuntu	192.168.9.133	128	256Gi	500GB	1TB	k8s-master

实战环境涉及软件版本信息：

• 操作系统：Ubuntu 20.04 LTS
• K8s：v1.28.8
• Containerd：1.7.13
• KubeKey: v3.1.1

使用kubekey部署k8s

1. 操作系统基础配置

主机名、DNS解析、时钟同步、防火墙、系统依赖软件等忽略

apt-get install -y install curl socat conntrack ebtables ipset ipvsadm

2. 安装部署 K8s

2.1 下载 KubeKey

https://github.com/kubesphere/kubekey

• 下载最新版的 KubeKey

root@VM-16-7-ubuntu:~# mkdir ~/kubekey
root@VM-16-7-ubuntu:~# cd ~/kubekey
root@VM-16-7-ubuntu:~/kubekey# curl -sfL https://get-kk.kubesphere.io | sh -Downloading kubekey v3.1.1 from https://github.com/kubesphere/kubekey/releases/download/v3.1.1/kubekey-v3.1.1-linux-amd64.tar.gz ...Kubekey v3.1.1 Download Complete!root@VM-16-7-ubuntu:~/kubekey# ll
total 116376
drwxr-xr-x 2 root root     4096 May  7 17:50 ./
drwx------ 7 root root     4096 May  7 17:49 ../
-rwxr-xr-x 1 root root 81950729 Apr 16 12:30 kk*
-rw-r--r-- 1 root root 37206726 May  7 17:50 kubekey-v3.1.1-linux-amd64.tar.gz

• 查看 KubeKey 支持的 Kubernetes 版本列表 ./kk version --show-supported-k8s

root@VM-16-7-ubuntu:~/kubekey# ./kk version --show-supported-k8s
...
v1.28.0
v1.28.1
v1.28.2
v1.28.3
v1.28.4
v1.28.5
v1.28.6
v1.28.7
v1.28.8
v1.29.0
v1.29.1
v1.29.2
v1.29.3
root@VM-16-7-ubuntu:~/kubekey# 
root@VM-16-7-ubuntu:~/kubekey# ./kk create config -f k8s-v1288.yaml --with-kubernetes v1.28.8
Generate KubeKey config file successfully

2.2 创建 K8s 集群部署配置文件

1. 创建集群配置文件

本文选择了 K8s v1.28.8。因此，指定配置文件名称为 k8s-v1288.yaml/

root@VM-16-7-ubuntu:~/kubekey# ./kk create cluster -f k8s-v1288.yaml _   __      _          _   __           
| | / /     | |        | | / /           
| |/ / _   _| |__   ___| |/ /  ___ _   _ 
|    \| | | | '_ \ / _ \    \ / _ \ | | |
| |\  \ |_| | |_) |  __/ |\  \  __/ |_| |
\_| \_/\__,_|_.__/ \___\_| \_/\___|\__, |__/ ||___/17:56:57 CST [GreetingsModule] Greetings
17:56:58 CST message: [VM-16-16-ubuntu]
Greetings, KubeKey!
17:56:58 CST message: [VM-16-7-ubuntu]
Greetings, KubeKey!
17:56:58 CST message: [VM-16-11-ubuntu]
Greetings, KubeKey!
17:56:58 CST success: [VM-16-16-ubuntu]
17:56:58 CST success: [VM-16-7-ubuntu]
17:56:58 CST success: [VM-16-11-ubuntu]
17:56:58 CST [NodePreCheckModule] A pre-check on nodes
17:56:58 CST success: [VM-16-16-ubuntu]
17:56:58 CST success: [VM-16-7-ubuntu]
17:56:58 CST success: [VM-16-11-ubuntu]
17:56:58 CST [ConfirmModule] Display confirmation form
+-----------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| name            | sudo | curl | openssl | ebtables | socat | ipset | ipvsadm | conntrack | chrony | docker | containerd | nfs client | ceph client | glusterfs client | time         |
+-----------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+
| VM-16-7-ubuntu  | y    | y    | y       | y        | y     | y     | y       | y         |        |        |            | y          |             |                  | CST 17:56:58 |
| VM-16-11-ubuntu | y    | y    | y       | y        | y     | y     | y       | y         |        |        |            | y          |             |                  | CST 17:56:58 |
| VM-16-16-ubuntu | y    | y    | y       | y        | y     | y     | y       | y         |        |        |            | y          |             |                  | CST 17:56:58 |
+-----------------+------+------+---------+----------+-------+-------+---------+-----------+--------+--------+------------+------------+-------------+------------------+--------------+This is a simple check of your environment.
Before installation, ensure that your machines meet all requirements specified at
https://github.com/kubesphere/kubekey#requirements-and-recommendationsContinue this installation? [yes/no]: yes
17:57:06 CST success: [LocalHost]
17:57:06 CST [NodeBinariesModule] Download installation binaries
17:57:06 CST message: [localhost]
downloading amd64 kubeadm v1.28.8 ...

修改配置文件
采用 3 个节点同时作为 control-plane、etcd 和 worker 节点。

编辑配置文件 k8s-v1288.yaml，主要修改 kind: Cluster 小节的相关配置

修改 kind: Cluster 小节中 hosts 和 roleGroups 等信息，修改说明如下。

• hosts：指定节点的 IP、ssh 用户、ssh 密码、ssh 端口
• roleGroups：指定 3 个 etcd、control-plane 节点，复用相同的机器作为 3 个 worker 节点
• internalLoadbalancer：启用内置的 HAProxy 负载均衡器
• domain：可自定义域名
• clusterName：可使用默认值 cluster.local
• autoRenewCerts：该参数可以实现证书到期自动续期，默认为 true
• containerManager：使用 containerd

部署完成需要大约 10-20 分钟左右，会在终端上看到类似于下面的输出。

daemonset.apps/calico-node created
deployment.apps/calico-kube-controllers created
18:12:37 CST skipped: [vm-16-16-ubuntu]
18:12:37 CST skipped: [vm-16-11-ubuntu]
18:12:37 CST success: [vm-16-7-ubuntu]
18:12:37 CST [ConfigureKubernetesModule] Configure kubernetes
18:12:37 CST success: [vm-16-7-ubuntu]
18:12:37 CST skipped: [vm-16-11-ubuntu]
18:12:37 CST skipped: [vm-16-16-ubuntu]
18:12:37 CST [ChownModule] Chown user $HOME/.kube dir
18:12:37 CST success: [vm-16-16-ubuntu]
18:12:37 CST success: [vm-16-11-ubuntu]
18:12:37 CST success: [vm-16-7-ubuntu]
18:12:37 CST [AutoRenewCertsModule] Generate k8s certs renew script
18:12:37 CST success: [vm-16-16-ubuntu]
18:12:37 CST success: [vm-16-11-ubuntu]
18:12:37 CST success: [vm-16-7-ubuntu]
18:12:37 CST [AutoRenewCertsModule] Generate k8s certs renew service
18:12:37 CST success: [vm-16-16-ubuntu]
18:12:37 CST success: [vm-16-7-ubuntu]
18:12:37 CST success: [vm-16-11-ubuntu]
18:12:37 CST [AutoRenewCertsModule] Generate k8s certs renew timer
18:12:38 CST success: [vm-16-16-ubuntu]
18:12:38 CST success: [vm-16-7-ubuntu]
18:12:38 CST success: [vm-16-11-ubuntu]
18:12:38 CST [AutoRenewCertsModule] Enable k8s certs renew service
18:12:38 CST success: [vm-16-16-ubuntu]
18:12:38 CST success: [vm-16-11-ubuntu]
18:12:38 CST success: [vm-16-7-ubuntu]
18:12:38 CST [SaveKubeConfigModule] Save kube config as a configmap
18:12:38 CST success: [LocalHost]
18:12:38 CST [AddonsModule] Install addons
18:12:38 CST success: [LocalHost]
18:12:38 CST Pipeline[CreateClusterPipeline] execute successfully
Installation is complete.Please check the result using the command:kubectl get pod -A

3. 验证 K8s 集群

3.1 验证集群状态

• 查看集群节点信息

root@VM-16-7-ubuntu:~# kubectl get nodes -owide
NAME              STATUS   ROLES                  AGE     VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION      CONTAINER-RUNTIME
vm-16-11-ubuntu   Ready    control-plane,worker   9m46s   v1.28.8   172.19.16.11   <none>        Ubuntu 20.04 LTS   5.4.0-174-generic   containerd://1.7.13
vm-16-16-ubuntu   Ready    control-plane,worker   9m43s   v1.28.8   172.19.16.16   <none>        Ubuntu 20.04 LTS   5.4.0-174-generic   containerd://1.7.13
vm-16-7-ubuntu    Ready    control-plane,worker   10m     v1.28.8   172.19.16.7    <none>        Ubuntu 20.04 LTS   5.4.0-174-generic   containerd://1.7.13

4. 部署测试资源

root@VM-16-7-ubuntu:~# kubectl create deployment nginx --image=nginx:alpine --replicas=2
deployment.apps/nginx created
root@VM-16-7-ubuntu:~# kubectl create service nodeport nginx --tcp=80:80
service/nginx created
root@VM-16-7-ubuntu:~# kubectl get deployment -o wide
NAME    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES         SELECTOR
nginx   2/2     2            2           21s   nginx        nginx:alpine   app=nginx
root@VM-16-7-ubuntu:~# kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP             NODE              NOMINATED NODE   READINESS GATES
nginx-b4ccb96c6-q5c52   1/1     Running   0          21s   10.233.123.1   vm-16-7-ubuntu    <none>           <none>
nginx-b4ccb96c6-wgjh4   1/1     Running   0          21s   10.233.96.1    vm-16-16-ubuntu   <none>           <none>

5.验证服务

• 验证直接访问 Pod

root@VM-16-7-ubuntu:~# curl -I http://10.233.96.1
HTTP/1.1 200 OK
Server: nginx/1.25.5
Date: Tue, 07 May 2024 10:24:00 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 16 Apr 2024 15:47:06 GMT
Connection: keep-alive
ETag: "661e9d7a-267"
Accept-Ranges: bytesroot@VM-16-7-ubuntu:~# curl -I http://10.233.123.1
HTTP/1.1 200 OK
Server: nginx/1.25.5
Date: Tue, 07 May 2024 10:24:10 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 16 Apr 2024 15:47:06 GMT
Connection: keep-alive
ETag: "661e9d7a-267"
Accept-Ranges: bytes

• 验证访问 Nodeport

root@VM-16-7-ubuntu:~# curl http://172.19.16.7:31377
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>