k8s环境部署配置

目录

一.虚拟机准备

二.基础环境配置（各个节点都做）

1.IP和hosts解析

2.防火墙和selinux

3.安装基本软件

4.配置时间同步

5.禁用swap分区

6.修改内核参数并重载

7.配置ipvs

三.docker环境（各个节点都做）

1.配置软件源并安装docker-ce

2.配置docker加速

四.cri环境配置（各个节点都做）

1.下载查看版本

2.配置服务启动

五.harbor环境（只有register节点做）

1.安装harbor配置

2.验证

 六.集群初始化

1.下载k8s所需软件（各个节点都做）

2.在harbor上新建项目，并执行脚本

3.master节点初始化 （只在master节点做）

4.执行成功后出现此界面后进行权限设置 

5.各个节点根据master初始化成功后的参数来加入（各个节点都做）

七.收尾工作（master节点做）

1.命令补全配置

2.网络配置

（1）下载

（2）执行脚本文件

（3）应用配置文件并验证

---

一.虚拟机准备

主机（我的主机名）	IP
master(node1-190.com)	192.168.2.190
node1(node2-191.com)	192.168.2.191
node2(node3-192.com)	192.168.2.192
node3(node4-193.com)	192.168.2.193
register(node5-196.com)	192.168.2.196

二.基础环境配置（各个节点都做）

1.IP和hosts解析

[root@node1-190 ~]# vim /etc/hosts192.168.2.190 node1-190.com
192.168.2.191 node2-191.com
192.168.2.192 node3-192.com
192.168.2.193 node4-193.com
192.168.2.196 node5-196.com

2.防火墙和selinux

[root@node1-190 ~]# systemctl status firewalld.service;getenforce 
● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)Active: inactive (dead)Docs: man:firewalld(1)
Disabled#临时
systemctl stop firewalld
setenforce 0
#禁用
systemctl disable firewalld
sed -i '/^SELINUX=/ c SELINUX=disabled' /etc/selinux/config

3.安装基本软件

yum install -y wget tree bash-completion lrzsz psmisc net-tools vim chrony

4.配置时间同步

[root@node1-190 ~]# vim /etc/chrony.conf
:3,6 s/^/#      #注释掉原有行
server ntp1.aliyun.com iburst[root@node1-190 ~]# systemctl restart chronyd[root@node1-190 ~]# chronyc sources
210 Number of sources = 1
MS Name/IP address         Stratum Poll Reach LastRx Last sample               
===============================================================================
^* 120.25.115.20                 2   8   341   431   -357us[ -771us] +/-   20ms

5.禁用swap分区

[root@node1-190 ~]# swapoff -a && sed -i 's/.*swap.*/#&/' /etc/fstab && free -mtotal        used        free      shared  buff/cache   available
Mem:          10376         943        8875          11         557        9178
Swap:             0           0           0

6.修改内核参数并重载

[root@node1-190 ~]# cat >> /etc/sysctl.d/k8s.conf << EOF
> vm.swappiness=0
> net.bridge.bridge-nf-call-ip6tables = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.ipv4.ip_forward = 1
> EOF[root@node1-190 ~]# modprobe br_netfilter &&  modprobe overlay && sysctl -p /etc/sysctl.d/k8s.conf
vm.swappiness = 0
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

7.配置ipvs

[root@node1-190 ~]# yum install ipset ipvsadm -y[root@node1-190 ~]# cat <<EOF >  /etc/sysconfig/modules/ipvs.modules
> #!/bin/bash
> modprobe -- ip_vs
> modprobe -- ip_vs_rr
> modprobe -- ip_vs_wrr
> modprobe -- ip_vs_sh
> modprobe -- nf_conntrack
> EOF[root@node1-190 ~]# chmod +x /etc/sysconfig/modules/ipvs.modules && /bin/bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
nf_conntrack_ipv4      15053  2 
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip_vs_sh               12688  0 
ip_vs_wrr              12697  0 
ip_vs_rr               12600  0 
ip_vs                 145458  6 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack          139264  7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
libcrc32c              12644  4 xfs,ip_vs,nf_nat,nf_conntrack

三.docker环境（各个节点都做）

1.配置软件源并安装docker-ce

[root@node1-190 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2[root@node1-190 ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo[root@node1-190 ~]# yum install -y docker-ce

2.配置docker加速

[root@node1-190 ~]# cat >> /etc/docker/daemon.json <<-EOF
> {
>  "registry-mirrors": [
> "http://74f21445.m.daocloud.io",
> "https://registry.docker-cn.com",
> "http://hub-mirror.c.163.com",
> "https://docker.mirrors.ustc.edu.cn"
>  ], 
>   "insecure-registries": ["node5-196.com"],    #写register节点的域名，各个节点的这个文件都写register的域名
>   "exec-opts": ["native.cgroupdriver=systemd"]
> }
> EOF[root@node1-190 ~]# systemctl daemon-reload && systemctl start docker

四.cri环境配置（各个节点都做）

1.下载查看版本

链接：https://pan.baidu.com/s/1rRIqV-GO6ETe4N9glNs2cg 
提取码：km17

[root@node1-190 ~]# tar -xf cri-dockerd-0.3.4.amd64.tgz -C /usr/local/
[root@node1-190 local]# ll
total 0
drwxr-xr-x. 2 root    root     6 Apr 11  2018 bin
drwxrwxr-x  2 sulibao sulibao 25 Jun 30 03:31 cri-dockerd
drwxr-xr-x. 2 root    root     6 Apr 11  2018 etc
drwxr-xr-x. 2 root    root     6 Apr 11  2018 games
drwxr-xr-x. 2 root    root     6 Apr 11  2018 include
drwxr-xr-x. 2 root    root     6 Apr 11  2018 lib
drwxr-xr-x. 2 root    root     6 Apr 11  2018 lib64
drwxr-xr-x. 2 root    root     6 Apr 11  2018 libexec
drwxr-xr-x. 2 root    root     6 Apr 11  2018 sbin
drwxr-xr-x. 5 root    root    49 Jul 23 08:59 share
drwxr-xr-x. 2 root    root     6 Apr 11  2018 src[root@node1-190 local]# mv cri-dockerd/cri-dockerd /usr/local/bin/[root@node1-190 local]# cri-dockerd --version
cri-dockerd 0.3.4 (e88b1605)

2.配置服务启动

[root@node1-190 local]# vim /etc/systemd/system/cri-dockerd.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target[Service]
Type=notify
ExecStart=/usr/local/bin/cri-dockerd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9 --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --cri-dockerd-root-directory=/var/lib/dockershim --docker-endpoint=unix:///var/run/docker.sock --cri-dockerd-root-directory=/var/lib/docker
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target[root@node1-190 local]# vim /etc/systemd/system/cri-dockerd.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service
[Socket]
ListenStream=/var/run/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker
[Install]
WantedBy=sockets.target[root@node1-190 ~]# systemctl daemon-reload
[root@node1-190 ~]# systemctl start cri-dockerd.service
[root@node1-190 ~]# systemctl enable cri-dockerd.service

五.harbor环境（只有register节点做）

1.安装harbor配置

[root@node5-196 ~]# mv docker-compose-linux-x86_64 /usr/local/bin/docker-compose[root@node5-196 ~]# chmod +x /usr/local/bin/docker-compose [root@node5-196 ~]# tar -xf harbor-offline-installer-v2.8.4.tgz -C /usr/local/
[root@node5-196 ~]# cd /usr/local/
[root@node5-196 local]# ll[root@node5-196 local]# cd harbor/
[root@node5-196 harbor]# ll
total 597536
-rw-r--r-- 1 root root      3639 Aug 15 17:53 common.sh
-rw-r--r-- 1 root root 611834153 Aug 15 17:54 harbor.v2.8.4.tar.gz
-rw-r--r-- 1 root root     12499 Aug 15 17:53 harbor.yml.tmpl
-rwxr-xr-x 1 root root      2725 Aug 15 17:53 install.sh
-rw-r--r-- 1 root root     11347 Aug 15 17:53 LICENSE
-rwxr-xr-x 1 root root      1881 Aug 15 17:53 prepare[root@node5-196 harbor]# cp harbor.yml.tmpl harbor.yml[root@node5-196 harbor]# vim harbor.yml
# Configuration file of Harbor# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: node5-196.com  #修改为自己的主机名# http related config
http:# port for http, default is 80. If https enabled, this port will redirect to https portport: 80    #注意端口# https related config
#https:      #https区块全部注释# https port for harbor, default is 443#port: 443# The path of cert and key files for nginx#certificate: /your/certificate/path#private_key: /your/private/key/path[root@node5-196 harbor]# ./install.sh   #安装
[root@node5-196 harbor]# docker images
REPOSITORY                      TAG       IMAGE ID       CREATED       SIZE
goharbor/harbor-exporter        v2.8.4    b8d33e28ec68   2 weeks ago   97.7MB
goharbor/redis-photon           v2.8.4    7b7324d651ca   2 weeks ago   120MB
goharbor/trivy-adapter-photon   v2.8.4    91d8e9f0b21a   2 weeks ago   464MB
goharbor/notary-server-photon   v2.8.4    a46f91560454   2 weeks ago   113MB
goharbor/notary-signer-photon   v2.8.4    da66bd8d944b   2 weeks ago   110MB
goharbor/harbor-registryctl     v2.8.4    805b38ca6bee   2 weeks ago   141MB
goharbor/registry-photon        v2.8.4    756769e94123   2 weeks ago   79MB
goharbor/nginx-photon           v2.8.4    375018db778b   2 weeks ago   116MB
goharbor/harbor-log             v2.8.4    8a2045fb24d2   2 weeks ago   124MB
goharbor/harbor-jobservice      v2.8.4    97808fc10f64   2 weeks ago   141MB
goharbor/harbor-core            v2.8.4    c26fcd0714d8   2 weeks ago   164MB
goharbor/harbor-portal          v2.8.4    4a8b0205c0f9   2 weeks ago   124MB
goharbor/harbor-db              v2.8.4    5b8af16d7420   2 weeks ago   174MB
goharbor/prepare                v2.8.4    bdbf974d86ce   2 weeks ago   166MB[root@node5-196 harbor]# cat /etc/docker/daemon.json 
{
"registry-mirrors": [
"http://74f21445.m.daocloud.io",
"https://registry.docker-cn.com",
"http://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn"],"insecure-registries": ["192.168.2.196"], "insecure-registries": ["node5-196.com"],"exec-opts": ["native.cgroupdriver=systemd"]
}
[root@node5-196 harbor]# tail -1 /etc/hosts
192.168.2.196 node5-196.com[root@node5-196 harbor]# systemctl restart docker
[root@node5-196 harbor]# docker-compose up -d

2.验证

[root@node5-196 harbor]# docker login http://node5-196.com   #进行harbor上镜像的推送和拉取验证
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded[root@node5-196 harbor]# docker tag busybox:latest node5-196.com/library/busybox:latest[root@node5-196 harbor]# docker push node5-196.com/library/busybox:latest 
The push refers to repository [node5-196.com/library/busybox]
3d24ee258efc: Layer already exists 
latest: digest: sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee size: 528[root@node5-196 harbor]# docker pull node5-196.com/library/busybox:latest 
latest: Pulling from library/busybox
Digest: sha256:023917ec6a886d0e8e15f28fb543515a5fcd8d938edb091e8147db4efed388ee
Status: Image is up to date for node5-196.com/library/busybox:latest
node5-196.com/library/busybox:latest
[root@node5-196 harbor]# docker images
REPOSITORY                      TAG       IMAGE ID       CREATED       SIZE
goharbor/harbor-exporter        v2.8.4    b8d33e28ec68   2 weeks ago   97.7MB
goharbor/redis-photon           v2.8.4    7b7324d651ca   2 weeks ago   120MB
goharbor/trivy-adapter-photon   v2.8.4    91d8e9f0b21a   2 weeks ago   464MB
goharbor/notary-server-photon   v2.8.4    a46f91560454   2 weeks ago   113MB
goharbor/notary-signer-photon   v2.8.4    da66bd8d944b   2 weeks ago   110MB
goharbor/harbor-registryctl     v2.8.4    805b38ca6bee   2 weeks ago   141MB
goharbor/registry-photon        v2.8.4    756769e94123   2 weeks ago   79MB
goharbor/nginx-photon           v2.8.4    375018db778b   2 weeks ago   116MB
goharbor/harbor-log             v2.8.4    8a2045fb24d2   2 weeks ago   124MB
goharbor/harbor-jobservice      v2.8.4    97808fc10f64   2 weeks ago   141MB
goharbor/harbor-core            v2.8.4    c26fcd0714d8   2 weeks ago   164MB
goharbor/harbor-portal          v2.8.4    4a8b0205c0f9   2 weeks ago   124MB
goharbor/harbor-db              v2.8.4    5b8af16d7420   2 weeks ago   174MB
goharbor/prepare                v2.8.4    bdbf974d86ce   2 weeks ago   166MB
busybox                         latest    a416a98b71e2   6 weeks ago   4.26MB
node5-196.com/library/busybox   latest    a416a98b71e2   6 weeks ago   4.26MB

windows端作域名映射

Windows PowerShell
版权所有（C） Microsoft Corporation。保留所有权利。安装最新的 PowerShell，了解新功能和改进！https://aka.ms/PSWindowsPS C:\WINDOWS\system32> cd .\drivers\etc\
PS C:\WINDOWS\system32\drivers\etc> notepad .\hosts
PS C:\WINDOWS\system32\drivers\etc>

 

推送后的结果 

 六.集群初始化

1.下载k8s所需软件（各个节点都做）

[root@node4-193 ~]# cat > /etc/yum.repos.d/kubernetes.repo << EOF
> [kubernetes]
> name=Kubernetes
> baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 
> enabled=1
> gpgcheck=0
> repo_gpgcheck=0
> gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
> https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
> EOF
[root@node4-193 ~]# 
[root@node4-193 ~]# yum install kubeadm kubectl kubelet -y
[root@node1-190 ~]# systemctl start kubelet

2.在harbor上新建项目，并执行脚本

[root@node1-190 ~]# vim dockerimages.sh 
#!/bin/bash
images=$(kubeadm config images list --kubernetes-version=1.28.0 | awk -F '/' '{print $NF}')
for i in ${images}
dodocker pull registry.aliyuncs.com/google_containers/$idocker tag registry.aliyuncs.com/google_containers/$i node5-196.com/google_containers/$i    #需要修改为刚创建的harbor的仓库地址docker push node5-196.com/google_containers/$idocker rmi registry.aliyuncs.com/google_containers/$i
done[root@node1-190 ~]# docker login node5-196.com
Username: admin 
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded[root@node1-190 ~]# bash dockerimages.sh[root@node1-190 ~]# docker images
REPOSITORY                                                TAG       IMAGE ID       CREATED         SIZE
node5-196.com/google_containers/kube-apiserver            v1.28.0   bb5e0dde9054   2 weeks ago     126MB
node5-196.com/google_containers/kube-scheduler            v1.28.0   f6f496300a2a   2 weeks ago     60.1MB
node5-196.com/google_containers/kube-controller-manager   v1.28.0   4be79c38a4ba   2 weeks ago     122MB
node5-196.com/google_containers/kube-proxy                v1.28.0   ea1030da44aa   2 weeks ago     73.1MB
node5-196.com/google_containers/etcd                      3.5.9-0   73deb9a3f702   3 months ago    294MB
node5-196.com/google_containers/coredns                   v1.10.1   ead0a4a53df8   6 months ago    53.6MB
node5-196.com/google_containers/pause                     3.9       e6f181688397   10 months ago   744kB

3.master节点初始化 （只在master节点做）

执行完脚本无误后执行以下命令

kubeadm init --kubernetes-version=1.28.0 \
--apiserver-advertise-address=192.168.2.190 \   #修改为master节点的IP
--image-repository node5-196.com/google_containers \   #修改为harbor仓库的地址
--service-cidr=10.96.0.0/12 \
--pod-network-cidr=10.244.0.0/16 \
--ignore-preflight-errors=Swap \
--cri-socket=unix:///var/run/cri-dockerd.sock

4.执行成功后出现此界面后进行权限设置 

[root@node1-190 ~]# mkdir -p $HOME/.kube && cp -i /etc/kubernetes/admin.conf $HOME/.kube/config && chown $(id -u):$(id -g) $HOME/.kube/config && export KUBECONFIG=/etc/kubernetes/admin.conf

5.各个节点根据master初始化成功后的参数来加入（各个节点都做）

kubeadm join 192.168.2.190:6443 --token m5mkth.rb2e9rb3892fsu37 \--discovery-token-ca-cert-hash sha256:1209d580742d411fb4406953185ef9f27522b3ba150b8726fd39f364628d0942 \--cri-socket=unix:///var/run/cri-dockerd.sock 

出现此界面即成功 

[root@node1-190 ~]# kubectl get nodes
NAME            STATUS     ROLES           AGE    VERSION
node1-190.com   NotReady   control-plane   121m   v1.28.1
node2-191.com   NotReady   <none>          104m   v1.28.1
node3-192.com   NotReady   <none>          96m    v1.28.1
node4-193.com   NotReady   <none>          11m    v1.28.1
node5-196.com   NotReady   <none>          9s     v1.28.1

七.收尾工作（master节点做）

1.命令补全配置

[root@node1-190 ~]# echo "source <(kubectl completion bash)" >> ~/.bashrc && echo "source <(kubeadm completion bash)" >> ~/.bashrc && source ~/.bashrc

2.网络配置

（1）下载

链接：https://pan.baidu.com/s/1TRUyPxbBZRcyyyqojquhBw 
提取码：wvap

（2）执行脚本文件

[root@node1-190 ~]# vim flannel.sh
#!/bin/bash
for i in $(grep image kube-flannel.yml | grep -v '#' | awk -F '/' '{print $NF}')
dodocker pull flannel/$idocker tag flannel/$i node5-196.com/google_containers/$i   #需要修改为刚创建的harbor的仓库地址docker push node5-196.com/google_containers/$idocker rmi flannel/$i
done
[root@node1-190 ~]# bash flannel.sh
[root@node1-190 ~]# docker images
REPOSITORY                                                  TAG       IMAGE ID       CREATED         SIZE
node5-196.com/google_containers/flannel                     v0.22.2   d73868a08083   13 days ago     70.2MB
node5-196.com/google_containers/kube-apiserver              v1.28.0   bb5e0dde9054   2 weeks ago     126MB
node5-196.com/google_containers/kube-controller-manager     v1.28.0   4be79c38a4ba   2 weeks ago     122MB
node5-196.com/google_containers/kube-scheduler              v1.28.0   f6f496300a2a   2 weeks ago     60.1MB
node5-196.com/google_containers/kube-proxy                  v1.28.0   ea1030da44aa   2 weeks ago     73.1MB
node5-196.com/google_containers/flannel-cni-plugin          v1.2.0    a55d1bad692b   5 weeks ago     8.04MB
node5-196.com/google_containers/etcd                        3.5.9-0   73deb9a3f702   3 months ago    294MB
node5-196.com/google_containers/coredns                     v1.10.1   ead0a4a53df8   6 months ago    53.6MB
node5-196.com/google_containers/pause                       3.9       e6f181688397   10 months ago   744kB
registry.cn-hangzhou.aliyuncs.com/google_containers/pause   3.9       e6f181688397   10 months ago   744kB[root@node1-190 ~]# sed -i '/ image:/s#docker.io/flannel#node5-196.com/google_containers#' kube-flannel.yml

（3）应用配置文件并验证

[root@node1-190 ~]# kubectl apply -f kube-flannel.yml   #应用
[root@node1-190 ~]# kubectl get pod -n kube-system   #验证
NAME                                    READY   STATUS    RESTARTS      AGE
coredns-6845c5578-g6dnk                 1/1     Running   0             159m
coredns-6845c5578-z5cj4                 1/1     Running   0             159m
etcd-node1-190.com                      1/1     Running   0             160m
kube-apiserver-node1-190.com            1/1     Running   0             160m
kube-controller-manager-node1-190.com   1/1     Running   0             160m
kube-proxy-5k8gg                        1/1     Running   0             134m
kube-proxy-b7kcg                        1/1     Running   0             159m
kube-proxy-h5zrz                        1/1     Running   0             142m
kube-proxy-h9gs5                        1/1     Running   1 (35m ago)   38m
kube-proxy-vfx9x                        1/1     Running   0             49m
kube-scheduler-node1-190.com            1/1     Running   0             160m