ubuntu离线安装k8s

目录

一、前期准备

二、安装前配置

三、安装docker

四、安装cri-dockerd

五、部署k8s master节点

六、整合kubectl与cri-dockerd

七、网络等插件安装

八、常见问题及解决方法

---

一、前期准备

①ubuntu系统

        本地已安装ubuntu系统，lsb_release -a命令查看版本信息：

②安装包

相关资源也可以从这里下载：

https://download.csdn.net/download/qq_41061437/88806777

https://download.csdn.net/download/qq_41061437/88806805

https://download.csdn.net/download/qq_41061437/88806805

国内很多镜像可以在这里找：docker-ce-linux安装包下载_开源镜像站-阿里云docker-ce-linux安装包是阿里云官方提供的开源镜像免费下载服务，每天下载量过亿，阿里巴巴开源镜像站为包含docker-ce-linux安装包的几百个操作系统镜像和依赖包镜像进行免费CDN加速，更新频率高、稳定安全。https://mirrors.aliyun.com/docker-ce/linux/?spm=a2c6h.25603864.0.0.5a9461d5d8WvU6

docker：

containerd.io_1.6.22-1_amd64.deb   

下载地址：Index of linux/ubuntu/dists/focal/pool/stable/amd64/https://download.docker.com/linux/ubuntu/dists/focal/pool/stable/amd64/docker.io_24.0.5-0ubuntu1_20.04.1_amd64.deb

下载地址：

https://ubuntu.pkgs.org/20.04/ubuntu-updates-universe-amd64/docker.io_24.0.5-0ubuntu1~20.04.1_amd64.deb.htmlhttps://ubuntu.pkgs.org/20.04/ubuntu-updates-universe-amd64/docker.io_24.0.5-0ubuntu1~20.04.1_amd64.deb.htmlcri-dockerd：

cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

下载地址：

Releases · Mirantis/cri-dockerd · GitHubdockerd as a compliant Container Runtime Interface for Kubernetes - Releases · Mirantis/cri-dockerdhttps://github.com/Mirantis/cri-dockerd/releasesk8s：

kubeadm_1.27.4-00_amd64.deb

kubectl_1.27.4-00_amd64.deb

kubelet_1.27.4-00_amd64.deb

下载地址：

Index of /kubernetes/apt/pool/http://mirrors.ustc.edu.cn/kubernetes/apt/pool/

其它：

ipset_7.5-1ubuntu0.20.04.1_amd64.deb

https://ubuntu.pkgs.org/20.04/ubuntu-main-arm64/ipset_7.5-1~exp1_arm64.deb.htmlhttps://ubuntu.pkgs.org/20.04/ubuntu-main-arm64/ipset_7.5-1~exp1_arm64.deb.htmlipvsadm_1:1.31-1_amd64.deb

ipvsadm package : Ubuntuhttps://launchpad.net/ubuntu/+source/ipvsadmlibipset13_7.5-1ubuntu0.20.04.1_amd64.deb

7.5-1ubuntu0.20.04.1 : libipset13 : amd64 : Focal (20.04) : UbuntuIP sets are a framework inside the Linux 2.4.x and 2.6.x kernel which can be administered by the ipset(8) utility. Depending on the type, currently an IP set may store IP addresses, (TCP/UDP) port numbers or IP addresses with MAC addresses in a way which ensures lightning speed when matching an entry against a set. . If you want to . * store multiple IP addresses or port numbers and match against the entire collection using a single iptables rule. * dynamically update iptable...https://bugs.launchpad.net/ubuntu/focal/amd64/libipset13/7.5-1ubuntu0.20.04.1kubernetes-cni_1.2.0-00_s390x_86cdf4d82e3a59c3f6e12975b149a5e42afebff3fd342161abac520253237938.deb

cri-tools_1.26.0-00_amd64.de

Index of /kubernetes/apt/pool/http://mirrors.ustc.edu.cn/kubernetes/apt/pool/socat_1.7.3.3-2_amd64.deb

1.7.3.3-2 : socat : amd64 : Focal (20.04) : UbuntuSocat (for SOcket CAT) establishes two bidirectional byte streams and transfers data between them. Data channels may be files, pipes, devices (terminal or modem, etc.), or sockets (Unix, IPv4, IPv6, raw, UDP, TCP, SSL). It provides forking, logging and tracing, different modes for interprocess communication and many more options. . It can be used, for example, as a TCP relay (one-shot or daemon), as an external socksifier, as a shell interface to Unix sockets, as an IPv6 relay, as a ...https://launchpad.net/ubuntu/focal/amd64/socat/1.7.3.3-2ebtables_2.0.11-3build1_amd64.deb

https://ubuntu.pkgs.org/20.04/ubuntu-main-amd64/ebtables_2.0.11-3build1_amd64.deb.htmlhttps://ubuntu.pkgs.org/20.04/ubuntu-main-amd64/ebtables_2.0.11-3build1_amd64.deb.html

conntrack_1%3A1.4.5-2_amd64.deb

https://ubuntu.pkgs.org/20.04/ubuntu-main-amd64/conntrack_1.4.5-2_amd64.deb.htmlhttps://ubuntu.pkgs.org/20.04/ubuntu-main-amd64/conntrack_1.4.5-2_amd64.deb.html

二、安装前配置

①设置主机名hostname，管理节点设置主机名为master

# 需要设置其他主机名称时，可将 master 替换为正确的主机名node1、node2即可。
sudo hostnamectl set-hostname master

②编辑/etc/hosts 文件，添加域名解析

sudo vim /etc/hosts

cat <<EOF >>/etc/hosts
10.10.10.10 master
EOF

③关闭防火墙、selinux和swap

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo setenforce 0
sudo sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
sudo swapoff -a

④禁用selinux

sudo apt-get install selinux-utils

sudo apt --fix-broken install

sudo setenforce 0

⑤禁用swap分区

sudo swapoff -a

禁用后如需开启swap：

1、vim /etc/default/kubelet

#添加内容

KUBELET_EXTRA_ARGS="--fail-swap-on=false"

2、sudo vim /var/lib/kubelet/config.yaml

#请修改或追加以下内容

featureGates:

  NodeSwap: true

memorySwap:

  swapBehavior: UnlimitedSwap

3、sudo systemctl start kubelet

⑥关闭防火墙

查看当前的防火墙状态：sudo ufw status

关闭防火墙: sudo ufw disable

⑦设置服务启动参数

sudo vim /etc/sysctl.d/k8s.conf 

参数内容如下

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

net.ipv4.ip_forward=1

vm.swappiness=0

vm.overcommit_memory=1

vm.panic_on_oom=0

fs.inotify.max_user_instances=8192

fs.inotify.max_user_watches=1048576

fs.file-max=52706963

fs.nr_open=52706963

net.ipv6.conf.all.disable_ipv6=1

net.netfilter.nf_conntrack_max=2310720

使配置生效

sudo sysctl -p /etc/sysctl.d/k8s.conf

如果执行报错见：八①

⑧安装ipvs内核模块

由于ubuntu系统默认已经加载ipvs内核模块，执行验证

lsmod | grep ip_vs

如果返回 为空，表示没有加载，执行下面命令重新加载 ：

sudo  modprobe  ip_vs

安装 ipvsadm ipset

sudo dpkg -i ipset_7.5-1ubuntu0.20.04.1_amd64.deb

sudo dpkg -i ipvsadm_1:1.31-1_amd64.deb 

sudo dpkg -i libipset13_7.5-1ubuntu0.20.04.1_amd64.deb

三、安装docker

①需要用到的离线包

containerd.io_1.6.22-1_amd64.deb

docker.io_24.0.5-0ubuntu1_20.04.1_amd64.deb

执行命令安装：

sudo dpkg -i containerd.io_1.6.22-1_amd64.deb

sudo dpkg -i docker.io_24.0.5-0ubuntu1_20.04.1_amd64.deb

sudo systemctl start containerd

sudo systemctl start docker

查看状态：

sudo systemctl status docker

②配置用户组

把需要使用docker命令的用户，添加到用户组中：

sudo groupadd docker

sudo usermod -aG docker $USER

 $USER是环境变量，指当前用户

配置docker开机启动

sudo systemctl enable docker

如果配置开机启动报错，见：八②

③配置私有镜像仓库以及日志切分配置

修改daemon.json文件

sudo vim /etc/docker/daemon.json

{
"registry-mirrors": [

"https://registry.docker-cn.com"

],

"insecure-registry": [

"registry.docker-cn.com"

],

"log-driver":"json-file",

    "log-opts":{

        "max-size":"100m",

        "max-file":"3"

    }

}

四、安装cri-dockerd

cri-docker是一个支持CRI标准的shim（垫片），一边通过CRI跟kubelet交互，另一边跟docker api交互，从而间接的实现了kubernetes以docker作为容器运行。

需要用到的离线包：cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

执行命令：

sudo dpkg -i cri-dockerd_0.3.4.3-0.ubuntu-jammy_amd64.deb

安装完cri-docker后，对应的服务会自动启动，命令查看：

systemctl status cri-docker

五、部署k8s master节点

①需要用到的离线包

kubeadm_1.27.4-00_amd64.deb

kubelet_1.27.4-00_amd64.deb 

kubectl_1.27.4-00_amd64.deb

执行命令：

sudo dpkg -i kubeadm_1.27.4-00_amd64.deb

sudo dpkg -i kubelet_1.27.4-00_amd64.deb

sudo dpkg -i kubectl_1.27.4-00_amd64.deb

如果报错，见：八③

导入k8s集群安装所需镜像

docker load -i k8s_images.tar

验证镜像是否存在：

docker image ls

六、整合kubectl与cri-dockerd

①修改配置文件

sudo vim /lib/systemd/system/cri-docker.service

内容如下：

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

修改后的cri-docker.service文件内容：

[Unit]

Description=CRI Interface for Docker Application Container Engine

Documentation=https://docs.mirantis.com

After=network-online.target firewalld.service docker.service

Wants=network-online.target

Requires=cri-docker.socket

[Service]

Type=notify

ExecStart=/usr/bin/cri-dockerd --container-runtime-endpoint fd:// --network-plugin=cni --cni-bin-dir=/opt/cni/bin --cni-cache-dir=/var/lib/cni/cache --cni-conf-dir=/etc/cni/net.d --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9

ExecReload=/bin/kill -s HUP $MAINPID

TimeoutSec=0

RestartSec=2

Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.

# Both the old, and new location are accepted by systemd 229 and up, so using the old location

# to make them work for either version of systemd.

StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.

# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make

# this option work for either version of systemd.

StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.

# Only systemd 226 and above support this option.

TasksMax=infinity

Delegate=yes

KillMode=process

[Install]

WantedBy=multi-user.target

在/usr/lib/systemd/system/cri-docker.service文件中添加上如上配置；

--network-plugin：指定网络插件规范的类型，这里要使用CNI；

--cni-bin-dir：指定CNI插件二进制程序文件的搜索目录；

--cni-cache-dir：CNI插件使用的缓存目录；

--cni-conf-dir：CNI插件加载配置文件的目录；

--pod-infra-container-image:指定pause镜像 这个一定要配置，不然systemctl status cri-docker会报错

②重启cri-dockerd服务

sudo  systemctl daemon-reload && sudo systemctl restart cri-docker

七、网络等插件安装

①kubeadm初始化

sudo kubeadm init --kubernetes-version=v1.27.4 --apiserver-advertise-address=xxxxxx --apiserver-bind-port=6443 --image-repository=registry.aliyuncs.com/google_containers --service-cidr=10.96.0.0/12 --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=Swap --cri-socket=unix:///run/cri-dockerd.sock --v=5

其中piserver-advertise-address是实际机器的ip地址

如果报错，见：八④

②配置kubectl工具

root用户：

sudo mkdir -p /root/.kube
sudo cp /etc/kubernetes/admin.conf /root/.kube/config

普通用户：

##创建自己的kube
mkdir -p $HOME/.kube
##复制root用户的kubectl配置到家目录下
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
##修改kubectl配置文件的权限
sudo chown $(id -u):$(id -g) $HOME/.kube/config

测试集群：

kubectl get nodes

kubectl get cs

kubectl get pods --all-namespaces

我这里是已经部署完毕并已经成功运行pod后的截图：

③安装calico

需要用到calico.yaml,详见：

https://docs.projectcalico.org/v3.20/manifests/calico.yamlhttps://docs.projectcalico.org/v3.20/manifests/calico.yaml执行命令：

kubectl apply -f  calico.yml

部署完毕之后：coredns的两个pod变为runing状态

执行命令查看：

kubectl get pod -n kube-system

④设置允许master调度pod

先执行命令查看：

sudo kubectl describe node master | grep Taints

会显示：

Taints:             node-role.kubernetes.io/control-plane:NoSchedule

执行命令：

kubectl taint node master node-role.kubernetes.io/control-plane:NoSchedule-

如果执行kubectl get nodes出现node没有ready，重启container和kubectl：

systemctl restart containerd

systemctl restart kubelet

如果报错见：八⑤

八、常见问题及解决方法

①sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No Such file or directory

        如果出现这个错误说明没有先加载内核模块br_netfilter，bridge-nf 使 netfilter 可以对 Linux 网桥上的 IPv4/ARP/IPv6 包过滤。比如设置net.bridge.bridge-nf-call-iptables=1后，二层的网桥在转发包时也会被 iptables的 FORWARD 规则所过滤。

解决方案：

1.加载overlay和br_netfilter两个内核模块

sudo modprobe overlay && sudo modprobe br_netfilter

持久化加载上述两个模块，避免重启失效。

$ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf

overlay

br_netfilter

EOF

通过运行 lsmod | grep br_netfilter 来验证 br_netfilter 模块是否已加载

通过运行 lsmod | grep overlay 来验证 overlay模块是否已加载

再次执行：sudo sysctl -p /etc/sysctl.d/k8s.conf

如果报错：sysctl: cannot stat /proc/sys/net/netfilter/nf_conntrack_max: No such file or directory，可能是 conntrack没有加载，执行：lsmod | grep conntrack

如果返回为空，表示没有加载，执行下面命令

重新加载：sudo modprobe ip_conntrack sudo sysctl -p /etc/sysctl.d/k8s.conf

②配置docker开机启动报错：Failed to enable unit: Unit file docker.service does not exist.

新建docker.service文件：sudo vim  /lib/systemd/system/docker.service

[Unit]

Description=Docker Application Container Engine

Documentation=https://docs.docker.com

After=network-online.target docker.socket firewalld.service containerd.service time-set.target

Wants=network-online.target containerd.service

Requires=docker.socket

[Service]

Type=notify

# the default is not to use systemd for cgroups because the delegate issues still

# exists and systemd currently does not support the cgroup feature set required

# for containers run by docker

ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

ExecReload=/bin/kill -s HUP $MAINPID

TimeoutStartSec=0

RestartSec=2

Restart=always

# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.

# Both the old, and new location are accepted by systemd 229 and up, so using the old location

# to make them work for either version of systemd.

StartLimitBurst=3

# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.

# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make

# this option work for either version of systemd.

StartLimitInterval=60s

# Having non-zero Limit*s causes performance problems due to accounting overhead

# in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity

LimitNPROC=infinity

LimitCORE=infinity

# Comment TasksMax if your systemd version does not support it.

# Only systemd 226 and above support this option.

TasksMax=infinity

# set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

# kill only the docker process, not all processes in the cgroup

KillMode=process

OOMScoreAdjust=-500

[Install]

WantedBy=multi-user.target

③安装kubeadm、kubelet、kubectl报错

如果报错：

Selecting previously unselected package kubeadm.

(Reading database ... 106973 files and directories currently installed.)

Preparing to unpack kubeadm_1.27.4-00_amd64.deb ...

Unpacking kubeadm (1.27.4-00) ...

Selecting previously unselected package kubelet.

Preparing to unpack kubelet_1.27.4-00_amd64.deb ...

Unpacking kubelet (1.27.4-00) ...

Selecting previously unselected package kubectl.

Preparing to unpack kubectl_1.27.4-00_amd64.deb ...

Unpacking kubectl (1.27.4-00) ...

dpkg: dependency problems prevent configuration of kubeadm:

 kubeadm depends on kubernetes-cni (>= 1.1.1); however:

  Package kubernetes-cni is not installed.

 kubeadm depends on cri-tools (>= 1.25.0); however:

  Package cri-tools is not installed.

dpkg: error processing package kubeadm (--install):

 dependency problems - leaving unconfigured

dpkg: dependency problems prevent configuration of kubelet:

 kubelet depends on kubernetes-cni (>= 1.1.1); however:

  Package kubernetes-cni is not installed.

 kubelet depends on socat; however:

  Package socat is not installed.

 kubelet depends on ebtables; however:

  Package ebtables is not installed.

 kubelet depends on conntrack; however:

  Package conntrack is not installed.

dpkg: error processing package kubelet (--install):

 dependency problems - leaving unconfigured

Setting up kubectl (1.27.4-00) ...

Errors were encountered while processing:

 kubeadm

 kubelet

需要安装依赖：kubernetes-cni  cri-tools  socat  ebtables  conntrack

sudo dpkg -i kubernetes-cni_1.2.0-00_s390x_86cdf4d82e3a59c3f6e12975b149a5e42afebff3fd342161abac520253237938.deb

sudo dpkg -i cri-tools_1.26.0-00_amd64.deb

sudo dpkg -i socat_1.7.3.3-2_amd64.deb

sudo dpkg -i ebtables_2.0.11-3build1_amd64.deb

sudo dpkg -i conntrack_1%3A1.4.5-2_amd64.deb

如果安装继续报错：

dpkg: error processing archive kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb (--install):

 package architecture (arm64) does not match system (amd64)

Errors were encountered while processing:

 kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb

halos@bgi:/mnt/test-halos/upgradeworkspace/deb$ sudo dpkg --add-architecture arm64

再次安装kubernetes-cni_1.2.0-00_arm64_5d61b8d04701612640667c1da13b616529ded1fed0b7405382d8d08eaa5b5af7.deb

sudo dpkg -i cri-tools_1.26.0-00_arm64_be3fa6bdc17ab229b45222887c442ae1a601b3b2bc3e011c9e7235767e7269c4.deb

④kubeadm初始化报错

如果报错：

[ERROR Port-10250]: Port 10250 is in use

要解除kubelet.service的mask状态，请使用以下命令：

sudo systemctl unmask kubelet.service

驱动一致还报错执行以下命令：

sudo kubeadm reset -f
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X

 如果继续报错：

[init] Using Kubernetes version: v1.24.4

[preflight] Running pre-flight checks

error execution phase preflight: [preflight] Some fatal errors occurred:

        [ERROR CRI]: container runtime is not running: output: E1107 11:17:24.937456   31126 remote_runtime.go:948] "Status from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

time="2022-11-07T11:17:24+08:00" level=fatal msg="getting status of runtime: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"

, error: exit status 1

[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

To see the stack trace of this error execute with --v=5 or higher

 执行以下命令：

sudo rm-rf /etc/containerd/config.toml

sudo systemctl daemon-reload

sudo systemctl restart containerd

执行完之后再重新执行kubeadm init命令

⑤设置允许master调度pod报错

kubectl taint node master node-role.kubernetes.io/control-plane:NoSchedule-
error: taint "node-role.kubernetes.io/control-plane:NoSchedule" not found

出现该错误，会导致pod节点无法调度，命令df -h查看系统磁盘空间，我这里的问题主要就是磁盘空间不足导致。