k8s二进制（ETCD的部署安装）

角色	ip	组件
k8s-master	192.168.11.169	kube-apiserver,kube-controller-manager,kube-scheduler,etcd
k8s-node1	192.168.11.164	kubelet,kube-proxy,docker,etcd
k8s-node2	192.168.11.166	kubelet,kube-proxy,docker,etcd

1、为etcd签发证书

1、证书的下载(任意机器上执行都可以)

#工作目录
mkdir -vp /usr/local/kubernetes/{etcd,cfssl}
#进入带cfssl证书目录下
mkdir -vp /opt/etcd/{bin,ssl,cfg}
cd /usr/local/kubernetes/cfssl#github二进制包下载地址：https://github.com/cloudflare/cfssl/releases
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl_1.6.1_linux_amd64
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssljson_1.6.1_linux_amd64
#wget https://github.com/cloudflare/cfssl/releases/download/v1.6.1/cfssl-certinfo_1.6.1_linux_amd64# 生成证书
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
# 利用Json生成证书
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
# 查看证书信息的工具
https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
#其他版本下载地址：https://github.com/cloudflare/cfssl/tags
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
#\cp -R  cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64 /usr/local/bin/
#mv cfssl_linux-amd64 /usr/local/bin/cfssl
#mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
#mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
\cp -R cfssl_linux-amd64 /usr/local/bin/cfssl
\cp -R cfssljson_linux-amd64 /usr/local/bin/cfssljson
\cp -R cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo#查看版本信息
cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6

2、生成自签证书颁发机构（CA）

1、配置CA证书请求文件

cd /usr/local/kubernetes/etcd
#创建ca-csr.json
cat > ca-csr.json <<"EOF"
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "kubemsb","OU": "CN"}],"ca": {"expiry": "87600h"}
}
EOF

2、生成CA

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
#查看是否生成成功
ls *pem ca*csr
#显示ca-key.pem ca.pem这2个文件即生成成功

3、配置ca证书策略

#创建ca-config.json
cat > ca-config.json << EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOF

• server auth 表示client可以对使用该ca对server提供证书进行验证
• client auth 表示server可以使用该ca对client提供的证书进行验证
  

3、使用自签CA 签发Etcd HTTPS 证书

1、创建证书申请文件

cd /usr/local/kubernetes/etcd
cat > etcd-csr.json << EOF
{"CN": "etcd","hosts": ["127.0.0.1","192.168.11.169","192.168.11.166","192.168.11.164"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Beijing","L": "Beijing","O": "kubemsb","OU": "CN"}]
}
EOF

以上ip修改为自己集群环境的ip

2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
#查看证书是否生成成功
ls etcd*pem

2、部署etcd集群

1、下载etcd

cd /usr/local/kubernetes/etcd/
#下载etcd文件
wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz
#wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz
#系统对应的版本https://github.com/etcd-io/etcd/releases/download/v3.5.7/etcd-v3.5.7-linux-amd64.tar.gz
#解压缩
tar zxvf etcd-v3.5.2-linux-amd64.tar.gz
#将下载下来的二进制可执行文件复制到etcd的工作目录中(用于复制到其他服务器上，部署使用)
\cp -R etcd-v3.5.2-linux-amd64/{etcd,etcdctl} /opt/etcd/bin

核心是复制etcd,etcdctl可执行文件到指定目录中,进行后续统一的远程传递。以及etcd集群的搭建

2、创建etcd.conf配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#填写当前节点的ip地址即可
ETCD_LISTEN_PEER_URLS="https://192.168.11.169:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.11.169:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.11.169:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.11.169:2379"
#配置所有etcd集群对应的ip服务配置项
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.11.169:2380,etcd-2=https://192.168.11.166:2380,etcd-3=https://192.168.11.164:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

• 配置配置选项说明

ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing 表示加入已有集群

3、systemd 管理etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/etcd.pem \
--key-file=/opt/etcd/ssl/etcd-key.pem \
--peer-cert-file=/opt/etcd/ssl/etcd.pem \
--peer-key-file=/opt/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-client-cert-auth \
--client-cert-auth \
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

4、拷贝生成的证书到ssl工作目录下

cd /usr/local/kubernetes/etcd
#复制ca办法机构和证书文件到etcd的ssl目录下准备生成etcd集群
\cp -R /usr/local/kubernetes/etcd/ca*.pem etcd*.pem  /opt/etcd/ssl/
cd /opt/etcd
tree .
[root@k8s-master1 etcd]# tree .
.
├── bin
│   ├── etcd
│   └── etcdctl
├── cfg
│   └── etcd.conf
└── ssl├── ca-key.pem├── ca.pem├── etcd-key.pem└── etcd.pem3 directories, 7 files
[root@k8s-master1 etcd]#

5、将节点etcd-1上的文件拷贝到节点2、3上

• 说明
  1、 etcd-1上面的文件是已经生成好的,直接传输到需要部署etcd的服务器上即可
  2、一定记得修改对应的ip配置等信息
  3、节点etcd-2和etcd-3文件一直，需要更改为自己的ip地址

1、复制文件

#在master其他节点创建工作目录
#mkdir -vp /opt/etcd
#节点1复制到节点2上
scp -r /opt/etcd/* root@192.168.11.166:/opt/etcd
scp /usr/lib/systemd/system/etcd.service root@192.168.11.166:/usr/lib/systemd/system/
#节点1复制到节点3上
scp -r /opt/etcd/* root@192.168.11.164:/opt/etcd
scp /usr/lib/systemd/system/etcd.service root@192.168.11.164:/usr/lib/systemd/system/

2、修改对应的ip地址

vi /opt/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1" # 修改此处，节点2 改为etcd-2，节点3 改为etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
#将192.168.11.169修改为当前服务器的ip地址
ETCD_LISTEN_PEER_URLS="https://192.168.11.169:2380" # 修改此处为当前服务器IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.11.169:2379" # 修改此处为当前服务器IP
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.11.169:2380" # 修改此处为当前服务器IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.11.169:2379" # 修改此处为当前服务器IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.11.169:2380,etcd-2=https://192.168.11.166:2380,etcd-3=https://192.168.11.164:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

6、启动并设置开机启动

节点都执行启动命令

systemctl daemon-reload
#启动etcd
systemctl start etcd
#开机启动
systemctl enable etcd
#查看启动状态
systemctl status etcd

7、记录错误

1、某一台服务一直启动不起起来

request sent was ignored (cluster ID mismatch: remote[9e5055697e

解决办法是：在配置的ETCD_DATA_DIR删除下面的文件

8、集群状态查看

• 里面的ip由于做了很多次、ip修改为自己的正确ip

1、验证etcd集群状态

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 endpoint health

2、集群状态性能

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 check perf

3、集群状态成员列表

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 member list

4、查看集群状态(主从状态)

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --write-out=table --cacert=/opt/etcd/ssl/ca.pem \
--cert=/opt/etcd/ssl/etcd.pem --key=/opt/etcd/ssl/etcd-key.pem \
--endpoints=https://192.168.11.173:2379,https://192.168.11.174:2379,https://192.168.11.175:2379 endpoint status