ETCD集群搭建（实践可用）

概述

    etcd 是兼具一致性和高可用性的键值数据库，可以作为保存 Kubernetes 所有集群数据的后台数据库。

- 官方网址：

    Documentation versions | etcd

准备cfssl证书生成工具

    cfssl是一个开源的证书管理工具，使用json文件生成证书.

在任意一台服务器上操作，这里选择k8s01

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

创建etcd相关目录

$ mkdir -pv /opt/kubernetes/etcd/{bin,cfg,ssl,data}

创建证书

4.1 创建ca证书json文件

$ cd /opt/kubernetes/etcd/ssl

# 创建ca-config

$ vim ca-config.json
{"signing": {"default": {"expiry": "87600h"},"profiles": {"www": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}

# 创建ca-csr

$ vim ca-csr.json 
{"CN": "etcd CA","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing"}]
}

4.2 生成ca证书

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

4.3 使用自签ca签发etcd证书

创建证书申请文件：

$ cd /opt/kubernetes/etcd/ssl

# 注意hosts内容，etcd集群内的ip都要写上，可以预留几个，为以后扩容使用

{"CN": "etcd","hosts": ["10.10.21.73","10.10.21.74","10.10.21.75"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing"}]
}

生成证书

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

# 会生成server.pem和server-key.pem文件 

$ ll |grep server
-rw-r--r-- 1 root root 1013 Sep 14 15:06 server.csr
-rw-r--r-- 1 root root  290 Sep 14 15:05 server-csr.json
-rw------- 1 root root 1679 Sep 14 15:06 server-key.pem
-rw-r--r-- 1 root root 1338 Sep 14 15:06 server.pem

下载etcd二进制文件

下载地址

https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz

部署ETCD集群

6.1 复制二进制文件到指定文件

tar xf etcd-v3.5.0-linux-amd64.tar.gz
cp etcd-v3.5.0-linux-amd64/{etcd,etcdctl,etcdutl} /opt/kubernetes/etcd/bin

6.2 创建etcd配置文件

$vim /opt/kubernetes/etcd/cfg/etcd.conf
#[Member]
ETCD_NAME="etcd-1"    # k8s01为etcd-1，k8s02为etcd-2。。。每个节点唯一标识符
ETCD_DATA_DIR="/opt/kubernetes/etcd/data/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.241:2380"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.241:2379"    # 修改对应ip，k8s01为241，k8s02为242...
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.1.241:2380,etcd-2=https://192.168.1.242:2380,etcd-3=https://192.168.1.243:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

注释：

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_ADVERTISE_PEERURLS：集群通告地址

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIALCLUSTER_TOKEN：集群Token

ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

6.3 创建systemd文件

$ vim /etc/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.targe[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/etcd/cfg/etcd.conf
ExecStart=/opt/kubernetes/etcd/bin/etcd \
--cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/kubernetes/etcd/ssl/server.pem \
--peer-key-file=/opt/kubernetes/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/kubernetes/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.target

分发etcd文件

node02

$ scp -r /opt/kubernetes k8s02:/opt/
$ scp /etc/systemd/system/etcd.service k8s02:/etc/systemd/system/
# 记得修改etcd配置文件

node03

$ scp -r /opt/kubernetes k8s03:/opt/
$ scp /etc/systemd/system/etcd.service k8s03:/etc/systemd/system/
# 记得修改etcd配置文件

8. 分别启动etcd服务

$ systemctl start etcd.service

查看集群状态

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" endpoint health --write-out=table+--------------------------+--------+-------------+-------+
|         ENDPOINT         | HEALTH |    TOOK     | ERROR |
+--------------------------+--------+-------------+-------+
| https://10.10.21.75:2379 |   true | 13.407895ms |       |
| https://10.10.21.74:2379 |   true |  13.61133ms |       |
| https://10.10.21.73:2379 |   true | 14.868649ms |       |
+--------------------------+--------+-------------+-------+

9. 测试

写入数据

[root@kubenode01 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" put foo bar
OK

读取数据

[root@kubenode02 etcd]# ./bin/etcdctl --cacert=/opt/kubernetes/etcd/ssl/ca.pem --cert=/opt/kubernetes/etcd/ssl/server.pem --key=/opt/kubernetes/etcd/ssl/server-key.pem --endpoints="https://10.10.21.73:2379,https://10.10.21.74:2379,https://10.10.21.75:2379" get foo
foo
bar