一起学习ETCD系列——运维操作之etcdctl使用

概要

本文主要用来总结ETCD客户端ctcdctl的命令操作，在运维过程中可能常常用到的。

一、命令

etcd工具
etcdctl官方命令示例

[root@test etcd-v3.5.11]# ./etcdctl -h
NAME:etcdctl - A simple command line client for etcd3.USAGE:etcdctl [flags]VERSION:3.5.11API VERSION:3.5COMMANDS:  #命令alarm disarm            Disarms all alarms  #关闭告警alarm list              Lists all alarms    #列出告警信息auth disable            Disables authentication #关闭认证auth enable             Enables authentication #开启认证auth status             Returns authentication status #查看认证状态check datascale         Check the memory usage of holding data for different workloads on a given server endpoint. #检查给定实例的内存使用情况check perf              Check the performance of the etcd cluster #检查etcd集群的性能compaction              Compacts the event history in etcd  #压缩etcd的历史事件defrag                  Defragments the storage of the etcd members with given endpoints #整理给定实例的存储碎片del                     Removes the specified key or range of keys [key, range_end) #删除keyelect                   Observes and participates in leader election #加入leader选举endpoint hashkv         Prints the KV history hash for each endpoint in --endpoints  #打印给定若干实例KV history hashendpoint health         Checks the healthiness of endpoints specified in `--endpoints` flag #检查给定若干实例健康endpoint status         Prints out the status of endpoints specified in `--endpoints` flag  #打印给定若干实例状态get                     Gets the key or a range of keys #获取key值help                    Help about any command  #打印命令帮助信息lease grant             Creates leases  #创建一个租约lease keep-alive        Keeps leases alive (renew) #续约lease list              List all active leases  #列出所有租约lease revoke            Revokes leases   #撤销租约lease timetolive        Get lease information #获取租约详情，比如剩余有效期lock                    Acquires a named lock #获取命名锁make-mirror             Makes a mirror at the destination etcd cluster #指定一个etcd集群作为镜像集群member add              Adds a member into the cluster #添加一个成员到etcd集群member list             Lists all members in the cluster #列出etcd集群全部成员member promote          Promotes a non-voting member in the cluster #将成员从Follower状态到Candidate状态member remove           Removes a member from the cluster #移除成员member update           Updates a member in the cluster #更新成员move-leader             Transfers leadership to another etcd cluster member. #指定成员成为Leader状态put                     Puts the given key into the store #插入或更新keyrole add                Adds a new role #添加新角色role delete             Deletes a role  #删除角色role get                Gets detailed information of a role #获取角色的详细信息role grant-permission   Grants a key to a role #授予角色对key的权限，比如读写role list               Lists all roles  #列出全部的角色role revoke-permission  Revokes a key from a role #撤销角色对指定key的权限snapshot restore        Restores an etcd member snapshot to an etcd directory  #恢复快照到指定数据目录snapshot save           Stores an etcd node backend snapshot to a given file #保存快照到指定文件snapshot status         [deprecated] Gets backend snapshot status of a given file #[弃用] 获取给定快照文件的状态txn                     Txn processes all the requests in one transaction #开启事务user add                Adds a new user   #添加账户user delete             Deletes a user  #删除账户user get                Gets detailed information of a user #获取账户详情user grant-role         Grants a role to a user  #指定账户角色user list               Lists all users    #列出所有账户user passwd             Changes password of user  #修改账户密码user revoke-role        Revokes a role from a user #撤销账户角色version                 Prints the version of etcdctl  #打印ectd当前版本watch                   Watches events stream on keys or prefixes #监听一些key或共同前缀key的事件【】OPTIONS:  #命令的可选参数--cacert=""                               verify certificates of TLS-enabled secure servers using this CA bundle #服务端使用https时，使用ca文件进行验证--cert=""                                 identify secure client using this TLS certificate file #指定HTTPS的TLS证书文件--command-timeout=5s                      timeout for short running command (excluding dial timeout) #命令执行超时时间--debug[=false]                           enable client-side debug logging #启用客户端调试日志，默认false--dial-timeout=2s                         dial timeout for client connections #指定连接超时时间-d, --discovery-srv=""                        domain name to query for SRV records describing cluster endpoints #用于查询描述集群端点的srv记录的域名--discovery-srv-name=""                   service name to query when using DNS discovery #使用DNS发现时要查询的服务名称--endpoints=[127.0.0.1:2379]              gRPC endpoints #指定实例访问ip+端口-h, --help[=false]                            help for etcdctl--hex[=false]                             print byte strings as hex encoded strings #将字节字符串打印为十六进制编码字符串--insecure-discovery[=true]               accept insecure SRV records describing cluster endpoints #接受描述集群端点的不安全的srv记录--insecure-skip-tls-verify[=false]        skip server certificate verification (CAUTION: this option should be enabled only for testing purposes)#跳过服务器证书验证（注意：此选项应仅用于测试目的）--insecure-transport[=true]               disable transport security for client connections #客户端禁用安全传输--keepalive-time=2s                       keepalive time for client connections #客户端连接的keepalive时间--keepalive-timeout=6s                    keepalive timeout for client connections #客户端连接的keepalive超时时间--key=""                                  identify secure client using this TLS key file #使用此 TLS 密钥文件识别安全客户端--password=""                             password for authentication (if this option is used, --user option shouldn't include password) #用于身份验证的密码（如果使用此选项，--user 选项不应包含密码）--user=""                                 username[:password] for authentication (prompt if password is not supplied)#用于身份验证（如果未提供密码则提示）-w, --write-out="simple"                      set the output format (fields, json, protobuf, simple, table) #输出格式,默认simple，个人感觉table格式看着最舒服

二、实操

etcdctl [command] -h 可以查看具体命令的使用说明：

[root@test etcd-v3.5.11]# ./etcdctl lease -h
NAME:lease - Lease related commands
USAGE:etcdctl lease <subcommand> [flags]
API VERSION:3.5
COMMANDS:grant           Creates leaseskeep-alive      Keeps leases alive (renew)list            List all active leasesrevoke          Revokes leasestimetolive      Get lease information
OPTIONS:-h, --help[=false]    help for lease[root@test etcd-v3.5.11]# ./etcdctl lease grant -h
NAME:lease grant - Creates leases
USAGE:etcdctl lease grant <ttl> [flags]  #详细使用说明
OPTIONS:-h, --help[=false]    help for grant

2.1、基本操作

#插入key
[root@test etcd-v3.5.11]# ./etcdctl put /test/key1 value1
OK
#创建500s的租约，其id为43e58d2bb5a3102d
[root@test etcd-v3.5.11]# ./etcdctl lease grant 500
lease 43e58d2bb5a3102d granted with TTL(500s)
#获取租约详情
[root@test etcd-v3.5.11]# ./etcdctl lease timetolive 43e58d2bb5a3102d
lease 43e58d2bb5a3102d granted with TTL(500s), remaining(470s)
#插入key，并指定租约
[root@test etcd-v3.5.11]# ./etcdctl put /test/key2 value2 --lease=43e58d2bb5a3102d
OK
#获取
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1
/test/key1
value1
[root@test etcd-v3.5.11]# ./etcdctl get /test/key2
/test/key2
value2
#删除
[root@test etcd-v3.5.11]# ./etcdctl del /test/key2
1
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1 -w josn
{"header":{"cluster_id":11073961802084797819,"member_id":8518302759694844901,"revision":10769,"raft_term":15},"kvs":[{"key":"L3Rlc3Qva2V5MQ==","create_revision":10759,"mod_revision":10761,"version":3,"value":"dmFsdWUxNA=="}],"count":1}
#cluster_id：请求的etcd集群ID。
#member_id：请求的etcd节点ID。
#revision：etcd服务端当前全局数据版本号。对任一key的put或delete操作都会使revision自增1。revision=1是etcd的保留版本号，因此用户的key版本号将从2开始。
#raft_term：etcd当前raft主节点任期号。
#create_revision：当前key创建时全局数据版本号revision的值。
#mod_revision：当前key最后一次修改时全局数据版本号revision的值。
#version：当前key的数据版本号。key创建时version为1，对当前key进行put操作会使version自增1，将key删除后，重新创建，version又会从1开始计数。

2.2、watch

监听key，可以看到key的事件

[root@test etcd-v3.5.11]# ./etcdctl watch /test/key3
PUT
/test/key3
value3
PUT
/test/key3
value31
PUT
/test/key3
value33
DELETE
/test/key3
PUT
/test/key3
value35

操作key

[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value3
OK
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value31
OK
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value33
OK
[root@test etcd-v3.5.11]# ./etcdctl del /test/key3
1
[root@test etcd-v3.5.11]# ./etcdctl put /test/key3 value35
OK

2.3、租约

#创建500s的租约，其id为43e58d2bb5a3102d
[root@test etcd-v3.5.11]# ./etcdctl lease grant 500
lease 43e58d2bb5a3102d granted with TTL(500s)
#获取租约详情
[root@test etcd-v3.5.11]# ./etcdctl lease timetolive 43e58d2bb5a3102d
lease 43e58d2bb5a3102d granted with TTL(500s), remaining(470s)
#只续约一次
[root@test etcd-v3.5.11]# ./etcdctl lease keep-alive --once=true 43e58d2bb5a3102d
#一直自动续约
[root@test etcd-v3.5.11]# ./etcdctl lease keep-alive 43e58d2bb5a3102d
#删除租约
[root@test etcd-v3.5.11]# ./etcdctl lease revoke 43e58d2bb5a3102d

2.4、分布式锁

etcd中分布式锁的结构是：lock_key/lease_id。
实现过程（源码）如下：

1. 准备
   客户端连接 Etcd，以 /lock/mylock 为前缀创建全局唯一的 key，假设第一个客户端对应的 key=“/lock/mylock/UUID1”，第二个为 key=“/lock/mylock/UUID2”；客户端分别为自己的 key 创建租约 - Lease，租约的长度根据业务耗时确定，假设为 15s；
2. 创建租约
   当一个客户端持有锁期间，其它客户端只能等待，为了避免等待期间租约失效，客户端需创建一个定时任务作为“心跳”进行续约。此外，如果持有锁期间客户端崩溃，心跳停止，key 将因租约到期而被删除，从而锁释放，避免死锁。
3. 客户端写入自己全局唯一的 key
   通过 PUT 操作，将步骤 1 中创建的 key 绑定租约写入 Etcd，根据 Etcd 的 Revision 机制，假设两个客户端 put 操作返回的 Revision 分别为 1、2，客户端需记录 Revision 用以接下来判断自己是否获得锁。
4. 客户端判断是否获得锁
   客户端以前缀 /lock/mylock 读取 keyValue 列表（keyValue 中带有 key 对应的 Revision），判断自己 key 的 Revision 是否为当前列表中最小的，如果是则认为获得锁；否则监听列表中前一个 Revision 比自己小的 key 的删除事件，一旦监听到删除事件或者因租约失效而删除的事件，则自己获得锁。
5. 执行业务
   获得锁后，操作共享资源，执行业务代码。
6. 释放锁
   完成业务流程后，释放锁,通过删除对应的key完成。

测试：
开三个窗口

#获取锁
[root@test etcd-v3.5.11]# ./etcdctl lock /lock/test --ttl=5
/lock/test/43e58d437ddbc834

#等待锁
[root@test etcd-v3.5.11]# ./etcdctl lock /lock/test --ttl=5

#watch监控/lock/test为前缀的key变化
[root@kdzl etcd-v3.5.11]# ./etcdctl watch /lock/test --prefix
PUT
/lock/test/43e58d437ddbc834
PUT
/lock/test/43e58d437ddbc838

2.5、角色

[root@test etcd-v3.5.11]# ./etcdctl role add book
Role book created
[root@test etcd-v3.5.11]# ./etcdctl role add man
Role man created
[root@test etcd-v3.5.11]# ./etcdctl role list
book
man
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:
KV Write:
#给key授权角色
#etcdctl role grant-permission [options] <role name> <permission type> <key> [endkey] [flags]
[root@test etcd-v3.5.11]# ./etcdctl role grant-permission man read /test
Role man updated
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:/test
KV Write:
#移除key角色
[root@test etcd-v3.5.11]# ./etcdctl role revoke-permission man /test
Permission of key /test is revoked from role man
[root@test etcd-v3.5.11]# ./etcdctl role get man
Role man
KV Read:
KV Write:

2.6、用户

#创建用户
[root@test etcd-v3.5.11]# ./etcdctl user add root:root
User root created
[root@test etcd-v3.5.11]# ./etcdctl user add test --new-user-password=12345
User test created
[root@test etcd-v3.5.11]# ./etcdctl user add pick:25823
User pick created
#列出全部的用户
[root@test etcd-v3.5.11]# ./etcdctl user list
root
test
[root@test etcd-v3.5.11]# ./etcdctl user get test
User: test
Roles:
#给账号一个角色,一个账户可以有多个角色
[root@test etcd-v3.5.11]# ./etcdctl user grant-role test man
Role man is granted to user test
[root@test etcd-v3.5.11]# ./etcdctl user get test
User: test
Roles: man
#删除角色
[root@test etcd-v3.5.11]# ./etcdctl user del pick
User pick deleted
#撤销用户角色
[root@test etcd-v3.5.11]# ./etcdctl user revoke-role test man
Role man is revoked from user test

2.7、认证

启用auth需要一个root用户，并且root用户有root角色

[root@test etcd-v3.5.11]# ./etcdctl role add root
Role root created
[root@test etcd-v3.5.11]# ./etcdctl role grant-permission --prefix=true root readwrite /
Role root updated
[root@test etcd-v3.5.11]# ./etcdctl user grant-role root root
Role root is granted to user root
[root@test etcd-v3.5.11]# ./etcdctl auth enable
Authentication Enabled
[root@test etcd-v3.5.11]# ./etcdctl auth status --user=root:root
Authentication Status: true
AuthRevision: 15

换成 test 用户操作

[root@test etcd-v3.5.11]# ./etcdctl role grant-permission --prefix=true man read /test --user=root:root  #对/test前缀的key有只读权限
Role man updated
[root@test etcd-v3.5.11]# ./etcdctl role get man --user=root:root
Role man
KV Read:/test
KV Write:
[root@test etcd-v3.5.11]# ./etcdctl user grant-role test man --user=root:root
Role man is granted to user test
[root@test etcd-v3.5.11]# ./etcdctl get /test/key1 --user=test:12345
/test/key1
value14
[root@test etcd-v3.5.11]# ./etcdctl put /test/key5 value5 --user=test:12345
{"level":"warn","ts":"2024-01-25T23:38:54.753123+0800","logger":"etcd-client","caller":"v3@v3.5.11/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000362e00/127.0.0.1:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}
Error: etcdserver: permission denied

可以看到权限生效

2.8、集群

1：查看每个实例状态,即检查etcd集群状态

[root@test etcd-v3.5.11]# ./etcdctl endpoint status -w table --endpoints"=127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359"
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 7637173a60d743e5 |  3.5.11 |   21 MB |      true |      false |        15 |      10923 |              10923 |        |
| 127.0.0.1:2369 | ce6b0efed2935560 |  3.5.11 |   21 MB |     false |      false |        15 |      10923 |              10923 |        |
| 127.0.0.1:2359 | 6ea1523f71bf92f0 |  3.5.11 |   21 MB |     false |      false |        15 |      10923 |              10923 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

2:查看健康状态

#TOOK表示心跳
[root@test etcd-v3.5.11]# ./etcdctl endpoint health --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+--------+-------------+-------+
|    ENDPOINT    | HEALTH |    TOOK     | ERROR |
+----------------+--------+-------------+-------+
| 127.0.0.1:2369 |   true | 16.178532ms |       |
| 127.0.0.1:2379 |   true | 17.826437ms |       |
| 127.0.0.1:2359 |   true | 19.841588ms |       |
+----------------+--------+-------------+-------+
[root@test etcd-v3.5.11]# ./etcdctl endpoint hashkv --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+------------+
|    ENDPOINT    |    HASH    |
+----------------+------------+
| 127.0.0.1:2379 | 2618695019 |
| 127.0.0.1:2369 | 2618695019 |
| 127.0.0.1:2359 | 2618695019 |
+----------------+------------+

3：切换集群leader
原来集群leader实例是127.0.0.1:2379,现在通过move-leader命令将leader切换成实例127.0.0.1:2359

[root@test etcd-v3.5.11]# ./etcdctl move-leader 6ea1523f71bf92f0 --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359"
Leadership transferred from 7637173a60d743e5 to 6ea1523f71bf92f0
[root@test etcd-v3.5.11]# ./etcdctl endpoint status --endpoints="127.0.0.1:2379,127.0.0.1:2369,127.0.0.1:2359" -w table
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 7637173a60d743e5 |  3.5.11 |   21 MB |     false |      false |        19 |      10933 |              10933 |        |
| 127.0.0.1:2369 | ce6b0efed2935560 |  3.5.11 |   21 MB |     false |      false |        19 |      10933 |              10933 |        |
| 127.0.0.1:2359 | 6ea1523f71bf92f0 |  3.5.11 |   21 MB |      true |      false |        19 |      10933 |              10933 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

4：查看集群成员

[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+

5：添加成员（一个learner成员，不参与投票）

1）etcdctl member add 命令将新成员添加到集群

[root@testl etcd-v3.5.11]# ./etcdctl  member add  etcd04 --peer-urls="http://127.0.0.16:2350" --learner=true
Member 690b31e00158f43a added to cluster 99ae9d962d824d7bETCD_NAME="etcd4"
ETCD_INITIAL_CLUSTER="etcd4=http://127.0.0.1:2350,etcd3=http://127.0.0.1:2360,etcd1=http://127.0.0.1:2380,etcd2=http://127.0.0.1:2370"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://127.0.0.1:2350"
ETCD_INITIAL_CLUSTER_STATE="existing"
#可以看到该成员尚未启动
[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+-----------+--------+----------------------------+-----------------------+------------+
|        ID        |  STATUS   |  NAME  |         PEER ADDRS         |     CLIENT ADDRS      | IS LEARNER |
+------------------+-----------+--------+----------------------------+-----------------------+------------+
| 690b31e00158f43a | unstarted |       | http://127.0.0.1:2350 |                       |       true |
| 6ea1523f71bf92f0 |   started | etcd3 |      http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 |   started | etcd1 |      http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| ce6b0efed2935560 |   started | etcd2 |      http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+-----------+--------+----------------------------+-----------------------+------------+

2）启动新成员
回溯集群安装篇
配置文件如下：

mkdir -p /usr/local/etcd-v3.5.11/etcd4/data
vim /usr/local/etcd-v3.5.11/etcd4/conf.yml

name: etcd4
data-dir: /usr/local/etcd-v3.5.11/etcd4/data
initial-advertise-peer-urls: http://127.0.0.1:2350
listen-peer-urls: http://127.0.0.1:2350
listen-client-urls: http://172.20.101.222:2349,http://127.0.0.1:2349
advertise-client-urls: http://127.0.0.1:2349
initial-cluster-token: test-etcd-cluster
initial-cluster: etcd1=http://127.0.0.1:2380,etcd2=http://127.0.0.1:2370,etcd3=http://127.0.0.1:2360,etcd4=http://127.0.0.1:2350
initial-cluster-state: existing   #注意，这里一定要标记为existing   

/usr/local/etcd-v3.5.11/etcd --config-file /usr/local/etcd-v3.5.11/etcd4/conf.yml

可以看到已经成功加入集群

[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| c64a0c36c4a5869f | started | etcd4 | http://127.0.0.1:2350 | http://127.0.0.1:2349 |       true |
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+

6：learner成员提升为follower成员
实例http://127.0.0.1:2350还是learner，没有投票权，用member promote命令提升下

[root@test etcd-v3.5.11]# ./etcdctl  member  promote c64a0c36c4a5869f
Member c64a0c36c4a5869f promoted in cluster 99ae9d962d824d7b
[root@test etcd-v3.5.11]# ./etcdctl  member list -w table
+------------------+---------+--------+-----------------------+-----------------------+------------+
|        ID        | STATUS  |  NAME  |      PEER ADDRS       |     CLIENT ADDRS      | IS LEARNER |
+------------------+---------+--------+-----------------------+-----------------------+------------+
| 6ea1523f71bf92f0 | started | etcd3 | http://127.0.0.1:2360 | http://127.0.0.1:2359 |      false |
| 7637173a60d743e5 | started | etcd1 | http://127.0.0.1:2380 | http://127.0.0.1:2379 |      false |
| c64a0c36c4a5869f | started | etcd4 | http://127.0.0.1:2350 | http://127.0.0.1:2349 |      false |  #可以看到已经不是learner了
| ce6b0efed2935560 | started | etcd2 | http://127.0.0.1:2370 | http://127.0.0.1:2369 |      false |
+------------------+---------+--------+-----------------------+-----------------------+------------+