TiDB 优雅关闭

背景

今天使用tiup做实验的事后，将tidb节点从2个缩到1个，发现tiup返回成功但是tidb-server进程还在。

这就引发的我的好奇心，why？

实验复现

启动集群

#( 07/31/23@ 8:32下午 )( happy@ZBMAC-f298743e3 ):~/docker/tiup/tiproxytiup playground v6.4.0 --db 2 --kv 1 --pd 1 --tiflash 0 --without-monitor --db.config tidb.toml
tiup is checking updates for component playground ...
Starting component `playground`: /Users/happy/.tiup/components/playground/v1.12.5/tiup-playground v6.4.0 --db 2 --kv 1 --pd 1 --tiflash 0 --without-monitor --db.config tidb.toml
Start pd instance:v6.4.0
Start tikv instance:v6.4.0
Start tidb instance:v6.4.0
Start tidb instance:v6.4.0
Waiting for tidb instances ready
127.0.0.1:4000 ... Done
127.0.0.1:4001 ... Done🎉 TiDB Playground Cluster is started, enjoy!Connect TiDB:   mysql --comments --host 127.0.0.1 --port 4000 -u root
Connect TiDB:   mysql --comments --host 127.0.0.1 --port 4001 -u root
TiDB Dashboard: http://127.0.0.1:2379/dashboard

查看节点信息

#( 07/31/23@ 8:32下午 )( happy@ZBMAC-f298743e3 ):~tiup playground display
tiup is checking updates for component playground ...
Starting component `playground`: /Users/happy/.tiup/components/playground/v1.12.5/tiup-playground display
Pid    Role  Uptime
---    ----  ------
10113  pd    49.376485092s
10114  tikv  49.32262974s
10115  tidb  49.283144092s
10116  tidb  49.245069308s

缩掉一个tidb节点

#( 07/31/23@ 8:34下午 )( happy@ZBMAC-f298743e3 ):~tiup playground scale-in --pid 10115
tiup is checking updates for component playground ...
Starting component `playground`: /Users/happy/.tiup/components/playground/v1.12.5/tiup-playground scale-in --pid 10115
scale in tidb success

这里可以看到已经返回了 scale in tidb success

查看进程

#( 07/31/23@ 8:34下午 )( happy@ZBMAC-f298743e3 ):~ps -ef | grep 10115502 11371 99718   0  8:34下午 ttys001    0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox 10115502 10115 10111   0  8:32下午 ttys005    0:04.29 /Users/happy/.tiup/components/tidb/v6.4.0/tidb-server -P 4000 --store=tikv --host=127.0.0.1 --status=10080 --path=127.0.0.1:2379 --log-file=/Users/happy/.tiup/data/TlaeoSj/tidb-0/tidb.log --config=/Users/happy/.tiup/data/TlaeoSj/tidb-0/tidb.toml

进程还是存在

分析

于是查看了 v6.4.0 的 tidb-server 代码。首先想到去main函数看下close的流程

main

func main() {//..signal.SetupSignalHandler(func(graceful bool) {svr.Close()cleanup(svr, storage, dom, graceful)cpuprofile.StopCPUProfiler()close(exited)})// ...
}

在这里发现两个重要的逻辑 svr.Close()，cleanup(svr, storage, dom, graceful)

svr.Close()

// Close closes the server.
func (s *Server) Close() {s.startShutdown()s.rwlock.Lock() // prevent new connectionsdefer s.rwlock.Unlock()if s.listener != nil {err := s.listener.Close()terror.Log(errors.Trace(err))s.listener = nil}if s.socket != nil {err := s.socket.Close()terror.Log(errors.Trace(err))s.socket = nil}if s.statusServer != nil {err := s.statusServer.Close()terror.Log(errors.Trace(err))s.statusServer = nil}if s.grpcServer != nil {s.grpcServer.Stop()s.grpcServer = nil}if s.autoIDService != nil {s.autoIDService.Close()}if s.authTokenCancelFunc != nil {s.authTokenCancelFunc()}s.wg.Wait()metrics.ServerEventCounter.WithLabelValues(metrics.EventClose).Inc()
}func (s *Server) startShutdown() {s.rwlock.RLock()logutil.BgLogger().Info("setting tidb-server to report unhealthy (shutting-down)")s.inShutdownMode = trues.rwlock.RUnlock()// give the load balancer a chance to receive a few unhealthy health reports// before acquiring the s.rwlock and blocking connections.waitTime := time.Duration(s.cfg.GracefulWaitBeforeShutdown) * time.Secondif waitTime > 0 {logutil.BgLogger().Info("waiting for stray connections before starting shutdown process", zap.Duration("waitTime", waitTime))time.Sleep(waitTime)}
}

从上面的逻辑可以看到，close的时候先startShutdown再进行资源回收。而在执行startShutdown的时候，居然有个time.Sleep(waitTime)。

然后研究下 graceful-wait-before-shutdown 参数，发现参数是0，不是此处导致的。

在 TiDB 等待服务器关闭期间，HTTP 状态会显示失败，使得负载均衡器可以重新路由流量
默认值：0
指定关闭服务器时 TiDB 等待的秒数，使得客户端有时间断开连接。

cleanup()

在 cleanup 中看到了 GracefulDown 和 TryGracefulDown 两个方法

func cleanup(svr *server.Server, storage kv.Storage, dom *domain.Domain, graceful bool) {if graceful {done := make(chan struct{})svr.GracefulDown(context.Background(), done)} else {svr.TryGracefulDown()}plugin.Shutdown(context.Background())closeDomainAndStorage(storage, dom)disk.CleanUp()topsql.Close()
}

TryGracefulDown

研究发现使用 SIGHUP 终止进程时使用 TryGracefulDown 方法，其他时候使用 GracefulDown。对比 TryGracefulDown 和 GracefulDown 实现， TryGracefulDown 只是多个15s的超时处理，底层逻辑还是 GracefulDown

var gracefulCloseConnectionsTimeout = 15 * time.Second// TryGracefulDown will try to gracefully close all connection first with timeout. if timeout, will close all connection directly.
func (s *Server) TryGracefulDown() {ctx, cancel := context.WithTimeout(context.Background(), gracefulCloseConnectionsTimeout)defer cancel()done := make(chan struct{})go func() {s.GracefulDown(ctx, done)}()select {case <-ctx.Done():s.KillAllConnections()case <-done:return}
}

GracefulDown

下面是 GracefulDown 实现，原来在这里会间隔1s，一直判断客户端连接是否存在，如果不存在才退出。

// GracefulDown waits all clients to close.
func (s *Server) GracefulDown(ctx context.Context, done chan struct{}) {logutil.Logger(ctx).Info("[server] graceful shutdown.")metrics.ServerEventCounter.WithLabelValues(metrics.EventGracefulDown).Inc()count := s.ConnectionCount()for i := 0; count > 0; i++ {s.kickIdleConnection()count = s.ConnectionCount()if count == 0 {break}// Print information for every 30s.if i%30 == 0 {logutil.Logger(ctx).Info("graceful shutdown...", zap.Int("conn count", count))}ticker := time.After(time.Second)select {case <-ctx.Done():returncase <-ticker:}}close(done)
}

ConnectionCount

判断连接个数的逻辑也很简单，就是对算下 s.clients 的 length

// ConnectionCount gets current connection count.
func (s *Server) ConnectionCount() int {s.rwlock.RLock()cnt := len(s.clients)s.rwlock.RUnlock()return cnt
}

其中还有一个奇怪的函数 kickIdleConnection，这个是做什么的？

kickIdleConnection

看逻辑是收集可以被close的会话然后close掉。

func (s *Server) kickIdleConnection() {var conns []*clientConns.rwlock.RLock()for _, cc := range s.clients {if cc.ShutdownOrNotify() {// Shutdowned conn will be closed by us, and notified conn will exist themselves.conns = append(conns, cc)}}s.rwlock.RUnlock()for _, cc := range conns {err := cc.Close()if err != nil {logutil.BgLogger().Error("close connection", zap.Error(err))}}
}

那么什么样的会话可以被close呢？

ShutdownOrNotify

有三类：

• client 状态处于 ServerStatusInTrans；
• 状态处于 connStatusReading
• 以及处于 connStatusDispatching 在 clientConn.Run 方法中被回收

// ShutdownOrNotify will Shutdown this client connection, or do its best to notify.
func (cc *clientConn) ShutdownOrNotify() bool {if (cc.ctx.Status() & mysql.ServerStatusInTrans) > 0 {return false}// If the client connection status is reading, it's safe to shutdown it.if atomic.CompareAndSwapInt32(&cc.status, connStatusReading, connStatusShutdown) {return true}// If the client connection status is dispatching, we can't shutdown it immediately,// so set the status to WaitShutdown as a notification, the loop in clientConn.Run// will detect it and then exit.atomic.StoreInt32(&cc.status, connStatusWaitShutdown)return false
}const (connStatusDispatching int32 = iotaconnStatusReadingconnStatusShutdown     // Closed by server.connStatusWaitShutdown // Notified by server to close.
)

破案

通过上面的分析，我们注意到了处于 ServerStatusInTrans 状态的连接不会被关闭，然后连接该节点执行show processlist发现的确有个处于事务中的会话

mysql> show processlist;
+---------------------+------+-----------------+------+---------+------+----------------------------+------------------+
| Id                  | User | Host            | db   | Command | Time | State                      | Info             |
+---------------------+------+-----------------+------+---------+------+----------------------------+------------------+
| 7794237818187809175 | root | 127.0.0.1:61293 | a    | Query   |    0 | in transaction; autocommit | show processlist |
+---------------------+------+-----------------+------+---------+------+----------------------------+------------------+
1 row in set (0.00 sec)

平时mysql使用的多，mysql在关闭的时候不管会话处于什么阶段，不管不顾直接停服，而tidb的这样处理着实让我想不到。

总结

本文简短的分析了下 tidb 进程关闭的处理流程，最终定位到进程没有及时关闭的原因。

对比于mysql的停服行为，让我们对tidb的处理方式有了不一样的理解。

对于 “graceful-wait-before-shutdown 参数”、“停服时等待事务结束的逻辑”的确需要在实践中才能积累。