【博客628】k8s pod访问集群外域名原理以及主机开启了systemd-resolved的不同情况

k8s pod访问集群外域名原理以及使用了systemd-resolved的不同情况

1、不同情况下的linux主机访问外部域名原理

没有使用systemd-resolved的linux主机上访问外部域名一般是按照以下步骤来的：

• 从dns缓存里查找域名与ip的映射关系

• 从/etc/hosts里查找域名与ip的映射关系

• 从/etc/resolv.conf里查找dns server，并发起解析请求

  /etc/resolv.conf的内容一般如下：
  nameserver 8.8.8.8

使用systemd-resolved的linux主机上访问外部域名一般是按照以下步骤来的：

• 从dns缓存里查找域名与ip的映射关系

• 从/etc/hosts里查找域名与ip的映射关系

• 将dns解析请求发给本地systemd-resolved，由其去代理处理，因为systemd-resolved修改了 /etc/resolv.conf，使得本地解析请求全部发到127.0.0.1:53

  此时/etc/resolv.conf的内容一般如下：

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.nameserver 127.0.0.53
options edns0 trust-ad

• 然后systemd-resolved再根据/run/systemd/resolve/resolv.conf里面的dns server去发起请求

  /run/systemd/resolve/resolv.conf记录的就是真正的后端dns server

cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.nameserver 8.8.8.8
nameserver 4.4.4.4

2、pod内访问集群内service域名

当pod启动的时候，一般用的是dnsPolicy: ClusterFirst，此时就会将pod的/etc/resolv.conf改为集群内coredns的地址，此时将解析请求发给coredns，由其代理处理：

集群内coredns的service ip：

kubectl get svc -n kube-systemNAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   14d

pod内的/etc/resolv.conf在启动的时候被指定dns server为coredns service ip：

# cat /etc/resolv.conf in pod
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

3、pod内访问集群外service域名

• 使用dnsPolicy: ClusterFirst时：

  pod内的/etc/resolv.conf在启动的时候被指定dns server为coredns service ip。
  coredns的默认配置如下，此时如果用在集群内找不到这个service域名，就会用forward去转发请求，此时默认配置的是使用coredns内的 /etc/resolv.conf文件里的dns server

coredns default config：

    .:53 {logerrorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {pods insecurefallthrough in-addr.arpa ip6.arpattl 30}prometheus :9153forward . /etc/resolv.conf {max_concurrent 1000}cache 30loopreloadloadbalance}

• 使用dnsPolicy: Default时：

  这种方式其实是让 kubelet 来决定使用何种 DNS 策略。而 kubelet 默认的方式，就是使用宿主机的 /etc/resolv.conf

  简述： pod将dns代理到coredns，coredns使用kubelet的resolv指定的conf里面的内容来解析集群外的ip

4、coredns pod内的/etc/resolv.conf为什么有时跟主机/etc/resolv.conf不一致

场景：

当主机使用systemd-resolved来代理dns解析请求的时候，此时coredns pod内的/etc/resolv.conf跟主机/etc/resolv.conf不一致。

coredns pod内的/etc/resolv.conf：

cat /run/systemd/resolve/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.nameserver 8.8.8.8
nameserver 4.4.4.4

主机/etc/resolv.conf：

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.nameserver 127.0.0.53
options edns0 trust-ad

coredns pod内的/etc/resolv.conf为什么有时跟主机/etc/resolv.conf不一致的原因：

coredns的pod使用的是dnsPolicy: Default，此时就会使用kubelet指定的resolvConf的地址，默认是/etc/resolv.conf，但是当主机使用了systemd-resolved，则kubelet的的resolvConf变成了：resolvConf: /run/systemd/resolve/resolv.conf，也就是systemd-resolved存放真实后端dns server的文件路径，所以其实coredns的pod使用的是/run/systemd/resolve/resolv.conf里的真实后端dns server

使用了systemd-resolved的主机，kubelet使用/run/systemd/resolve/resolv.conf，而不用/etc/resolv.conf的原因：

• 如果coredns也是用/etc/resolv.conf，则集群里的dns解析请求都要代理到systemd-resolved，如果systemd-resolved挂了或者更新，那上层k8s集群里的dns解析也全部受到影响

• 这里面会有循环依赖的问题，参考coredns的官方文档解析：

  Troubleshooting Loops In Kubernetes Clusters

  A common cause of forwarding loops in Kubernetes clusters is an interaction with a local DNS cache on the host node (e.g. systemd-resolved). For example, in certain configurations systemd-resolved will put the loopback address 127.0.0.53 as a nameserver into /etc/resolv.conf. Kubernetes (via kubelet) by default will pass this /etc/resolv.conf file to all Pods using the default dnsPolicy rendering them unable to make DNS lookups (this includes CoreDNS Pods). CoreDNS uses this /etc/resolv.conf as a list of upstreams to forward requests to. Since it contains a loopback address, CoreDNS ends up forwarding requests to itself.

  简述： coredns转到127.0.0.53，此时源目ip都是自己，自己在给自己转，就会有循环问题

5、在node上如何访问集群内的service域名

• 通用方法：直接修改网卡interface的配置文件，在里面配上DNS的解析server

• 不通用方法：linux没有使用systemd-resolved时：

  在/etc/resolv.conf里加入coredns的service ip

• 不通用方法：linux使用systemd-resolved时：

  • 在/etc/systemd/resolved.conf里加入coredns的service ip
    [Resolve]
    DNS=10.96.0.10
    #FallbackDNS=
    #Domains=
    #LLMNR=no
    #MulticastDNS=no
    #DNSSEC=no
    #DNSOverTLS=no
    #Cache=no-negative
    #DNSStubListener=yes
    #ReadEtcHosts=yes
  • systemctl restart systemd-resolved.service
  • systemd-resolve --status查看结果
    Global
    LLMNR setting: no
    MulticastDNS setting: no
    DNSOverTLS setting: no
    DNSSEC setting: no
    DNSSEC supported: no
    DNS Servers: 10.96.0.10
    DNSSEC NTA: 10.in-addr.arpa
    16.172.in-addr.arpa
    168.192.in-addr.arpa
    17.172.in-addr.arpa
    18.172.in-addr.arpa
    19.172.in-addr.arpa
    20.172.in-addr.arpa
    21.172.in-addr.arpa
    …
    …

• 注意：不可以直接改/etc/resolv.conf，否则重启后丢失配置，因为此时/etc/resolv.conf被systemd-resolved接管，每次重启由其来生成其中的内容

example：

# 指定使用coredns来解析集群内service：
root@:/home/ubuntu# nslookup vmselect-example-vmcluster-persistent.default.svc.cluster.local 10.96.0.10
Server:        10.96.0.10
Address:    10.96.0.10#53Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.0.5
Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.1.3
Name:    vmselect-example-vmcluster-persistent.default.svc.cluster.local
Address: 10.244.1.4# /etc/resolv.conf加入：
nameserver 10.96.0.10# node上直接访问集群内service域名，此时会转到coredns去解析
root@:/home/ubuntu# curl vmselect-example-vmcluster-persistent.default.svc.cluster.local:8481/metrics
...
flag{name="promscrape.suppressScrapeErrors", value="false", is_set="false"} 1
flag{name="promscrape.suppressScrapeErrorsDelay", value="0s", is_set="false"} 1
flag{name="promscrape.yandexcloudSDCheckInterval", value="30s", is_set="false"} 1
flag{name="pushmetrics.extraLabel", value="", is_set="false"} 1
flag{name="pushmetrics.interval", value="10s", is_set="false"} 1
flag{name="pushmetrics.url", value="secret", is_set="false"} 1
flag{name="replicationFactor", value="1", is_set="false"} 1
flag{name="search.cacheTimestampOffset", value="5m0s", is_set="false"} 1
flag{name="search.denyPartialResponse", value="false", is_set="false"} 1
...
...

6、使用了systemd-resolved的主机，如果关闭systemd-resolved则机器的dns解析都会不同，即使主机能通后端dns server

ubuntu@:~$ sudo systemctl stop systemd-resolved.service
ubuntu@:~$ nslookup www.baidu.com
^C
ubuntu@:~$ dig www.baidu.com
^C
ubuntu@:~$ host www.baidu.com
^C

7、主机如何修改dns server

• 没有systemd-resolved，直接修改/etc/resolv.conf
• 如果机器装了systemd-resolved，那么就不可以直接改/etc/resolv.conf，则改法如下：

以加上8.8.8.8为例

root@:/home/ubuntu# cat /etc/systemd/resolved.conf
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details[Resolve]
DNS=8.8.8.8
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSOverTLS=no
#Cache=no-negative
#DNSStubListener=yes
#ReadEtcHosts=yessystemctl restart systemd-resolved.serviceroot@:/home/ubuntu# systemd-resolve --status
GlobalLLMNR setting: no
MulticastDNS setting: noDNSOverTLS setting: noDNSSEC setting: noDNSSEC supported: noDNS Servers: 8.8.8.8DNSSEC NTA: 10.in-addr.arpa16.172.in-addr.arpa168.192.in-addr.arpa17.172.in-addr.arpa18.172.in-addr.arpa19.172.in-addr.arpa20.172.in-addr.arpa21.172.in-addr.arpa......