juju创建lxd容器时如何使用本地镜像(by quqi99)

作者：张华 发表于：2023-03-01
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

问题

没有外网，所以配置了一个local custom镜像库，也使用了container-image-metadata-url进行配置，但是用juju创建lxd容器时还是说找不着image.

相关搜索

关于container-image-metadata-url的代码如下：
https://github.com/juju/juju/pull/8578
https://github.com/juju/juju/blob/juju-2.9.35/container/lxd/manager.go#L282-L284
也有一个好帖子：
https://discourse.charmhub.io/t/local-lxd-image-server/3929/5

实验环境

1, 使用 juju创建一个focal的machine 0, 然后再machine 0上部署一个xenial的lxd容器。

juju add-model test
juju add-machine --series focal
juju model-config logging-config="<root>=DEBUG"
juju remove-application ceph-radosgw && juju deploy ceph-radosgw --series=xenial --to="lxd:0"

2, 在juju controller(juju ssh -m controller 0)与machine 0上运行下列iptables来模拟和cloud-images.ubuntu.com断网。这里我发现：

• machine 0的日志（/var/log/juju/machine-0.log）显示它好像是从juju controller处下载镜像的

2023-03-01 07:58:21 INFO juju.cloudconfig userdatacfg_unix.go:613 Fetching agent: curl -sSf --connect-timeout 20 --noproxy "*" --insecure -o $bin/tools.tar.gz <[https://10.5.0.31:17070/model/deb85179-10a6-4877-88f7-012ef768d726/tools/2.9.38-ubuntu-amd64 https://252.0.31.1:17070/model/deb85179-10a6-4877-88f7-012ef768d726/tools/2.9.38-ubuntu-amd64]>
2023-03-01 07:59:03 INFO juju.container.lxd container.go:256 starting new container "juju-68d726-0-lxd-2" (image "ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz")
2023-03-01 07:59:03 DEBUG juju.container.lxd container.go:257 new container has profiles [default]
2023-03-01 07:59:42 DEBUG juju.container.lxd container.go:286 created container "juju-68d726-0-lxd-2", waiting for start...

• 但如果不在machine 0上运行下载iptables，测试表明machine0也能直接扰开juju controller从cloud-images.ubuntu.com处下镜像.
• 似乎二者均相关，那就二者将运行下列iptables吧

dig cloud-images.ubuntu.com  #185.125.190.37 and 185.125.190.40
iptables -A OUTPUT -d 185.125.190.37 -j DROP
iptables -A OUTPUT -d 185.125.190.40 -j DROP

3, bastion上运行sstream-mirror将cloud-images.ubuntu.com中的xenial amd64镜像mirror了下来。

sudo apt -y install simplestreams -y
workdir=/home/ubuntu/simplestreams2
sudo sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg --progress --max=1 --path=streams/v1/index.json https://cloud-images.ubuntu.com/releases/ $workdir 'arch=amd64' 'release~(xenial)' 'ftype~(lxd.tar.xz|squashfs|root.tar.xz|root.tar.gz|disk1.img|.json|.sjson)'

然后用nginx为它配置了https:

#https://goharbor.io/docs/2.6.0/install-config/configure-https/
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=quqi.com" -key ca.key -out ca.crt
openssl genrsa -out quqi.com.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=quqi.com" -key quqi.com.key -out quqi.com.csr
#complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements to avoid 'x509: certificate relies on legacy Common Name field, use SANs instead'
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names[alt_names]
DNS.1=quqi.com
DNS.2=quqi
DNS.3=hostname
EOF
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in quqi.com.csr -out quqi.com.crt
#for docker, the Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.
openssl x509 -inform PEM -in quqi.com.crt -out quqi.com.cert
curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve quqi.com:443:10.5.0.126 https://quqi.com:443/streams/v1/index.json$ cat /etc/nginx/sites-available/default
server {listen 443 ssl http2;listen [::]:443 ssl http2;server_name quqi.com;ssl_certificate /home/ubuntu/ca/quqi.com.crt;ssl_certificate_key /home/ubuntu/ca/quqi.com.key;ssl_protocols TLSv1.2;ssl_prefer_server_ciphers on; location / {root /home/ubuntu/simplestreams2;index index.html;}  
}
# 注意：由于上面使用了一个新目录/home/ubuntu/simplestreams2作为root，那需要将/etc/nginx/nginx.conf中添加'user root;'来避免权限问题
#curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/images/streams/v1/index.json
curl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json

4, 配置juju中的container-image-metadata-url使用上面的https based local image mirror

juju model-config container-image-metadata-url=https://quqi.com:443
juju model-config image-metadata-url=https://quqi.com:443

5, juju controller由于访问local image mirror, 所以配置hosts与添加ca key

echo '10.5.0.126 quqi.com' >> /etc/hostscurl --resolve quqi.com:443:10.5.0.126 --cacert ~/ca/ca.crt https://quqi.com:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve quqi.com:443:10.5.0.126 https://quqi.com:443/streams/v1/index.json

6, 记得重新测试之前将machine 0上的image cache删除

juju ssh 0 -- sudo lxc image delete juju/xenial/amd64
juju remove-application ceph-radosgw

7, 重新测试

juju deploy ceph-radosgw --series=xenial --to="lxd:0"
sudo tail -f /var/log/juju/machine-0.log

能在machine 0的/var/log/juju/machine-0.log中观察下列日志:

2023-03-01 08:26:45 WARNING juju.worker.lxdprovisioner provisioner_task.go:1371 machine 0/lxd/3 failed to start: acquiring LXD image: no matching image found
2023-03-01 08:26:45 WARNING juju.worker.lxdprovisioner provisioner_task.go:1410 failed to start machine 0/lxd/3 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)

在juju controller上有时能搜到quqi, 有时候又不能，奇怪.

2023-02-23 07:33:52 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered "https://quqi.com:443/images/streams/v1/streams/v1/index.json": Get "https://quqi.com:443/images/streams/v1/streams/v1/index.json": dial tcp 49.234.171.74:443: i/o timeout while getting published images metadata from image-metadata-url
2023-03-01 08:52:56 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting "https://quqi.com:443/streams/v1/index.json": Get "https://quqi.com:443/streams/v1/index.json": x509: certificate relies on legacy Common Name field, use SANs instead

juju controller上仍然能看到cloud-images.ubuntu.com

2023-03-01 08:34:54 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": dial tcp 185.125.190.37:80: i/o timeout while getting published images metadata from default ubuntu cloud images

使用glance中的image来提供simplestreams

上面是使用来提供simplestreams, 我们现在换用glance中的image来提供simplestreams继续测试 （不确定是否这种只适用于创建juju controller， 还是说也可以用于VM/LXD创建，试一下)

mkdir -p ~/simplestreams/images
IMAGE_ID=26751c0e-4282-415e-b8dc-a7a21d2f781d
SERIES=xenial
juju metadata generate-image -d ~/simplestreams -i $IMAGE_ID -s $SERIES -r RegionOne -u $OS_AUTH_URL

然后修改/etc/nginx/sites-available/default将上面测试用的/home/ubuntu/simplestreams2改成/home/ubuntu/simplestreams， 重启nginx之后, 设置container-image-metadata-url (注意：此时后面链接多出了/images)

juju model-config container-image-metadata-url=https://quqi.com:443/images
juju model-config image-metadata-url=https://quqi.com:443/images

为保证测试环境干净，我也在controller与machine0上运行了下列命令：

systemctl restart jujud-machine-0.service

然后重复测试后，问题依旧， controller上看到下列日志：

2023-03-01 10:29:32 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting "https://streams.canonical.com/juju/tools/streams/v1/index.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index.sjson": dial tcp 185.125.190.37:443: i/o timeout
2023-03-01 10:29:36 INFO juju.state addmachine.go:505 new machine "0/lxd/11" has preferred addresses: private "", public ""
2023-03-01 10:29:37 WARNING juju.apiserver.instancemutater lxdprofilewatcher.go:206 unit ceph-radosgw/11 has no machine id, start watching when machine id assigned.
2023-03-01 10:29:41 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered index file has no data for cloud {stsstack http://10.230.19.53:5000/v3} not found while getting published images metadata from image-metadata-url
2023-03-01 10:30:11 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp 185.125.190.40:80: i/o timeout
2023-03-01 10:30:41 WARNING juju.environs.simplestreams datasource.go:184 Got error requesting "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": dial tcp 185.125.190.37:80: i/o timeout
2023-03-01 10:30:41 WARNING juju.apiserver.provisioner provisioninginfo.go:801 encountered "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson": dial tcp 185.125.190.37:80: i/o timeout while getting published images metadata from default ubuntu cloud images

看样子和simplestreams类型无关。

测试cloudinit-userdata

即然与simplestreams类型无关，那我们将nginx再恢复之前的/home/ubuntu/simplestreams2

juju model-config container-image-metadata-url=https://quqi.com:443/
juju model-config image-metadata-url=https://quqi.com:443/

然后测试cloudinit-userdata, 这个是没问题的，可以作workaround

cat << EOF |tee cloudinit-userdata.yaml
cloudinit-userdata: |postruncmd:- echo '10.5.0.126 quqi.com' >> /etc/hosts- if hostname |grep -qv lxd; then wget --tries=15 --retry-connrefused --timeout=15 --random-wait=on -O /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz https://quqi.com:443/server/releases/xenial/release-20211001/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz --no-check-certificate; wget --tries=15 --retry-connrefused --timeout=15 --random-wait=on -O /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64.squashfs https://quqi.com:443/server/releases/xenial/release-20211001/ubuntu-16.04-server-cloudimg-amd64.squashfs --no-check-certificate; fi- sleep 30- if hostname |grep -qv lxd; then lxc image import /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64-lxd.tar.xz /home/ubuntu/ubuntu-16.04-server-cloudimg-amd64.squashfs --alias juju/xenial/amd64; fi
EOF
juju model-config ./cloudinit-userdata.yaml
juju model-config cloudinit-userdata --format yaml
#juju model-config --reset cloudinit-userdata

注意：之前一直不work的原因是因为在postruncmd:后加了 | 的原因，找到答案的过程见下列的"调试cloud-init"一节。

调试cloud-init

最后发现用下面的是不work的：

cat << EOF |tee test.yaml
cloudinit-userdata: |postruncmd: |- echo '10.5.0.126 quqi.com' >> /etc/hosts- echo 'test' > /home/ubuntu/cloud-init.txt
EOF

需要改成下面的：

cat << EOF |tee test.yaml
cloudinit-userdata: |postruncmd:- bash -c 'echo 10.5.0.126 quqi.com >> /etc/hosts'- bash -c 'echo test > /home/ubuntu/cloud-init.txt'
EOF

下面的也不会work

cat << EOF |tee test.yaml
cloudinit-userdata: |postruncmd: |bash -c 'echo 10.5.0.126 quqi.com >> /etc/hosts'bash -c 'echo test > /home/ubuntu/cloud-init.txt'
EOF

下面的更不会work, 会直接报：ERROR json: unsupported type: map[interface {}]interface {}’

cat << EOF |tee test.yaml
cloudinit-userdata: |postruncmd:bash -c 'echo 10.5.0.126 quqi.com >> /etc/hosts'bash -c 'echo test > /home/ubuntu/cloud-init.txt'
EOF

其他调试方法如下：

juju add-model test
juju model-config ./test.yaml
juju model-config cloudinit-userdata --format yaml
juju add-machine --series focal1, check cloud-init log:    cloud-init collect-logs & tar -xf cloud-init.tar.gz
2, check cloud-init config: /etc/cloud/cloud.cfg
3, cloud-init is enabled: systemctl list-unit-files | grep cloud
4, /var/lib/cloud/instances/af2d721e-e38e-4937-81ad-7cc72a49c184/cloud-config.txt

lp bug 1797168

试图排除https://bugs.launchpad.net/juju/+bug/1797168

juju add-model test2
juju model-config container-image-metadata-url=https://quqi.com:443/
juju model-config image-metadata-url=https://quqi.com:443/
juju model-config logging-config="<root>=DEBUG"
juju add-machine --series focal
juju remove-application ceph-radosgw && juju deploy ceph-radosgw --series=xenial --to="lxd:0"

但是也不行. 晕了。