ceph 18.2.4二次开发，docker镜像制作

编译环境要求

#需要ubuntu 22.04版本 参考https://docs.ceph.com/en/reef/start/os-recommendations/
#磁盘空间最好大于200GB
#内存如果小于100GB 会有OOM的情况发生,需要重跑 目前遇到内存占用最高为92GB

替换阿里云ubuntu 22.04源

将下面内容写入/etc/apt/sources.list 文件

mv /etc/apt/sources.list /etc/apt/sources.list.bak
#内容见下方
vim /etc/apt/sources.list
#更新apt源
apt update

deb https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse# deb https://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiverse
# deb-src https://mirrors.aliyun.com/ubuntu/ jammy-proposed main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse
#deb-src https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse

拉取代码

git clone https://github.com/ceph/ceph.git
#切换tag
git checkout v18.2.4 -b v18.2.4
#下载子模块
git submodule update --init --recursive

安装依赖

sudo apt install curlexport PIP_INDEX_URL=https://mirrors.aliyun.com/pypi/simple/#安装依赖包
./install-deps.sh#检查当前的 GCC 版本 是否为c++ 11
#gcc --version
#g++ --versionsudo apt install python3-routes reprepro npm docker.iosudo apt install `cat doc_deps.deb.txt`
#npm阿里源
npm config set registry https://registry.npmmirror.com
#npm腾讯源
#npm config set registry https://mirrors.cloud.tencent.com/npm/

编译deb包

#脚本内容见下方,注意脚本中的注释
#test.sh的内容根据官方提供的make-debs.sh进行了微调整
#也可以直接运行make-debs.sh 替代这一步
touch test.shvim test.sh
#执行脚本 先打出来一个ceph的tar包, 手动将make-dict与exit这两行注释掉以后执行第二遍
sh test.sh#大概需要运行 2个小时左右
#运行成功后 默认会在/tmp/release/Ubuntu/WORKDIR生成对应的deb安装包
#运行过程中可能会有错误，大概率是因为从github.com上下载包没有下载成功导致
sh test.sh

#!/usr/bin/env bashset -xe. /etc/os-release
base=${1:-/tmp/release}
releasedir=$base/$NAME/WORKDIRmkdir -p $releasedir
#可以不打开
#清理工作目录中未被 Git 跟踪的文件和目录
#git clean -dxfvers=$(git describe --match "v*" | sed s/^v//)
#打包代码，依赖 会生成ceph-$vers.tar.bz2
#建议第一次执行在下面 添加exit 0 
./make-dist $vers
exit 0cp ceph-$vers.tar.bz2 $releasedir/ceph_$vers.orig.tar.bz2
tar -C $releasedir -jxf $releasedir/ceph_$vers.orig.tar.bz2cp -a debian $releasedir/ceph-$vers/debian
cd $releasedir
perl -ni -e 'print if(!(/^Package: .*-dbg$/../^$/))' ceph-$vers/debian/control
perl -pi -e 's/--dbg-package.*//' ceph-$vers/debian/rulesdvers="$vers-1"cd ceph-$vers# ceph (18.2.4-1) stable; urgency=medium 取版本 18.2.4-1
chvers=$(head -1 debian/changelog | perl -ne 's/.*\(//; s/\).*//; print')
# 如果两个版本号获取不同，更改版本号
if [ "$chvers" != "$dvers" ]; thenDEBEMAIL="[e-mail]" dch -D $VERSION_CODENAME --force-distribution -b -v "$dvers" "new version"
fi#获取一半的cpu核数 计算后续的job的数量 -jxx
: ${NPROC:=$(($(nproc) / 2))}
#嫌慢可以全部cpu跑起来
#: ${NPROC:=$(nproc)}if test $NPROC -gt 1 ; thenj=-j${NPROC}
fi#dpkg-buildpackage -j24 -uc -us -b
PATH=/usr/lib/ccache:$PATH dpkg-buildpackage $j -uc -uscd ../..mkdir -p $VERSION_CODENAME/conf
cat > $VERSION_CODENAME/conf/distributions <<EOF
Codename: $VERSION_CODENAME
Suite: stable
Components: main
Architectures: $(dpkg --print-architecture) source
EOFif [ ! -e conf ]; thenln -s $VERSION_CODENAME/conf conf
fi#reprepro --basedir /tmp/release/Ubuntu include focal WORKDIR/ceph_18.2.4-1_amd64.changesreprepro --basedir $(pwd) include $VERSION_CODENAME WORKDIR/*.changesecho $dvers > $VERSION_CODENAME/version

制作本机ubuntu源

#安装nginx
sudo apt install nginx#将制作好的安装包路径 加个软连到nginx root_path下面
ln -s /tmp/release/Ubuntu /var/www/html/ubuntu#修改nginx配置 添加autoindex on; 配置项 注意带分号
vi /etc/nginx/nginx.conf#重启nginx服务
sudo systemctl restart nginx#没有安装firewalld可以忽略这个步骤
#查看firewalld状态 如果是running 需要开放端口
sudo firewall-cmd --state
#开放80 443端口
sudo firewall-cmd --permanent --add-port=80/tcp
sudo firewall-cmd --permanent --add-port=443/tcp
#重新加载firewalld配置
sudo firewall-cmd --reload
#浏览器访问 http://主机IP/ubuntu 看看能不能正常访问#主机IP 需要根据自己的实际ip修改
echo "deb [arch=amd64 trusted=yes] http://主机IP/ubuntu jammy main" >> /etc/apt/sources.list
#更新源
sudo apt update#安装cephadm软件 注意看deb的路径是不是自己的本机
sudo apt install cephadm ceph-common

打包Docker镜像

cd ~mkdir docker && cd dockertouch Dockerfile
#内容见下方，记得将"主机IP"修改自己的刚刚配置的主机地址
vi Dockerfile#编译镜像
docker build -t 'my/ceph' .#推送远端hub
docker push my/ceph:latest

FROM ubuntu:jammyENV DEBIAN_FRONTEND=noninteractive#安装需要的软件
RUN apt-get update && apt-get install -y ca-certificates#调整apt源
RUN echo "deb https://mirrors.aliyun.com/ubuntu/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \echo "deb https://mirrors.aliyun.com/ubuntu/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \echo "deb [arch=amd64 trusted=yes] http://主机IP/ubuntu jammy main" >> /etc/apt/sources.list && \apt update#安装ceph
RUN apt install -y ceph ceph-volume ceph-exporter  python3-cffi udev#开放端口
EXPOSE 6789 6800 6801 6802 6803 6804 6805 80 5000# 设置内存锁限制
RUN echo '@ceph - memlock 204800' >> /etc/security/limits.conf && \echo '@root - memlock 204800' >> /etc/security/limits.conf# 记录初始磁盘使用量
RUN INITIAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')"# Postinstall 清理
RUN ( echo "apt clean" && apt-get clean && \echo "apt autoclean" && apt-get autoclean ) || \( retval=$? && cat /var/log/apt/history.log && exit $retval )# 清理不必要的 apt、deb、dpkg 数据
RUN rm -rf \/var/lib/apt/lists/* \/var/cache/debconf/* \/var/log/apt/ \/var/log/dpkg.log# 编辑 lvm.conf 文件
RUN echo "About to edit lvm.conf" && \sed -i -e 's/^\([[:space:]#]*udev_rules =\) 1$/\1 0/' -e 's/^\([[:space:]#]*udev_sync =\) 1$/\1 0/' -e 's/^\([[:space:]#]*obtain_device_list_from_udev =\) 1$/\1 0/' /etc/lvm/lvm.conf && \echo "About to validate lvm.conf edits" && \# validate the sed command worked as expectedgrep -sqo "udev_sync = 0" /etc/lvm/lvm.conf && \grep -sqo "udev_rules = 0" /etc/lvm/lvm.conf && \grep -sqo "obtain_device_list_from_udev = 0" /etc/lvm/lvm.conf && \echo "Edits validated OK" && \mkdir -p /var/run/ganesha# 清理不必要的文件和目录
RUN rm -rf \/etc/{selinux,systemd,udev} \/lib/{lsb,udev} \/tmp/* \/usr/lib{,64}/{locale,systemd,udev,dracut} \/usr/share/{doc,info,locale,man} \/usr/share/{bash-completion,pkgconfig/bash-completion.pc} \/var/log/* \/var/tmp/*# 清理 Python 编译文件
RUN find / -xdev -name "*.pyc" -o -name "*.pyo" -exec rm -f {} \;# 压缩 ceph-dencoder
RUN if [ -f /usr/bin/ceph-dencoder ]; then gzip -9 /usr/bin/ceph-dencoder; fi# 清理 radosgw 相关文件
RUN rm -f /usr/lib/ceph/mgr/dashboard/static/AdminLTE-*/plugins/datatables/extensions/TableTools/images/psd/*# 清空日志文件
RUN find /var/log/ -type f -exec truncate -s 0 {} \;# 记录最终磁盘使用量
RUN FINAL_SIZE="$(bash -c 'sz="$(du -sm --exclude=/proc /)" ; echo "${sz%*/}"')"
RUN REMOVED_SIZE=$((INITIAL_SIZE - FINAL_SIZE))
RUN echo "Cleaning process removed ${REMOVED_SIZE}MB"
RUN echo "Dropped container size from ${INITIAL_SIZE}MB to ${FINAL_SIZE}MB"# 验证安装的软件包
RUN apt-cache show \ca-certificates \e2fsprogs \ceph-common \ceph-mon \ceph-osd \ceph-mds \cephfs-mirror \cephfs-top \rbd-mirror \ceph-mgr \ceph-mgr-cephadm \ceph-mgr-dashboard \ceph-mgr-k8sevents \ceph-mgr-rook \ceph-grafana-dashboards \kmod \lvm2 \gdisk \smartmontools \nvme-cli \udev \sg3-utils \procps \hostname \radosgw \nfs-ganesha \nfs-ganesha-ceph \attr \ceph-fuse \rbd-nbd \${CRIMSON_PACKAGES} \ceph-immutable-object-cache \ceph-volume \ceph-exporter \ceph-node-proxyRUN echo 'Packages verified successfully'

遇到的报错

CMake Error at xsimd_ep-stamp/xsimd_ep-download-RELEASE.cmake:37 (message):

CMake Error at xsimd_ep-stamp/xsimd_ep-download-RELEASE.cmake:37 (message):Command failed: 1'/usr/bin/cmake' '-Dmake=' '-Dconfig=' '-P' '/tmp/release/Ubuntu/WORKDIR/ceph-18.2.4-35-gac3a54a4db5/obj-x86_64-linux-gnu/src/arrow/cpp/src/xsimd_ep-stamp/xsimd_ep-download-RELEASE-impl.cmake'See also/tmp/release/Ubuntu/WORKDIR/ceph-18.2.4-35-gac3a54a4db5/obj-x86_64-linux-gnu/src/arrow/cpp/src/xsimd_ep-stamp/xsimd_ep-download-*.log#网络问题 最好加个代理 重新执行

Could not find platform independent libraries Consider setting $PYTHONHOME to [:<exec_prefix>]

mv /var/lib/dpkg/info/ /var/lib/dpkg/info_old/
mkdir /var/lib/dpkg/info/
apt-get update
apt-get -f install
mv /var/lib/dpkg/info/* /var/lib/dpkg/info_old/
rm -rf /var/lib/dpkg/info
mv /var/lib/dpkg/info_old/ /var/lib/dpkg/info

make-dict执行的太慢

由于make-dict 中需要下载外网资源 可以先下载完毕后,放到执行脚本目录，修改makde-dict注释download_from相关函数调用

cephadm启动报错 /usr/bin/ceph: stderr Error EIO: Module ‘cephadm’ has experienced an error and cannot handle commands: ContainerInspectInfo(image_id=‘438bbc41efbb76533bdb38441b1b804159559d6e170ac7bb2404dd8f3d6b1e58’, ceph_version=‘ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)’, repo_digests=[‘’]) Unable to set up “admin” label; assuming older version of Ceph

#查看RepoDigests为空，这个需要推送一个hub仓库填充一下即可
docker inspect imagesid

2024-11-18 09:52:18,352 7f21e3e421c0 INFO /usr/bin/docker: stderr FileNotFoundError: [Errno 2] No such file or directory: ‘udevadm’

#缺少udevadm软件
udevadm --version 
#上面不能正常运行，进行安装操作
apt install udev