Zero to JupyterHub with Kubernetes 下篇 - Jupyterhub on k8s

前言：纯个人记录使用。

• 搭建 Zero to JupyterHub with Kubernetes 上篇 - Kubernetes 离线二进制部署。
• 搭建 Zero to JupyterHub with Kubernetes 中篇 - Kubernetes 常规使用记录。
• 搭建 Zero to JupyterHub with Kubernetes 下篇 - Jupyterhub on k8s。

官方文档：Zero to JupyterHub with Kubernetes

**版本对应：**This documentation is for Helm chart version 2.0.0 that deploys JupyterHub version 3.0.0 and other components versioned in hub/images/requirements.txt. The Helm chart requires Kubernetes version >=1.20.0 and Helm >=3.5

组件	版本
kubernetes	v1.20.4
jupyterhub-chart	2.0.0
helm	v3.12.3

第一部分: Setup Kubernetes

1、Setup Kubernetes

kubernetes-v1.20.4 离线二进制部署

[root@k8s-master /data/s0/kubernetes]$ kubectl version --short
Client Version: v1.20.4
Server Version: v1.20.4

2、Setting up helm

通过百度网盘分享的文件：helm-v3.12.3-linux-amd64.tar.gz
链接：https://pan.baidu.com/s/1f8xONKHWshHxieu7jEN4yA
提取码：1234

# 解压安装
[root@k8s-master /data/s0/kubernetes/helm]$ tar -xzvf helm-v3.12.3-linux-amd64.tar.gz
[root@k8s-master /data/s0/kubernetes/helm]$ ln -s /data/s0/kubernetes/helm/linux-amd64/helm /usr/local/bin
# 验证
[root@k8s-master /data/s0/kubernetes/helm]$ helm version
version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

第二部分: Setup JupyterHub

1、Installing JupyterHub

1.1 下载所需jupyterhub chart版本

​ JupyterHub’s Helm chart 仓库 --> jupyterhub-2.0.0.tgz

通过百度网盘分享的文件：jupyterhub-2.0.0.tgz
链接：https://pan.baidu.com/s/1ZrEHC9al29ye7n0W3UAi3g
提取码：1234

1.2 下载相关离线镜像

# 解压安装
[root@k8s-master /data/s0/kubernetes/helm]$ tar -xzvf jupyterhub-2.0.0.tgz   # jupyterhub chart 
# 查看所需镜像
[root@k8s-master /data/s0/kubernetes/helm]$ cat jupyterhub/Chart.yaml
annotations:artifacthub.io/images: |- image: jupyterhub/configurable-http-proxy:4.5.3name: configurable-http-proxy- image: jupyterhub/k8s-hub:2.0.0name: k8s-hub- image: jupyterhub/k8s-image-awaiter:2.0.0name: k8s-image-awaiter- image: jupyterhub/k8s-network-tools:2.0.0name: k8s-network-tools- image: jupyterhub/k8s-secret-sync:2.0.0name: k8s-secret-sync- image: jupyterhub/k8s-singleuser-sample:2.0.0name: k8s-singleuser-sample- image: k8s.gcr.io/kube-scheduler:v1.23.10  # helm upgrate 启动部署时，此版本有问题，改为v1.20.15，注意values.yaml中镜像名称修改，镜像保持一致name: kube-scheduler- image: k8s.gcr.io/pause:3.8  # 部署k8s时已下载安装，注意values.yaml中镜像名称修改，保持一致name: pause- image: k8s.gcr.io/pause:3.8name: pausd- image: traefik:v2.8.4name: traefik# 联网保存本地镜像
# 1. 下载保存 jupyterhub/configurable-http-proxy:4.5.3
> docker pull quay.io/jupyterhub/configurable-http-proxy:4.5.3
> docker tag quay.io/jupyterhub/configurable-http-proxy:4.5.3 jupyterhub/configurable-http-proxy:4.5.3
> docker save -o configurable-http-proxy:4.5.3.tar jupyterhub/configurable-http-proxy:4.5.3  # 2. 下载保存 jupyterhub/k8s-hub:2.0.0
> docker pull quay.io/jupyterhub/k8s-hub:2.0.0
> docker tag quay.io/jupyterhub/k8s-hub:2.0.0 jupyterhub/k8s-hub:2.0.0
> docker save -o k8s-hub:2.0.0.tar jupyterhub/k8s-hub:2.0.0# 3. 下载保存 jupyterhub/k8s-image-awaiter:2.0.0
> docker pull quay.io/jupyterhub/k8s-image-awaiter:2.0.0
> docker tag  quay.io/jupyterhub/k8s-image-awaiter:2.0.0 jupyterhub/k8s-image-awaiter:2.0.0> docker save -o k8s-image-awaiter:2.0.0.tar jupyterhub/k8s-image-awaiter:2.0.0
# 4. 下载保存 jupyterhub/k8s-network-tools:2.0.0
> docker pull quay.io/jupyterhub/k8s-network-tools:2.0.0
> docker tag  quay.io/jupyterhub/k8s-network-tools:2.0.0 jupyterhub/k8s-network-tools:2.0.0
> docker save -o k8s-network-tools:2.0.0.tar jupyterhub/k8s-network-tools:2.0.0# 5. 下载保存 jupyterhub/k8s-secret-sync:2.0.0
> docker pull quay.io/jupyterhub/k8s-secret-sync:2.0.0
> docker tag quay.io/jupyterhub/k8s-secret-sync:2.0.0 jupyterhub/k8s-secret-sync:2.0.0
> docker save -o k8s-secret-sync:2.0.0.tar jupyterhub/k8s-secret-sync:2.0.0# 6. 下载保存 jupyterhub/k8s-singleuser-sample:2.0.0
> docker pull m.daocloud.io/docker.io/jupyterhub/k8s-singleuser-sample:2.0.0
> docker tag m.daocloud.io/docker.io/jupyterhub/k8s-singleuser-sample:2.0.0 jupyterhub/k8s-singleuser-sample:2.0.0
> docker save -o k8s-singleuser-sample:2.0.0.tar jupyterhub/k8s-singleuser-sample:2.0.0# 7. 下载保存 k8s.gcr.io/kube-scheduler:v1.20.15
> docker pull k8s-gcr.m.daocloud.io/kube-scheduler:v1.20.15
> docker tag k8s-gcr.m.daocloud.io/kube-scheduler:v1.20.15  k8s.gcr.io/kube-scheduler:v1.20.15
> docker save -o kube-scheduler:v1.20.15.tar k8s.gcr.io/kube-scheduler:v1.20.15# 8. 下载保存 traefik:v2.8.4
> docker pull m.daocloud.io/docker.io/library/traefik:v2.8.4
> docker tag m.daocloud.io/docker.io/library/traefik:v2.8.4 traefik:v2.8.4
> docker save -o traefik:v2.8.4.tar traefik:v2.8.4## 9. 将离线镜像打包上传
> tar -czvf jupyterhub-chart-images.tgz ./*
> scp jupyterhub-chart-images.tgz k8s-master:/data/s0/kubernetes/helm

1.3 加载镜像

# ------------------ k8s-matser，k8s-node1、k8s-node2 ----------------------------
# 1. 加载镜像，node1、node2节点同理
[root@k8s-master /data/s0/kubernetes/helm]$ tar -xzvf jupyterhub-chart-images.tgz -C ./chart-images
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i configurable-http-proxy:4.5.3.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i k8s-hub:2.0.0.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i k8s-image-awaiter:2.0.0.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i k8s-network-tools:2.0.0.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i k8s-secret-sync:2.0.0.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i k8s-singleuser-sample:2.0.0.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i kube-scheduler:v1.20.15.tar
[root@k8s-master /data/s0/kubernetes/helm/chart-images]$ docker load -i traefik:v2.8.4.tar# 2.加载自定义用户科学环境；默认的单用户服务器jupyter镜像 k8s-singleuser-sample
# docker pull m.daocloud.io/docker.io/jupyter/datascience-notebook 默认拉取最新版本,最好指定版本，否则每次拉最新的
[root@k8s-master /data/s0/kubernetes/helm]$ docker load -i datascience-notebook.tar
# 注意：k8s在不指定镜像拉取策略imagePullPolicy的情况下，如果镜像标签tag：latest，imagePullPolicy默认值为“Always” 总是从镜像库拉取；
# 如果镜像标签tag不是：latest，imagePullPolicy默认值为“IfNotPresent” 本地有使用本地镜像，本地没有则拉取镜像库；
[root@k8s-master /data/s0/kubernetes/helm]$ docker tag jupyter/datascience-notebook:latest jupyter/datascience-notebook:2023.10.23

1.4 jupyterhub 配置

# jupyterhub 自定义配置
[root@datanode40 /data/s0/kubernetes/helm]$ touch config.yaml
[root@datanode40 /data/s0/kubernetes/helm]$ vim config.yaml

config.yaml 内容如下：

# 应用名称（deployment、service、pod等资源对象名称）
fullnameOverride: "jupyterhub"# 拉取镜像时，相关仓库身份认证（使用本机离线镜像）
imagePullSecret:create: falseautomaticReferenceInjection: false# hub服务pod配置(auth权限认证)
hub:revisionHistoryLimit: 1                  # Kubernetes 中保留的历史版本数量config:                                  # jupyterhub_cnfig.py 配置文件内容JupyterHub:admin_access: trueadmin_users: - zyp                             # 设置管理员用户authenticator_class: dummy           # 用户验证，测试采用虚拟验证service:type: ClusterIP                        ports:nodePort:db:type: sqlite-pvc                       #  JupyterHub 使用数据库,存储用户信息、服务器状态、活动记录等数据pvc:                                   # 需要预先创建对应pvaccessModes:- ReadWriteOncestorage: 2GisubPath: sqlite                      # PV存储卷子路径，默认根路径storageClassName: sqlite-pv          # 存储类别image:name: jupyterhub/k8s-hubtag: "2.0.0"pullPolicy: IfNotPresent#设置 chp(configurable-http-proxy)pod的代理、公网代理、https代理相关
proxy:service:type: NodePort                            # 公网代理服务                nodePorts:http: 30081chp:                                         # configurable-http-proxy (chp)配置revisionHistoryLimit: 1image:name: jupyterhub/configurable-http-proxytag: "4.5.3" pullPolicy: IfNotPresenthttps:enabled: false                              # 禁用https          # 单用户jupyter服务器
singleuser:networkTools:image:name: jupyterhub/k8s-network-toolstag: "2.0.0"pullPolicy: IfNotPresentstorage:                                       # 配置单用户环境存储type: static                                 # 静态挂载方式static:pvcName: notebook-pvc                      # 存储pvc名称，需手动创建pvc和pvsubPath: "{username}"capacity: 10GihomeMountPath: /home/jovyan                  # 容器中挂载主文件夹存储的位置     # Defines the default image  image:name: jupyterhub/k8s-singleuser-sample        # 可修改为自己的科学计算环境tag: "2.0.0"pullPolicy: IfNotPresent profileList:                                    # 用户科学环境选择- display_name: "sample environment"description: "To avoid too much bells and whistles: Python."default: true- display_name: "Datascience environment"description: "If you want the additional bells and whistles: Python, R, and Julia."kubespawner_override:image: jupyter/datascience-notebook:2023.10.23pullPolicy: IfNotPresentstartTimeout: 300cpu:limit:guarantee: 0.5memory:limit:guarantee: 1Gcmd: jupyterhub-singleuser       # 容器内，启动单用户服务器的命令defaultUrl: "/lab"               # 用户jupyter界面extraEnv:JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp"  # k8s 容器调度相关
scheduling:userScheduler:revisionHistoryLimit: 1image:name: k8s.gcr.io/kube-schedulertag: "v1.20.15" pullPolicy: IfNotPresentuserPlaceholder:image:name: k8s.gcr.io/pausetag: "3.8"pullPolicy: IfNotPresent# 镜像拉取器
prePuller:                               hook:enabled: false                       # 离线环境，本地镜像，无需拉取pullOnlyOnChanges: falsecontinuous:enabled: falsepullProfileListImages: false# 空闲进程杀死服务 
cull:enabled: trueusers: false # --cull-usersadminUsers: true # --cull-admin-usersremoveNamedServers: false # --remove-named-serverstimeout: 3600 # --timeoutevery: 600 # --cull-everyconcurrency: 10 # --concurrencymaxAge: 0 # --max-age

1.4.1 预先配置pv与pvc

pv 持久化参见 Kubernetes 常规使用记录。

# 配置sqlite存储的PV 和 单用户服务器存储的pv和PVC
[root@k8s-node1 /data/s0/kubernetes/nfs]$ mkdir pvs
[root@k8s-node1 /data/s0/kubernetes/nfs/pvs]$ vim pvs.yaml

# sqlite存储的PV
apiVersion: v1
kind: PersistentVolume
metadata:name: sqlite-pv1      
spec:nfs: path: /data/s0/kubernetes/nfs/pv1readOnly: false server: k8s-node1capacity: storage: 2GiaccessModes: - ReadWriteOncestorageClassName: sqlite-pv  persistentVolumeReclaimPolicy: Retain
---
# 单用户服务器pv
apiVersion: v1
kind: PersistentVolume
metadata:name: notebook-pv2      
spec:nfs: path: /data/s0/kubernetes/nfs/pv2readOnly: false server: k8s-node1capacity: storage: 200GiaccessModes: - ReadWriteMany storageClassName: single-notebook   persistentVolumeReclaimPolicy: Retain 
---  
# 单用户服务器pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: notebook-pvc                # 与配置文件对应 namespace: jhub 
spec:accessModes:- ReadWriteManystorageClassName: single-notebook  resources: requests:storage: 20Gi

1.5 启动jupyterhub

# 创建命名空间
[root@k8s-master /data/s0/kubernetes/helm]$ kubectl create ns jhub
# 启动预设pvc和pv 
[root@k8s-node1 /data/s0/kubernetes/nfs/pvs]$ kubectl apply -f pvs.yaml
# 启动jupyterhub
[root@k8s-master /data/s0/kubernetes/helm]$ helm upgrade --cleanup-on-fail \--install jupyterhub-release ./jupyterhub \--namespace jhub \--values config.yaml

# 验证pod运行状态（若存在pod 状态  Pending or ContainerCreating --》 kubectl --namespace=jhub describe pod <name of pod>）
[root@k8s-master /data/s0/kubernetes/helm]$ kubectl --namespace=jhub get pod
jupyterhub-hub-c87985f75-lkl4f               1/1     Running   0          5m18s
jupyterhub-proxy-5d95bb6786-87cqs            1/1     Running   0          5m18s
jupyterhub-user-scheduler-786c6759c7-2r24k   1/1     Running   0          5m18s
jupyterhub-user-scheduler-786c6759c7-6x5k6   1/1     Running   0          5m18s# 验证是否为k8s服务jupyterhub-proxy-public提供了外部IP
[root@k8s-master /data/s0/kubernetes/helm]$ kubectl --namespace=jhub get svc 
NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
jupyterhub-hub            ClusterIP   10.0.0.50    <none>        8081/TCP       90s
jupyterhub-proxy-api      ClusterIP   10.0.0.196   <none>        8001/TCP       90s
jupyterhub-proxy-public   NodePort    10.0.0.51    <none>        80:30081/TCP   90s

问题：Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole “jupyterhub-user-scheduler” in namespace “” exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key “meta.helm.sh/release-name” must equal “jupyterhub-release”: current value is “jupyterhub-v1”

解决：

​ kubectl delete clusterrole jupyterhub-user-scheduler

​ kubectl delete clusterrolebinding jupyterhub-user-scheduler

1.6 jupyterhub 服务验证

远程主机登录 http://k8s-matser:80081

• 用户登录界面
  

• 科学计算环境选择界面
  

• 用户分析操作界面

  

• 底层单用户容器

• 持久化存储查看

• k8s管理界面查看