Kubesphere中DevOps流水线无法部署/部署失败

摘要

总算能让devops运行以后，流水线却卡在了deploy这一步。碰到了两个比较大的问题，一个是无法使用k8sp自带的kubeconfig认证去部署；一个是部署好了以后但是没有办法解析镜像名。

版本信息

k8s：v1.21.5
k8sp：v3.3.0

流水线概览

Q问题描述

pipeline 在deploy 的阶段总是报各种错。

Q1.使用k8sp自带kube认证产生报错

      stage('deploy fail') {agent nonesteps {withCredentials([kubeconfigContent(credentialsId : 'kubeconfigger' ,variable : 'KUBECONFIGGER' ,)]) {kubernetesDeploy(enableConfigSubstitution: true, deleteResource: false, kubeconfigId: 'kubeconfigger', configs: 'hospital-manage/deploy/**')}}}

报错内容如下：

Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/redp5lk5/rose/hospital-manage/deploy/deploy.yml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: 
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.openapi.ApiException: at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiExceptionExceptNotFound(ResourceManager.java:180)at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:213)at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:201)at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:93)at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)at hudson.remoting.UserRequest.perform(UserRequest.java:211)at hudson.remoting.UserRequest.perform(UserRequest.java:54)at hudson.remoting.Request$2.run(Request.java:376)at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)at java.base/java.util.concurrent.FutureTask.run(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)at java.base/java.lang.Thread.run(Unknown Source)Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.81.183/10.233.81.183:49480at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)at hudson.remoting.Channel.call(Channel.java:1001)at hudson.FilePath.act(FilePath.java:1256)at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.openapi.ApiException: at io.kubernetes.client.openapi.ApiClient.handleResponse(ApiClient.java:979)at io.kubernetes.client.openapi.ApiClient.execute(ApiClient.java:895)at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeploymentWithHttpInfo(AppsV1Api.java:7299)at io.kubernetes.client.openapi.apis.AppsV1Api.readNamespacedDeployment(AppsV1Api.java:7275)at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$DeploymentUpdater.getCurrentResource(V1ResourceManager.java:210)... 16 more
Api call failed with code 400, detailed message: {"kind": "Status","apiVersion": "v1","metadata": {},"status": "Failure","message": "the export parameter, deprecated since v1.14, is no longer supported","reason": "BadRequest","code": 400
}
Kubernetes deployment ended with HasError

从message来看，已经不支持这个什么什么参数了。从与小伙伴的沟通看，以前的版本是可以用的。但是新版本不支持了。

解决方案

采用以下写法。
缺点：不支持图形化流水线编辑，点击编辑该凭证会闪退。

法1

需要在环境变量处声明 KUBECONFIG_CREDENTIAL_ID

    stage('deploy success') {agent nonesteps {container('maven') {withCredentials([kubeconfigFile(credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG') ]) {sh 'envsubst < hospital-manage/deploy/deploy.yml | kubectl apply -f -'}}}}

法2

与法1没什么区别。主要区别就是shell中的命令。

    stage('deploy hospital-manage to dev') {agent nonesteps {container('maven') {withCredentials([kubeconfigFile(credentialsId: env.KUBECONFIG_CREDENTIAL_ID, variable: 'KUBECONFIG')]) {sh 'kubectl apply -f hospital-manage/deploy/**'}}}}

Q2：无法解析镜像名

在deploy.yml文件中，有关镜像名的描述如下所示：

spec:containers:- image: $DOCKERHUB_NAMESPACE/server-gateway:SNAPSHOT-$BUILD_NUMBER

但是却出现了如下结果

状态信息
初始化完成
状态：True
容器组就绪
状态：False
原因：ContainersNotReady
消息：containers with unready status: [app]
所有容器就绪
状态：False
原因：ContainersNotReady
消息：containers with unready status: [app]
容器组调度完成
状态：True
无法解析镜像名称

无法解析镜像名，我点进去这个pod的yml以后，才发现，流水线没能解析出这些变量。才导致了无法解析镜像名。

解决方案

治标不治本方案

直接把镜像写死

belchance/ruoyi:hospital-manage_SNAPSHOT-10

替换环境变量方案

Deploy to kubernets阶段报错 - KubeSphere 开发者社区
来源于社区网友，方法就是使用 envsubst，把环境yml文件里的环境变量改了。
注意替换的字符要在JenkinsFile的环境变量区声明，deploy.yml的位置要准确。

  agent nonesteps {container('nodejs') {sh 'envsubst \\'${REGISTRY},${ALIYUNHUB_NAMESPACE},${BUILD_NUMBER}\\' < deploy/deploy.yml > deploy/deploy2.yml'sh 'cat deploy/deploy2.yml'}}
}

envsubst用法介绍：

envsubst '$DOCKERHUB_NAME,$NUMBER' < deploy.yml 
envsubst '需要替换的环境变量' < target.file
command < file     将输入重定向到 file。