Jenkins Pipeline脚本优化：为Kubernetes应用部署增加状态检测

引言

在软件部署的世界中，Jenkins已经成为自动化流程的代名词。不断变化的技术环境要求我们持续改进部署流程以满足现代应用部署的需要。在本篇博客中，作为一位资深运维工程师，我将分享如何将Jenkins Pipeline进化至不仅能支持部署应用直至Running状态检测，同时也能兼顾Deployment和StatefulSet资源的轮询更新，并详细介绍滚动更新策略的配置方法。

初始Jenkins Pipeline分析

参照前文：Jenkins Pipeline 脚本优化实践：从繁琐到简洁，初始化pipeline如下：

pipeline {
    agent none // Use none at the top level, each stage will define its own agent.

    environment {
        REGISTRY = "swr.cn-north-4.myhuaweicloud.com/master-metaspace"
        KUBE_CONFIG = "--namespace=master-metaspace --context=master"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        // Assume that 'data' is defined elsewhere or injected as a parameter.
        BASE_WORKSPACE = "xxxxxxx" // 定义一个基础工作空间路径
    }

    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader', value: 'true' }
                    steps {
                        buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
            }
        }
        
        stage('Development Deployment') {
            parallel {
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                    }
                }
                stage("Deploy dataloader-game-ucenter") {
                    when { environment name: 'dataloader', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("dataloader-game-ucenter")
                    }
                }
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "huaweiyun-xx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                    }
                }
            }
        }
    }
}

// Define methods outside pipeline to avoid repetition

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基础工作空间变量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'hw-registry', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}

def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

初始的Jenkins Pipeline定义了一个基本的CI/CD流程，涵盖了代码拉取、Docker镜像构建、推送及在Kubernetes环境中的部署。然而，流程中缺少了对部署状态的检查，这是在确保部署稳定性方面至关重要的一个环节。

进化 I：探针引入Deployment部署

现代应用部署不仅仅需要一个“部署到Kubernetes”的指令，更需要在部署后进行健康检查。对于Deployment类型的应用来说，我们需要在所有Pods运行并处于READY状态后才认为部署成功。

状态检测方法介绍

为此，我们引入了checkKubernetesResourceStatus方法来检查资源的状态。该方法通过kubectl的get命令和jsonpath查询输出来轮询检查ready副本数。如果指定时间内资源不达状态，则流程失败。

Jenkinsfile变更详解：

引入checkKubernetesResourceStatus方法来检测deployment各个阶段部署的状态。

def checkKubernetesResourceStatus(String deploymentName, String namespace) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the deployment status
        String statusCheck = sh (
            script: "kubectl get deployment ${deploymentName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and greater than 0
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "Deployment ${deploymentName} is ready."
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}"
            sleep sleepTime
        }
    }
    error "Deployment ${deploymentName} did not become ready after ${attempts} attempts"
}

以Deploy game-ucenter stage为例：

                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace")
                    }
                }

game-ucenter模板文件如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: game-ucenter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: game-ucenter
  template:
    metadata:
      labels:
        app: game-ucenter
    spec:
      containers:
        - name: game-ucenter
          image: xxxxxx/xxxx/game-ucenter:{data}
          envFrom:
          - configMapRef:
              name: deploy
          ports:
            - containerPort: 80
          resources:
            requests:
              memory: "4096M"
              cpu: "2000m"
            limits:
              memory: "4096M"
              cpu: "2000m" 
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              scheme: HTTP
              path: /test.html
              port: 80
            initialDelaySeconds: 20
            periodSeconds: 120
      imagePullSecrets:                                              
        - name: xxxx
---

apiVersion: v1
kind: Service
metadata:
  name: game-ucenter
  labels:
    app: game-ucenter
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: game-ucenter

尝试修改以下**replicas: 3 **重新运行以下pipeline：

也没有问题，pipeline 脚本有效！

进化 II：兼容StatefulSet的健康检查

考虑到某些应用可能采用StatefulSet作为工作负载类型，我们必须确保Jenkins Pipeline能够针对不同的工作负载类型执行健康检查。

状态检测兼容性改进

为了适配StatefulSet，我们对checkKubernetesResourceStatus方法做了略微修改，使其可以接受一个resourceType参数来区分资源类型，进而查询对应的状态字段，代码片段如下：

def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the resource status
        String statusCheck = sh (
            script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and equal to the desired number
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "${resourceType} ${resourceName} is ready."
            return
        } else {
            echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }
    error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}

修改game-ucenter stage:

                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "k8s-node-06" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
                    }
                }

创建一个statefulset datawriter-game-ucenter stage:

                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "xxxxx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                        checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                    }
                }

注意:我这里截图还是用了game-ucenter做的测试，其实我想用我的datawriter-game-ucenter，but这个服务是一个node应用没有没有livenessProbe readinessProbe，所以截图我还是使用了game-ucenter!

进化 III：引入滚动更新策略配置和检测

当我们更新Deployment资源时，通常会采用滚动更新策略，以逐步替换旧Pods，最小化部署时的中断。

更新策略检测逻辑

def checkDeploymentUpdateStatus(String deploymentName, String namespace) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    echo "Checking the update status of Deployment: ${deploymentName}"

    for (int i = 1; i <= attempts; i++) {
        String updateStatus = sh (
            script: "kubectl rollout status deployment/${deploymentName} --namespace=${namespace}",
            returnStdout: true
        ).trim()

        if (updateStatus.contains("successfully rolled out")) {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }

    error "Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts"
}

1. 我们增加了checkDeploymentUpdateStatus方法，该方法通过kubectl命令rollout status监控Deployment的更新状态。
2. 当检测到successfully rolled out时，表示滚动更新成功。
3. 如果在给定时间内更新没有成功，则流程将失败。

继续考虑一下如果statefulset多实例呢？不想写两个了整合成一个方法如下：

def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds

    if (!(resourceType in ["deployment", "statefulset"])) {
        error "Unknown resource type: ${resourceType}. Only 'deployment' and 'statefulset' are supported."
    }

    echo "Checking the update status of ${resourceType} '${resourceName}' in namespace '${namespace}'"

    for (int i = 1; i <= attempts; i++) {
        String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
        String updateStatus = sh (
            script: rolloutCommand,
            returnStdout: true
        ).trim()

        if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete"))  {
            echo "Update status: ${updateStatus}"
            return
        } else {
            echo "Waiting for ${resourceType} '${resourceName}' to successfully roll out. Attempt ${i}/${attempts}."
            sleep(sleepTime)
        }
    }

    error "${resourceType} '${resourceName}' did not successfully roll out after ${attempts} attempts in namespace '${namespace}'"
}

Jenkinsfile更新实现

经过上述进化，Jenkinsfile中现在包含了完整的部署状态检查逻辑，以应对不同类型资源的部署监控需求。

pipeline {
    agent none // Use none at the top level, each stage will define its own agent.

    environment {
        REGISTRY = "ccr.ccs.tencentyun.com/xxxxx"
        KUBE_CONFIG = "--namespace=master-metaspace"
        KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
        // Assume that 'data' is defined elsewhere or injected as a parameter.
        BASE_WORKSPACE = "xxxxxx" // 定义一个基础工作空间路径
    }

    stages {
        stage("GetCode") {
            agent { label "build01" }
            steps {
                script {
                    checkout scm: [
                        $class: 'GitSCM',
                        branches: [[name: env.branchName]],
                        extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
                        userRemoteConfigs: [[credentialsId: 'xxxxx', url: env.gitHttpURL]]
                    ]
                }
            }
        }
        
        stage("Docker Builds") {
            parallel {
                stage('Build dataloader-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'dataloader-game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build datawriter-game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
                stage('Build game-ucenter') {
                    agent { label "build01" }
                    when { environment name: 'game-ucenter', value: 'true' }
                    steps {
                        buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
                    }
                }
            }
        }
        
        stage('Development Deployment') {
            parallel {
                stage("Deploy datawriter-game-ucenter") {
                    when { environment name: 'datawriter-game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("datawriter-game-ucenter")
                        checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                        checkRolloutStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
                    }
                }
                stage("Deploy dataloader-game-ucenter") {
                    when { environment name: 'dataloader-game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("dataloader-game-ucenter")
                        checkKubernetesResourceStatus("dataloader-game-ucenter", "master-metaspace", "statefulset")
                    }
                }
                stage("Deploy game-ucenter") {
                    when { environment name: 'game-ucenter', value: 'true' }
                    agent { label  "xxxx" }
                    steps {
                        deployToKubernetes("game-ucenter")
                        checkRolloutStatus("game-ucenter", "master-metaspace", "deployment")
                        checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
                    }
                }
            }
        }
    }
}

// Define methods outside pipeline to avoid repetition

def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
    sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基础工作空间变量
    sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=game-ucenter -t $REGISTRY/$imageName:$tag ."
    withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
        sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
        sh "docker push $REGISTRY/$imageName:$tag"
    }
}

def deployToKubernetes(String kubernetesComponent) {
    String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
    String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
    sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
    sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}

def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // 设置重试次数
    int sleepTime = 10 // 设置重试间隔时间（秒）

    if (!(resourceType in ["deployment", "statefulset"])) {
        error "未知资源类型：${resourceType}。只支持 'deployment' 和 'statefulset' 。"
    }

    echo "正在检查${resourceType} '${resourceName}' 在命名空间 '${namespace}' 的更新状态"

    for (int i = 1; i <= attempts; i++) {
        String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
        try {
            String updateStatus = sh (
                script: rolloutCommand,
                returnStdout: true
            ).trim()

            // 添加对 "partitioned roll out complete" 状态的检查
            if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
                echo "更新状态：${updateStatus}"
                return
            } else {
                echo "等待 ${resourceType} '${resourceName}' 成功发布。尝试次数：${i}/${attempts}。"
                sleep(sleepTime)
            }
        } catch (Exception e) {
            echo "获取更新状态时发生错误：${e.getMessage()}。尝试次数：${i}/${attempts}。"
            sleep(sleepTime)
        }
    }

    error "${resourceType} '${resourceName}' 在命名空间 '${namespace}' 内未能在 ${attempts} 次尝试之后成功发布"
}
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
    int attempts = 30 // Set the number of retry attempts
    int sleepTime = 10 // Set the sleep time between attempts in seconds
    String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
    for (int i = 1; i <= attempts; i++) {
        // Check the resource status
        String statusCheck = sh (
            script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
            returnStdout: true
        ).trim()

        // If the number of ready replicas is not empty and equal to the desired number
        if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
            echo "${resourceType} ${resourceName} is ready."
            return
        } else {
            echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
            sleep(sleepTime)
        }
    }
    error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}// 更新后的Jenkins Pipeline代码详细定义参照本文开头给出的代码

总结

本篇博客通过对Jenkins Pipeline的进化过程展开讲解，展现了如何从简单的部署任务转变为一个健壮且兼顾各类工作负载状态监测的CI/CD流程。我们强化了状态检测的逻辑，引入了更新策略的检测，并保持了对不同Kubernetes资源类型的兼容性。这些改进确保了自动化流程能够与现代部署实践保持同步，给运维团队带来极大便利，并最大化地保障了部署的可靠性。

后记

由于篇幅限制，本篇文章未作其他更详细演示。然而，在实际应用中，运维团队可以根据自己的具体需求和环境进一步丰富和细化每个步骤的实现，确保Pipeline的健壮性和高可用性，以适应不断变化的技术挑战。