CI/CD Pipelines with Jenkins: An End-to-End Walkthrough

Jenkins remains one of the most flexible CI engines in production today, despite a generation of newer competitors (GitHub Actions, GitLab CI, CircleCI). The reason is simple: it runs anywhere, integrates with everything, and pipelines are code — a Jenkinsfile lives in the repo, is reviewed in PRs, and travels with the codebase.

This walkthrough builds a realistic five-stage pipeline (checkout → build → test → scan → deploy), then shows how to keep it maintainable as you add more services and more environments.

Why Jenkinsfiles instead of clicked-through jobs

Old Jenkins was full of "freestyle jobs" — configurations clicked together in the UI. They worked, but they were not versioned, not reviewable, and not reproducible across instances. A migration to a new Jenkins controller meant manually recreating dozens of jobs and praying nothing was forgotten.

Declarative pipelines fix this. The pipeline definition lives in your repo as Jenkinsfile. Jenkins discovers it via Multibranch Pipeline projects, runs it on every branch and PR, and re-creates the job automatically. Roll back the file, roll back the pipeline. Diff two branches, diff two pipelines.

A complete declarative pipeline

pipeline {
  agent any
  options {
    timestamps()
    ansiColor('xterm')
    timeout(time: 30, unit: 'MINUTES')
    buildDiscarder(logRotator(numToKeepStr: '50', artifactNumToKeepStr: '10'))
  }
  environment {
    REGISTRY = 'registry.example.com'
    APP      = 'myapp'
  }

  stages {
    stage('Checkout') {
      steps { checkout scm }
    }

    stage('Build') {
      steps {
        sh 'mvn -B -DskipTests package'
        sh "docker build -t ${REGISTRY}/${APP}:${env.GIT_COMMIT} ."
      }
    }

    stage('Test') {
      steps { sh 'mvn -B test' }
      post {
        always {
          junit 'target/surefire-reports/*.xml'
          archiveArtifacts artifacts: 'target/*.jar', fingerprint: true
        }
      }
    }

    stage('Scan') {
      parallel {
        stage('SonarQube') {
          steps {
            withSonarQubeEnv('sonar-prod') {
              sh 'mvn sonar:sonar -Dsonar.projectKey=myapp'
            }
          }
        }
        stage('Trivy') {
          steps {
            sh "trivy image --exit-code 1 --severity HIGH,CRITICAL ${REGISTRY}/${APP}:${env.GIT_COMMIT}"
          }
        }
      }
    }

    stage('Push') {
      when { branch 'main' }
      steps {
        withCredentials([usernamePassword(credentialsId: 'registry', usernameVariable: 'U', passwordVariable: 'P')]) {
          sh 'echo $P | docker login $REGISTRY -u $U --password-stdin'
          sh "docker push ${REGISTRY}/${APP}:${env.GIT_COMMIT}"
        }
      }
    }

    stage('Deploy to staging') {
      when { branch 'main' }
      steps { sh "./deploy.sh staging ${env.GIT_COMMIT}" }
    }

    stage('Promote to production?') {
      when { branch 'main' }
      steps {
        input message: 'Promote build to production?', ok: 'Promote'
        sh "./deploy.sh production ${env.GIT_COMMIT}"
      }
    }
  }

  post {
    success { slackSend channel: '#deploys', message: "Build green: ${env.JOB_NAME} #${env.BUILD_NUMBER}" }
    failure { slackSend channel: '#deploys', message: "Build failed: ${env.JOB_NAME} #${env.BUILD_NUMBER} (${env.BUILD_URL})" }
  }
}

That single file gives you reproducible builds, JUnit-aware test reporting, parallel security scanning, gated production promotion, and Slack notifications.

Stage by stage

Checkout. checkout scm uses the SCM configuration from the Multibranch Pipeline. No manual git URLs, no credentials hard-coded.

Build. Compile and package. Tag the resulting image with the commit SHA, never latest. The SHA gives you a guaranteed-unique, content-addressable identifier you can correlate to git history during incidents.

Test. Unit tests first, integration tests next. The junit step in post.always captures results even if the stage fails, so you can see which tests broke without rerunning.

Scan. SonarQube checks for code smells, bugs, and coverage thresholds. Trivy scans the built image for known CVEs. Both run in parallel because there is no dependency between them and the wall-clock time matters when developers are waiting.

Push and deploy. Only main pushes to the registry and deploys, gated by a credentials block that injects registry credentials safely. Staging deploys automatically; production requires a human input step.

Shared libraries — DRY across services

Once you have ten services with similar pipelines, the copy-paste problem becomes painful. Jenkins shared libraries solve this. Put reusable steps in a separate repo:

// vars/buildAndPushImage.groovy
def call(Map config) {
  sh "docker build -t ${config.registry}/${config.app}:${env.GIT_COMMIT} ."
  withCredentials([usernamePassword(credentialsId: 'registry', usernameVariable: 'U', passwordVariable: 'P')]) {
    sh "echo \$P | docker login ${config.registry} -u \$U --password-stdin"
    sh "docker push ${config.registry}/${config.app}:${env.GIT_COMMIT}"
  }
}

Then every Jenkinsfile collapses to:

@Library('platform-jenkins-lib@v3') _

pipeline {
  agent any
  stages {
    stage('Checkout') { steps { checkout scm } }
    stage('Build & Push') {
      steps { buildAndPushImage(registry: 'registry.example.com', app: 'myapp') }
    }
  }
}

A bug fix in the shared library propagates to every consumer the next time they build, with no per-repo PR.

Controller and agent architecture

Understanding how the controller and agents fit together is the difference between a reliable Jenkins instance and one that wedges every other week.

The controller is the brain. It runs the web UI, owns the configuration, holds credentials in an encrypted store, schedules jobs, persists build history under JENKINS_HOME, and runs every plugin you have installed. It must never run user builds — a misbehaving build script that fills the disk or pegs CPU on the controller takes the whole instance down with it.

Agents are dumb workers. They connect outward to the controller (over JNLP or SSH), receive build steps, run them, and stream logs back. When the build is done, they either go idle (static agents) or are torn down entirely (ephemeral agents). The diagram shows the modern pattern: every build gets its own short-lived pod, sized for the work it has to do, with exactly the toolchain it needs.

A few architectural rules pay back many times over:

• Treat JENKINS_HOME as the source of truth for state, but back it up. Job configurations, build history, plugin data, and credentials all live here. Snapshot it daily. Restore drills should be on the team calendar, not improvised at 3am.
• Manage Jenkins configuration as code. The Configuration as Code (JCasC) plugin lets you describe controllers, plugins, credentials, and global settings in YAML. A new controller can be rebuilt from git in minutes instead of clicked back together over days.
• Keep plugins minimal and pinned. Each plugin is code running on the controller, and plugin compatibility issues are a common upgrade hazard. List dependencies in JCasC and update on a schedule, not reactively.
• Scale agents, not the controller. A single controller can drive thousands of builds per day if those builds run on agents. Vertical scaling of the controller (more RAM) helps far less than people expect.
• Use distinct agent labels per toolchain. Tag pod templates with label: maven-jdk21, label: node-20, etc., and let agent { label 'maven-jdk21' } in the Jenkinsfile pick the right environment. This avoids generic "build" agents that bundle every tool ever needed.

For small teams, a single controller plus a Kubernetes cluster for agents is enough. For large organizations, a multi-controller pattern with shared agent infrastructure (or a Cloudbees Operations Center) keeps blast radius small.

Agents — keep them ephemeral

Long-lived static agents accumulate state. A leftover Maven cache speeds up your build until it silently masks a missing dependency that will fail when production tries to build the same artifact. The fix is ephemeral agents.

The Kubernetes plugin lets you define agent pod templates per build:

pipeline {
  agent {
    kubernetes {
      yaml '''
        apiVersion: v1
        kind: Pod
        spec:
          containers:
          - name: maven
            image: maven:3.9-eclipse-temurin-21
            command: [cat]
            tty: true
      '''
    }
  }
  stages {
    stage('Build') {
      steps { container('maven') { sh 'mvn -B package' } }
    }
  }
}

Every build gets a fresh container. Caching becomes an explicit, opt-in concern (mounted volumes, registry-side layer cache) rather than an accidental property.

Security checklist

• Store every credential in Jenkins Credentials, never in the Jenkinsfile.
• Run docker login with --password-stdin so the password is not visible in process listings.
• Restrict the "Approve" step on production deploys to a specific group.
• Pin shared library versions (@Library('lib@v3') _) so an accidental main push does not deploy untested pipeline code.
• Enable script approval for shared libraries and review changes carefully.
• Use the Configuration as Code plugin to manage Jenkins itself in git.

Common pipeline anti-patterns

• One giant stage — debugging is painful when half a dozen scripts run together. Split for visibility.
• Hard-coded environments — derive environment, registry URLs, and Kubernetes namespaces from env.BRANCH_NAME or parameters.
• Skipping tests on red — fix the test, do not bypass it. Every quality gate you skip eventually causes an outage.
• No timeouts — a hung pipeline blocks an executor for hours. Always set timeout().
• Leaving artifacts forever — buildDiscarder keeps disk usage bounded.

What a good pipeline gets you

• Reproducible builds. Anyone can replay the pipeline on a fresh agent and get the same artifact.
• Fast feedback. Parallel scan stages keep total wall time low.
• Quality gates. Sonar and Trivy fail the build before bad code reaches main.
• Branch-aware deploys. Only main triggers staging; only humans trigger production.
• Auditable history. Every build links commit, test result, scan, and deploy artifact.

Disaster recovery for Jenkins

Jenkins is stateful infrastructure. Treating it like cattle is the eventual goal, but until your entire configuration is in JCasC, plan for the controller to die and have a tested recovery path:

• Daily snapshots of JENKINS_HOME. Use restic, AWS Backup, Velero, or a plain tar | aws s3 cp cron job — the mechanism matters less than the discipline. Retain at least 14 days.
• Encrypted credential store backups. secrets/ inside JENKINS_HOME contains the master key. Without it, the encrypted credentials in your backup are useless. Back up the master key separately, in a secret manager.
• Test restores quarterly. Spin up a fresh controller, restore the most recent backup, and verify a representative subset of jobs builds. A backup you have never restored is a backup that does not work.
• Document plugin versions. A plugins.txt file in git, kept in sync with the running controller, lets you reproduce the exact plugin set on a fresh install. The Jenkins CLI can both export and import this list.
• Have a runbook. Step-by-step: provision new instance → install Java → install Jenkins → restore backup → start service → verify URL → DNS cutover. Stress-test the runbook on a calm afternoon, not during an outage.

A common shortcut is "we can rebuild from git via JCasC" — true in principle, in practice you will be missing build history, fingerprints, and any plugin state that JCasC does not cover. Backups are still the fastest path back to a working controller.

Migrating from freestyle jobs

If you are inheriting a Jenkins instance full of clicked-together freestyle jobs, do not try to convert them all in one weekend. A workable migration plan:

1. Pick the noisiest, most-built job first — usually a flagship service. Replace it with a Multibranch Pipeline pointing at a Jenkinsfile in the repo.
2. Run both the old and new jobs in parallel for a week. Compare results; fix any divergence.
3. Disable (do not delete) the old job. Keep it around for two weeks in case you need to reference its config.
4. Repeat for the next service. Reuse the shared library you build along the way.

Each migration takes a day or two but eliminates a perpetual source of "who changed this and when?" mysteries. Within a few months you will have moved your entire Jenkins instance into git.

A pipeline like this is the spine of a healthy delivery practice — every commit is testable, traceable, and one click away from production. The investment in writing it well repays itself the first time you can ship a critical fix on a Friday afternoon and trust the pipeline to catch the things you forgot to check by hand.

The best signal that your CI/CD is healthy is how often engineers reach for it without thinking. When pushing to a branch is the obvious way to "test in a production-like environment," when a green build is enough to trust a deploy, and when a red build is investigated within minutes — your pipeline is doing its job. When teams start working around CI, building locally, and deploying by hand, it usually means the pipeline is slow, flaky, or hard to debug. Treat those signals as bug reports.