Skip to content

DevSecOps & Supply-Chain Security

What you’ll learn. How to build security into the delivery pipeline instead of bolting it on: managing secrets properly (Vault, SOPS, external secrets); the testing taxonomy (SAST, SCA, DAST, IAST, secret scanning, IaC scanning); software supply-chain security (SBOMs, provenance, Sigstore/cosign signing, SLSA levels); least privilege; and hardening containers and Kubernetes. Builds on CI/CD, Docker, and Kubernetes Fundamentals.

DevSecOps extends shift-left to security: make the secure path the easy, automated path; give developers fast, actionable feedback in the PR; and keep security as a shared responsibility rather than a gate at the end.

Three framing ideas:

  • Threat model first. Who would attack you, what do they want, and what can they reach? A lightweight STRIDE pass (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) over a data-flow diagram finds more than a scanner.
  • Defense in depth. Assume any single control fails; layer them (identity, network, runtime, detection).
  • Assume breach. Limit blast radius, make theft of any one credential not catastrophic, and be able to detect and rotate quickly.

Your CI/CD system is among your most privileged assets: it holds credentials to deploy to production and runs third-party code. Attackers know this; the 2020 SolarWinds build compromise, the 2021 Codecov bash-uploader breach, and the 2024 xz-utils backdoor are the canonical reminders. Securing the pipeline and the supply chain is not optional.

A secret is anything that grants access: API keys, passwords, tokens, private keys, certificates, connection strings.

  1. Never commit secrets to Git (even private repos, even “temporarily”). History is forever and clones proliferate.
  2. Never bake secrets into images or pass them as plain build args (Docker).
  3. Prefer identity over secrets. Use workload identity and short-lived credentials (OIDC federation from CI, IAM roles, Kubernetes workload identity) so there is no static secret to steal (CI/CD, Cloud Fundamentals).
  4. When a secret is unavoidable: store in a dedicated secret manager, scope tightly, rotate regularly, audit access, and deliver at runtime.
  5. Assume leaks happen: detect (secret scanning) and rotate. A leaked secret is compromised no matter how fast you delete the commit.

Prevent and detect committed secrets:

Terminal window
gitleaks detect --source . --redact # scan history
gitleaks protect --staged # as a pre-commit hook
trufflehog git file://. --only-verified # verifies whether found creds are live

GitHub push protection and secret scanning block known token formats server-side. Run scanners in pre-commit and CI, because local hooks can be skipped (Git & Workflows).

Vault is a secrets and encryption service: central storage with fine-grained policies, audit logs, and, crucially, dynamic secrets (generated on demand with a TTL and revoked automatically). Like Terraform, Vault moved to the BSL license in 2023; OpenBao is the open-source fork under the Linux Foundation. Concepts apply to both.

  • Auth methods: how clients prove identity: Kubernetes service account JWT, cloud IAM, OIDC, AppRole, LDAP.
  • Secrets engines: KV (static), Database (dynamic per-client DB credentials), PKI (issue short-lived certs), AWS/GCP/Azure (dynamic cloud credentials), Transit (encryption as a service, keys never leave Vault).
  • Policies: path-based ACLs.
  • Seal/unseal: data at rest encrypted; master key protected by auto-unseal via a cloud KMS.
# policy: app can read its own secrets only
path "secret/data/myapp/*" {
capabilities = ["read"]
}
path "database/creds/myapp-readonly" {
capabilities = ["read"]
}
Terminal window
vault kv put secret/myapp/config api_key="s3cr3t"
vault read database/creds/myapp-readonly
# Key Value
# lease_id database/creds/myapp-readonly/abc123
# lease_duration 1h
# username v-k8s-myapp-xyz
# password A1b2... <- exists for 1 hour, then revoked

Dynamic secrets change the game: a stolen credential expires in an hour and each consumer has a unique identity for auditing. The cost is operating Vault (high availability, backups of its storage, unsealing). Managed alternatives: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HCP Vault.

Kubernetes Secrets are base64, not encryption (Kubernetes Fundamentals). Options:

  • External Secrets Operator (ESO): syncs from a cloud/Vault secret store into Kubernetes Secrets on a refresh interval.
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: {name: api-secrets, namespace: prod}
spec:
refreshInterval: 1h
secretStoreRef: {kind: ClusterSecretStore, name: aws-secrets-manager}
target: {name: api-secrets}
data:
- secretKey: db-password
remoteRef: {key: prod/api/db, property: password}
  • Secrets Store CSI driver: mounts secrets as volumes directly from the provider (optionally without creating a Kubernetes Secret).
  • Vault Agent / Vault Secrets Operator injection.
  • Sealed Secrets or SOPS: encrypted secrets committed to Git, decrypted in cluster (GitOps-friendly, see Kubernetes in Production).
  • Also enable encryption at rest for etcd (KMS provider).

SOPS (Mozilla, now CNCF) encrypts the values in YAML/JSON/ENV files while leaving keys readable, so diffs and reviews still work, using a KMS (AWS KMS, GCP KMS, Azure Key Vault), age, or PGP keys.

.sops.yaml
creation_rules:
- path_regex: secrets/.*\.yaml$
kms: arn:aws:kms:eu-west-1:123456789012:key/abcd-1234
# or: age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
Terminal window
sops --encrypt --in-place secrets/prod.yaml # values become ENC[AES256_GCM,...]
sops secrets/prod.yaml # decrypt in $EDITOR, re-encrypt on save
sops --decrypt secrets/prod.yaml | kubectl apply -f -

Access is controlled by who can use the KMS key, so audit and rotation come from the KMS. Flux and Argo CD (via plugins) can decrypt SOPS in-cluster. Terraform has a SOPS provider. The model suits small teams that want secrets in Git without running Vault.

Different tools find different bug classes at different stages; use several.

Technique What it analyzes Finds When Examples
SAST (static) Your source code Injection patterns, unsafe APIs, hardcoded creds PR/CI Semgrep, CodeQL, SonarQube, Bandit, gosec
SCA (composition) Third-party dependencies Known-vulnerable libraries (CVEs), license issues PR/CI, scheduled Dependabot, Renovate, Trivy, Grype, OSV-Scanner, Snyk
Secret scanning Code, history, config Leaked credentials pre-commit, CI gitleaks, trufflehog
IaC scanning Terraform, Kubernetes, Dockerfiles Misconfigurations PR/CI Checkov, Trivy config, KICS (IaC)
Container scanning Image contents OS/package CVEs Build, registry, runtime Trivy, Grype, Docker Scout
DAST (dynamic) A running app, black-box Runtime issues: XSS, auth flaws, misconfigured headers Staging, nightly OWASP ZAP, Burp, Nuclei
IAST Instrumented app during tests Precise runtime-confirmed flaws Test envs Contrast, vendor agents
Fuzzing Inputs to parsers/APIs Crashes, memory bugs CI/continuous AFL++, libFuzzer, Atheris, OSS-Fuzz

Scanners drown teams in noise. Practices:

  • Fail the build only on high-confidence, high-severity, fixable findings; report the rest as tickets. A gate everyone bypasses is worse than none.
  • Diff-aware scanning: report only issues introduced by the PR, so developers are not blamed for legacy debt.
  • Prioritize by exploitability and reachability (is the vulnerable function actually called? is the service internet-facing?), using EPSS scores and CISA’s Known Exploited Vulnerabilities list rather than raw CVSS. VEX (Vulnerability Exploitability eXchange) documents let you declare “affected / not affected”.
  • Automate dependency updates (Dependabot/Renovate) with CI to merge safe patches quickly. Old dependencies are the most common exposure.
  • Triage SLAs (critical within days, high within weeks) with owners and tracking.
.github/workflows/security.yml
name: security
on:
pull_request:
schedule: [{cron: "0 4 * * *"}]
permissions: {contents: read}
jobs:
scan:
runs-on: ubuntu-latest
permissions: {contents: read, security-events: write}
steps:
- uses: actions/checkout@v4
- name: Secret scan
uses: gitleaks/gitleaks-action@v2
env: {GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"}
- name: SAST (Semgrep)
run: |
pipx install semgrep
semgrep scan --config auto --error --severity ERROR
- name: SCA + IaC (Trivy filesystem scan)
uses: aquasecurity/trivy-action@0.28.0
with:
scan-type: fs
scanners: vuln,secret,misconfig
severity: CRITICAL,HIGH
ignore-unfixed: true
exit-code: "1"

(Pin actions to commit SHAs in real use; see CI/CD.)

Modern software is mostly other people’s code. Supply-chain attacks target what you consume (malicious or typosquatted packages, compromised maintainers, dependency confusion), what you build with (compromised CI, poisoned actions, build tools), and what you ship (tampered artifacts). The questions to answer: what is in this artifact, where did it come from, who built it, and has it been tampered with?

A Software Bill of Materials is an inventory of components in an artifact (name, version, supplier, hashes, licenses, dependency relationships). Standard formats: SPDX (Linux Foundation/ISO 5962) and CycloneDX (OWASP). Uses: respond fast to new CVEs (“are we affected by the next Log4Shell?”), license compliance, and regulatory requirements (US Executive Order 14028 guidance, EU Cyber Resilience Act).

Terminal window
syft ghcr.io/me/api:1.4.2 -o cyclonedx-json > sbom.cdx.json
grype sbom:sbom.cdx.json --fail-on high # scan the SBOM for known vulns
trivy image --format cyclonedx --output sbom.json ghcr.io/me/api:1.4.2

Generate the SBOM at build time (from the lockfile/build graph, most accurate), attach it to the image in the registry as an attestation, and store it for later querying (Dependency-Track is an open-source platform for this). An SBOM is only useful if you act on it.

A digital signature binds an artifact to an identity so consumers can verify authenticity and integrity. Traditional signing (GPG, long-lived keys) fails in practice: key management is hard, and keys leak or are lost.

Sigstore (OpenSSF) enables keyless signing:

  • cosign: signs and verifies container images and other artifacts (signature stored in the OCI registry).
  • Fulcio: a certificate authority that issues a short-lived certificate (about 10 minutes) bound to an OIDC identity (e.g., a GitHub Actions workflow identity or a person’s email).
  • Rekor: a public, append-only transparency log recording signatures, so signing events are auditable and tamper-evident.
CI job --(OIDC token: "workflow X in repo Y")--> Fulcio --> short-lived signing cert
CI job: sign image digest with ephemeral key --> signature + cert --> registry + Rekor log
Verifier: checks signature, cert identity matches expected workflow, log inclusion

There is no long-lived private key to steal; trust is anchored in which identity signed. Verification must specify the expected identity and issuer, otherwise “signed by anyone” proves nothing.

# in a GitHub Actions job with permissions: id-token: write, packages: write
- uses: sigstore/cosign-installer@v3
- name: Sign image by digest
run: cosign sign --yes ghcr.io/${{ github.repository }}@${{ steps.push.outputs.digest }}
- name: Attach SBOM attestation
run: |
cosign attest --yes --type cyclonedx --predicate sbom.cdx.json \
ghcr.io/${{ github.repository }}@${{ steps.push.outputs.digest }}
Terminal window
cosign verify ghcr.io/myorg/api@sha256:... \
--certificate-identity-regexp '^https://github.com/myorg/api/\.github/workflows/release\.yml@refs/heads/main$' \
--certificate-oidc-issuer https://token.actions.githubusercontent.com

Signing is worthless if nobody verifies. Enforce verification at admission in Kubernetes with Kyverno verifyImages, Sigstore policy-controller, or Ratify, so unsigned or foreign images are rejected. Always sign and deploy by digest, not tag. Signed Git commits and tags (gitsign, SSH signing) cover the source side (Git & Workflows).

Provenance is a signed statement of how an artifact was built: source repository and commit, builder, build parameters, timestamps. SLSA (“salsa”, Supply-chain Levels for Software Artifacts; OpenSSF) is a framework of levels describing build integrity. In the current (v1.0) specification the Build track has:

Level Meaning (summarized)
Build L0 No guarantees
Build L1 Provenance exists (documents how it was built), but may be forgeable
Build L2 Hosted build platform generates and signs provenance; harder to forge
Build L3 Hardened build platform: isolated builds, provenance that the tenant’s build steps cannot forge or tamper with

The idea: even if a developer’s account or a build script is malicious, a trustworthy platform produces unforgeable evidence of what really happened. In GitHub Actions, the slsa-github-generator reusable workflows and GitHub’s artifact attestations (actions/attest-build-provenance) generate provenance that can be verified (gh attestation verify, cosign verify-attestation, or slsa-verifier). Check the current spec for details; the framework is evolving (additional tracks are in development).

  • Pin and lock: lockfiles with hashes (pip install --require-hashes, npm ci), digests for base images, commit SHAs for CI actions.
  • Reduce dependencies; vet new ones (maintainers, activity, install scripts). Beware typosquatting and dependency confusion: scope internal packages to your registry and use a proxy/artifact repository (Artifactory, Nexus, Artifact Registry) that mediates and caches upstream.
  • OpenSSF Scorecard scores a repo’s security practices; npm audit signatures/provenance and PyPI trusted publishing (OIDC publishing without API tokens) improve registry-side trust.
  • Build hermetically and reproducibly where possible: if two independent builds from the same source produce identical bits, tampering becomes detectable. Tools: Bazel, Nix, SOURCE_DATE_EPOCH, ko/Jib for deterministic images.
  • Ephemeral, isolated runners; no persistent credentials on build machines; separate build and deploy permissions; protect the main branch and require review (two-person rule) for changes to pipeline definitions.
  • Pipeline injection (untrusted PR titles/branch names in run: steps, pull_request_target) is a top CI risk; see CI/CD.

The principle: every identity (human, service, pipeline) gets only the permissions needed, only when needed, only for as long as needed.

  • Cloud IAM: narrow policies; role per workload; permission boundaries; review with access analyzers; time-bound, approved elevation (just-in-time access) for humans instead of standing admin (Cloud Fundamentals).
  • CI/CD: read-only default GITHUB_TOKEN; separate roles per environment; plan role is read-only, apply role is gated (IaC).
  • Kubernetes RBAC: namespaced Roles over ClusterRoles, no wildcards, one ServiceAccount per workload.
  • Network: default-deny, allow-listing (security groups, NetworkPolicy).
  • Data: column-level or row-level access, separate credentials for read-only analytics.
  • Break-glass: documented emergency access, heavily logged and alerted.

Audit regularly and remove unused permissions; permissions only ever grow unless someone prunes them.

Minimal base images (distroless), non-root user, no shell in production images, read-only root filesystem, dropped capabilities, no privileged mode, pinned and scanned images, signed images.

Use Pod Security Admission with the restricted profile (Pod Security Standards: privileged, baseline, restricted), which enforces non-root, no privilege escalation, dropped capabilities, seccomp RuntimeDefault, and no hostPath/hostNetwork.

apiVersion: v1
kind: Namespace
metadata:
name: prod
labels:
pod-security.kubernetes.io/enforce: restricted
pod-security.kubernetes.io/enforce-version: latest

A hardened container securityContext:

securityContext:
runAsNonRoot: true
runAsUser: 10001
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: {drop: ["ALL"]}
seccompProfile: {type: RuntimeDefault}

Also: automountServiceAccountToken: false where the API is not needed; resource limits to prevent noisy-neighbor DoS; NetworkPolicies default-deny (Kubernetes in Production); no use of hostPath, hostPID, or docker socket mounts; user namespaces for pods where supported.

  • Managed control plane with private or restricted API endpoint; SSO with short-lived credentials for humans; audit logging enabled and shipped off-cluster.
  • Admission policy (Kyverno, OPA Gatekeeper, ValidatingAdmissionPolicy) to enforce: required labels, no latest tags, only approved registries, image signatures, no privileged pods (IaC policy-as-code).
  • Keep Kubernetes and nodes patched (Kubernetes in Production upgrades); use hardened, minimal, immutable node OS (Bottlerocket, Talos, Flatcar).
  • Benchmark with CIS Kubernetes Benchmark (kube-bench), and scan live clusters (Trivy Operator, Kubescape).
  • Runtime detection: Falco or Tetragon (eBPF) watch syscalls for suspicious behavior (shell spawned in a container, unexpected outbound connection, reading /etc/shadow). For stronger isolation of untrusted workloads use gVisor or Kata Containers.
  • Protect etcd (encryption at rest, restricted access) and the kubelet API.
  • Egress control: most attacks need to call home; restrict outbound traffic.

Prevention fails eventually. Centralize audit logs (cloud trail, K8s audit, auth logs), alert on anomalous events (new admin, key creation, disabled logging), and rehearse credential rotation and revocation (Incident & Reliability). Know how to answer “which secrets could this compromised pod read?” fast.

If starting from little, prioritize by payoff:

  1. Remove static cloud keys from CI (OIDC) and enable MFA/SSO everywhere.
  2. Secret scanning with push protection; rotate any found secret.
  3. Dependency updates automated (Dependabot/Renovate) and SCA in CI.
  4. Minimal, non-root images with scanning; Pod Security restricted.
  5. Least-privilege IAM and RBAC; separate accounts/environments.
  6. SAST on PRs with a tuned ruleset; IaC scanning.
  7. Build provenance + signing + SBOM; verify at admission.
  8. Runtime detection and incident rehearsals.
  • Security is a delivery-pipeline property: automate checks early, give actionable feedback, and tune gates to avoid alert fatigue.
  • Best secret is no secret: use identity federation and short-lived credentials; otherwise use a secret manager (Vault/cloud), SOPS or ESO for GitOps, and rotate on leak.
  • SAST, SCA, secret, IaC, container, DAST, and fuzzing each catch different problems; prioritize by exploitability, not raw CVSS.
  • SBOMs inventory what you ship; cosign/Sigstore keyless signing binds artifacts to build identities; SLSA provenance proves how they were built; verify at admission or the signing is theater.
  • Least privilege, everywhere, with auditing and just-in-time elevation.
  • Harden containers and clusters with non-root, dropped capabilities, read-only filesystems, Pod Security restricted, NetworkPolicies, admission policy, and runtime detection.
  1. Hands-on. Add gitleaks as a pre-commit hook and CI step. Commit a fake AWS key in a test branch to confirm it is blocked; then explain why you must still rotate a key that was pushed.
  2. Hands-on. Build an image in GitHub Actions, sign it with keyless cosign, and verify it with a strict --certificate-identity. Try verifying with a wrong identity and observe the failure.
  3. Hands-on. Generate an SBOM for an image with Syft and scan it with Grype. Pick one finding and decide, with evidence, whether it is reachable.
  4. Conceptual. Why is “we scan images for CVEs” insufficient as a supply-chain defense? List three attack types it misses.
  5. Design. Write Kyverno (or ValidatingAdmissionPolicy) rules that reject Pods that run as root, use the latest tag, or pull from outside your registry.
Solution notes (4)

CVE scanning finds only known vulnerabilities in known components. It misses a malicious package or backdoor with no CVE (xz-style), a compromised build system or poisoned CI action that tampers with the artifact after tests, typosquatted/dependency-confusion packages, stolen registry or signing credentials, and misconfigurations (over-broad IAM, exposed secrets). Provenance, signing with verification, pinning, and least privilege address these.

  • OWASP resources: Top 10, Application Security Verification Standard (ASVS), and the CI/CD Top 10 risks.
  • Liz Rice, Container Security (2020); Kubernetes docs on Pod Security Standards and the security checklist.
  • SLSA specification (slsa.dev) and OpenSSF Scorecard documentation; Sigstore documentation (cosign, Fulcio, Rekor).
  • NIST SP 800-218 (Secure Software Development Framework, SSDF) and CISA guidance on SBOMs.
  • Vault and OpenBao documentation; SOPS documentation.
  • Adam Shostack, Threat Modeling: Designing for Security (2014).