DevSecOps & Supply-Chain Security
What you’ll learn. How to build security into the delivery pipeline instead of bolting it on: managing secrets properly (Vault, SOPS, external secrets); the testing taxonomy (SAST, SCA, DAST, IAST, secret scanning, IaC scanning); software supply-chain security (SBOMs, provenance, Sigstore/cosign signing, SLSA levels); least privilege; and hardening containers and Kubernetes. Builds on CI/CD, Docker, and Kubernetes Fundamentals.
The mindset
Section titled “The mindset”DevSecOps extends shift-left to security: make the secure path the easy, automated path; give developers fast, actionable feedback in the PR; and keep security as a shared responsibility rather than a gate at the end.
Three framing ideas:
- Threat model first. Who would attack you, what do they want, and what can they reach? A lightweight STRIDE pass (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege) over a data-flow diagram finds more than a scanner.
- Defense in depth. Assume any single control fails; layer them (identity, network, runtime, detection).
- Assume breach. Limit blast radius, make theft of any one credential not catastrophic, and be able to detect and rotate quickly.
Your CI/CD system is among your most privileged assets: it holds credentials to deploy to production and runs third-party code. Attackers know this; the 2020 SolarWinds build compromise, the 2021 Codecov bash-uploader breach, and the 2024 xz-utils backdoor are the canonical reminders. Securing the pipeline and the supply chain is not optional.
Secrets management
Section titled “Secrets management”A secret is anything that grants access: API keys, passwords, tokens, private keys, certificates, connection strings.
The rules
Section titled “The rules”- Never commit secrets to Git (even private repos, even “temporarily”). History is forever and clones proliferate.
- Never bake secrets into images or pass them as plain build args (Docker).
- Prefer identity over secrets. Use workload identity and short-lived credentials (OIDC federation from CI, IAM roles, Kubernetes workload identity) so there is no static secret to steal (CI/CD, Cloud Fundamentals).
- When a secret is unavoidable: store in a dedicated secret manager, scope tightly, rotate regularly, audit access, and deliver at runtime.
- Assume leaks happen: detect (secret scanning) and rotate. A leaked secret is compromised no matter how fast you delete the commit.
Secret scanning
Section titled “Secret scanning”Prevent and detect committed secrets:
gitleaks detect --source . --redact # scan historygitleaks protect --staged # as a pre-commit hooktrufflehog git file://. --only-verified # verifies whether found creds are liveGitHub push protection and secret scanning block known token formats server-side. Run scanners in pre-commit and CI, because local hooks can be skipped (Git & Workflows).
HashiCorp Vault (and OpenBao)
Section titled “HashiCorp Vault (and OpenBao)”Vault is a secrets and encryption service: central storage with fine-grained policies, audit logs, and, crucially, dynamic secrets (generated on demand with a TTL and revoked automatically). Like Terraform, Vault moved to the BSL license in 2023; OpenBao is the open-source fork under the Linux Foundation. Concepts apply to both.
- Auth methods: how clients prove identity: Kubernetes service account JWT, cloud IAM, OIDC, AppRole, LDAP.
- Secrets engines: KV (static), Database (dynamic per-client DB credentials), PKI (issue short-lived certs), AWS/GCP/Azure (dynamic cloud credentials), Transit (encryption as a service, keys never leave Vault).
- Policies: path-based ACLs.
- Seal/unseal: data at rest encrypted; master key protected by auto-unseal via a cloud KMS.
# policy: app can read its own secrets onlypath "secret/data/myapp/*" { capabilities = ["read"]}path "database/creds/myapp-readonly" { capabilities = ["read"]}vault kv put secret/myapp/config api_key="s3cr3t"vault read database/creds/myapp-readonly# Key Value# lease_id database/creds/myapp-readonly/abc123# lease_duration 1h# username v-k8s-myapp-xyz# password A1b2... <- exists for 1 hour, then revokedDynamic secrets change the game: a stolen credential expires in an hour and each consumer has a unique identity for auditing. The cost is operating Vault (high availability, backups of its storage, unsealing). Managed alternatives: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, HCP Vault.
Getting secrets into Kubernetes
Section titled “Getting secrets into Kubernetes”Kubernetes Secrets are base64, not encryption (Kubernetes Fundamentals). Options:
- External Secrets Operator (ESO): syncs from a cloud/Vault secret store into Kubernetes Secrets on a refresh interval.
apiVersion: external-secrets.io/v1beta1kind: ExternalSecretmetadata: {name: api-secrets, namespace: prod}spec: refreshInterval: 1h secretStoreRef: {kind: ClusterSecretStore, name: aws-secrets-manager} target: {name: api-secrets} data: - secretKey: db-password remoteRef: {key: prod/api/db, property: password}- Secrets Store CSI driver: mounts secrets as volumes directly from the provider (optionally without creating a Kubernetes Secret).
- Vault Agent / Vault Secrets Operator injection.
- Sealed Secrets or SOPS: encrypted secrets committed to Git, decrypted in cluster (GitOps-friendly, see Kubernetes in Production).
- Also enable encryption at rest for etcd (KMS provider).
SOPS (Mozilla, now CNCF) encrypts the values in YAML/JSON/ENV files while leaving keys readable, so diffs and reviews still work, using a KMS (AWS KMS, GCP KMS, Azure Key Vault), age, or PGP keys.
creation_rules: - path_regex: secrets/.*\.yaml$ kms: arn:aws:kms:eu-west-1:123456789012:key/abcd-1234 # or: age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8psops --encrypt --in-place secrets/prod.yaml # values become ENC[AES256_GCM,...]sops secrets/prod.yaml # decrypt in $EDITOR, re-encrypt on savesops --decrypt secrets/prod.yaml | kubectl apply -f -Access is controlled by who can use the KMS key, so audit and rotation come from the KMS. Flux and Argo CD (via plugins) can decrypt SOPS in-cluster. Terraform has a SOPS provider. The model suits small teams that want secrets in Git without running Vault.
Application security testing
Section titled “Application security testing”Different tools find different bug classes at different stages; use several.
| Technique | What it analyzes | Finds | When | Examples |
|---|---|---|---|---|
| SAST (static) | Your source code | Injection patterns, unsafe APIs, hardcoded creds | PR/CI | Semgrep, CodeQL, SonarQube, Bandit, gosec |
| SCA (composition) | Third-party dependencies | Known-vulnerable libraries (CVEs), license issues | PR/CI, scheduled | Dependabot, Renovate, Trivy, Grype, OSV-Scanner, Snyk |
| Secret scanning | Code, history, config | Leaked credentials | pre-commit, CI | gitleaks, trufflehog |
| IaC scanning | Terraform, Kubernetes, Dockerfiles | Misconfigurations | PR/CI | Checkov, Trivy config, KICS (IaC) |
| Container scanning | Image contents | OS/package CVEs | Build, registry, runtime | Trivy, Grype, Docker Scout |
| DAST (dynamic) | A running app, black-box | Runtime issues: XSS, auth flaws, misconfigured headers | Staging, nightly | OWASP ZAP, Burp, Nuclei |
| IAST | Instrumented app during tests | Precise runtime-confirmed flaws | Test envs | Contrast, vendor agents |
| Fuzzing | Inputs to parsers/APIs | Crashes, memory bugs | CI/continuous | AFL++, libFuzzer, Atheris, OSS-Fuzz |
Making scans useful
Section titled “Making scans useful”Scanners drown teams in noise. Practices:
- Fail the build only on high-confidence, high-severity, fixable findings; report the rest as tickets. A gate everyone bypasses is worse than none.
- Diff-aware scanning: report only issues introduced by the PR, so developers are not blamed for legacy debt.
- Prioritize by exploitability and reachability (is the vulnerable function actually called? is the service internet-facing?), using EPSS scores and CISA’s Known Exploited Vulnerabilities list rather than raw CVSS. VEX (Vulnerability Exploitability eXchange) documents let you declare “affected / not affected”.
- Automate dependency updates (Dependabot/Renovate) with CI to merge safe patches quickly. Old dependencies are the most common exposure.
- Triage SLAs (critical within days, high within weeks) with owners and tracking.
name: securityon: pull_request: schedule: [{cron: "0 4 * * *"}]permissions: {contents: read}jobs: scan: runs-on: ubuntu-latest permissions: {contents: read, security-events: write} steps: - uses: actions/checkout@v4 - name: Secret scan uses: gitleaks/gitleaks-action@v2 env: {GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"} - name: SAST (Semgrep) run: | pipx install semgrep semgrep scan --config auto --error --severity ERROR - name: SCA + IaC (Trivy filesystem scan) uses: aquasecurity/trivy-action@0.28.0 with: scan-type: fs scanners: vuln,secret,misconfig severity: CRITICAL,HIGH ignore-unfixed: true exit-code: "1"(Pin actions to commit SHAs in real use; see CI/CD.)
Supply-chain security
Section titled “Supply-chain security”Modern software is mostly other people’s code. Supply-chain attacks target what you consume (malicious or typosquatted packages, compromised maintainers, dependency confusion), what you build with (compromised CI, poisoned actions, build tools), and what you ship (tampered artifacts). The questions to answer: what is in this artifact, where did it come from, who built it, and has it been tampered with?
A Software Bill of Materials is an inventory of components in an artifact (name, version, supplier, hashes, licenses, dependency relationships). Standard formats: SPDX (Linux Foundation/ISO 5962) and CycloneDX (OWASP). Uses: respond fast to new CVEs (“are we affected by the next Log4Shell?”), license compliance, and regulatory requirements (US Executive Order 14028 guidance, EU Cyber Resilience Act).
syft ghcr.io/me/api:1.4.2 -o cyclonedx-json > sbom.cdx.jsongrype sbom:sbom.cdx.json --fail-on high # scan the SBOM for known vulnstrivy image --format cyclonedx --output sbom.json ghcr.io/me/api:1.4.2Generate the SBOM at build time (from the lockfile/build graph, most accurate), attach it to the image in the registry as an attestation, and store it for later querying (Dependency-Track is an open-source platform for this). An SBOM is only useful if you act on it.
Signing and Sigstore
Section titled “Signing and Sigstore”A digital signature binds an artifact to an identity so consumers can verify authenticity and integrity. Traditional signing (GPG, long-lived keys) fails in practice: key management is hard, and keys leak or are lost.
Sigstore (OpenSSF) enables keyless signing:
- cosign: signs and verifies container images and other artifacts (signature stored in the OCI registry).
- Fulcio: a certificate authority that issues a short-lived certificate (about 10 minutes) bound to an OIDC identity (e.g., a GitHub Actions workflow identity or a person’s email).
- Rekor: a public, append-only transparency log recording signatures, so signing events are auditable and tamper-evident.
CI job --(OIDC token: "workflow X in repo Y")--> Fulcio --> short-lived signing certCI job: sign image digest with ephemeral key --> signature + cert --> registry + Rekor logVerifier: checks signature, cert identity matches expected workflow, log inclusionThere is no long-lived private key to steal; trust is anchored in which identity signed. Verification must specify the expected identity and issuer, otherwise “signed by anyone” proves nothing.
# in a GitHub Actions job with permissions: id-token: write, packages: write- uses: sigstore/cosign-installer@v3- name: Sign image by digest run: cosign sign --yes ghcr.io/${{ github.repository }}@${{ steps.push.outputs.digest }}- name: Attach SBOM attestation run: | cosign attest --yes --type cyclonedx --predicate sbom.cdx.json \ ghcr.io/${{ github.repository }}@${{ steps.push.outputs.digest }}cosign verify ghcr.io/myorg/api@sha256:... \ --certificate-identity-regexp '^https://github.com/myorg/api/\.github/workflows/release\.yml@refs/heads/main$' \ --certificate-oidc-issuer https://token.actions.githubusercontent.comSigning is worthless if nobody verifies. Enforce verification at admission in Kubernetes with Kyverno verifyImages, Sigstore policy-controller, or Ratify, so unsigned or foreign images are rejected. Always sign and deploy by digest, not tag. Signed Git commits and tags (gitsign, SSH signing) cover the source side (Git & Workflows).
Provenance and SLSA
Section titled “Provenance and SLSA”Provenance is a signed statement of how an artifact was built: source repository and commit, builder, build parameters, timestamps. SLSA (“salsa”, Supply-chain Levels for Software Artifacts; OpenSSF) is a framework of levels describing build integrity. In the current (v1.0) specification the Build track has:
| Level | Meaning (summarized) |
|---|---|
| Build L0 | No guarantees |
| Build L1 | Provenance exists (documents how it was built), but may be forgeable |
| Build L2 | Hosted build platform generates and signs provenance; harder to forge |
| Build L3 | Hardened build platform: isolated builds, provenance that the tenant’s build steps cannot forge or tamper with |
The idea: even if a developer’s account or a build script is malicious, a trustworthy platform produces unforgeable evidence of what really happened. In GitHub Actions, the slsa-github-generator reusable workflows and GitHub’s artifact attestations (actions/attest-build-provenance) generate provenance that can be verified (gh attestation verify, cosign verify-attestation, or slsa-verifier). Check the current spec for details; the framework is evolving (additional tracks are in development).
Hardening the dependency and build chain
Section titled “Hardening the dependency and build chain”- Pin and lock: lockfiles with hashes (
pip install --require-hashes,npm ci), digests for base images, commit SHAs for CI actions. - Reduce dependencies; vet new ones (maintainers, activity, install scripts). Beware typosquatting and dependency confusion: scope internal packages to your registry and use a proxy/artifact repository (Artifactory, Nexus, Artifact Registry) that mediates and caches upstream.
- OpenSSF Scorecard scores a repo’s security practices;
npm audit signatures/provenance and PyPI trusted publishing (OIDC publishing without API tokens) improve registry-side trust. - Build hermetically and reproducibly where possible: if two independent builds from the same source produce identical bits, tampering becomes detectable. Tools: Bazel, Nix,
SOURCE_DATE_EPOCH, ko/Jib for deterministic images. - Ephemeral, isolated runners; no persistent credentials on build machines; separate build and deploy permissions; protect the main branch and require review (two-person rule) for changes to pipeline definitions.
- Pipeline injection (untrusted PR titles/branch names in
run:steps,pull_request_target) is a top CI risk; see CI/CD.
Least privilege
Section titled “Least privilege”The principle: every identity (human, service, pipeline) gets only the permissions needed, only when needed, only for as long as needed.
- Cloud IAM: narrow policies; role per workload; permission boundaries; review with access analyzers; time-bound, approved elevation (just-in-time access) for humans instead of standing admin (Cloud Fundamentals).
- CI/CD: read-only default
GITHUB_TOKEN; separate roles per environment; plan role is read-only, apply role is gated (IaC). - Kubernetes RBAC: namespaced Roles over ClusterRoles, no wildcards, one ServiceAccount per workload.
- Network: default-deny, allow-listing (security groups, NetworkPolicy).
- Data: column-level or row-level access, separate credentials for read-only analytics.
- Break-glass: documented emergency access, heavily logged and alerted.
Audit regularly and remove unused permissions; permissions only ever grow unless someone prunes them.
Container and Kubernetes hardening
Section titled “Container and Kubernetes hardening”Image and container level (Docker)
Section titled “Image and container level (Docker)”Minimal base images (distroless), non-root user, no shell in production images, read-only root filesystem, dropped capabilities, no privileged mode, pinned and scanned images, signed images.
Kubernetes pod security
Section titled “Kubernetes pod security”Use Pod Security Admission with the restricted profile (Pod Security Standards: privileged, baseline, restricted), which enforces non-root, no privilege escalation, dropped capabilities, seccomp RuntimeDefault, and no hostPath/hostNetwork.
apiVersion: v1kind: Namespacemetadata: name: prod labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/enforce-version: latestA hardened container securityContext:
securityContext: runAsNonRoot: true runAsUser: 10001 allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: {drop: ["ALL"]} seccompProfile: {type: RuntimeDefault}Also: automountServiceAccountToken: false where the API is not needed; resource limits to prevent noisy-neighbor DoS; NetworkPolicies default-deny (Kubernetes in Production); no use of hostPath, hostPID, or docker socket mounts; user namespaces for pods where supported.
Cluster level
Section titled “Cluster level”- Managed control plane with private or restricted API endpoint; SSO with short-lived credentials for humans; audit logging enabled and shipped off-cluster.
- Admission policy (Kyverno, OPA Gatekeeper, ValidatingAdmissionPolicy) to enforce: required labels, no
latesttags, only approved registries, image signatures, no privileged pods (IaC policy-as-code). - Keep Kubernetes and nodes patched (Kubernetes in Production upgrades); use hardened, minimal, immutable node OS (Bottlerocket, Talos, Flatcar).
- Benchmark with CIS Kubernetes Benchmark (
kube-bench), and scan live clusters (Trivy Operator, Kubescape). - Runtime detection: Falco or Tetragon (eBPF) watch syscalls for suspicious behavior (shell spawned in a container, unexpected outbound connection, reading
/etc/shadow). For stronger isolation of untrusted workloads use gVisor or Kata Containers. - Protect etcd (encryption at rest, restricted access) and the kubelet API.
- Egress control: most attacks need to call home; restrict outbound traffic.
Detection and response
Section titled “Detection and response”Prevention fails eventually. Centralize audit logs (cloud trail, K8s audit, auth logs), alert on anomalous events (new admin, key creation, disabled logging), and rehearse credential rotation and revocation (Incident & Reliability). Know how to answer “which secrets could this compromised pod read?” fast.
A pragmatic order of operations
Section titled “A pragmatic order of operations”If starting from little, prioritize by payoff:
- Remove static cloud keys from CI (OIDC) and enable MFA/SSO everywhere.
- Secret scanning with push protection; rotate any found secret.
- Dependency updates automated (Dependabot/Renovate) and SCA in CI.
- Minimal, non-root images with scanning; Pod Security
restricted. - Least-privilege IAM and RBAC; separate accounts/environments.
- SAST on PRs with a tuned ruleset; IaC scanning.
- Build provenance + signing + SBOM; verify at admission.
- Runtime detection and incident rehearsals.
Key takeaways
Section titled “Key takeaways”- Security is a delivery-pipeline property: automate checks early, give actionable feedback, and tune gates to avoid alert fatigue.
- Best secret is no secret: use identity federation and short-lived credentials; otherwise use a secret manager (Vault/cloud), SOPS or ESO for GitOps, and rotate on leak.
- SAST, SCA, secret, IaC, container, DAST, and fuzzing each catch different problems; prioritize by exploitability, not raw CVSS.
- SBOMs inventory what you ship; cosign/Sigstore keyless signing binds artifacts to build identities; SLSA provenance proves how they were built; verify at admission or the signing is theater.
- Least privilege, everywhere, with auditing and just-in-time elevation.
- Harden containers and clusters with non-root, dropped capabilities, read-only filesystems, Pod Security
restricted, NetworkPolicies, admission policy, and runtime detection.
Exercises
Section titled “Exercises”- Hands-on. Add gitleaks as a pre-commit hook and CI step. Commit a fake AWS key in a test branch to confirm it is blocked; then explain why you must still rotate a key that was pushed.
- Hands-on. Build an image in GitHub Actions, sign it with keyless cosign, and verify it with a strict
--certificate-identity. Try verifying with a wrong identity and observe the failure. - Hands-on. Generate an SBOM for an image with Syft and scan it with Grype. Pick one finding and decide, with evidence, whether it is reachable.
- Conceptual. Why is “we scan images for CVEs” insufficient as a supply-chain defense? List three attack types it misses.
- Design. Write Kyverno (or ValidatingAdmissionPolicy) rules that reject Pods that run as root, use the
latesttag, or pull from outside your registry.
Solution notes (4)
CVE scanning finds only known vulnerabilities in known components. It misses a malicious package or backdoor with no CVE (xz-style), a compromised build system or poisoned CI action that tampers with the artifact after tests, typosquatted/dependency-confusion packages, stolen registry or signing credentials, and misconfigurations (over-broad IAM, exposed secrets). Provenance, signing with verification, pinning, and least privilege address these.
Further reading
Section titled “Further reading”- OWASP resources: Top 10, Application Security Verification Standard (ASVS), and the CI/CD Top 10 risks.
- Liz Rice, Container Security (2020); Kubernetes docs on Pod Security Standards and the security checklist.
- SLSA specification (slsa.dev) and OpenSSF Scorecard documentation; Sigstore documentation (cosign, Fulcio, Rekor).
- NIST SP 800-218 (Secure Software Development Framework, SSDF) and CISA guidance on SBOMs.
- Vault and OpenBao documentation; SOPS documentation.
- Adam Shostack, Threat Modeling: Designing for Security (2014).