# Security in CI/CD Pipelines > **CNPA Domain:** Platform Observability, Security, and Conformance (20%) **Topic:** Security in CI/CD Pipelines ## Overview CI/CD pipelines have become a high-value attack target. A compromised pipeline can push malicious code to production, steal secrets, or poison artifacts used by downstream teams. **Supply chain security** — ensuring the integrity of code and artifacts from commit to production — is a top priority for platform engineering teams in 2025 and beyond. *** ## The CI/CD Attack Surface ``` Developer CI Pipeline Registry Production │ │ │ │ │─── push ──▶ │ │ │ │ [build + test] │ │ │ [image scan] │ │ │ [sign artifact] ──push──▶ │ │ │ [verify sig] ──deploy──▶│ Attack vectors: ↑ Code injection ↑ Dependency confusion ↑ Image tampering ↑ Secrets theft ↑ Malicious dependencies ↑ Registry hijack ``` *** ## Secrets Management in Pipelines ### Never Store Secrets as Plain Text ```yaml # ❌ Bad: Secret hardcoded - name: Push image run: docker push --password "mysupersecretpassword" ... # ❌ Bad: Secret stored in repository env: DB_PASSWORD: ${{ secrets.DB_PASSWORD }} # OK if GitHub Secrets DB_PASSWORD: "hardcoded-value" # NEVER # ✅ Good: OIDC - no long-lived credentials - name: Configure AWS credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::123456789:role/github-ci-role aws-region: us-east-1 ``` ### OIDC for Cloud Authentication Modern CI systems support **OpenID Connect (OIDC)** — the CI runner proves its identity to cloud providers using a short-lived JWT token. No long-lived credentials needed. ``` GitHub Actions runner → gets short-lived JWT from GitHub OIDC provider → presents JWT to AWS/GCP/Azure → cloud verifies JWT signature against GitHub's public keys → issues temporary cloud credentials → runner uses temporary credentials (expires in ~1 hour) ``` ```yaml # GitHub Actions OIDC with AWS permissions: id-token: write # Required for OIDC contents: read jobs: deploy: runs-on: ubuntu-latest steps: - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: ${{ vars.AWS_ROLE_ARN }} aws-region: us-east-1 ``` *** ## Container Image Scanning Every container image must be scanned for known vulnerabilities before being pushed to production. ### Trivy in CI ```yaml # GitHub Actions: scan image before push - name: Build image run: docker build -t myapp:${{ github.sha }} . - name: Scan with Trivy uses: aquasecurity/trivy-action@master with: image-ref: myapp:${{ github.sha }} format: sarif output: trivy-results.sarif severity: CRITICAL,HIGH exit-code: '1' # Fail pipeline on critical/high CVEs - name: Upload to GitHub Security uses: github/codeql-action/upload-sarif@v3 with: sarif_file: trivy-results.sarif ``` ### Scanning at Multiple Stages | Stage | What | Tool | | -------------- | --------------------- | ------------------- | | **Pre-commit** | IaC misconfigurations | Checkov, tfsec | | **CI build** | Dependencies (SCA) | Trivy, Snyk, Grype | | **CI build** | Container image | Trivy, Grype, Clair | | **Registry** | Continuous after push | Prisma Cloud, Snyk | | **Runtime** | Live threat detection | Falco | *** ## Software Bill of Materials (SBOM) An **SBOM** is a machine-readable inventory of all components in a software artifact — like a supply chain manifest. ```bash # Generate SBOM with Syft syft myapp:abc1234 -o cyclonedx-json > sbom.json syft myapp:abc1234 -o spdx-json > sbom.spdx.json # Attest SBOM to the image cosign attest --predicate sbom.json \ --type cyclonedx \ myapp:abc1234 ``` Consumers can query the SBOM to quickly determine if a vulnerable library (e.g., Log4j) is present in any image. *** ## Image Signing and Verification (Cosign / Sigstore) [Cosign](https://docs.sigstore.dev/cosign/) (part of the [Sigstore](https://www.sigstore.dev/) project) provides tools for signing and verifying container images. ### Signing in CI ```yaml - name: Sign image env: COSIGN_PRIVATE_KEY: ${{ secrets.COSIGN_PRIVATE_KEY }} run: | cosign sign --key env://COSIGN_PRIVATE_KEY \ registry.example.com/myapp:${{ github.sha }} ``` ### Keyless Signing with OIDC Sigstore supports **keyless signing** — no private key to manage. The CI identity (GitHub Actions OIDC) is recorded in a public transparency log: ```bash # Keyless signing (GitHub Actions) cosign sign --yes registry.example.com/myapp:abc1234 ``` ### Verifying Signatures with Kyverno ```yaml # Kyverno: reject unsigned images apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: verify-image-signatures spec: validationFailureAction: Enforce rules: - name: check-image-signature match: any: - resources: kinds: [Pod] verifyImages: - imageReferences: - "registry.example.com/*" attestors: - entries: - keyless: subject: "https://github.com/example/myapp/.github/workflows/ci.yaml@refs/heads/main" issuer: "https://token.actions.githubusercontent.com" ``` *** ## SLSA Framework [SLSA (Supply-chain Levels for Software Artifacts)](https://slsa.dev/) is a security framework defining levels of supply chain integrity: | Level | Requirements | Protection | | ---------- | -------------------------------------------------- | --------------------------- | | **SLSA 1** | Build process documented | Basic provenance | | **SLSA 2** | Version controlled, authenticated build service | Tamper evidence | | **SLSA 3** | Hardened build, isolated, non-forgeable provenance | Stronger tamper protection | | **SLSA 4** | Two-person review, hermetic builds | Full supply chain integrity | ```yaml # Generate SLSA provenance with GitHub Actions - uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1 with: image: registry.example.com/myapp digest: ${{ steps.build.outputs.digest }} ``` *** ## Dependency Security ### Pinning Dependencies ```dockerfile # ❌ Unpinned — vulnerable to supply chain attacks FROM node:20 RUN npm install # ✅ Pinned base image with digest FROM node:20.11.0@sha256:abc123... COPY package-lock.json . RUN npm ci # Uses lockfile exactly ``` ### Dependency Confusion Attacks Configure npm/pip/Maven to use internal registries first: ```bash # npm: use private registry, fallback public npm config set registry https://npm.internal.example.com npm config set '@company:registry' https://npm.internal.example.com ``` *** ## Pipeline Security Best Practices | Practice | Description | | -------------------------- | -------------------------------------------------- | | **Pin action versions** | Use commit SHA in GitHub Actions, not mutable tags | | **Minimal permissions** | Use OIDC; scope IAM roles to specific actions | | **Ephemeral runners** | Fresh runner per job, no persistent credentials | | **Secret rotation** | Rotate all long-lived secrets regularly | | **Audit pipeline changes** | Require PR review for workflow file changes | | **Block `eval` patterns** | Prevent shell injection in pipeline expressions | *** ## Key Takeaways * Use **OIDC** instead of long-lived secrets in CI pipelines * **Scan container images** with Trivy/Grype at build time; fail the pipeline on critical CVEs * Generate **SBOMs** for every artifact to enable rapid vulnerability assessment * **Sign images** with Cosign/Sigstore and verify signatures in Kubernetes via Kyverno * Follow the **SLSA framework** to progressively improve supply chain integrity * Pin base images to digests and use `npm ci` / `pip install --require-hashes` to lock dependencies *** ## Further Reading * [Sigstore / Cosign Documentation](https://docs.sigstore.dev/) * [SLSA Framework](https://slsa.dev/) * [CNCF Security TAG — Supply Chain Security](https://github.com/cncf/tag-security/blob/main/supply-chain-security/supply-chain-security-paper/CNCF_SSCP_v1.pdf) * [Trivy Documentation](https://aquasecurity.github.io/trivy/) * → Next: [Platform Observability and Metrics](https://blog.htunnthuthu.com/getting-started/fundamentals/platform-engineering-101/platform-engineering-101-observability)