Article 10: CI/CD Fundamentals

Introduction

Continuous Integration and Continuous Delivery (CI/CD) automate the process of testing, building, and deploying your code. Through implementing pipelines for numerous projects, I've learned that good CI/CD isn't just about automation—it's about building confidence in every change.

This article covers GitHub Actions for Python projects, including automated testing, linting, and deployment workflows.

What is CI/CD?

Term	Description
Continuous Integration	Automatically test and validate every code change
Continuous Delivery	Automatically prepare releases for deployment
Continuous Deployment	Automatically deploy every passing change

Benefits

• Catch bugs early - Tests run on every push

• Consistent quality - Same checks for every change

• Faster feedback - Know if your code works in minutes

• Reduced manual work - No manual testing or deployment

• Documentation - Pipeline shows what's needed to ship

GitHub Actions Basics

Workflow Structure

# .github/workflows/ci.yml
name: CI Pipeline              # Workflow name

on:                            # Trigger events
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:                          # Jobs to run
  test:                        # Job name
    runs-on: ubuntu-latest     # Runner environment
    
    steps:                     # Steps in the job
      - uses: actions/checkout@v4
      - name: Run tests
        run: pytest

Triggers

on:
  # On push to specific branches
  push:
    branches: [main, develop]
    paths:
      - 'src/**'
      - 'tests/**'
      - 'pyproject.toml'
  
  # On pull requests
  pull_request:
    branches: [main]
  
  # On schedule (cron syntax)
  schedule:
    - cron: '0 0 * * *'  # Daily at midnight
  
  # Manual trigger
  workflow_dispatch:
    inputs:
      environment:
        description: 'Environment to deploy to'
        required: true
        default: 'staging'
  
  # On release
  release:
    types: [published]

Basic Python Workflow

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -e ".[dev]"
      
      - name: Run tests
        run: pytest -v

Comprehensive CI Pipeline

Full Python CI Workflow

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  PYTHON_VERSION: '3.11'

jobs:
  # Linting job
  lint:
    name: Lint
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      
      - name: Install dependencies
        run: |
          pip install ruff mypy
          pip install -e ".[dev]"
      
      - name: Run Ruff
        run: ruff check src tests
      
      - name: Run Ruff Format Check
        run: ruff format --check src tests
      
      - name: Run mypy
        run: mypy src

  # Testing job
  test:
    name: Test (Python ${{ matrix.python-version }})
    runs-on: ubuntu-latest
    
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      
      - name: Cache pip packages
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }}
          restore-keys: |
            ${{ runner.os }}-pip-
      
      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          pip install -e ".[dev]"
      
      - name: Run tests with coverage
        run: |
          pytest --cov=src --cov-report=xml --cov-report=term-missing
      
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage.xml
          fail_ci_if_error: true

  # Security scanning
  security:
    name: Security Scan
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      
      - name: Install dependencies
        run: pip install bandit safety
      
      - name: Run Bandit
        run: bandit -r src -ll
      
      - name: Check dependencies for vulnerabilities
        run: safety check

  # Build job (only on main)
  build:
    name: Build Package
    runs-on: ubuntu-latest
    needs: [lint, test, security]
    if: github.ref == 'refs/heads/main'
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ env.PYTHON_VERSION }}
      
      - name: Install build tools
        run: pip install build twine
      
      - name: Build package
        run: python -m build
      
      - name: Check package
        run: twine check dist/*
      
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/

Matrix Testing

jobs:
  test:
    runs-on: ${{ matrix.os }}
    
    strategy:
      fail-fast: false  # Don't cancel other jobs if one fails
      matrix:
        os: [ubuntu-latest, macos-latest, windows-latest]
        python-version: ['3.10', '3.11', '3.12']
        exclude:
          - os: macos-latest
            python-version: '3.10'
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
      
      - name: Run tests
        run: pytest

Caching and Optimization

Dependency Caching

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'  # Built-in pip caching
      
      # Alternative: explicit caching
      - name: Cache pip packages
        uses: actions/cache@v4
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml', 'requirements*.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
      
      - name: Install dependencies
        run: pip install -e ".[dev]"

Parallel Jobs

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: ruff check .
  
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest
  
  # Runs after both complete
  deploy:
    needs: [lint, test]  # Dependencies
    runs-on: ubuntu-latest
    steps:
      - run: echo "Deploying..."

Secrets and Environment Variables

Using Secrets

jobs:
  deploy:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
        run: |
          pip install twine
          twine upload dist/*
      
      - name: Notify Slack
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK_URL }}
        run: |
          curl -X POST -H 'Content-type: application/json' \
            --data '{"text":"Deployment successful!"}' \
            $SLACK_WEBHOOK

Environment Configuration

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production  # Requires approval
    
    steps:
      - name: Deploy
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          API_KEY: ${{ secrets.API_KEY }}
        run: ./deploy.sh

Deployment Workflows

Deploy on Release

# .github/workflows/release.yml
name: Release

on:
  release:
    types: [published]

jobs:
  publish-pypi:
    name: Publish to PyPI
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install build dependencies
        run: pip install build twine
      
      - name: Build package
        run: python -m build
      
      - name: Publish to PyPI
        env:
          TWINE_USERNAME: __token__
          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
        run: twine upload dist/*

  publish-docker:
    name: Publish Docker Image
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Log in to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}
      
      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: |
            username/app:${{ github.ref_name }}
            username/app:latest

Deploy to Cloud Platforms

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    name: Deploy to Staging
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - uses: actions/checkout@v4
      
      # Example: Deploy to AWS
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1
      
      - name: Deploy to ECS
        run: |
          aws ecs update-service \
            --cluster production \
            --service app-service \
            --force-new-deployment

  deploy-production:
    name: Deploy to Production
    runs-on: ubuntu-latest
    needs: deploy-staging
    environment: production
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Deploy to production
        run: ./scripts/deploy-production.sh

Workflow Patterns

Reusable Workflows

# .github/workflows/reusable-test.yml
name: Reusable Test Workflow

on:
  workflow_call:
    inputs:
      python-version:
        required: true
        type: string
    secrets:
      codecov-token:
        required: false

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ inputs.python-version }}
      
      - name: Run tests
        run: pytest --cov
      
      - name: Upload coverage
        if: ${{ secrets.codecov-token }}
        uses: codecov/codecov-action@v4
        with:
          token: ${{ secrets.codecov-token }}

# .github/workflows/ci.yml
name: CI

on:
  push:
    branches: [main]

jobs:
  test-311:
    uses: ./.github/workflows/reusable-test.yml
    with:
      python-version: '3.11'
    secrets:
      codecov-token: ${{ secrets.CODECOV_TOKEN }}
  
  test-312:
    uses: ./.github/workflows/reusable-test.yml
    with:
      python-version: '3.12'

Conditional Jobs

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      src: ${{ steps.filter.outputs.src }}
      docs: ${{ steps.filter.outputs.docs }}
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            src:
              - 'src/**'
              - 'tests/**'
            docs:
              - 'docs/**'
  
  test:
    needs: changes
    if: needs.changes.outputs.src == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: pytest
  
  build-docs:
    needs: changes
    if: needs.changes.outputs.docs == 'true'
    runs-on: ubuntu-latest
    steps:
      - run: mkdocs build

Pull Request Workflows

PR Checks

# .github/workflows/pr.yml
name: PR Checks

on:
  pull_request:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - run: pip install ruff
      - run: ruff check --output-format=github .
  
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - run: pip install -e ".[dev]"
      - run: pytest --tb=short
  
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - run: pip install -e ".[dev]"
      - run: pytest --cov=src --cov-fail-under=80

Auto-merge Dependabot PRs

# .github/workflows/dependabot-automerge.yml
name: Dependabot Auto-merge

on:
  pull_request:

permissions:
  contents: write
  pull-requests: write

jobs:
  automerge:
    runs-on: ubuntu-latest
    if: github.actor == 'dependabot[bot]'
    
    steps:
      - name: Fetch Dependabot metadata
        id: metadata
        uses: dependabot/fetch-metadata@v2
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Auto-merge minor updates
        if: steps.metadata.outputs.update-type == 'version-update:semver-minor'
        run: gh pr merge --auto --squash "$PR_URL"
        env:
          PR_URL: ${{ github.event.pull_request.html_url }}
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Practical Exercise

Exercise 1: Create a Complete CI Pipeline

Create a CI workflow that:

1. Runs linting (Ruff)

2. Runs tests with coverage

3. Checks types with mypy

4. Builds the package

5. Publishes to PyPI on release

# .github/workflows/ci.yml
name: CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
  release:
    types: [published]

jobs:
  quality:
    name: Code Quality
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install dependencies
        run: pip install ruff mypy
      
      - name: Run Ruff
        run: ruff check .
      
      - name: Check formatting
        run: ruff format --check .
      
      - name: Run mypy
        run: mypy src

  test:
    name: Test
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install dependencies
        run: pip install -e ".[dev]"
      
      - name: Run tests
        run: pytest --cov=src --cov-report=xml --cov-fail-under=80
      
      - name: Upload coverage
        uses: codecov/codecov-action@v4
        with:
          files: ./coverage.xml

  build:
    name: Build
    runs-on: ubuntu-latest
    needs: [quality, test]
    
    steps:
      - uses: actions/checkout@v4
      
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Build package
        run: |
          pip install build
          python -m build
      
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: dist
          path: dist/

  publish:
    name: Publish
    runs-on: ubuntu-latest
    needs: build
    if: github.event_name == 'release'
    
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: dist
          path: dist/
      
      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        with:
          password: ${{ secrets.PYPI_API_TOKEN }}

Exercise 2: Add Branch Protection

Configure branch protection rules for main:

• Require status checks to pass

• Require pull request reviews

• Require up-to-date branches

CI/CD Best Practices

Do's

• ✅ Run tests on every PR

• ✅ Cache dependencies

• ✅ Use matrix testing for multiple versions

• ✅ Keep secrets in GitHub Secrets

• ✅ Use environment protection rules

• ✅ Fail fast on obvious issues

Don'ts

• ❌ Store secrets in code

• ❌ Skip tests for "small changes"

• ❌ Deploy without tests passing

• ❌ Use overly complex workflows

• ❌ Ignore flaky tests

Key Takeaways

1. Automate everything - If you do it twice, automate it

2. Fast feedback - Run quick checks first

3. Test on multiple versions - Matrix builds catch compatibility issues

4. Secure your secrets - Use GitHub Secrets, never commit credentials

5. Progressive deployment - Stage → Production with approvals

What's Next?

With CI/CD automating your quality checks, let's focus on human collaboration. In Article 11: Code Review Practices, we'll cover effective code review strategies and PR workflows.

---

This article is part of the Software Engineering 101 series.