# Part 5: Testing with pytest and Modern Packaging

## Introduction

`ansible-inspec` uses `pytest` with async support, fixture-based test isolation, and `httpx.AsyncClient` for API integration tests. All of this is configured through `pyproject.toml`. This part covers the test patterns I actually use and how the project is packaged for PyPI.

***

## pytest Basics

### Installation

```bash
pip install pytest pytest-asyncio httpx
```

### Minimal test

```python
# tests/unit/test_converter.py

def add(a: int, b: int) -> int:
    return a + b

def test_add() -> None:
    assert add(2, 3) == 5

def test_add_negative() -> None:
    assert add(-1, 1) == 0
```

Run:

```bash
pytest                          # run all tests
pytest tests/unit/              # run only unit tests
pytest -v                       # verbose output
pytest -k "test_add"            # run tests matching name
pytest --tb=short               # shorter tracebacks
```

### Assertions — just use `assert`

```python
def test_assertions() -> None:
    # Equality
    assert 1 + 1 == 2

    # Truthy / falsy
    assert "hello"
    assert not ""

    # Containers
    hosts = ["web-01", "web-02"]
    assert "web-01" in hosts
    assert len(hosts) == 2

    # Exceptions
    import pytest
    with pytest.raises(ValueError, match="Port must be"):
        from ansible_inspec.models import HostConfig
        HostConfig(hostname="web-01", port=99999)
```

***

## Fixtures

Fixtures are pytest's dependency injection system. They set up (and tear down) resources for tests.

### Basic fixture

```python
import pytest
from ansible_inspec.models import JobTemplate

@pytest.fixture
def linux_baseline_template() -> JobTemplate:
    return JobTemplate(
        name="linux-baseline",
        profile="dev-sec/linux-baseline",
        timeout=120,
    )

def test_template_name(linux_baseline_template: JobTemplate) -> None:
    assert linux_baseline_template.name == "linux-baseline"

def test_template_default_timeout() -> None:
    t = JobTemplate(name="test", profile="test/profile")
    assert t.timeout == 300
```

### Fixture scope

```python
import pytest

@pytest.fixture(scope="session")
def shared_config() -> dict:
    """Created once per test session — expensive to initialise."""
    return {"base_url": "http://localhost:8080", "timeout": 30}

@pytest.fixture(scope="module")
def module_templates() -> list[dict]:
    """Created once per test module file."""
    return [
        {"name": "t1", "profile": "p1"},
        {"name": "t2", "profile": "p2"},
    ]

@pytest.fixture   # default scope="function" — recreated for every test
def fresh_store() -> dict:
    return {}
```

### Fixture with teardown

```python
import pytest
import tempfile
import os

@pytest.fixture
def temp_profile_dir():
    """Create a temporary directory with profile files, clean up after."""
    with tempfile.TemporaryDirectory() as tmpdir:
        # Write a minimal InSpec control file
        control_path = os.path.join(tmpdir, "ssh.rb")
        with open(control_path, "w") as f:
            f.write('control "sshd-01" do\n  title "SSH root login disabled"\nend\n')
        yield tmpdir   # test runs with this path
    # Cleanup happens automatically (TemporaryDirectory context manager)

def test_load_profile(temp_profile_dir: str) -> None:
    files = os.listdir(temp_profile_dir)
    assert "ssh.rb" in files
```

### `conftest.py` — shared fixtures across files

Pytest automatically loads `conftest.py` from the test directory. Put shared fixtures there:

```python
# tests/conftest.py
import pytest
from fastapi.testclient import TestClient
from ansible_inspec.server.app import app

@pytest.fixture(scope="session")
def client() -> TestClient:
    return TestClient(app)

@pytest.fixture
def sample_template_payload() -> dict:
    return {
        "name": "ssh-baseline",
        "profile": "dev-sec/ssh-baseline",
        "timeout": 120,
    }
```

***

## Async Tests with `pytest-asyncio`

All async tests need `pytest-asyncio`. Configure it in `pyproject.toml`:

```toml
[tool.pytest.ini_options]
asyncio_mode = "auto"    # auto-detect async tests without explicit markers
testpaths = ["tests"]
```

With `asyncio_mode = "auto"`, any `async def test_*` function is automatically treated as an async test:

```python
import asyncio
import pytest

async def fetch_status(host: str) -> dict:
    await asyncio.sleep(0.01)    # simulate I/O
    return {"host": host, "status": "ok"}

async def test_fetch_single() -> None:
    result = await fetch_status("web-01")
    assert result["status"] == "ok"

async def test_fetch_concurrent() -> None:
    hosts = ["web-01", "web-02", "db-01"]
    results = await asyncio.gather(*[fetch_status(h) for h in hosts])
    assert len(results) == 3
    assert all(r["status"] == "ok" for r in results)
```

### Async fixtures

```python
import pytest
import httpx

@pytest.fixture
async def async_client():
    async with httpx.AsyncClient(base_url="http://test") as client:
        yield client

async def test_with_async_client(async_client: httpx.AsyncClient) -> None:
    # In real tests this would use the ASGI transport
    assert async_client is not None
```

***

## Testing FastAPI with `httpx.AsyncClient`

The cleanest way to integration-test FastAPI without a real server:

```python
# tests/integration/test_templates_api.py
import pytest
import httpx
from fastapi import FastAPI
from ansible_inspec.server.app import app   # import the real app

@pytest.fixture
async def api_client():
    async with httpx.AsyncClient(
        transport=httpx.ASGITransport(app=app),
        base_url="http://test",
    ) as client:
        yield client

async def test_health(api_client: httpx.AsyncClient) -> None:
    response = await api_client.get("/health")
    assert response.status_code == 200
    assert response.json()["status"] == "ok"

async def test_create_template(api_client: httpx.AsyncClient) -> None:
    payload = {
        "name": "ssh-baseline",
        "profile": "dev-sec/ssh-baseline",
        "timeout": 120,
    }
    response = await api_client.post("/api/v1/templates", json=payload)
    assert response.status_code == 201
    data = response.json()
    assert data["name"] == "ssh-baseline"
    assert "id" in data

async def test_create_duplicate_template(api_client: httpx.AsyncClient) -> None:
    payload = {"name": "dup-test", "profile": "dev-sec/linux-baseline"}
    await api_client.post("/api/v1/templates", json=payload)
    response = await api_client.post("/api/v1/templates", json=payload)
    assert response.status_code == 409

async def test_get_nonexistent_template(api_client: httpx.AsyncClient) -> None:
    response = await api_client.get("/api/v1/templates/does-not-exist")
    assert response.status_code == 404
```

***

## Mocking with `unittest.mock`

For tests that shouldn't hit real external systems (Git, SSH, file system):

```python
from unittest.mock import AsyncMock, MagicMock, patch

# Patch a module-level function
def test_load_inventory_file_not_found() -> None:
    from ansible_inspec.adapters import load_inventory
    with patch("builtins.open", side_effect=FileNotFoundError("no file")):
        with pytest.raises(FileNotFoundError):
            load_inventory("/nonexistent/inventory.yml")

# Patch an async function
async def test_job_executor_timeout() -> None:
    from ansible_inspec.executor import run_compliance

    with patch(
        "ansible_inspec.executor.check_host",
        new_callable=AsyncMock,
        side_effect=asyncio.TimeoutError,
    ):
        result = await run_compliance(["web-01"], timeout=1.0)
        assert result["web-01"]["status"] == "timeout"

# MagicMock for class instances
def test_converter_calls_load() -> None:
    from ansible_inspec.converter import AnsibleCollectionConverter
    converter = AnsibleCollectionConverter("/profiles", "example", "test")
    converter._parse_file = MagicMock()
    converter.load()
    # Verify _parse_file was called for any .rb files found
```

### `pytest-mock` — cleaner mock API

```bash
pip install pytest-mock
```

```python
def test_with_mocker(mocker) -> None:
    mock_fn = mocker.patch("ansible_inspec.executor.subprocess.check_output")
    mock_fn.return_value = b"ansible 2.15"

    from ansible_inspec.executor import get_ansible_version
    version = get_ansible_version()
    mock_fn.assert_called_once()
    assert "ansible" in version
```

***

## Parametrize — test multiple inputs

```python
import pytest

@pytest.mark.parametrize("hostname,expected", [
    ("web-01", True),
    ("192.168.1.10", True),
    ("bad host!", False),
    ("", False),
    ("web-01.example.com", True),
])
def test_validate_hostname(hostname: str, expected: bool) -> None:
    from ansible_inspec.models import HostConfig
    import pydantic
    if expected:
        config = HostConfig(hostname=hostname)
        assert config.hostname == hostname.lower()
    else:
        with pytest.raises(pydantic.ValidationError):
            HostConfig(hostname=hostname)

@pytest.mark.parametrize("reporter_type", ["json", "html", "junit"])
async def test_reporter_formats(
    reporter_type: str, api_client: httpx.AsyncClient
) -> None:
    response = await api_client.get(f"/api/v1/reports?format={reporter_type}")
    assert response.status_code == 200
```

***

## Test Organisation

Structure mirrors the source tree:

```
tests/
├── conftest.py              # shared fixtures
├── unit/
│   ├── test_converter.py
│   ├── test_models.py
│   └── test_executor.py
├── integration/
│   ├── test_templates_api.py
│   ├── test_jobs_api.py
│   └── test_auth.py
└── e2e/
    └── test_full_workflow.py
```

Run by layer:

```bash
pytest tests/unit/          # fast, no I/O
pytest tests/integration/   # tests with in-process API
pytest tests/e2e/           # full stack (requires running server)
```

***

## Code Quality Tools

### `ruff` — linter + formatter (replaces flake8 + black)

```bash
pip install ruff

# Lint
ruff check lib/ tests/

# Format
ruff format lib/ tests/

# Fix auto-fixable lint issues
ruff check --fix lib/
```

`pyproject.toml` config:

```toml
[tool.ruff]
line-length = 100
target-version = "py312"
select = ["E", "F", "I", "UP", "B", "SIM"]
ignore = ["E501"]

[tool.ruff.isort]
known-first-party = ["ansible_inspec"]
```

### `mypy` — static type checking

```bash
pip install mypy

mypy lib/ --strict
```

`pyproject.toml` config:

```toml
[tool.mypy]
python_version = "3.12"
strict = true
ignore_missing_imports = true
```

### Makefile bringing it together

```makefile
.PHONY: test lint format typecheck ci

test:
	pytest -v

lint:
	ruff check lib/ tests/

format:
	ruff format lib/ tests/

typecheck:
	mypy lib/ --strict

ci: lint typecheck test
```

***

## GitHub Actions CI

`ansible-inspec` runs tests on every push:

```yaml
# .github/workflows/test.yml
name: Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          pip install -e ".[dev]"

      - name: Lint
        run: ruff check lib/ tests/

      - name: Type check
        run: mypy lib/ --strict

      - name: Run tests
        run: pytest -v --tb=short
```

***

## Publishing to PyPI

`ansible-inspec` publishes automatically on a git tag via GitHub Actions:

```yaml
# .github/workflows/publish.yml
name: Publish to PyPI

on:
  push:
    tags: ["v*"]

jobs:
  publish:
    runs-on: ubuntu-latest
    environment: pypi
    permissions:
      id-token: write   # for trusted publishing (no API key needed)

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Build
        run: |
          pip install build
          python -m build   # creates dist/*.whl and dist/*.tar.gz

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1
```

Manual build and publish:

```bash
# Build
pip install build
python -m build
ls dist/
# ansible_inspec-0.2.12-py3-none-any.whl
# ansible_inspec-0.2.12.tar.gz

# Upload to PyPI (install twine first)
pip install twine
twine upload dist/*
```

***

## Full `pyproject.toml` Reference

```toml
[build-system]
requires = ["setuptools>=65.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "ansible-inspec"
version = "0.2.12"
description = "Compliance testing with Ansible and InSpec integration"
readme = "README.md"
authors = [{ name = "Htunn Thu Thu" }]
license = { text = "GPL-3.0-or-later" }
requires-python = ">=3.8"
classifiers = [
    "Programming Language :: Python :: 3.12",
    "Topic :: Security",
    "Topic :: System :: Systems Administration",
]
dependencies = [
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.30.0",
    "pydantic>=2.0.0",
    "pydantic-settings>=2.0.0",
    "pyyaml>=6.0",
    "httpx>=0.27.0",
]

[project.optional-dependencies]
server = ["prisma>=0.15.0", "passlib[bcrypt]>=1.7.4"]
dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.23.0",
    "pytest-mock>=3.14.0",
    "httpx>=0.27.0",
    "ruff>=0.4.0",
    "mypy>=1.10.0",
]

[project.scripts]
ansible-inspec = "ansible_inspec.cli.main:main"

[tool.setuptools.packages.find]
where = ["lib"]

[tool.ruff]
line-length = 100
target-version = "py312"
select = ["E", "F", "I", "UP", "B", "SIM"]

[tool.mypy]
python_version = "3.12"
strict = true
ignore_missing_imports = true

[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]
```

***

## Summary

| Concept                    | Key point                                                    |
| -------------------------- | ------------------------------------------------------------ |
| `pytest` fixtures          | Reusable, scoped setup/teardown via `@pytest.fixture`        |
| `conftest.py`              | Shared fixtures visible to all test files in the directory   |
| `pytest-asyncio`           | `asyncio_mode = "auto"` — no extra markers needed            |
| `httpx.AsyncClient`        | In-process API tests with `ASGITransport` — no server needed |
| `unittest.mock`            | Patch external systems; `AsyncMock` for async functions      |
| `@pytest.mark.parametrize` | Test multiple inputs in one test function                    |
| `ruff`                     | Linter + formatter replacing flake8 + black                  |
| `mypy --strict`            | Catch type errors before runtime                             |
| `python -m build`          | Build sdist + wheel from `pyproject.toml`                    |
| Trusted publishing         | PyPI publish via OIDC — no API token to manage               |

***

## Series Complete

This series covered Python 3.12 from first setup through production-ready packaging:

| Part | Topic                                                        |
| ---- | ------------------------------------------------------------ |
| 1    | Environment, `pyproject.toml`, Python 3.12 language features |
| 2    | Data structures, type hints, Pydantic v2                     |
| 3    | OOP, `@dataclass`, `Protocol`, ABC                           |
| 4    | `asyncio`, `async/await`, FastAPI                            |
| 5    | `pytest`, async tests, CI, PyPI packaging                    |

All code patterns in this series are drawn from [ansible-inspec](https://github.com/Htunn/ansible-inspec) — a real open-source tool worth exploring if you want to see these patterns at scale.