Part 5: Testing with pytest and Modern Packaging

Introduction

ansible-inspec uses pytest with async support, fixture-based test isolation, and httpx.AsyncClient for API integration tests. All of this is configured through pyproject.toml. This part covers the test patterns I actually use and how the project is packaged for PyPI.

---

pytest Basics

Installation

pip install pytest pytest-asyncio httpx

Minimal test

# tests/unit/test_converter.py

def add(a: int, b: int) -> int:
    return a + b

def test_add() -> None:
    assert add(2, 3) == 5

def test_add_negative() -> None:
    assert add(-1, 1) == 0

Run:

pytest                          # run all tests
pytest tests/unit/              # run only unit tests
pytest -v                       # verbose output
pytest -k "test_add"            # run tests matching name
pytest --tb=short               # shorter tracebacks

Assertions — just use assert

def test_assertions() -> None:
    # Equality
    assert 1 + 1 == 2

    # Truthy / falsy
    assert "hello"
    assert not ""

    # Containers
    hosts = ["web-01", "web-02"]
    assert "web-01" in hosts
    assert len(hosts) == 2

    # Exceptions
    import pytest
    with pytest.raises(ValueError, match="Port must be"):
        from ansible_inspec.models import HostConfig
        HostConfig(hostname="web-01", port=99999)

---

Fixtures

Fixtures are pytest's dependency injection system. They set up (and tear down) resources for tests.

Basic fixture

import pytest
from ansible_inspec.models import JobTemplate

@pytest.fixture
def linux_baseline_template() -> JobTemplate:
    return JobTemplate(
        name="linux-baseline",
        profile="dev-sec/linux-baseline",
        timeout=120,
    )

def test_template_name(linux_baseline_template: JobTemplate) -> None:
    assert linux_baseline_template.name == "linux-baseline"

def test_template_default_timeout() -> None:
    t = JobTemplate(name="test", profile="test/profile")
    assert t.timeout == 300

Fixture scope

import pytest

@pytest.fixture(scope="session")
def shared_config() -> dict:
    """Created once per test session — expensive to initialise."""
    return {"base_url": "http://localhost:8080", "timeout": 30}

@pytest.fixture(scope="module")
def module_templates() -> list[dict]:
    """Created once per test module file."""
    return [
        {"name": "t1", "profile": "p1"},
        {"name": "t2", "profile": "p2"},
    ]

@pytest.fixture   # default scope="function" — recreated for every test
def fresh_store() -> dict:
    return {}

Fixture with teardown

import pytest
import tempfile
import os

@pytest.fixture
def temp_profile_dir():
    """Create a temporary directory with profile files, clean up after."""
    with tempfile.TemporaryDirectory() as tmpdir:
        # Write a minimal InSpec control file
        control_path = os.path.join(tmpdir, "ssh.rb")
        with open(control_path, "w") as f:
            f.write('control "sshd-01" do\n  title "SSH root login disabled"\nend\n')
        yield tmpdir   # test runs with this path
    # Cleanup happens automatically (TemporaryDirectory context manager)

def test_load_profile(temp_profile_dir: str) -> None:
    files = os.listdir(temp_profile_dir)
    assert "ssh.rb" in files

conftest.py — shared fixtures across files

Pytest automatically loads conftest.py from the test directory. Put shared fixtures there:

# tests/conftest.py
import pytest
from fastapi.testclient import TestClient
from ansible_inspec.server.app import app

@pytest.fixture(scope="session")
def client() -> TestClient:
    return TestClient(app)

@pytest.fixture
def sample_template_payload() -> dict:
    return {
        "name": "ssh-baseline",
        "profile": "dev-sec/ssh-baseline",
        "timeout": 120,
    }

---

Async Tests with pytest-asyncio

All async tests need pytest-asyncio. Configure it in pyproject.toml:

[tool.pytest.ini_options]
asyncio_mode = "auto"    # auto-detect async tests without explicit markers
testpaths = ["tests"]

With asyncio_mode = "auto", any async def test_* function is automatically treated as an async test:

import asyncio
import pytest

async def fetch_status(host: str) -> dict:
    await asyncio.sleep(0.01)    # simulate I/O
    return {"host": host, "status": "ok"}

async def test_fetch_single() -> None:
    result = await fetch_status("web-01")
    assert result["status"] == "ok"

async def test_fetch_concurrent() -> None:
    hosts = ["web-01", "web-02", "db-01"]
    results = await asyncio.gather(*[fetch_status(h) for h in hosts])
    assert len(results) == 3
    assert all(r["status"] == "ok" for r in results)

Async fixtures

import pytest
import httpx

@pytest.fixture
async def async_client():
    async with httpx.AsyncClient(base_url="http://test") as client:
        yield client

async def test_with_async_client(async_client: httpx.AsyncClient) -> None:
    # In real tests this would use the ASGI transport
    assert async_client is not None

---

Testing FastAPI with httpx.AsyncClient

The cleanest way to integration-test FastAPI without a real server:

# tests/integration/test_templates_api.py
import pytest
import httpx
from fastapi import FastAPI
from ansible_inspec.server.app import app   # import the real app

@pytest.fixture
async def api_client():
    async with httpx.AsyncClient(
        transport=httpx.ASGITransport(app=app),
        base_url="http://test",
    ) as client:
        yield client

async def test_health(api_client: httpx.AsyncClient) -> None:
    response = await api_client.get("/health")
    assert response.status_code == 200
    assert response.json()["status"] == "ok"

async def test_create_template(api_client: httpx.AsyncClient) -> None:
    payload = {
        "name": "ssh-baseline",
        "profile": "dev-sec/ssh-baseline",
        "timeout": 120,
    }
    response = await api_client.post("/api/v1/templates", json=payload)
    assert response.status_code == 201
    data = response.json()
    assert data["name"] == "ssh-baseline"
    assert "id" in data

async def test_create_duplicate_template(api_client: httpx.AsyncClient) -> None:
    payload = {"name": "dup-test", "profile": "dev-sec/linux-baseline"}
    await api_client.post("/api/v1/templates", json=payload)
    response = await api_client.post("/api/v1/templates", json=payload)
    assert response.status_code == 409

async def test_get_nonexistent_template(api_client: httpx.AsyncClient) -> None:
    response = await api_client.get("/api/v1/templates/does-not-exist")
    assert response.status_code == 404

---

Mocking with unittest.mock

For tests that shouldn't hit real external systems (Git, SSH, file system):

from unittest.mock import AsyncMock, MagicMock, patch

# Patch a module-level function
def test_load_inventory_file_not_found() -> None:
    from ansible_inspec.adapters import load_inventory
    with patch("builtins.open", side_effect=FileNotFoundError("no file")):
        with pytest.raises(FileNotFoundError):
            load_inventory("/nonexistent/inventory.yml")

# Patch an async function
async def test_job_executor_timeout() -> None:
    from ansible_inspec.executor import run_compliance

    with patch(
        "ansible_inspec.executor.check_host",
        new_callable=AsyncMock,
        side_effect=asyncio.TimeoutError,
    ):
        result = await run_compliance(["web-01"], timeout=1.0)
        assert result["web-01"]["status"] == "timeout"

# MagicMock for class instances
def test_converter_calls_load() -> None:
    from ansible_inspec.converter import AnsibleCollectionConverter
    converter = AnsibleCollectionConverter("/profiles", "example", "test")
    converter._parse_file = MagicMock()
    converter.load()
    # Verify _parse_file was called for any .rb files found

pytest-mock — cleaner mock API

pip install pytest-mock

def test_with_mocker(mocker) -> None:
    mock_fn = mocker.patch("ansible_inspec.executor.subprocess.check_output")
    mock_fn.return_value = b"ansible 2.15"

    from ansible_inspec.executor import get_ansible_version
    version = get_ansible_version()
    mock_fn.assert_called_once()
    assert "ansible" in version

---

Parametrize — test multiple inputs

import pytest

@pytest.mark.parametrize("hostname,expected", [
    ("web-01", True),
    ("192.168.1.10", True),
    ("bad host!", False),
    ("", False),
    ("web-01.example.com", True),
])
def test_validate_hostname(hostname: str, expected: bool) -> None:
    from ansible_inspec.models import HostConfig
    import pydantic
    if expected:
        config = HostConfig(hostname=hostname)
        assert config.hostname == hostname.lower()
    else:
        with pytest.raises(pydantic.ValidationError):
            HostConfig(hostname=hostname)

@pytest.mark.parametrize("reporter_type", ["json", "html", "junit"])
async def test_reporter_formats(
    reporter_type: str, api_client: httpx.AsyncClient
) -> None:
    response = await api_client.get(f"/api/v1/reports?format={reporter_type}")
    assert response.status_code == 200

---

Test Organisation

Structure mirrors the source tree:

tests/
├── conftest.py              # shared fixtures
├── unit/
│   ├── test_converter.py
│   ├── test_models.py
│   └── test_executor.py
├── integration/
│   ├── test_templates_api.py
│   ├── test_jobs_api.py
│   └── test_auth.py
└── e2e/
    └── test_full_workflow.py

Run by layer:

pytest tests/unit/          # fast, no I/O
pytest tests/integration/   # tests with in-process API
pytest tests/e2e/           # full stack (requires running server)

---

Code Quality Tools

ruff — linter + formatter (replaces flake8 + black)

pip install ruff

# Lint
ruff check lib/ tests/

# Format
ruff format lib/ tests/

# Fix auto-fixable lint issues
ruff check --fix lib/

pyproject.toml config:

[tool.ruff]
line-length = 100
target-version = "py312"
select = ["E", "F", "I", "UP", "B", "SIM"]
ignore = ["E501"]

[tool.ruff.isort]
known-first-party = ["ansible_inspec"]

mypy — static type checking

pip install mypy

mypy lib/ --strict

pyproject.toml config:

[tool.mypy]
python_version = "3.12"
strict = true
ignore_missing_imports = true

Makefile bringing it together

.PHONY: test lint format typecheck ci

test:
	pytest -v

lint:
	ruff check lib/ tests/

format:
	ruff format lib/ tests/

typecheck:
	mypy lib/ --strict

ci: lint typecheck test

---

GitHub Actions CI

ansible-inspec runs tests on every push:

# .github/workflows/test.yml
name: Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12"]

    steps:
      - uses: actions/checkout@v4

      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install dependencies
        run: |
          pip install -e ".[dev]"

      - name: Lint
        run: ruff check lib/ tests/

      - name: Type check
        run: mypy lib/ --strict

      - name: Run tests
        run: pytest -v --tb=short

---

Publishing to PyPI

ansible-inspec publishes automatically on a git tag via GitHub Actions:

# .github/workflows/publish.yml
name: Publish to PyPI

on:
  push:
    tags: ["v*"]

jobs:
  publish:
    runs-on: ubuntu-latest
    environment: pypi
    permissions:
      id-token: write   # for trusted publishing (no API key needed)

    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Build
        run: |
          pip install build
          python -m build   # creates dist/*.whl and dist/*.tar.gz

      - name: Publish to PyPI
        uses: pypa/gh-action-pypi-publish@release/v1

Manual build and publish:

# Build
pip install build
python -m build
ls dist/
# ansible_inspec-0.2.12-py3-none-any.whl
# ansible_inspec-0.2.12.tar.gz

# Upload to PyPI (install twine first)
pip install twine
twine upload dist/*

---

Full pyproject.toml Reference

[build-system]
requires = ["setuptools>=65.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "ansible-inspec"
version = "0.2.12"
description = "Compliance testing with Ansible and InSpec integration"
readme = "README.md"
authors = [{ name = "Htunn Thu Thu" }]
license = { text = "GPL-3.0-or-later" }
requires-python = ">=3.8"
classifiers = [
    "Programming Language :: Python :: 3.12",
    "Topic :: Security",
    "Topic :: System :: Systems Administration",
]
dependencies = [
    "fastapi>=0.115.0",
    "uvicorn[standard]>=0.30.0",
    "pydantic>=2.0.0",
    "pydantic-settings>=2.0.0",
    "pyyaml>=6.0",
    "httpx>=0.27.0",
]

[project.optional-dependencies]
server = ["prisma>=0.15.0", "passlib[bcrypt]>=1.7.4"]
dev = [
    "pytest>=8.0.0",
    "pytest-asyncio>=0.23.0",
    "pytest-mock>=3.14.0",
    "httpx>=0.27.0",
    "ruff>=0.4.0",
    "mypy>=1.10.0",
]

[project.scripts]
ansible-inspec = "ansible_inspec.cli.main:main"

[tool.setuptools.packages.find]
where = ["lib"]

[tool.ruff]
line-length = 100
target-version = "py312"
select = ["E", "F", "I", "UP", "B", "SIM"]

[tool.mypy]
python_version = "3.12"
strict = true
ignore_missing_imports = true

[tool.pytest.ini_options]
asyncio_mode = "auto"
testpaths = ["tests"]

---

Summary

Concept	Key point
pytest fixtures	Reusable, scoped setup/teardown via @pytest.fixture
conftest.py	Shared fixtures visible to all test files in the directory
pytest-asyncio	asyncio_mode = "auto" — no extra markers needed
httpx.AsyncClient	In-process API tests with ASGITransport — no server needed
unittest.mock	Patch external systems; AsyncMock for async functions
@pytest.mark.parametrize	Test multiple inputs in one test function
ruff	Linter + formatter replacing flake8 + black
mypy --strict	Catch type errors before runtime
python -m build	Build sdist + wheel from pyproject.toml
Trusted publishing	PyPI publish via OIDC — no API token to manage

---

Series Complete

This series covered Python 3.12 from first setup through production-ready packaging:

Part	Topic
1	Environment, pyproject.toml, Python 3.12 language features
2	Data structures, type hints, Pydantic v2
3	OOP, @dataclass, Protocol, ABC
4	asyncio, async/await, FastAPI
5	pytest, async tests, CI, PyPI packaging

All code patterns in this series are drawn from ansible-inspec — a real open-source tool worth exploring if you want to see these patterns at scale.