Part 5: Prompt Engineering for Production Systems

Prompt Engineering Is Software Engineering

When I first heard "prompt engineering," I thought it was about cleverly phrasing questions to get better answers from ChatGPT. After building production systems that depend on LLM outputs, I've come to see it differently. Prompt engineering is designing the interface between your code and the language model — and it requires the same discipline as designing any other software interface.

A prompt in a production system isn't a one-off question. It's a template that runs thousands of times with different inputs. It needs to handle edge cases, produce consistent output formats, and degrade gracefully when the input is unexpected. The skills are less "creative writing" and more "API contract design."

This article covers the prompt engineering patterns I use in my own projects — from basic structure to Pydantic-validated templates and defensive techniques.

---

The Anatomy of a Production Prompt

Every prompt I write for a production system has three layers:

┌─────────────────────────────────┐
│         System Prompt            │  ← Who the model is, rules, output format
├─────────────────────────────────┤
│         Context                  │  ← Retrieved documents, user data, history
├─────────────────────────────────┤
│         User Message             │  ← The actual question or instruction
└─────────────────────────────────┘

Here's how that looks in code:

# A complete prompt for my RAG service
SYSTEM_PROMPT = """You are a technical documentation assistant. You answer questions
based ONLY on the provided context. Follow these rules:

1. If the context contains the answer, provide it with specific details.
2. If the context does not contain the answer, say "I don't have information about this
   in my knowledge base."
3. Never make up information that isn't in the context.
4. Reference the source document when possible.
5. Keep answers concise — aim for 2-4 paragraphs maximum.

Output format:
- Start with a direct answer to the question
- Follow with supporting details from the context
- End with source references if applicable"""


def build_rag_prompt(
    question: str,
    context_chunks: list[dict[str, str]],
) -> list[dict[str, str]]:
    """Build a complete prompt for RAG question-answering."""
    # Format retrieved context
    context_parts = []
    for i, chunk in enumerate(context_chunks, 1):
        context_parts.append(
            f"[Source {i}: {chunk['title']}]\n{chunk['content']}"
        )
    context_text = "\n\n---\n\n".join(context_parts)

    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": f"Context:\n{context_text}\n\nQuestion: {question}",
        },
    ]

Why This Structure Matters

• System prompt sets behavioral constraints that apply to every request. I put output format rules here because they don't change between requests.

• Context is dynamic per request — different retrieved chunks, different user data. Separating it from the system prompt means I can cache the system prompt tokens.

• User message is the actual input. Keeping it last gives it the strongest attention from the model (the "recency effect" from Part 3).

---

System Prompt Design

The system prompt is the most important part of a production prompt. It's the contract between your code and the model. Here are the patterns I've found effective:

Pattern 1: Role + Rules + Format

EXTRACTION_SYSTEM_PROMPT = """You are a structured data extraction system.

Your task: Extract specific fields from the provided text and return them as JSON.

Rules:
- Extract ONLY the fields specified in the schema below
- If a field is not present in the text, use null
- Do not infer or guess values that aren't explicitly stated
- Dates should be in ISO 8601 format (YYYY-MM-DD)
- Numbers should be numeric types, not strings

Output schema:
{
    "title": "string or null",
    "date": "ISO date string or null",
    "author": "string or null",
    "tags": ["array of strings"],
    "summary": "string, max 100 words"
}

Respond with ONLY the JSON object. No explanation, no markdown fences."""

Pattern 2: Explicit Boundaries

When I built my DevOps monitoring agent, I needed the LLM to analyze alerts but not take dangerous actions:

MONITORING_SYSTEM_PROMPT = """You are a Kubernetes monitoring assistant that analyzes
cluster alerts and suggests remediation steps.

You CAN:
- Analyze alert content and explain likely root causes
- Suggest kubectl commands for investigation
- Recommend remediation steps
- Explain Kubernetes concepts relevant to the alert

You CANNOT:
- Execute any commands directly
- Access the cluster directly
- Modify any resources
- Suggest destructive operations (delete pods, scale to 0, drain nodes)
  without explicit warnings

When suggesting commands, always format them as:
```bash
# Explanation of what this does
kubectl command here

Always start with investigation commands before suggesting changes."""

### Pattern 3: Few-Shot Examples

For tasks where the output format is critical, I include examples directly in the system prompt:

```python
CLASSIFICATION_SYSTEM_PROMPT = """You classify support messages into categories.

Categories: billing, technical, account, general

Examples:

Input: "I was charged twice for my subscription last month"
Output: {"category": "billing", "confidence": 0.95}

Input: "The API is returning 500 errors on the /users endpoint"
Output: {"category": "technical", "confidence": 0.92}

Input: "How do I change my password?"
Output: {"category": "account", "confidence": 0.88}

Input: "What are your business hours?"
Output: {"category": "general", "confidence": 0.90}

Respond with ONLY a JSON object containing "category" and "confidence" (0.0 to 1.0)."""

Few-shot examples are the most reliable way I've found to control output format. The model learns the pattern from examples better than from instructions alone.

---

Prompt Templates with Pydantic

In my projects, I never construct prompts with raw string concatenation. Instead, I use Pydantic models to define and validate prompt templates:

# src/ai_engineer/prompts/templates.py
from pydantic import BaseModel, Field


class RAGPromptInput(BaseModel):
    """Validated input for the RAG prompt template."""

    question: str = Field(..., min_length=1, max_length=2000)
    context_chunks: list[dict[str, str]] = Field(
        ..., min_length=1, max_length=10
    )
    max_context_tokens: int = Field(default=3000, ge=500, le=8000)


class ExtractionPromptInput(BaseModel):
    """Validated input for data extraction."""

    text: str = Field(..., min_length=10, max_length=10000)
    fields: list[str] = Field(..., min_length=1)
    output_format: str = Field(default="json")


class PromptBuilder:
    """Build validated prompts from templates."""

    @staticmethod
    def build_rag_prompt(input_data: RAGPromptInput) -> list[dict[str, str]]:
        """Build a RAG prompt with validated inputs."""
        context_parts = []
        total_chars = 0

        for i, chunk in enumerate(input_data.context_chunks, 1):
            chunk_text = f"[Source {i}: {chunk['title']}]\n{chunk['content']}"

            # Rough token estimate: 1 token ≈ 4 chars
            estimated_tokens = len(chunk_text) / 4
            if total_chars / 4 + estimated_tokens > input_data.max_context_tokens:
                break

            context_parts.append(chunk_text)
            total_chars += len(chunk_text)

        context_text = "\n\n---\n\n".join(context_parts)

        return [
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": f"Context:\n{context_text}\n\nQuestion: {input_data.question}",
            },
        ]

    @staticmethod
    def build_extraction_prompt(
        input_data: ExtractionPromptInput,
    ) -> list[dict[str, str]]:
        """Build a data extraction prompt with validated inputs."""
        fields_str = ", ".join(input_data.fields)

        return [
            {
                "role": "system",
                "content": f"Extract the following fields from the text: {fields_str}.\n"
                f"Return as {input_data.output_format}. Use null for missing fields.",
            },
            {"role": "user", "content": input_data.text},
        ]

Why Pydantic for Prompts?

1. Validation at construction time. If someone passes an empty question or 50 context chunks, the model catches it before the LLM call. I've had cases where upstream bugs produced empty strings that created nonsensical prompts — Pydantic catches these immediately.

2. Self-documenting. The model fields tell you exactly what inputs the prompt expects, with constraints.

3. Testable. I can write unit tests for prompt construction without calling the LLM:

# tests/test_prompts.py
import pytest
from pydantic import ValidationError
from ai_engineer.prompts.templates import RAGPromptInput, PromptBuilder


def test_rag_prompt_valid():
    input_data = RAGPromptInput(
        question="What is pgvector?",
        context_chunks=[
            {"title": "pgvector Setup", "content": "pgvector is a PostgreSQL extension..."}
        ],
    )
    messages = PromptBuilder.build_rag_prompt(input_data)

    assert len(messages) == 2
    assert messages[0]["role"] == "system"
    assert "pgvector" in messages[1]["content"]


def test_rag_prompt_empty_question():
    with pytest.raises(ValidationError):
        RAGPromptInput(
            question="",
            context_chunks=[{"title": "Test", "content": "Content"}],
        )


def test_rag_prompt_no_context():
    with pytest.raises(ValidationError):
        RAGPromptInput(
            question="What is pgvector?",
            context_chunks=[],
        )

---

Structured Output — Getting JSON from LLMs

One of the most common tasks in AI engineering is getting the model to return structured data. I've gone through several approaches:

Approach 1: Prompt Instructions (Fragile)

# This works ~90% of the time — not enough for production
prompt = """Extract the person's name and email from this text.
Return as JSON with "name" and "email" fields.

Text: Contact John Smith at [email protected] for details."""

The model might return {"name": "John Smith", "email": "[email protected]"} — or it might wrap it in markdown fences, add an explanation, or use different field names.

Approach 2: JSON Mode (Better)

Most API providers now support a JSON mode:

import httpx

async def extract_structured(
    prompt: str,
    api_key: str,
) -> dict:
    async with httpx.AsyncClient(timeout=30.0) as client:
        response = await client.post(
            "https://models.inference.ai.azure.com/chat/completions",
            headers={"Authorization": f"Bearer {api_key}"},
            json={
                "model": "gpt-4o",
                "messages": [{"role": "user", "content": prompt}],
                "response_format": {"type": "json_object"},
                "temperature": 0,
            },
        )
        response.raise_for_status()
        content = response.json()["choices"][0]["message"]["content"]
        return json.loads(content)

JSON mode guarantees valid JSON but doesn't guarantee the schema matches what you expect.

Approach 3: Pydantic Validation on Output (My Preferred Approach)

I combine JSON mode with Pydantic validation to get reliable structured output:

# src/ai_engineer/llm/structured.py
import json
from typing import TypeVar

from pydantic import BaseModel, ValidationError

from ai_engineer.llm.base import LLMProvider

T = TypeVar("T", bound=BaseModel)


async def generate_structured(
    provider: LLMProvider,
    prompt: str,
    response_model: type[T],
    *,
    max_retries: int = 2,
) -> T:
    """Generate a structured response validated against a Pydantic model.

    Retries with error feedback if validation fails.
    """
    # Include the schema in the prompt
    schema_str = json.dumps(response_model.model_json_schema(), indent=2)
    full_prompt = (
        f"{prompt}\n\n"
        f"Respond with a JSON object matching this schema:\n{schema_str}\n\n"
        f"Return ONLY the JSON object."
    )

    last_error = None
    for attempt in range(max_retries + 1):
        if attempt > 0 and last_error:
            # Add error feedback for retry
            full_prompt += f"\n\nPrevious attempt failed with: {last_error}\nPlease fix and try again."

        raw_response = await provider.generate(
            full_prompt,
            temperature=0.0,
            max_tokens=1024,
        )

        # Clean up common issues
        cleaned = raw_response.strip()
        if cleaned.startswith("```"):
            # Strip markdown code fences
            lines = cleaned.split("\n")
            cleaned = "\n".join(lines[1:-1])

        try:
            data = json.loads(cleaned)
            return response_model.model_validate(data)
        except (json.JSONDecodeError, ValidationError) as e:
            last_error = str(e)

    raise ValueError(
        f"Failed to get valid structured output after {max_retries + 1} attempts. "
        f"Last error: {last_error}"
    )

Usage:

from pydantic import BaseModel, Field


class ExtractedContact(BaseModel):
    name: str = Field(..., min_length=1)
    email: str = Field(..., pattern=r"^[\w.+-]+@[\w-]+\.[\w.]+$")
    phone: str | None = None


# This will retry with error feedback if the model returns invalid email format
contact = await generate_structured(
    provider=llm_provider,
    prompt="Extract contact info from: Call John Smith at [email protected] or 555-0123",
    response_model=ExtractedContact,
)
print(contact)  # ExtractedContact(name='John Smith', email='[email protected]', phone='555-0123')

This pattern has been reliable in my production code. The retry with error feedback handles the ~5% of cases where the model's first attempt doesn't match the schema.

---

Defensive Prompting

Production prompts need to handle adversarial and unexpected inputs. Here are the techniques I use:

Input Sanitization

# src/ai_engineer/prompts/sanitize.py
import re


def sanitize_user_input(text: str) -> str:
    """Sanitize user input before including it in a prompt.

    Prevents basic prompt injection attempts.
    """
    # Remove potential prompt injection patterns
    # These are heuristic — not a security boundary
    sanitized = text

    # Remove attempts to override system instructions
    injection_patterns = [
        r"ignore (?:all )?(?:previous |above )?instructions",
        r"you are now",
        r"new instructions:",
        r"system:\s",
        r"<\|(?:im_start|system)\|>",
    ]

    for pattern in injection_patterns:
        sanitized = re.sub(pattern, "[filtered]", sanitized, flags=re.IGNORECASE)

    # Truncate to reasonable length
    max_length = 2000
    if len(sanitized) > max_length:
        sanitized = sanitized[:max_length] + "... [truncated]"

    return sanitized

Output Validation

Never trust LLM output. Always validate:

async def ask_with_validation(
    question: str,
    context_chunks: list[dict],
    provider: LLMProvider,
) -> dict:
    """Ask a question with input sanitization and output validation."""
    # Sanitize input
    clean_question = sanitize_user_input(question)

    # Build prompt
    prompt_input = RAGPromptInput(
        question=clean_question,
        context_chunks=context_chunks,
    )
    messages = PromptBuilder.build_rag_prompt(prompt_input)

    # Generate response
    raw_answer = await provider.generate(
        messages[-1]["content"],
        temperature=0.1,
        max_tokens=1024,
    )

    # Validate output
    answer = raw_answer.strip()
    if not answer:
        answer = "I was unable to generate a response. Please try rephrasing your question."

    if len(answer) > 5000:
        answer = answer[:5000] + "\n\n[Response truncated]"

    return {
        "answer": answer,
        "question": clean_question,
        "sources_used": len(context_chunks),
    }

Handling Edge Cases

Through my own testing, I've identified common edge cases that need handling:

# Edge cases my RAG system handles

# 1. Question outside knowledge base
# System prompt instruction: "say 'I don't have information about this'"

# 2. Ambiguous question
# Handle by asking for clarification in the response

# 3. Question about multiple topics
# Retrieve context for the primary topic, note limitations

# 4. Very short question ("what?", "help")
# Return a helpful default: "Could you provide more detail about what you're looking for?"

# 5. Non-English input
# The system prompt doesn't restrict language, so the model handles it naturally
# But embeddings may be less accurate for non-English text

MIN_QUESTION_LENGTH = 3

def validate_question(question: str) -> str | None:
    """Validate a question before processing. Returns error message or None."""
    stripped = question.strip()
    if len(stripped) < MIN_QUESTION_LENGTH:
        return "Please provide a more detailed question (at least a few words)."
    if len(stripped) > 2000:
        return "Question is too long. Please keep it under 2000 characters."
    return None

---

Prompt Versioning

In my projects, I version prompts the same way I version code. When I change a system prompt, I want to track what changed and measure the impact:

# src/ai_engineer/prompts/registry.py
from dataclasses import dataclass


@dataclass(frozen=True)
class PromptVersion:
    name: str
    version: str
    system_prompt: str
    description: str


# Registry of all prompt versions
PROMPTS: dict[str, PromptVersion] = {
    "rag-v1": PromptVersion(
        name="rag",
        version="v1",
        system_prompt="""You are a documentation assistant. Answer based on the provided context.
If the answer is not in the context, say so.""",
        description="Initial RAG prompt — simple and direct",
    ),
    "rag-v2": PromptVersion(
        name="rag",
        version="v2",
        system_prompt="""You are a technical documentation assistant. You answer questions
based ONLY on the provided context. Follow these rules:

1. If the context contains the answer, provide it with specific details.
2. If the context does not contain the answer, say "I don't have information about this
   in my knowledge base."
3. Never make up information that isn't in the context.
4. Reference the source document when possible.
5. Keep answers concise — aim for 2-4 paragraphs maximum.

Output format:
- Start with a direct answer to the question
- Follow with supporting details from the context
- End with source references if applicable""",
        description="Added explicit rules, output format, and boundary behavior",
    ),
}


def get_prompt(name: str, version: str | None = None) -> PromptVersion:
    """Get a prompt by name and optional version."""
    if version:
        key = f"{name}-{version}"
        if key not in PROMPTS:
            raise KeyError(f"Prompt '{key}' not found")
        return PROMPTS[key]

    # Get the latest version
    matching = [
        (k, v) for k, v in PROMPTS.items() if v.name == name
    ]
    if not matching:
        raise KeyError(f"No prompts found with name '{name}'")
    return max(matching, key=lambda x: x[1].version)[1]

The v1 → v2 change happened after I noticed the model was sometimes making up information when the context didn't contain the answer. Adding explicit rule #2 ("say I don't have information") cut hallucinations in my testing by about 80%.

---

Key Takeaways

1. System prompts are interfaces. Design them with the same care as an API contract: define inputs, outputs, constraints, and error behavior.

2. Use Pydantic for prompt inputs AND outputs. Validation at both ends catches bugs before they become user-visible problems.

3. Few-shot examples are more reliable than instructions. When output format matters, show the model what you want rather than describing it.

4. Never trust LLM output. Always validate, truncate, and handle malformed responses gracefully.

5. Version your prompts. A prompt change can affect every response your system produces. Track changes and measure impact.

6. Temperature=0 for structured output. Deterministic decoding dramatically improves format consistency.

---

Previous: Part 4 — Embeddings and Vector Search

Next: Part 6 — Building AI-Powered APIs with FastAPI