Part 2: Planning, Architecture, and Project Setup

Introduction

The most expensive code is the code you have to rewrite. In my early projects, I'd jump straight into coding—only to refactor everything weeks later when requirements evolved. Now, I spend 20% of project time on planning, which saves me 80% of headaches later.

This part covers my battle-tested approach to starting projects right.

Planning & Architecture

The Real-World Problem

Let's work through a concrete example I built last year: a webhook processing service for handling Stripe payment events.

Requirements:

• Receive webhooks from Stripe

• Verify signatures

• Process events asynchronously

• Retry failed processing

• Monitor success/failure rates

• Handle 1000+ events/hour

My Architecture Process

Step 1: Component Identification

I start with a simple component diagram:

Why this architecture?

• API decoupled from processing: Stripe gets fast 200 response

• Queue for reliability: If worker fails, event isn't lost

• Redis cache: Prevent duplicate processing

• Separate monitoring: See failures before customers do

Step 2: Define Boundaries

For my webhook service, I defined clear boundaries:

Lesson learned: When I tried making the webhook handler do everything, it became impossible to test and deploy. Separation saved me.

Step 3: Choose Tech Stack

My decision matrix for the webhook service:

Requirement	Options Considered	Choice	Reason
Language	Python, Node.js, Go	Node.js	Team expertise, async I/O
Framework	Express, Fastify, Nest.js	Fastify	Performance, built-in validation
Queue	RabbitMQ, SQS, Redis	Redis	Already using, simple setup
Database	PostgreSQL, MongoDB	PostgreSQL	ACID compliance needed
Deployment	ECS, Lambda, K8s	ECS	Right balance of simplicity/control

Anti-pattern I avoided: Using the "hottest" tech. Kubernetes was overkill for this service. ECS was perfect.

Step 4: Design Data Flow

Here's the actual flow from my implementation:

Critical decision: Transaction boundary includes webhook log but NOT email. Why? Email failure shouldn't rollback payment update.

Step 5: Plan for Failure

I use a failure modes table:

What Can Fail	Impact	Detection	Mitigation
Stripe signature invalid	Medium	Immediate	Log, alert, return 401
Redis down	High	Health check	Fallback to direct processing
Worker crash	Low	Process monitor	Restart, event stays in queue
Database deadlock	Medium	Metrics	Retry with exponential backoff
Duplicate event	Low	Redis cache	Idempotency check

Real incident: Redis went down during Black Friday. Because I planned for it, we fell back to direct processing. No lost events.

Step 6: Document Decisions

I write Architecture Decision Records (ADRs):

# ADR-001: Use Redis for Webhook Queue

## Status
Accepted

## Context
Need reliable queue for webhook events. Must handle 1000+ events/hour.

## Decision
Use Redis Lists as queue with LPUSH/BRPOP pattern.

## Consequences
Positive:
- Simple setup, team knows Redis
- Persistence enabled = don't lose events
- Sub-millisecond latency

Negative:
- Not a "true" message queue
- Need manual retry logic
- Limited to single Redis instance scale

## Alternatives Considered
- RabbitMQ: Too complex for our needs
- SQS: Vendor lock-in, higher latency

Why ADRs saved me: Six months later, new team member asked "why not use SQS?" I just linked the ADR.

Project Setup & Scaffolding

The Scaffolding Strategy

I maintain templates for common project types. Here's my Node.js TypeScript service template:

webhook-service/
├── src/
│   ├── api/              # HTTP handlers
│   ├── workers/          # Background jobs
│   ├── services/         # Business logic
│   ├── models/           # Data models
│   ├── utils/            # Helpers
│   └── index.ts          # Entry point
├── tests/
│   ├── unit/
│   ├── integration/
│   └── fixtures/
├── deployments/
│   ├── docker/
│   ├── k8s/
│   └── terraform/
├── docs/
│   ├── architecture/
│   ├── api/
│   └── runbooks/
├── .github/
│   └── workflows/        # CI/CD
├── package.json
├── tsconfig.json
├── Dockerfile
└── README.md

Automated Setup

Instead of manually creating files, I use my initialization script:

#!/bin/bash
# init-service.sh

SERVICE_NAME=$1
SERVICE_TYPE=${2:-api}  # api, worker, or lib

echo "🚀 Creating $SERVICE_NAME ($SERVICE_TYPE)"

# Create directory structure
mkdir -p $SERVICE_NAME/{src/{api,workers,services,models,utils},tests/{unit,integration,fixtures},deployments/{docker,k8s,terraform},docs/{architecture,api,runbooks},.github/workflows}

# Generate package.json
cat > $SERVICE_NAME/package.json << EOF
{
  "name": "$SERVICE_NAME",
  "version": "1.0.0",
  "scripts": {
    "dev": "tsx watch src/index.ts",
    "build": "tsc",
    "test": "jest",
    "test:watch": "jest --watch",
    "lint": "eslint src --ext .ts",
    "format": "prettier --write 'src/**/*.ts'"
  },
  "dependencies": {
    "fastify": "^4.26.0",
    "@fastify/cors": "^8.4.2",
    "zod": "^3.22.4"
  },
  "devDependencies": {
    "typescript": "^5.3.3",
    "tsx": "^4.7.0",
    "@types/node": "^20.11.0",
    "jest": "^29.7.0",
    "@types/jest": "^29.5.11",
    "eslint": "^8.56.0",
    "prettier": "^3.2.4"
  }
}
EOF

# Generate tsconfig.json
cat > $SERVICE_NAME/tsconfig.json << EOF
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "commonjs",
    "lib": ["ES2022"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true,
    "declaration": true,
    "declarationMap": true,
    "sourceMap": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules", "dist", "tests"]
}
EOF

# Generate Dockerfile
cat > $SERVICE_NAME/Dockerfile << EOF
FROM node:20-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build

FROM node:20-alpine

WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY package*.json ./

USER node
EXPOSE 3000

CMD ["node", "dist/index.js"]
EOF

# Generate basic index.ts
cat > $SERVICE_NAME/src/index.ts << EOF
import Fastify from 'fastify';
import cors from '@fastify/cors';

const fastify = Fastify({
  logger: true
});

fastify.register(cors);

fastify.get('/health', async () => {
  return { status: 'healthy' };
});

const start = async () => {
  try {
    await fastify.listen({ port: 3000, host: '0.0.0.0' });
  } catch (err) {
    fastify.log.error(err);
    process.exit(1);
  }
};

start();
EOF

# Generate GitHub Actions workflow
cat > $SERVICE_NAME/.github/workflows/ci.yml << EOF
name: CI

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test
      - run: npm run lint
      
  build:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: false
          tags: $SERVICE_NAME:latest
EOF

# Initialize git
cd $SERVICE_NAME
git init
git add .
git commit -m "Initial commit: $SERVICE_NAME scaffolding"

echo "✅ $SERVICE_NAME created successfully!"
echo "📝 Next steps:"
echo "   cd $SERVICE_NAME"
echo "   npm install"
echo "   npm run dev"

Usage:

./init-service.sh webhook-service api
cd webhook-service
npm install
npm run dev

Time saved: What used to take 2 hours (copying files, updating configs) now takes 30 seconds.

Environment Configuration

I use a .env.example file that documents all required variables:

# .env.example - Copy to .env and update values

# Application
NODE_ENV=development
PORT=3000
LOG_LEVEL=debug

# Database
DATABASE_URL=postgresql://user:pass@localhost:5432/webhooks
DATABASE_POOL_MIN=2
DATABASE_POOL_MAX=10

# Redis
REDIS_URL=redis://localhost:6379
REDIS_PREFIX=webhooks:

# Stripe
STRIPE_WEBHOOK_SECRET=whsec_...
STRIPE_API_KEY=sk_test_...

# Monitoring
SENTRY_DSN=https://...
METRICS_PORT=9090

# Feature Flags
ENABLE_RETRY=true
MAX_RETRY_ATTEMPTS=3
RETRY_DELAY_MS=30000

Validation on startup:

import { z } from 'zod';

const envSchema = z.object({
  NODE_ENV: z.enum(['development', 'production', 'test']),
  PORT: z.coerce.number().min(1000).max(65535),
  DATABASE_URL: z.string().url(),
  REDIS_URL: z.string().url(),
  STRIPE_WEBHOOK_SECRET: z.string().startsWith('whsec_'),
});

export const env = envSchema.parse(process.env);

Why this matters: App crashes immediately with clear error if config is wrong. No mysterious production failures.

Git & Commands

My Git Workflow

Essential Git Commands I Use Daily

1. Clean Commit History

# Before creating PR, cleanup commits
git rebase -i HEAD~5

# In editor, squash and reword:
pick abc1234 Add webhook handler
squash def5678 Fix typo
squash ghi9012 Add more tests
reword jkl3456 Add retry logic

# Results in clean history:
# - Add webhook handler with tests
# - Add retry logic with exponential backoff

2. Cherry-Pick Specific Fixes

# Production bug fix on main
git checkout main
git cherry-pick abc1234  # The fix commit from develop

# Deploy immediately without other develop changes

Real scenario: Production bug in payment validation. I cherry-picked just the fix commit without pulling in work-in-progress features.

3. Find When Bug Was Introduced

# Binary search through history
git bisect start
git bisect bad                    # Current commit is bad
git bisect good v1.2.0            # v1.2.0 was working

# Git checks out middle commit, you test
npm test
git bisect bad  # or: git bisect good

# Continues until finding exact commit

Saved me 4 hours tracking down a regression in JWT validation.

4. Stash Work-in-Progress

# Urgent bug, need to switch branches
git stash push -m "WIP: webhook validation refactor"

# Fix bug, come back
git stash list
git stash pop stash@{0}

5. Rewrite History (Carefully!)

# Remove sensitive data accidentally committed
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch config/production.env' \
  --prune-empty --tag-name-filter cat -- --all

# Force push (dangerous, coordinate with team!)
git push origin --force --all

Lesson learned: I committed an API key once. This saved me from rotating 50+ client secrets.

Branch Strategy

For my webhook service:

Branch naming convention:

• feature/short-description - New functionality

• bugfix/short-description - Bug fixes

• hotfix/short-description - Production emergencies

• chore/short-description - Maintenance (deps, docs)

Commit Message Format

I follow conventional commits:

type(scope): subject

body

footer

Examples from my webhook service:

# Feature
feat(webhook): add signature verification
- Implement HMAC-SHA256 validation
- Add test fixtures for Stripe signatures
- Document verification flow

# Bug fix  
fix(worker): prevent duplicate event processing
- Add Redis lock with 5min TTL
- Race condition found in stress test
- Closes #123

# Documentation
docs(api): add webhook endpoint examples
- cURL examples for testing
- Response status code table
- Error handling scenarios

# Performance
perf(queue): optimize Redis connection pooling
- Reduce connections from 50 to 10
- Implement lazy connection init
- 40% reduction in memory usage

Why this format?

• Auto-generate changelogs

• Filter commits by type

• Clear communication with team

Git Hooks for Code Quality

I use Husky for git hooks:

# .husky/pre-commit
#!/bin/sh
. "$(dirname "$0")/_/husky.sh"

# Run linter
npm run lint

# Run type check
npm run type-check

# Run tests
npm test -- --findRelatedTests

# Check commit message format
npx commitlint --edit $1

What this prevents:

• ❌ Committing code that doesn't lint

• ❌ Committing code that doesn't type-check

• ❌ Committing code that breaks tests

• ❌ Committing with bad commit messages

Trade-off: Commits take 10-30 seconds longer, but saves hours in code review and debugging.

Practical Git Troubleshooting

Scenario 1: Accidentally Committed to Main

# Oh no, committed directly to main!
git reset --soft HEAD~1       # Undo commit, keep changes
git stash                     # Stash changes
git checkout -b feature/my-feature  # Create feature branch
git stash pop                # Apply changes
git add .
git commit -m "feat: my feature"

Scenario 2: Need to Undo Public Commit

# Don't rewrite public history! Use revert:
git revert abc1234
git push origin main

# Creates new commit that undoes abc1234

Scenario 3: Merge Conflict

# During rebase or merge
git status  # See conflicted files

# Edit files, resolve conflicts
# Then:
git add resolved-file.ts
git rebase --continue

# Or abort:
git rebase --abort

My conflict resolution workflow:

// File with conflict:
<<<<<<< HEAD
const result = await processPayment(order);
=======
const result = await validateAndProcessPayment(order);
>>>>>>> feature/validation

// My resolution:
const validatedOrder = await validateOrder(order);
const result = await processPayment(validatedOrder);

Keep both approaches, combine them logically.

Development Environment Setup

My Standard Dev Container

I use Docker Compose for consistent environments:

# docker-compose.yml
version: '3.8'

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - .:/app
      - /app/node_modules  # Don't override node_modules
    ports:
      - "3000:3000"
      - "9229:9229"  # Node.js debugger
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgresql://postgres:password@db:5432/webhooks
      - REDIS_URL=redis://redis:6379
    depends_on:
      - db
      - redis
    command: npm run dev

  db:
    image: postgres:16-alpine
    environment:
      - POSTGRES_DB=webhooks
      - POSTGRES_PASSWORD=password
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

  adminer:  # Database GUI
    image: adminer
    ports:
      - "8080:8080"

volumes:
  postgres_data:
  redis_data:

Start everything:

docker-compose up -d

Why this approach?

• ✅ Same environment for all developers

• ✅ No "works on my machine" issues

• ✅ Easy onboarding for new team members

• ✅ Matches production architecture

VS Code Configuration

My .vscode/settings.json:

{
  "editor.formatOnSave": true,
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  },
  "typescript.tsdk": "node_modules/typescript/lib",
  "typescript.enablePromptUseWorkspaceTsdk": true,
  "files.exclude": {
    "**/node_modules": true,
    "**/dist": true,
    "**/.git": true
  },
  "search.exclude": {
    "**/node_modules": true,
    "**/dist": true,
    "**/*.log": true
  }
}

Debugging Setup

My .vscode/launch.json:

{
  "version": "0.2.0",
  "configurations": [
    {
      "type": "node",
      "request": "launch",
      "name": "Debug App",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["run", "dev"],
      "port": 9229,
      "skipFiles": ["<node_internals>/**"],
      "console": "integratedTerminal"
    },
    {
      "type": "node",
      "request": "launch",
      "name": "Debug Tests",
      "runtimeExecutable": "npm",
      "runtimeArgs": ["test", "--", "--runInBand"],
      "port": 9229,
      "skipFiles": ["<node_internals>/**"]
    }
  ]
}

Usage: Press F5 in VS Code, set breakpoints, step through code.

Time saved: Beats console.log() debugging by miles.

Complete Setup Flow

Here's my actual workflow starting a new service:

Timeline (webhook service):

• Planning: 2 hours (architecture, ADRs)

• Scaffolding: 5 minutes (automated script)

• Git setup: 10 minutes (repo, hooks, CI)

• Dev env: 15 minutes (Docker Compose)

• First test: 20 minutes (test framework setup)

Total: ~3 hours to writing first line of business logic

Compare to my early days: Would spend 2-3 days just setting up tooling.

Key Takeaways

1. Plan before coding: 20% planning time saves 80% refactoring time

2. Automate scaffolding: Don't manually create project structure

3. Document decisions: ADRs prevent repeated discussions

4. Clean Git history: Makes code review and debugging easier

5. Consistent environment: Docker Compose eliminates "works on my machine"

What's Next

In Part 3, we'll actually write code:

• Development workflow

• Testing strategies

• Debugging techniques

• Browser automation

We'll implement the webhook handler with proper error handling, tests, and observability.

---

Ready to write some code? Part 3 is where theory meets practice.