Part 5: Advanced Version Control and Production Workflows

Introduction

The most stressful moment in my development career was when I had to deploy a hotfix to production at 2 AM because a critical payment processing bug was causing transaction failures. I learned that night that having solid production workflows and version control practices isn't just nice to have—it's essential for survival in production environments.

Since then, I've refined my approach to release management, hotfix procedures, and recovery strategies. In this final part of the series, I'll share advanced Git workflows, production best practices, and troubleshooting techniques I've learned from managing TypeScript microservices in production.

Release Management and Versioning

Semantic Versioning (SemVer)

I follow Semantic Versioning for all my projects: MAJOR.MINOR.PATCH

Example: 2.3.1

• MAJOR (2): Incompatible API changes

• MINOR (3): New features (backward compatible)

• PATCH (1): Bug fixes (backward compatible)

Real-World Versioning Example

// Version 1.0.0 - Initial release
interface PaymentRequest {
  amount: number;
  currency: string;
}

// Version 1.1.0 - Added optional field (MINOR bump)
interface PaymentRequest {
  amount: number;
  currency: string;
  description?: string;  // New optional field - backward compatible
}

// Version 1.1.1 - Bug fix (PATCH bump)
// Fixed currency validation bug
// No interface changes

// Version 2.0.0 - Breaking change (MAJOR bump)
interface PaymentRequest {
  amount: number;
  currency: Currency;  // Changed from string to enum - BREAKING
  description?: string;
  metadata: Record<string, any>;  // New required field - BREAKING
}

Creating Releases on GitHub

Manual Release Process:

# 1. Update version in package.json
npm version minor  # 1.2.3 → 1.3.0

# 2. Update CHANGELOG.md

# Changelog

## [1.3.0] - 2024-02-07

### Added
- Multi-currency support for international payments
- New `/api/currencies` endpoint
- Currency conversion service

### Changed
- Updated payment processing to handle different currencies
- Improved error messages for invalid currency codes

### Fixed
- Fixed race condition in payment webhook handler
- Corrected tax calculation for EU countries

### Security
- Updated dependencies to patch security vulnerabilities

# 3. Commit changes
git add package.json package-lock.json CHANGELOG.md
git commit -m "chore: bump version to 1.3.0"

# 4. Create and push tag
git tag -a v1.3.0 -m "Release version 1.3.0: Multi-currency support"
git push origin main --tags

# 5. Create release on GitHub
# Go to Releases → Draft a new release
# Select tag v1.3.0
# Copy changelog content
# Publish release

Automated Release with GitHub Actions:

# .github/workflows/release.yml
name: Release

on:
  push:
    tags:
      - 'v*'

jobs:
  release:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build
        run: npm run build
      
      - name: Run tests
        run: npm test
      
      - name: Generate changelog
        id: changelog
        uses: metcalfc/[email protected]
        with:
          myToken: ${{ secrets.GITHUB_TOKEN }}
      
      - name: Create GitHub Release
        uses: actions/create-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          tag_name: ${{ github.ref }}
          release_name: Release ${{ github.ref }}
          body: ${{ steps.changelog.outputs.changelog }}
          draft: false
          prerelease: false
      
      - name: Publish to npm
        run: npm publish
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
      
      - name: Deploy to production
        run: npm run deploy:production
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      
      - name: Notify team
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: |
            🚀 New release deployed to production!
            Version: ${{ github.ref }}
            Changelog: ${{ steps.changelog.outputs.changelog }}
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

To trigger release:

git tag -a v1.3.0 -m "Release 1.3.0"
git push origin v1.3.0

# GitHub Actions automatically:
# - Builds the project
# - Runs tests
# - Creates GitHub release
# - Publishes to npm
# - Deploys to production
# - Notifies team on Slack

Hotfix Procedures

When Production is Broken

Scenario: Critical bug discovered in production—payment processing is failing.

Git Flow Hotfix Process:

# 1. Create hotfix branch from main (production)
git checkout main
git pull origin main
git checkout -b hotfix/payment-timeout

# 2. Fix the bug

// src/services/payment.service.ts

// Before (bug):
async processPayment(amount: number): Promise<Payment> {
  const response = await axios.post(PAYMENT_API, { amount }, {
    timeout: 5000  // Too short! Payments timing out
  });
  return response.data;
}

// After (fix):
async processPayment(amount: number): Promise<Payment> {
  const response = await axios.post(PAYMENT_API, { amount }, {
    timeout: 30000,  // Increased to 30 seconds
    retry: 3,        // Added retry logic
  });
  return response.data;
}

# 3. Commit the fix
git add src/services/payment.service.ts
git commit -m "fix: increase payment API timeout to 30s

Payment API was timing out after 5s causing transaction failures.
Increased timeout to 30s and added retry logic.

Fixes #567"

# 4. Test thoroughly
npm test
npm run test:integration

# 5. Create PR for review (even for hotfixes!)
git push -u origin hotfix/payment-timeout

# Create PR on GitHub with:
# - Clear description of bug
# - Testing performed
# - Impact assessment
# Request expedited review

# 6. After approval, merge to main
git checkout main
git merge hotfix/payment-timeout

# 7. Tag the hotfix version
npm version patch  # 1.3.0 → 1.3.1
git tag -a v1.3.1 -m "Hotfix: Payment API timeout"

# 8. Deploy to production immediately
git push origin main --tags
# Triggers automated deployment

# 9. Merge back to develop (if using Git Flow)
git checkout develop
git merge hotfix/payment-timeout
git push origin develop

# 10. Delete hotfix branch
git branch -d hotfix/payment-timeout
git push origin --delete hotfix/payment-timeout

# 11. Monitor production
# Watch dashboards and error logs to confirm fix

GitHub Flow Hotfix (Simplified)

# 1. Create branch from main
git checkout main
git pull origin main
git checkout -b fix-payment-timeout

# 2. Fix bug and commit
git add src/services/payment.service.ts
git commit -m "fix: increase payment API timeout to 30s"

# 3. Push and create PR
git push -u origin fix-payment-timeout

# 4. After review, merge and deploy
# GitHub Actions automatically deploys

# 5. Monitor production

Emergency Deployment Checklist

I keep this checklist for production emergencies:

## Emergency Deployment Checklist

### Before Deploying
- [ ] Bug is confirmed and reproducible
- [ ] Root cause identified
- [ ] Fix is tested locally
- [ ] Integration tests pass
- [ ] At least one team member reviewed the fix
- [ ] Rollback plan is ready

### During Deployment
- [ ] Create backup of production database (if schema changes)
- [ ] Notify team in Slack #incidents channel
- [ ] Deploy fix
- [ ] Monitor error rates in real-time
- [ ] Check key metrics (latency, throughput, errors)

### After Deployment
- [ ] Verify fix resolves the issue
- [ ] Monitor for 30 minutes
- [ ] Update incident post-mortem
- [ ] Create follow-up tasks for permanent fix (if temporary)
- [ ] Document lessons learned

Git Hooks for Automation

Git hooks are scripts that run automatically at certain points in the Git workflow.

Pre-Commit Hook: Prevent Bad Commits

# .git/hooks/pre-commit
#!/bin/sh

echo "Running pre-commit checks..."

# 1. Run linter
npm run lint
if [ $? -ne 0 ]; then
  echo "❌ Linting failed. Please fix errors before committing."
  exit 1
fi

# 2. Run type check
npm run type-check
if [ $? -ne 0 ]; then
  echo "❌ Type check failed. Please fix errors before committing."
  exit 1
fi

# 3. Run tests
npm test
if [ $? -ne 0 ]; then
  echo "❌ Tests failed. Please fix failing tests before committing."
  exit 1
fi

# 4. Check for console.log
if git diff --cached | grep -q "console.log"; then
  echo "⚠️  Warning: Found console.log in staged files"
  echo "Remove debugging statements before committing"
  exit 1
fi

# 5. Check for TODO/FIXME
if git diff --cached | grep -q -E "TODO|FIXME"; then
  echo "⚠️  Warning: Found TODO/FIXME in staged files"
  echo "Consider creating an issue instead"
  # Don't exit - just warn
fi

echo "✅ All pre-commit checks passed!"
exit 0

Using Husky for Git Hooks

Install Husky:

npm install --save-dev husky
npx husky install

# Add to package.json scripts
npm set-script prepare "husky install"

Create Pre-Commit Hook:

npx husky add .husky/pre-commit "npm run lint-staged"

// package.json
{
  "scripts": {
    "prepare": "husky install",
    "lint-staged": "lint-staged"
  },
  "lint-staged": {
    "*.ts": [
      "eslint --fix",
      "prettier --write",
      "git add"
    ],
    "*.{json,md}": [
      "prettier --write",
      "git add"
    ]
  },
  "devDependencies": {
    "husky": "^8.0.0",
    "lint-staged": "^13.0.0"
  }
}

Commit Message Hook: Enforce Convention

# .husky/commit-msg
#!/bin/sh

commit_msg_file=$1
commit_msg=$(cat "$commit_msg_file")

# Check if commit message follows Conventional Commits format
if ! echo "$commit_msg" | grep -qE "^(feat|fix|docs|style|refactor|test|chore|perf|ci|build|revert)(\(.+\))?: .{1,}"; then
  echo "❌ Invalid commit message format!"
  echo ""
  echo "Commit message must follow Conventional Commits format:"
  echo "  <type>: <description>"
  echo ""
  echo "Examples:"
  echo "  feat: add user authentication"
  echo "  fix: resolve null pointer in payment service"
  echo "  docs: update API documentation"
  echo ""
  echo "Valid types: feat, fix, docs, style, refactor, test, chore, perf, ci, build, revert"
  exit 1
fi

echo "✅ Commit message format is valid"
exit 0

# Now commits must follow format:
git commit -m "feat: add email notifications"  # ✅ Valid
git commit -m "fixed bug"                       # ❌ Invalid
git commit -m "update stuff"                    # ❌ Invalid

Advanced Git Techniques

Interactive Rebase: Clean Up History

Scenario: Made messy commits while developing, want to clean up before PR.

# My commit history:
git log --oneline
a1b2c3d fix typo
b2c3d4e WIP
c3d4e5f add tests
d4e5f6g implement feature
e5f6g7h fix linting

# Clean it up with interactive rebase
git rebase -i HEAD~5

# Opens editor:
pick d4e5f6g implement feature
pick c3d4e5f add tests
pick e5f6g7h fix linting
pick b2c3d4e WIP
pick a1b2c3d fix typo

# Change to:
pick d4e5f6g implement feature
squash c3d4e5f add tests
squash e5f6g7h fix linting
squash b2c3d4e WIP
squash a1b2c3d fix typo

# Save and close editor
# Git prompts for new commit message:

feat: add email notifications with template support

- Implemented EmailService with template rendering
- Added comprehensive test coverage
- Fixed linting issues

# Result: 5 messy commits → 1 clean commit
git log --oneline
d4e5f6g feat: add email notifications with template support

Cherry-Pick: Apply Specific Commits

Scenario: Need to apply a specific fix from one branch to another.

# Bug fix is in feature branch but needed in main immediately
git checkout feature/new-dashboard
git log --oneline
a1b2c3d feat: add new charts
b2c3d4e fix: resolve date formatting bug  # Need this fix!
c3d4e5f feat: add export functionality

# Apply just the bug fix to main
git checkout main
git cherry-pick b2c3d4e

# Commit b2c3d4e is now in main
git log --oneline
b2c3d4e fix: resolve date formatting bug
...

Git Bisect: Find Bug Introduction

Scenario: Bug exists in production, but not sure when it was introduced.

# Start bisect
git bisect start

# Mark current commit as bad (has bug)
git bisect bad

# Mark a known good commit (from 2 weeks ago)
git bisect good v1.2.0

# Git checks out a commit halfway between
# Test if bug exists...
npm test

# If bug exists:
git bisect bad

# If bug doesn't exist:
git bisect good

# Git continues binary search
# Eventually identifies the problematic commit:
# b2c3d4e is the first bad commit
# commit b2c3d4e
# Author: John Doe
# Date: Thu Feb 1 10:00:00 2024
#
#   feat: optimize database queries

# Reset
git bisect reset

Git Stash: Temporarily Save Work

# Working on feature, need to switch to hotfix urgently
git status
# Modified: src/services/email.service.ts
# Modified: src/templates/welcome.html

# Save work temporarily
git stash save "WIP: email notification implementation"

# Working directory is now clean
git status
# nothing to commit, working tree clean

# Switch to hotfix
git checkout main
git checkout -b hotfix/critical-bug
# ... fix bug ...

# Return to feature work
git checkout feature/email-notifications
git stash list
# stash@{0}: On feature/email-notifications: WIP: email notification implementation

# Restore work
git stash pop

# Or apply without removing from stash
git stash apply stash@{0}

Troubleshooting Common Issues

Issue 1: Accidentally Committed to Wrong Branch

# Oh no! Committed to main instead of feature branch
git log --oneline
a1b2c3d feat: add new feature  # This shouldn't be on main!

# Solution: Move commit to correct branch
git reset HEAD~1 --soft  # Undo commit, keep changes
git stash                 # Save changes
git checkout -b feature/new-feature  # Create correct branch
git stash pop            # Restore changes
git add .
git commit -m "feat: add new feature"

# Main is back to previous state, commit is now on feature branch

Issue 2: Need to Undo a Pushed Commit

# Pushed bad commit to main
git log --oneline
a1b2c3d Bad commit
b2c3d4e Good commit

# Option 1: Revert (creates new commit undoing changes)
git revert a1b2c3d
git push origin main

# Option 2: Reset (rewrites history - dangerous!)
# Only if no one else has pulled this commit!
git reset --hard b2c3d4e
git push --force origin main  # ⚠️ Use with extreme caution!

Issue 3: Merge Conflict Resolution

# Merge conflict during pull
git pull origin main
# Auto-merging src/services/user.service.ts
# CONFLICT (content): Merge conflict in src/services/user.service.ts

# Open file

// src/services/user.service.ts
export class UserService {
  async getUser(id: string): Promise<User> {
<<<<<<< HEAD
    // Your changes
    return await this.prisma.user.findUnique({
      where: { id },
      include: { 
        profile: true, 
        posts: true,
        comments: true 
      }
    });
=======
    // Their changes
    return await this.prisma.user.findUnique({
      where: { id },
      include: { 
        profile: true,
        settings: true 
      }
    });
>>>>>>> origin/main
  }
}

# Resolve manually by keeping both includes

export class UserService {
  async getUser(id: string): Promise<User> {
    return await this.prisma.user.findUnique({
      where: { id },
      include: { 
        profile: true, 
        posts: true,
        comments: true,
        settings: true 
      }
    });
  }
}

# Mark as resolved and complete merge
git add src/services/user.service.ts
git commit -m "merge: resolve conflict in user.service.ts"
git push

Issue 4: Large File Accidentally Committed

# Committed large file (video.mp4 - 100MB)
git add .
git commit -m "Add demo video"
git push
# error: file too large (max 50MB)

# Remove from git but keep locally
git rm --cached video.mp4

# Add to .gitignore
echo "*.mp4" >> .gitignore

# Commit removal
git add .gitignore
git commit -m "Remove large video file, add to gitignore"

# If already pushed, need to remove from history
git filter-branch --tree-filter 'rm -f video.mp4' HEAD
git push --force

Issue 5: Recover Deleted Branch

# Accidentally deleted branch
git branch -D feature/important-work
# Deleted branch feature/important-work

# Find commit SHA of deleted branch
git reflog
# a1b2c3d HEAD@{0}: checkout: moving from feature/important-work to main
# b2c3d4e HEAD@{1}: commit: Important work

# Recreate branch from that commit
git checkout -b feature/important-work b2c3d4e

# Branch recovered!

Production Deployment Best Practices

Blue-Green Deployment with Git Tags

# Deploy v1.3.0 to production
git tag -a v1.3.0-blue -m "Production deployment (blue)"
git push origin v1.3.0-blue

# Deployment runs in parallel environment
# Test in blue environment
# Switch traffic: blue → green
# green now serves v1.3.0

# If issues, quick rollback
git tag -a v1.2.9-blue -m "Rollback to v1.2.9"
git push origin v1.2.9-blue
# Switch traffic back

Deployment Freeze During Critical Periods

# .github/DEPLOYMENT_FREEZE.md

## Deployment Freeze Schedule

### Holiday Season (Nov 20 - Jan 5)
- No deployments to production Friday-Sunday
- Emergency hotfixes only during freeze
- All changes must be approved by tech lead

### End of Month (Last 3 days)
- No major releases
- Bug fixes and hotfixes allowed
- Database migrations require extra approval

Monitoring After Deployment

# After deploying v1.3.0

# 1. Check error rates
# Grafana dashboard: https://grafana.company.com/d/errors

# 2. Monitor key metrics
watch -n 5 'curl -s https://api.company.com/health | jq'

# 3. Check logs
kubectl logs -f deployment/payment-service --tail=100

# 4. Monitor for 30 minutes before declaring success

# 5. If issues detected, rollback immediately
git tag -a v1.2.9-rollback -m "Rollback due to increased error rate"
git push origin v1.2.9-rollback

Conclusion

Advanced version control workflows are essential for managing production systems reliably. Throughout this series, we've covered:

Part 1: Git fundamentals, essential commands, and basic workflows

Part 2: Branching strategies (Git Flow, GitHub Flow, Trunk-Based Development)

Part 3: Monorepo vs Polyrepo approaches with real examples

Part 4: GitHub collaboration, pull requests, and code reviews

Part 5: Release management, hotfix procedures, Git hooks, and troubleshooting

Key Lessons from Production Experience

1. Always have a rollback plan before deploying

2. Automate what you can (CI/CD, releases, checks)

3. Use semantic versioning for clear communication

4. Tag releases for easy rollback and tracking

5. Monitor deployments for at least 30 minutes

6. Document hotfix procedures before you need them

7. Use Git hooks to prevent common mistakes

8. Keep commit history clean for easier debugging

9. Test thoroughly before production deployment

10. Learn from incidents and improve processes

My Production Workflow Summary

# Feature development
git checkout -b feature/my-feature
# ... develop, commit, push ...
# Create PR, get review, merge

# Release
npm version minor
git tag -a v1.3.0 -m "Release 1.3.0"
git push origin main --tags
# Automated deployment via GitHub Actions

# Monitor production
# Check metrics, logs, errors for 30 minutes

# If issues: Hotfix
git checkout -b hotfix/critical-issue
# ... fix, test ...
git push, create PR, expedited review
npm version patch
git tag -a v1.3.1 -m "Hotfix 1.3.1"
git push origin main --tags
# Deploy, monitor

Further Learning

• Pro Git Book - Comprehensive Git guide

• GitHub Skills - Interactive GitHub tutorials

• Conventional Commits - Commit message standard

• Semantic Versioning - Versioning specification

---

Final Thoughts:

Version control is more than just a tool—it's a mindset. It's about:

• Collaboration over individual work

• Automation over manual processes

• Safety over speed (though automation enables both)

• Learning from every deployment

The best time to implement proper version control practices was at the start of your project. The second-best time is now.

Thank you for following this series! May your commits be atomic, your merges conflict-free, and your deployments successful! 🚀

---

Version Control 101 Series Complete

All parts:

1. Introduction to Version Control and Git Fundamentals

2. Branching Strategies and Workflows

3. Monorepo vs Polyrepo Strategies

4. GitHub Best Practices and Collaboration

5. Advanced Version Control and Production Workflows

Remember: Good version control practices separate professional development from chaos!