Part 5: HTTP Clients and Real-World Integration - Complete Application

Introduction

Building a security tool isn't just about the core logic—it needs HTTP clients, JSON parsing, CLI arguments, and user-friendly output. This final part shows how I integrated everything to make simple-waf-scanner a complete, production-ready tool.

We'll cover HTTP with reqwest, JSON with serde, CLI with clap, colored terminal output, and publishing to crates.io.

HTTP Requests with Reqwest

Setting Up the Client

use reqwest::{Client, Response, header};
use std::time::Duration;

fn build_client() -> Result<Client, reqwest::Error> {
    Client::builder()
        .timeout(Duration::from_secs(10))
        .user_agent("WAF-Scanner/0.1.0")
        .danger_accept_invalid_certs(true)  // For testing environments
        .redirect(reqwest::redirect::Policy::limited(5))
        .build()
}

Making GET Requests

async fn test_get_request(
    client: &Client,
    url: &str,
    payload: &str,
) -> Result<Response, reqwest::Error> {
    client
        .get(url)
        .query(&[("q", payload)])
        .header(header::ACCEPT, "application/json")
        .send()
        .await
}

Making POST Requests

use serde_json::json;

async fn test_post_request(
    client: &Client,
    url: &str,
    payload: &str,
) -> Result<Response, reqwest::Error> {
    let body = json!({
        "query": payload,
        "type": "search"
    });
    
    client
        .post(url)
        .json(&body)
        .send()
        .await
}

Custom Headers

async fn test_with_custom_headers(
    client: &Client,
    url: &str,
) -> Result<Response, reqwest::Error> {
    client
        .get(url)
        .header("X-Original-URL", "/admin")
        .header("X-Rewrite-URL", "/admin")
        .header("X-Forwarded-For", "127.0.0.1")
        .send()
        .await
}

JSON Serialization with Serde

Defining Data Structures

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
pub struct Payload {
    pub id: String,
    pub value: String,
    pub category: String,
    pub severity: Severity,
}

#[derive(Debug, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
pub enum Severity {
    Critical,
    High,
    Medium,
    Low,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct Finding {
    pub payload_id: String,
    pub url: String,
    pub status_code: u16,
    pub response_size: usize,
    pub timestamp: String,
}

Loading JSON from Files

use std::fs;

fn load_payloads(path: &str) -> Result<Vec<Payload>, Box<dyn std::error::Error>> {
    let content = fs::read_to_string(path)?;
    let payloads: Vec<Payload> = serde_json::from_str(&content)?;
    Ok(payloads)
}

Saving Results as JSON

fn save_findings(
    findings: &[Finding],
    path: &str,
) -> Result<(), Box<dyn std::error::Error>> {
    let json = serde_json::to_string_pretty(findings)?;
    fs::write(path, json)?;
    Ok(())
}

Working with API Responses

#[derive(Debug, Deserialize)]
struct ApiResponse {
    success: bool,
    data: Option<serde_json::Value>,
    error: Option<String>,
}

async fn parse_api_response(
    response: Response,
) -> Result<ApiResponse, reqwest::Error> {
    response.json::<ApiResponse>().await
}

Building the CLI with Clap

Add Dependencies

[dependencies]
clap = { version = "4", features = ["derive"] }

Define CLI Structure

use clap::Parser;

#[derive(Parser, Debug)]
#[command(name = "waf-scanner")]
#[command(author = "Your Name")]
#[command(version = "0.1.0")]
#[command(about = "WAF bypass testing tool", long_about = None)]
struct Cli {
    /// Target URL to scan
    #[arg(short, long)]
    url: String,
    
    /// Path to payloads file
    #[arg(short, long, default_value = "payloads.json")]
    payloads: String,
    
    /// Maximum concurrent requests
    #[arg(short, long, default_value_t = 10)]
    concurrency: usize,
    
    /// Delay between requests (ms)
    #[arg(short, long, default_value_t = 0)]
    delay: u64,
    
    /// Output file for findings
    #[arg(short, long)]
    output: Option<String>,
    
    /// Verbose output
    #[arg(short, long)]
    verbose: bool,
}

Using CLI Arguments

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    
    if cli.verbose {
        println!("Target URL: {}", cli.url);
        println!("Concurrency: {}", cli.concurrency);
    }
    
    let scanner = Scanner::new(ScanConfig {
        target_url: cli.url,
        concurrency: cli.concurrency,
        delay_ms: cli.delay,
        timeout_secs: 10,
    });
    
    let payloads = load_payloads(&cli.payloads)?;
    let findings = scanner.scan(payloads).await;
    
    if let Some(output_path) = cli.output {
        save_findings(&findings, &output_path)?;
    } else {
        print_findings(&findings);
    }
    
    Ok(())
}

Subcommands

use clap::{Parser, Subcommand};

#[derive(Parser)]
#[command(name = "waf-scanner")]
struct Cli {
    #[command(subcommand)]
    command: Commands,
}

#[derive(Subcommand)]
enum Commands {
    /// Scan a target URL
    Scan {
        #[arg(short, long)]
        url: String,
        
        #[arg(short, long)]
        payloads: String,
    },
    
    /// Detect WAF type
    Detect {
        #[arg(short, long)]
        url: String,
    },
    
    /// List available payloads
    List {
        #[arg(short, long)]
        category: Option<String>,
    },
}

fn main() {
    let cli = Cli::parse();
    
    match cli.command {
        Commands::Scan { url, payloads } => {
            println!("Scanning {} with {}", url, payloads);
        }
        Commands::Detect { url } => {
            println!("Detecting WAF at {}", url);
        }
        Commands::List { category } => {
            println!("Listing payloads: {:?}", category);
        }
    }
}

Colored Terminal Output

Add Dependency

[dependencies]
colored = "2"

Using Colors

use colored::*;

fn print_banner() {
    println!("{}", "═══════════════════════════════════".cyan().bold());
    println!("{}", "   WAF Bypass Scanner v0.1.0      ".cyan().bold());
    println!("{}", "═══════════════════════════════════".cyan().bold());
}

fn print_finding(finding: &Finding) {
    println!("{} {}", 
        "✓".green().bold(),
        format!("Bypass found: {}", finding.payload_id).white()
    );
    println!("  {} {}", "URL:".yellow(), finding.url);
    println!("  {} {}", "Status:".yellow(), finding.status_code);
}

fn print_error(message: &str) {
    eprintln!("{} {}", "✗".red().bold(), message.red());
}

fn print_summary(total: usize, found: usize) {
    println!();
    println!("{}", "Summary:".cyan().bold());
    println!("  Total payloads tested: {}", total.to_string().white());
    println!("  Bypasses found: {}", 
        if found > 0 {
            found.to_string().green().bold()
        } else {
            found.to_string().yellow()
        }
    );
}

Progress Indication

Using indicatif

[dependencies]
indicatif = "0.17"

Progress Bar Implementation

use indicatif::{ProgressBar, ProgressStyle};

async fn scan_with_progress(
    scanner: &Scanner,
    payloads: Vec<Payload>,
) -> Vec<Finding> {
    let pb = ProgressBar::new(payloads.len() as u64);
    pb.set_style(
        ProgressStyle::default_bar()
            .template("{spinner:.green} [{elapsed_precise}] [{bar:40.cyan/blue}] {pos}/{len} ({eta})")
            .unwrap()
            .progress_chars("#>-")
    );
    
    let (tx, mut rx) = mpsc::channel(100);
    
    // Spawn scanning tasks
    for payload in payloads {
        let tx = tx.clone();
        let scanner = scanner.clone();
        tokio::spawn(async move {
            let result = scanner.test_payload(&payload).await;
            tx.send((payload, result)).await.unwrap();
        });
    }
    drop(tx);
    
    // Collect results with progress updates
    let mut findings = Vec::new();
    while let Some((payload, result)) = rx.recv().await {
        pb.inc(1);
        if let Ok(Some(finding)) = result {
            pb.println(format!("{} Found: {}", "✓".green(), payload.id));
            findings.push(finding);
        }
    }
    
    pb.finish_with_message("Scan complete");
    findings
}

Complete Application Structure

// src/main.rs
use clap::Parser;
use colored::*;

mod scanner;
mod payload;
mod config;

use scanner::Scanner;
use payload::load_payloads;
use config::ScanConfig;

#[derive(Parser)]
#[command(version, about)]
struct Cli {
    #[arg(short, long)]
    url: String,
    
    #[arg(short, long, default_value = "payloads.json")]
    payloads: String,
    
    #[arg(short, long, default_value_t = 10)]
    concurrency: usize,
    
    #[arg(short, long, default_value_t = 0)]
    delay: u64,
    
    #[arg(short, long)]
    output: Option<String>,
    
    #[arg(short, long)]
    verbose: bool,
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    
    print_banner();
    
    let config = ScanConfig {
        target_url: cli.url.clone(),
        concurrency: cli.concurrency,
        delay_ms: cli.delay,
        timeout_secs: 10,
        verbose: cli.verbose,
    };
    
    if config.verbose {
        print_config(&config);
    }
    
    let scanner = Scanner::new(config);
    let payloads = load_payloads(&cli.payloads)?;
    
    println!("{} Loaded {} payloads", "→".cyan(), payloads.len());
    println!("{} Starting scan...\n", "→".cyan());
    
    let findings = scanner.scan(payloads).await;
    
    println!();
    print_summary(&findings);
    
    if let Some(output_path) = cli.output {
        save_findings(&findings, &output_path)?;
        println!("{} Saved findings to {}", "→".cyan(), output_path);
    }
    
    Ok(())
}

fn print_banner() {
    println!("{}", "═════════════════════════════════════".cyan().bold());
    println!("{}", "   Simple WAF Scanner v0.1.2        ".cyan().bold());
    println!("{}", "═════════════════════════════════════".cyan().bold());
    println!();
}

fn print_config(config: &ScanConfig) {
    println!("{}", "Configuration:".yellow().bold());
    println!("  Target:      {}", config.target_url);
    println!("  Concurrency: {}", config.concurrency);
    println!("  Delay:       {}ms", config.delay_ms);
    println!();
}

fn print_summary(findings: &[Finding]) {
    println!("{}", "═════════════════════════════════════".cyan());
    println!("{}", "Summary:".cyan().bold());
    println!("  Bypasses found: {}", 
        if findings.is_empty() {
            findings.len().to_string().yellow()
        } else {
            findings.len().to_string().green().bold()
        }
    );
    
    for finding in findings {
        println!("\n  {} {}", "✓".green().bold(), finding.payload_id.white());
        println!("    Status: {}", finding.status_code);
        println!("    Size:   {} bytes", finding.response_size);
    }
    
    println!("{}", "═════════════════════════════════════".cyan());
}

Real Scanner Module

// src/scanner.rs
use reqwest::Client;
use std::sync::Arc;
use tokio::sync::Semaphore;
use futures::stream::{self, StreamExt};

use crate::config::ScanConfig;
use crate::payload::Payload;

#[derive(Clone)]
pub struct Scanner {
    client: Arc<Client>,
    config: ScanConfig,
}

impl Scanner {
    pub fn new(config: ScanConfig) -> Self {
        let client = Client::builder()
            .timeout(std::time::Duration::from_secs(config.timeout_secs))
            .danger_accept_invalid_certs(true)
            .build()
            .unwrap();
        
        Scanner {
            client: Arc::new(client),
            config,
        }
    }
    
    pub async fn scan(&self, payloads: Vec<Payload>) -> Vec<Finding> {
        let semaphore = Arc::new(Semaphore::new(self.config.concurrency));
        
        stream::iter(payloads)
            .map(|payload| {
                let client = self.client.clone();
                let url = self.config.target_url.clone();
                let delay = self.config.delay_ms;
                let permit = semaphore.clone();
                
                async move {
                    let _permit = permit.acquire().await.unwrap();
                    
                    let result = test_payload(&client, &url, &payload).await;
                    
                    if delay > 0 {
                        tokio::time::sleep(
                            std::time::Duration::from_millis(delay)
                        ).await;
                    }
                    
                    result.ok().flatten()
                }
            })
            .buffer_unordered(self.config.concurrency)
            .filter_map(|x| async { x })
            .collect()
            .await
    }
}

async fn test_payload(
    client: &Client,
    url: &str,
    payload: &Payload,
) -> Result<Option<Finding>, reqwest::Error> {
    let response = client
        .get(url)
        .query(&[("q", &payload.value)])
        .send()
        .await?;
    
    let status = response.status().as_u16();
    let size = response.bytes().await?.len();
    
    if is_bypass(status) {
        Ok(Some(Finding {
            payload_id: payload.id.clone(),
            url: url.to_string(),
            status_code: status,
            response_size: size,
            timestamp: chrono::Utc::now().to_rfc3339(),
        }))
    } else {
        Ok(None)
    }
}

fn is_bypass(status: u16) -> bool {
    matches!(status, 200..=299)
}

#[derive(Debug, serde::Serialize)]
pub struct Finding {
    pub payload_id: String,
    pub url: String,
    pub status_code: u16,
    pub response_size: usize,
    pub timestamp: String,
}

Publishing to Crates.io

Prepare Cargo.toml

[package]
name = "simple-waf-scanner"
version = "0.1.2"
edition = "2021"
authors = ["Your Name <[email protected]>"]
description = "A lightweight WAF bypass testing tool"
license = "MIT"
repository = "https://github.com/yourusername/simple-waf-scanner"
keywords = ["security", "waf", "scanner", "bypass"]
categories = ["command-line-utilities", "web-programming"]
readme = "README.md"

[dependencies]
tokio = { version = "1", features = ["full"] }
reqwest = { version = "0.11", features = ["json"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
clap = { version = "4", features = ["derive"] }
colored = "2"
futures = "0.3"

Create README.md

# Simple WAF Scanner

A lightweight, concurrent WAF bypass testing tool written in Rust.

## Features

- 280+ security payloads
- Concurrent testing with configurable limits
- Rate limiting support
- JSON output
- Colored terminal output

## Installation

```bash
cargo install simple-waf-scanner

Usage

waf-scanner -u https://example.com -p payloads.json -c 10

License

MIT

### Publish to Crates.io

```bash
# Login (one-time)
cargo login

# Build and test
cargo build --release
cargo test

# Package (dry run)
cargo package --allow-dirty

# Publish
cargo publish

Version Updates

# Update version in Cargo.toml
vim Cargo.toml  # Change version = "0.1.3"

# Tag release
git tag v0.1.3
git push origin v0.1.3

# Publish new version
cargo publish

Integration Testing

Testing the Complete Application

#[cfg(test)]
mod tests {
    use super::*;
    
    #[tokio::test]
    async fn test_full_scan() {
        let config = ScanConfig {
            target_url: "https://httpbin.org/get".to_string(),
            concurrency: 5,
            delay_ms: 100,
            timeout_secs: 10,
            verbose: false,
        };
        
        let scanner = Scanner::new(config);
        let payloads = vec![
            Payload::new("test1", "<script>alert(1)</script>"),
            Payload::new("test2", "' OR '1'='1"),
        ];
        
        let findings = scanner.scan(payloads).await;
        
        // Should find bypasses for test payloads
        assert!(!findings.is_empty());
    }
}

Environment Configuration

Using .env Files

[dependencies]
dotenv = "0.15"

use dotenv::dotenv;
use std::env;

fn main() {
    dotenv().ok();
    
    let api_key = env::var("API_KEY")
        .expect("API_KEY must be set");
    
    let timeout = env::var("TIMEOUT_SECS")
        .unwrap_or_else(|_| "10".to_string())
        .parse()
        .unwrap();
}

Logging

Add structured logging

[dependencies]
tracing = "0.1"
tracing-subscriber = "0.3"

use tracing::{info, warn, error, debug};
use tracing_subscriber;

fn main() {
    tracing_subscriber::fmt::init();
    
    info!("Starting scanner");
    debug!("Config: {:?}", config);
    warn!("Rate limiting enabled");
    error!("Failed to connect: {}", err);
}

Key Takeaways

1. Reqwest: Powerful HTTP client with async support

2. Serde: Effortless JSON serialization/deserialization

3. Clap: Beautiful CLI parsing with derive macros

4. Colored: Easy terminal output styling

5. Crates.io: Simple publishing process

6. Integration: Combine libraries for complete solutions

Common Patterns

// HTTP request with retries
async fn request_with_retry(
    client: &Client,
    url: &str,
    retries: u32,
) -> Result<Response, reqwest::Error> {
    for attempt in 0..retries {
        match client.get(url).send().await {
            Ok(response) => return Ok(response),
            Err(e) if attempt == retries - 1 => return Err(e),
            Err(_) => tokio::time::sleep(Duration::from_secs(1)).await,
        }
    }
    unreachable!()
}

// JSON with default values
#[derive(Deserialize)]
struct Config {
    #[serde(default = "default_timeout")]
    timeout: u64,
    
    #[serde(default)]
    verbose: bool,  // defaults to false
}

fn default_timeout() -> u64 { 10 }

// CLI with validation
fn validate_url(url: &str) -> Result<(), String> {
    if url.starts_with("http") {
        Ok(())
    } else {
        Err("URL must start with http:// or https://".to_string())
    }
}

Series Conclusion

You've now built a complete Rust application from scratch:

• Part 1: Setup, basics, types, structs

• Part 2: Ownership, borrowing, lifetimes

• Part 3: Error handling with Result

• Part 4: Async programming with Tokio

• Part 5: HTTP, JSON, CLI, publishing

The simple-waf-scanner project demonstrates all these concepts in a real production tool with 280+ payloads and 11+ WAF detections.

---

Project available at github.com/Htunn/simple-waf-scanner Published on crates.io as simple-waf-scanner