Part 3: Natural Language Processing — NLP, NLU, and NLG

Part of the AI Fundamentals 101 Series

Why NLP Is the Backbone of Modern AI

Every time you interact with a modern AI system — asking Claude a question, using Google Translate, dictating a message to Siri, or getting autocomplete suggestions in your IDE — you're using Natural Language Processing.

NLP is the bridge between human language and machine computation. Without it, AI would be limited to numbers, images, and structured data. NLP is what makes AI conversational.

I first appreciated NLP when I built a log analysis tool for my home lab. Kubernetes error logs are technically "text," but they're messy: stack traces mixed with timestamps, pod names, cryptic error codes. Teaching a system to extract meaningful information from that unstructured text is an NLP problem. And understanding the distinction between NLP, NLU, and NLG is what helped me architect that system correctly.

---

What is NLP?

Natural Language Processing (NLP) is the field of AI that deals with the interaction between computers and human language. It covers everything from basic text manipulation to understanding meaning to generating coherent text.

NLP is actually an umbrella term that contains two major subfields:

Natural Language Processing (NLP)
├── NLU — Natural Language Understanding
│   (Input: text → Output: meaning/intent/entities)
└── NLG — Natural Language Generation
    (Input: meaning/data → Output: text)

NLU vs NLG — The Two Halves

• NLU (Natural Language Understanding): Taking text and extracting meaning. "What does this text say?"

• NLG (Natural Language Generation): Taking structured data or intent and producing text. "How do I express this?"

# NLU: Text → Structured Information
text = "Deploy the payment-service to production on Friday at 3pm"

nlu_output = {
    "intent": "deploy",
    "entities": {
        "service": "payment-service",
        "environment": "production",
        "date": "Friday",
        "time": "3pm"
    },
    "sentiment": "neutral",
    "urgency": "scheduled"
}

# NLG: Structured Information → Text
deployment_data = {
    "service": "payment-service",
    "status": "completed",
    "duration": "45 seconds",
    "replicas": 3
}

nlg_output = (
    "Deployment of payment-service completed successfully in 45 seconds. "
    "3 replicas are now running in production."
)

Every conversational AI system uses both:

1. NLU to understand what you said

2. Processing/reasoning in the middle

3. NLG to formulate a response

---

The NLP Pipeline: From Raw Text to Understanding

Processing natural language involves multiple stages. Here's the pipeline that most NLP systems use, starting from raw text:

Step 1: Tokenization — Breaking Text into Pieces

Tokenization splits text into smaller units (tokens) that the model can process.

# Simple word tokenization
text = "The Kubernetes pod crashed with OOMKilled error at 2024-03-15T14:30:00Z"

# Basic approach: split on whitespace
basic_tokens = text.split()
print(f"Basic tokens ({len(basic_tokens)}): {basic_tokens}")
# ['The', 'Kubernetes', 'pod', 'crashed', 'with', 'OOMKilled', 'error', 'at', '2024-03-15T14:30:00Z']

# Better: NLTK tokenizer handles punctuation and special cases
import nltk
nltk.download("punkt_tab", quiet=True)
from nltk.tokenize import word_tokenize

nltk_tokens = word_tokenize(text)
print(f"NLTK tokens ({len(nltk_tokens)}): {nltk_tokens}")
# ['The', 'Kubernetes', 'pod', 'crashed', 'with', 'OOMKilled', 'error', 'at', '2024-03-15T14:30:00Z']

LLMs use subword tokenization, which is different:

# How LLMs tokenize — subword level (BPE / SentencePiece)
# "unhappiness" might become: ["un", "happi", "ness"]
# This lets the model handle words it has never seen before

# You can see how OpenAI tokenizes text:
# Every ~4 characters ≈ 1 token (rough approximation)
text = "The Kubernetes pod crashed with OOMKilled error"
estimated_tokens = len(text) / 4
print(f"Approximate tokens: {estimated_tokens:.0f}")  # ~12 tokens

Step 2: Text Preprocessing — Cleaning the Data

import re

def preprocess_text(text: str) -> str:
    """Clean and normalize text for NLP processing."""
    # Lowercase
    text = text.lower()
    # Remove timestamps
    text = re.sub(r'\d{4}-\d{2}-\d{2}t\d{2}:\d{2}:\d{2}z?', '', text)
    # Remove extra whitespace
    text = re.sub(r'\s+', ' ', text).strip()
    return text

raw_logs = [
    "2024-03-15T14:30:00Z ERROR Pod payment-svc-7d8b OOMKilled",
    "2024-03-15T14:30:01Z WARN  Memory usage at 95% for payment-svc",
    "2024-03-15T14:30:02Z INFO  Restarting pod payment-svc-7d8b",
]

for log in raw_logs:
    print(preprocess_text(log))
# error pod payment-svc-7d8b oomkilled
# warn memory usage at 95% for payment-svc
# info restarting pod payment-svc-7d8b

Step 3: Stopword Removal and Stemming

from nltk.corpus import stopwords
from nltk.stem import PorterStemmer

nltk.download("stopwords", quiet=True)

stop_words = set(stopwords.words("english"))
stemmer = PorterStemmer()

text = "The servers are running but the deployments are failing repeatedly"
tokens = word_tokenize(text.lower())

# Remove stopwords (common words that don't carry meaning)
filtered = [t for t in tokens if t not in stop_words]
print(f"Without stopwords: {filtered}")
# ['servers', 'running', 'deployments', 'failing', 'repeatedly']

# Stemming: reduce words to their root form
stemmed = [stemmer.stem(t) for t in filtered]
print(f"Stemmed: {stemmed}")
# ['server', 'run', 'deploy', 'fail', 'repeatedli']

Note: Stemming is a crude approach — "repeatedly" becomes "repeatedli" which isn't a real word. Modern approaches use lemmatization (which produces real words) or skip this step entirely and let the model handle it.

Step 4: Feature Extraction — Turning Text into Numbers

Models need numbers, not words. There are several ways to convert text to numerical features:

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

documents = [
    "pod crashed with OOMKilled error",
    "deployment successful all replicas healthy",
    "pod restarting due to memory limit exceeded",
    "service endpoint responding normally",
    "container killed out of memory resources"
]

# Approach 1: Bag of Words — count word occurrences
count_vec = CountVectorizer()
bow_matrix = count_vec.fit_transform(documents)
print("Vocabulary:", count_vec.get_feature_names_out()[:10])
print(f"Shape: {bow_matrix.shape}")  # (5 documents, N unique words)

# Approach 2: TF-IDF — words weighted by importance
# TF (Term Frequency): how often a word appears in the document
# IDF (Inverse Document Frequency): how rare it is across all documents
# Words that appear in many documents are downweighted
tfidf_vec = TfidfVectorizer()
tfidf_matrix = tfidf_vec.fit_transform(documents)

# "the" appears everywhere → low TF-IDF score
# "OOMKilled" appears in one document → high TF-IDF score
print(f"\nTF-IDF shape: {tfidf_matrix.shape}")

Modern approach: embeddings. Instead of counting words, we use neural networks to create dense vector representations where similar meanings are close together. We'll cover this in detail in Part 4.

---

Named Entity Recognition (NER)

NER identifies and classifies entities in text — names, organizations, locations, dates, etc.

# Simple rule-based NER for infrastructure logs
import re

def extract_infrastructure_entities(text: str) -> dict:
    """Extract infrastructure-related entities from log text."""
    entities = {}

    # Pod names (pattern: name-hash)
    pod_match = re.findall(r'[\w]+-[\w]+-[\w]+', text)
    if pod_match:
        entities["pods"] = pod_match

    # Namespaces (after "namespace/" or "ns=")
    ns_match = re.findall(r'(?:namespace/|ns=)([\w-]+)', text)
    if ns_match:
        entities["namespaces"] = ns_match

    # Error types
    error_patterns = ["OOMKilled", "CrashLoopBackOff", "ImagePullBackOff",
                      "ErrImagePull", "CreateContainerError"]
    found_errors = [e for e in error_patterns if e.lower() in text.lower()]
    if found_errors:
        entities["error_types"] = found_errors

    # Percentages
    pct_match = re.findall(r'(\d+(?:\.\d+)?%)', text)
    if pct_match:
        entities["metrics"] = pct_match

    return entities

logs = [
    "Pod payment-svc-7d8b in namespace/production: OOMKilled (memory at 98%)",
    "CrashLoopBackOff for api-gateway-abc1 in ns=staging",
]

for log in logs:
    print(f"Log: {log}")
    print(f"Entities: {extract_infrastructure_entities(log)}\n")

Output:

Log: Pod payment-svc-7d8b in namespace/production: OOMKilled (memory at 98%)
Entities: {'pods': ['payment-svc-7d8b'], 'namespaces': ['production'], 'error_types': ['OOMKilled'], 'metrics': ['98%']}

Log: CrashLoopBackOff for api-gateway-abc1 in ns=staging
Entities: {'pods': ['api-gateway-abc1'], 'namespaces': ['staging'], 'error_types': ['CrashLoopBackOff']}

This is a simple rule-based NER. Production systems use trained models (spaCy, transformers) that can recognize entities they've never seen before.

---

Sentiment Analysis

Sentiment analysis determines whether text expresses positive, negative, or neutral sentiment.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import Pipeline
from sklearn.model_selection import cross_val_score
import numpy as np

# Training data: log messages with sentiment labels
# In my monitoring project, I use this to classify alert severity
messages = [
    # Positive/Normal (0)
    "Deployment completed successfully",
    "All health checks passing",
    "Service scaled to 5 replicas",
    "Certificate renewed without issues",
    "Backup completed in 2 minutes",
    "All pods running and healthy",
    "Traffic routing updated successfully",
    "Database migration completed",
    # Negative/Problem (1)
    "Pod crashed with OOMKilled",
    "Connection timeout after 30 seconds",
    "Disk usage critical at 95%",
    "Multiple pods in CrashLoopBackOff",
    "Database replication lag exceeding threshold",
    "Memory leak detected in worker process",
    "SSL certificate expiring in 2 days",
    "Service returning 500 errors",
]

labels = [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1]

# Build a simple sentiment classifier
sentiment_pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),  # Include word pairs
    ("clf", MultinomialNB())
])

# Evaluate with cross-validation
scores = cross_val_score(sentiment_pipeline, messages, labels, cv=4)
print(f"Accuracy: {scores.mean():.2%} (+/- {scores.std():.2%})")

# Train on all data and test
sentiment_pipeline.fit(messages, labels)

test_messages = [
    "Deployment failed with timeout error",
    "New version rolling out smoothly",
    "High memory usage detected on node-3",
]

for msg in test_messages:
    prediction = sentiment_pipeline.predict([msg])[0]
    label = "Problem" if prediction == 1 else "Normal"
    print(f"  [{label}] {msg}")

---

Text Classification

Beyond sentiment, NLP can classify text into any categories you define:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
import numpy as np

# Multi-class classification: categorize infrastructure alerts
alert_texts = [
    # Category 0: Compute
    "CPU utilization at 95% on node-1",
    "Pod OOMKilled due to memory limit",
    "Container using excessive CPU resources",
    "Worker process consuming all available memory",
    # Category 1: Network
    "Connection refused on port 5432",
    "DNS resolution failing for service endpoint",
    "Network latency spike to 500ms",
    "Load balancer returning 502 Bad Gateway",
    # Category 2: Storage
    "Disk usage at 90% on /data volume",
    "PVC provisioning failed for database",
    "Backup storage quota exceeded",
    "IO latency on persistent volume exceeding 100ms",
]

categories = [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]
category_names = {0: "Compute", 1: "Network", 2: "Storage"}

# Build classifier
classifier = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000))
])

classifier.fit(alert_texts, categories)

# Classify new alerts
new_alerts = [
    "Node running out of memory, pods being evicted",
    "Unable to connect to Redis on port 6379",
    "Persistent volume claim stuck in pending state",
]

for alert in new_alerts:
    prediction = classifier.predict([alert])[0]
    print(f"  [{category_names[prediction]}] {alert}")
# [Compute] Node running out of memory, pods being evicted
# [Network] Unable to connect to Redis on port 6379
# [Storage] Persistent volume claim stuck in pending state

---

From Rule-Based Chatbots to LLM-Powered Assistants

The evolution of chatbots perfectly illustrates the progression of NLP technology.

Generation 1: Rule-Based (Pattern Matching)

import re

def rule_based_chatbot(message: str) -> str:
    """The simplest chatbot: pattern matching."""
    message = message.lower().strip()

    patterns = [
        (r"(hi|hello|hey)", "Hello! How can I help you?"),
        (r"(status|health).*cluster", "All 3 nodes are healthy. 42 pods running."),
        (r"(deploy|release).*(production|prod)", "Last production deployment: 2 hours ago, v2.3.1"),
        (r"(cpu|memory|disk).*usage", "CPU: 45%, Memory: 62%, Disk: 38%"),
        (r"(restart|bounce).*pod", "Which pod would you like to restart?"),
        (r"(help|what can you do)", "I can check cluster status, deployment info, and resource usage."),
    ]

    for pattern, response in patterns:
        if re.search(pattern, message):
            return response

    return "I don't understand. Try 'help' to see what I can do."

# Test it
test_messages = [
    "Hey there",
    "What's the cluster status?",
    "Show me CPU usage",
    "Can you write me a poem?",  # Falls through to default
]

for msg in test_messages:
    print(f"User: {msg}")
    print(f"Bot:  {rule_based_chatbot(msg)}\n")

Limitations: You must anticipate every possible phrasing. "How's the cluster doing?" won't match "status.*cluster". This approach doesn't scale.

Generation 2: Intent Classification + Entity Extraction

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
import re

class IntentChatbot:
    """Chatbot that classifies intent, then acts."""

    def __init__(self):
        # Training data for intent classification
        training_data = [
            ("check cluster status", "cluster_status"),
            ("how is the cluster", "cluster_status"),
            ("is everything running", "cluster_status"),
            ("show resource usage", "resource_usage"),
            ("what's the cpu at", "resource_usage"),
            ("memory utilization", "resource_usage"),
            ("deploy to production", "deploy"),
            ("release new version", "deploy"),
            ("push to prod", "deploy"),
            ("restart the pod", "restart_pod"),
            ("bounce the service", "restart_pod"),
            ("kill and restart", "restart_pod"),
        ]

        texts, intents = zip(*training_data)
        self.vectorizer = TfidfVectorizer()
        X = self.vectorizer.fit_transform(texts)

        self.classifier = LogisticRegression(max_iter=1000)
        self.classifier.fit(X, intents)

        self.handlers = {
            "cluster_status": self._handle_status,
            "resource_usage": self._handle_resources,
            "deploy": self._handle_deploy,
            "restart_pod": self._handle_restart,
        }

    def respond(self, message: str) -> str:
        X = self.vectorizer.transform([message])
        intent = self.classifier.predict(X)[0]
        confidence = self.classifier.predict_proba(X).max()

        if confidence < 0.4:
            return "I'm not sure what you're asking. Could you rephrase?"

        handler = self.handlers.get(intent, self._handle_unknown)
        return f"[Intent: {intent}, Confidence: {confidence:.0%}]\n{handler(message)}"

    def _handle_status(self, msg: str) -> str:
        return "Cluster: healthy | Nodes: 3/3 | Pods: 42/42 running"

    def _handle_resources(self, msg: str) -> str:
        return "CPU: 45% | Memory: 62% | Disk: 38%"

    def _handle_deploy(self, msg: str) -> str:
        return "Ready to deploy. Which service and version?"

    def _handle_restart(self, msg: str) -> str:
        return "Which pod should I restart? (e.g., payment-svc-7d8b)"

    def _handle_unknown(self, msg: str) -> str:
        return "I don't have a handler for that intent."

bot = IntentChatbot()
for msg in ["How's the cluster doing?", "What's the memory usage?", "Restart api-gateway"]:
    print(f"User: {msg}")
    print(f"Bot:  {bot.respond(msg)}\n")

Better: Handles paraphrasing because the classifier learned the patterns. But still limited to predefined intents.

Generation 3: LLM-Powered (Modern)

# The modern approach: just describe what you want

system_prompt = """You are a DevOps assistant for a Kubernetes cluster.
You have access to these tools:
- get_cluster_status() → Returns node and pod health
- get_resource_usage() → Returns CPU, memory, disk metrics
- deploy_service(name, version) → Triggers deployment
- restart_pod(pod_name) → Restarts a specific pod

Answer questions naturally using the available tools.
If the user asks something outside your scope, say so politely."""

# With an LLM, the user can say ANYTHING:
# "Hey, my colleague mentioned the cluster was acting up. Can you check?"
# "What pods are using the most memory and should I restart any?"
# "Schedule a canary deployment of payment-service v3.2 to staging first"

# The LLM understands intent, extracts entities, reasons about the request,
# and generates a natural response — no predefined patterns needed.

The leap: LLMs handle arbitrary phrasing, multi-step reasoning, and natural conversation without any intent classification training data.

---

NLP in Practice: A Log Analysis Pipeline

Here's a practical example combining multiple NLP techniques — similar to what I built for my home lab monitoring:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from collections import Counter
import re

# Simulated Kubernetes logs
logs = [
    "2024-03-15 14:30:01 ERROR payment-svc-7d8b OOMKilled container exceeded memory limit 512Mi",
    "2024-03-15 14:30:02 ERROR payment-svc-9f2a OOMKilled container exceeded memory limit 512Mi",
    "2024-03-15 14:31:00 WARN  api-gateway-abc1 high latency 500ms on /api/checkout",
    "2024-03-15 14:31:01 WARN  api-gateway-def2 high latency 480ms on /api/checkout",
    "2024-03-15 14:31:02 WARN  api-gateway-ghi3 high latency 520ms on /api/checkout",
    "2024-03-15 14:32:00 ERROR db-replica-001 replication lag 45s exceeding threshold 30s",
    "2024-03-15 14:32:30 INFO  scheduler scaled payment-svc to 5 replicas",
    "2024-03-15 14:33:00 INFO  certificate-manager renewed cert for api.example.com",
    "2024-03-15 14:33:30 ERROR payment-svc-x1y2 OOMKilled container exceeded memory limit 512Mi",
    "2024-03-15 14:34:00 WARN  api-gateway-jkl4 high latency 510ms on /api/orders",
]

# Step 1: Extract severity
def extract_severity(log: str) -> str:
    for level in ["ERROR", "WARN", "INFO", "DEBUG"]:
        if level in log:
            return level
    return "UNKNOWN"

# Step 2: Clean logs for clustering
def clean_log(log: str) -> str:
    """Remove timestamps, pod hashes, and specific values for clustering."""
    log = re.sub(r'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}', '', log)
    log = re.sub(r'[\w]+-[\w]{4}', 'POD', log)  # Normalize pod names
    log = re.sub(r'\d+(?:ms|Mi|s)', 'VALUE', log)  # Normalize metrics
    return log.strip()

cleaned = [clean_log(log) for log in logs]

# Step 3: Cluster similar log messages
vectorizer = TfidfVectorizer(stop_words="english")
X = vectorizer.fit_transform(cleaned)

kmeans = KMeans(n_clusters=4, random_state=42, n_init=10)
clusters = kmeans.fit_predict(X)

# Step 4: Summarize findings
print("Log Analysis Report")
print("=" * 60)

# Severity distribution
severities = [extract_severity(log) for log in logs]
severity_counts = Counter(severities)
print(f"\nSeverity Distribution:")
for level, count in severity_counts.most_common():
    print(f"  {level}: {count}")

# Cluster summary
print(f"\nIdentified {len(set(clusters))} distinct issue patterns:")
for cluster_id in sorted(set(clusters)):
    cluster_logs = [logs[i] for i in range(len(logs)) if clusters[i] == cluster_id]
    print(f"\n  Pattern {cluster_id + 1} ({len(cluster_logs)} occurrences):")
    print(f"    Example: {cluster_logs[0][:80]}...")

Output:

Log Analysis Report
============================================================

Severity Distribution:
  ERROR: 4
  WARN: 4
  INFO: 2

Identified 4 distinct issue patterns:

  Pattern 1 (3 occurrences):
    Example: 2024-03-15 14:30:01 ERROR payment-svc-7d8b OOMKilled container exceeded...

  Pattern 2 (4 occurrences):
    Example: 2024-03-15 14:31:00 WARN  api-gateway-abc1 high latency 500ms on /api/ch...

  Pattern 3 (2 occurrences):
    Example: 2024-03-15 14:32:30 INFO  scheduler scaled payment-svc to 5 replicas...

  Pattern 4 (1 occurrences):
    Example: 2024-03-15 14:32:00 ERROR db-replica-001 replication lag 45s exceeding th...

This pipeline: tokenizes → cleans → vectorizes → clusters → summarizes. Each step is a core NLP operation.

---

Key NLP Concepts Summary

Concept	What It Does	Example
Tokenization	Splits text into tokens	"Hello world" → ["Hello", "world"]
Stemming	Reduces words to root form	"running" → "run"
Lemmatization	Reduces to dictionary form	"better" → "good"
Stopword Removal	Removes common words	"The cat is on the mat" → "cat mat"
TF-IDF	Weights words by importance	Rare words get higher scores
NER	Finds named entities	"Deploy to US-East" → {region: "US-East"}
Sentiment Analysis	Detects positive/negative	"System crashed" → Negative
Text Classification	Assigns categories	"OOMKilled" → Category: Memory
Embeddings	Dense vector representations	"king" → [0.2, -0.1, 0.8, ...]

---

The Shift from Classical NLP to LLMs

Classical NLP (everything we've covered above) required:

• Feature engineering (TF-IDF, n-grams, hand-crafted rules)

• Task-specific models (one model for sentiment, another for NER, another for classification)

• Labeled training data for each task

Modern LLM-based NLP requires:

• A good prompt

• That's it.

# Classical NLP: build a pipeline for each task
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB

# Task 1: Sentiment → train a model
# Task 2: NER → train a different model  
# Task 3: Classification → train another model
# Each needs labeled data, feature engineering, evaluation

# Modern NLP: one model, prompt-driven
prompt = """Analyze this log message and return:
1. Sentiment (positive/negative/neutral)
2. Named entities (pods, services, namespaces)
3. Category (compute/network/storage/deployment)
4. Severity (critical/warning/info)
5. Suggested action

Log: "Pod payment-svc-7d8b in namespace production OOMKilled, memory at 98%"
"""
# One API call handles all five tasks

But classical NLP isn't dead. When you need:

• Fast inference (microseconds, not seconds)

• Low cost (no API calls)

• Deterministic behavior (same input → same output)

• To process millions of documents

...classical NLP with scikit-learn is still the right tool. I use TF-IDF + Naive Bayes for my initial log classification (fast, cheap, handles volume) and then route only the interesting cases to an LLM for deeper analysis.

---

What's Next

Now that you understand how machines process human language, we'll explore the technology that revolutionized NLP: Large Language Models and Generative AI — how transformers work, what makes models "large," and why generative AI is both powerful and limited.

---

Next: Part 4 — Large Language Models and Generative AI

---

← Part 2: ML, DL, and Foundation Models · Series Overview · Next →