Part 4: Advanced Features and Techniques

Part of the Hugging Face Transformers 101 Series

Beyond Basic Fine-tuning

After fine-tuning dozens of models, I hit performance and resource challenges:

• Training large models required expensive GPUs

• Inference was too slow for real-time applications

• Fine-tuning updated millions of parameters (time + cost)

• Memory constraints limited model sizes

Advanced techniques solved these problems:

• PEFT/LoRA: Fine-tune with < 1% of parameters

• Quantization: Reduce model size by 75%

• Text Generation Strategies: Control output quality

• Multi-modal Models: Process text + images

Let me share what I learned applying these in production.

Parameter-Efficient Fine-Tuning (PEFT)

Traditional fine-tuning: Update all model parameters (millions to billions) PEFT: Update small subset of parameters, freeze rest

Why Use PEFT?

My experience:

• Less memory: 4x smaller GPU requirements

• Faster training: 2-3x speedup

• Better generalization: Less prone to overfitting

• Easier deployment: Can serve multiple adapters on one base model

LoRA (Low-Rank Adaptation)

LoRA is the most popular PEFT method. Instead of updating full weight matrices, it learns small adapter matrices.

# Install PEFT library
# pip install peft

from transformers import AutoModelForSequenceClassification, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model, TaskType
from datasets import load_dataset

# Load base model
model_name = "facebook/opt-350m"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2
)

# Configure LoRA
lora_config = LoraConfig(
    task_type=TaskType.SEQ_CLS,  # Sequence classification
    r=8,  # Rank of adaptation matrices
    lora_alpha=32,  # Scaling factor
    lora_dropout=0.1,  # Dropout probability
    target_modules=["q_proj", "v_proj"],  # Which layers to adapt
)

# Wrap model with LoRA
model = get_peft_model(model, lora_config)

# Show trainable parameters
model.print_trainable_parameters()
# Output: trainable params: 294,912 || all params: 332,769,280 || trainable%: 0.09%

0.09% of parameters! Massive savings.

# Train with Trainer API (same as before)
training_args = TrainingArguments(
    output_dir="./lora-model",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=1e-4,  # Can use higher LR with LoRA
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

# Save LoRA adapters (very small - few MB)
model.save_pretrained("./lora-adapters")

Loading LoRA Models

from transformers import AutoModelForSequenceClassification
from peft import PeftModel

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    "facebook/opt-350m",
    num_labels=2
)

# Load LoRA adapters
model = PeftModel.from_pretrained(base_model, "./lora-adapters")

# Use for inference
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-350m")
inputs = tokenizer("This is amazing!", return_tensors="pt")
outputs = model(**inputs)
prediction = torch.argmax(outputs.logits, dim=-1)

LoRA for Text Generation

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType

# Load LLM
model = AutoModelForCausalLM.from_pretrained(
    "gpt2",
    device_map="auto"
)

# LoRA config for causal LM
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["c_attn"],  # GPT-2 attention modules
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# Fine-tune on your dataset
# ... (same Trainer workflow)

# Generate text
tokenizer = AutoTokenizer.from_pretrained("gpt2")
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

Other PEFT Methods

Prefix Tuning:

from peft import PrefixTuningConfig, get_peft_model

config = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,
)

model = get_peft_model(model, config)

P-Tuning:

from peft import PromptEncoderConfig, get_peft_model

config = PromptEncoderConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,
    encoder_hidden_size=128,
)

model = get_peft_model(model, config)

I use LoRA 90% of the time - great balance of performance and simplicity.

Model Quantization

Quantization reduces model size by using lower precision (int8 instead of float32).

Benefits

• 4x smaller models: 1GB → 250MB

• Faster inference: Less memory bandwidth

• Same accuracy: Minimal quality loss

int8 Quantization

# Install bitsandbytes
# pip install bitsandbytes

from transformers import AutoModelForCausalLM

# Load in 8-bit
model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-1.3b",
    load_in_8bit=True,
    device_map="auto"
)

# Use normally
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("facebook/opt-1.3b")
inputs = tokenizer("Hello world", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

I run 1.3B parameter models on consumer GPUs with 8-bit quantization.

4-bit Quantization (QLoRA)

Even more aggressive - 1/8 the size:

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",  # NormalFloat4
    bnb_4bit_use_double_quant=True,  # Nested quantization
    bnb_4bit_compute_dtype=torch.bfloat16,  # Compute in bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=quant_config,
    device_map="auto"
)

# Combine with LoRA for efficient fine-tuning (QLoRA)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

# Prepare for training
model = prepare_model_for_kbit_training(model)

# Add LoRA
lora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)

model = get_peft_model(model, lora_config)

# Fine-tune 7B model on consumer GPU!

QLoRA = 4-bit quantization + LoRA. Game-changer for fine-tuning large models.

Dynamic Quantization (Post-training)

import torch
from transformers import AutoModelForSequenceClassification

# Load model
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")

# Dynamic quantization
quantized_model = torch.quantization.quantize_dynamic(
    model,
    {torch.nn.Linear},  # Quantize linear layers
    dtype=torch.qint8
)

# Save quantized model
torch.save(quantized_model.state_dict(), "quantized_model.pth")

# 3-4x smaller, faster inference

Text Generation Strategies

Controlling how models generate text is crucial for quality.

Basic Generation

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")

prompt = "The future of AI is"
inputs = tokenizer(prompt, return_tensors="pt")

# Greedy decoding (default)
outputs = model.generate(
    **inputs,
    max_length=50
)

print(tokenizer.decode(outputs[0]))

Sampling Strategies

Temperature Sampling:

# Lower temperature = more focused, deterministic
# Higher temperature = more random, creative

# Conservative (temperature=0.7)
outputs = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    temperature=0.7,
)

# Creative (temperature=1.5)
outputs = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    temperature=1.5,
)

Top-k Sampling:

# Sample from top k most likely tokens
outputs = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    top_k=50,  # Consider top 50 tokens
    temperature=0.9,
)

Top-p (Nucleus) Sampling:

# Sample from smallest set of tokens with cumulative probability > p
outputs = model.generate(
    **inputs,
    max_length=50,
    do_sample=True,
    top_p=0.95,  # Nucleus sampling
    temperature=0.9,
)

Combined Strategy (Best Results):

# My go-to settings for balanced generation
outputs = model.generate(
    **inputs,
    max_length=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
    repetition_penalty=1.2,  # Penalize repetition
    no_repeat_ngram_size=3,  # Don't repeat 3-grams
    num_return_sequences=1,
)

Beam Search

Better quality but slower:

# Beam search with 5 beams
outputs = model.generate(
    **inputs,
    max_length=50,
    num_beams=5,  # Number of beams
    early_stopping=True,  # Stop when all beams finish
    no_repeat_ngram_size=2,
)

# Returns best sequence
print(tokenizer.decode(outputs[0]))

Constrained Generation

Force specific outputs:

# Force generation to include specific words
from transformers import LogitsProcessor, LogitsProcessorList

class ForceWordLogitsProcessor(LogitsProcessor):
    def __init__(self, force_word_ids):
        self.force_word_ids = force_word_ids
    
    def __call__(self, input_ids, scores):
        # Boost probability of desired words
        for word_id in self.force_word_ids:
            scores[:, word_id] += 10.0
        return scores

# Use it
force_words = tokenizer.encode("artificial intelligence", add_special_tokens=False)
processor = ForceWordLogitsProcessor(force_words)

outputs = model.generate(
    **inputs,
    max_length=50,
    logits_processor=LogitsProcessorList([processor])
)

Streaming Generation

For real-time applications:

from transformers import TextIteratorStreamer
from threading import Thread

streamer = TextIteratorStreamer(tokenizer, skip_special_tokens=True)

# Generate in background thread
generation_kwargs = dict(
    **inputs,
    max_length=100,
    streamer=streamer,
)

thread = Thread(target=model.generate, kwargs=generation_kwargs)
thread.start()

# Stream tokens as they're generated
for text in streamer:
    print(text, end="", flush=True)

thread.join()

I use streaming for chatbot interfaces - much better UX.

Multi-modal Models

Process multiple modalities (text + images, text + audio).

Vision-Language Models (CLIP)

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import requests

# Load model
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

# Load image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"
image = Image.open(requests.get(url, stream=True).raw)

# Define labels
labels = ["a cat", "a dog", "a bird", "a car"]

# Process inputs
inputs = processor(
    text=labels,
    images=image,
    return_tensors="pt",
    padding=True
)

# Get similarities
outputs = model(**inputs)
logits_per_image = outputs.logits_per_image
probs = logits_per_image.softmax(dim=1)

# Show results
for label, prob in zip(labels, probs[0]):
    print(f"{label}: {prob:.4f}")

Output:

a cat: 0.9876
a dog: 0.0084
a bird: 0.0032
a car: 0.0008

Zero-shot image classification! No fine-tuning needed.

Image Captioning

from transformers import VisionEncoderDecoderModel, ViTImageProcessor, AutoTokenizer
from PIL import Image
import requests

# Load model
model = VisionEncoderDecoderModel.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
processor = ViTImageProcessor.from_pretrained("nlpconnect/vit-gpt2-image-captioning")
tokenizer = AutoTokenizer.from_pretrained("nlpconnect/vit-gpt2-image-captioning")

# Load image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"
image = Image.open(requests.get(url, stream=True).raw)

# Generate caption
pixel_values = processor(images=image, return_tensors="pt").pixel_values

generated_ids = model.generate(
    pixel_values,
    max_length=50,
    num_beams=4,
)

caption = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
print("Caption:", caption)
# Output: Caption: two cats laying on a couch

Visual Question Answering

from transformers import ViltProcessor, ViltForQuestionAnswering
from PIL import Image
import requests

# Load model
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")

# Load image
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/cats.png"
image = Image.open(requests.get(url, stream=True).raw)

# Ask question
question = "How many cats are there?"

# Process
inputs = processor(image, question, return_tensors="pt")

# Get answer
outputs = model(**inputs)
logits = outputs.logits
idx = logits.argmax(-1).item()
answer = model.config.id2label[idx]

print(f"Question: {question}")
print(f"Answer: {answer}")

Whisper (Speech Recognition)

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")

# Load audio
audio, sr = librosa.load("audio.wav", sr=16000)

# Process
inputs = processor(audio, sampling_rate=sr, return_tensors="pt")

# Transcribe
generated_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]

print("Transcription:", transcription)

I use Whisper for meeting transcriptions - incredibly accurate.

Custom Model Architectures

Building custom models on top of Transformers.

Custom Classification Head

import torch.nn as nn
from transformers import BertModel

class CustomBertClassifier(nn.Module):
    """BERT with custom classification head."""
    
    def __init__(self, num_labels, dropout=0.3):
        super().__init__()
        self.bert = BertModel.from_pretrained("bert-base-uncased")
        
        # Custom head
        self.dropout = nn.Dropout(dropout)
        self.classifier = nn.Sequential(
            nn.Linear(768, 512),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(512, 256),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(256, num_labels)
        )
    
    def forward(self, input_ids, attention_mask):
        # BERT outputs
        outputs = self.bert(
            input_ids=input_ids,
            attention_mask=attention_mask
        )
        
        # Use pooled output
        pooled = outputs.pooler_output
        
        # Custom classification
        pooled = self.dropout(pooled)
        logits = self.classifier(pooled)
        
        return logits

# Usage
model = CustomBertClassifier(num_labels=3)

Multi-task Learning

class MultiTaskModel(nn.Module):
    """Single model for multiple tasks."""
    
    def __init__(self, num_labels_task1, num_labels_task2):
        super().__init__()
        self.bert = BertModel.from_pretrained("bert-base-uncased")
        
        # Task 1: Classification
        self.classifier = nn.Linear(768, num_labels_task1)
        
        # Task 2: Regression
        self.regressor = nn.Linear(768, 1)
    
    def forward(self, input_ids, attention_mask, task="task1"):
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        pooled = outputs.pooler_output
        
        if task == "task1":
            return self.classifier(pooled)
        else:
            return self.regressor(pooled)

# Train separately or jointly

Model Ensembles

Combine multiple models for better predictions.

from transformers import pipeline

# Load multiple models
model1 = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
model2 = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment")
model3 = pipeline("sentiment-analysis", model="nlptown/bert-base-multilingual-uncased-sentiment")

def ensemble_predict(text, models):
    """Ensemble prediction with voting."""
    predictions = []
    
    for model in models:
        result = model(text)[0]
        predictions.append({
            'label': result['label'],
            'score': result['score']
        })
    
    # Majority voting
    from collections import Counter
    votes = Counter([p['label'] for p in predictions])
    final_label = votes.most_common(1)[0][0]
    
    # Average confidence
    avg_score = sum(p['score'] for p in predictions) / len(predictions)
    
    return {
        'label': final_label,
        'confidence': avg_score,
        'individual_predictions': predictions
    }

# Use ensemble
text = "This product is amazing!"
result = ensemble_predict(text, [model1, model2, model3])
print(result)

Best Practices

From my experience with advanced techniques:

PEFT/LoRA:

• Start with r=8 or r=16, increase if needed

• Target all attention layers for best results

• Use higher learning rates (1e-4 instead of 2e-5)

• Save adapters separately - easy to swap

Quantization:

• 8-bit for most use cases - minimal quality loss

• 4-bit for very large models (7B+)

• Test quantized models thoroughly - edge cases can differ

• Combine with LoRA for efficient fine-tuning (QLoRA)

Text Generation:

• Temperature 0.7-0.8 for balanced outputs

• Use top-p=0.95 + top-k=50 together

• Add repetition_penalty=1.2 to avoid loops

• Beam search for quality, sampling for diversity

Multi-modal:

• CLIP for zero-shot image classification

• Whisper for speech recognition (state-of-the-art)

• Image captioning models for accessibility

What's Next?

You now know advanced techniques for optimizing and extending Transformers. In Part 5, we'll cover production deployment: serving models efficiently, monitoring, and scaling.

Next: Part 5 - Production Deployment and Optimization

---

Previous: Part 3 - Fine-tuning and Training with Trainer

This article is part of the Hugging Face Transformers 101 series. Check out the series overview for more content.