Fine-Tuning LLMs: Complete Guide to Instruction Tuning and LoRA

Last updated: March 2026

Fine-tuning is not the first thing you should try when an LLM underperforms on your task — it is the last thing. Improve your prompt first, add few-shot examples, try RAG if you need domain knowledge. Fine-tuning is powerful but expensive, slow to iterate, and genuinely necessary for a narrower set of problems than developers typically assume. When it is the right tool, though, it produces results that no amount of prompt engineering can match: consistent output format, deep domain adaptation, dramatic latency reductions by distilling large model behavior into smaller models. This guide covers when to fine-tune, how to prepare data correctly, and how to train efficiently with LoRA and QLoRA.

When Fine-Tuning is the Right Choice

Before starting a fine-tuning project, work through this decision table honestly:

The most common waste in fine-tuning projects: spending two weeks preparing a dataset and training runs to achieve what a better system prompt would have done in an afternoon.

Types of Fine-Tuning

Full Fine-Tuning

Update all model parameters on your training data. This has the highest performance ceiling but requires enormous compute — proportionally similar to the original pre-training. Impractical for models above 1B parameters without a dedicated GPU cluster.

Supervised Fine-Tuning (SFT / Instruction Tuning)

Fine-tune on instruction-response pairs. This is how GPT-3 became InstructGPT — showing the model examples of good instructions and responses makes it dramatically better at following instructions. Most fine-tuning projects use SFT.

RLHF (Reinforcement Learning from Human Feedback)

Fine-tune using human preference comparisons. Humans rate two model responses, the model learns to prefer what humans prefer. This is how ChatGPT, Claude, and other aligned models are created. Very complex and expensive — not applicable to most developer projects.

LoRA (Low-Rank Adaptation)

Adds small trainable adapter matrices to frozen model layers. Reduces trainable parameters by 100–1000x compared to full fine-tuning. Near-equivalent quality at a tiny fraction of the compute cost. This is the standard approach for most fine-tuning projects today.

QLoRA (Quantized LoRA)

Quantizes the base model to 4-bit format, then applies LoRA adapters. Enables fine-tuning a 70B model on a single 48GB GPU. The practical default for developer-accessible hardware. See QLoRA explained for the technique in depth.

Dataset Preparation

Dataset quality is the most important factor in fine-tuning quality. 500 carefully curated examples consistently outperforms 10,000 scraped examples with noisy labels.

The Instruction-Following Format

Modern LLMs use a chat template format with system, user, and assistant turns:

# Standard example format — adjust to match your base model's template
{
    "messages": [
        {"role": "system",    "content": "You are a SQL expert who writes clean, commented SQL."},
        {"role": "user",      "content": "Get all customers who made a purchase in the last 30 days"},
        {"role": "assistant", "content": "SELECT DISTINCT c.customer_id, c.name, c.email\nFROM customers c\nINNER JOIN orders o ON c.customer_id = o.customer_id\nWHERE o.created_at >= CURRENT_DATE - INTERVAL '30 days'\nORDER BY c.name;"}
    ]
}

Build and Save a Training Dataset

import json

dataset = [
    {
        "messages": [
            {"role": "system",    "content": "You are a SQL expert."},
            {"role": "user",      "content": "Find all users where age > 30"},
            {"role": "assistant", "content": "SELECT * FROM users WHERE age > 30;"}
        ]
    },
    {
        "messages": [
            {"role": "system",    "content": "You are a SQL expert."},
            {"role": "user",      "content": "Count orders by status"},
            {"role": "assistant", "content": "SELECT status, COUNT(*) AS order_count\nFROM orders\nGROUP BY status\nORDER BY order_count DESC;"}
        ]
    },
    # ... more examples
]

# Save as JSONL (one JSON object per line)
with open("train.jsonl", "w") as f:
    for item in dataset:
        f.write(json.dumps(item) + "\n")

# Create a small validation split (10–20% of data)
split = int(len(dataset) * 0.8)
with open("val.jsonl", "w") as f:
    for item in dataset[split:]:
        f.write(json.dumps(item) + "\n")

print(f"Training: {split} examples, Validation: {len(dataset) - split} examples")

Data quality checklist:

• Every response is correct (have a domain expert review a sample)
• Responses have consistent format and tone
• Coverage is diverse — include edge cases and difficult examples
• Length is appropriate — don't include artificially short or long responses
• System prompt matches what you will use at inference

Fine-Tuning with LoRA and QLoRA

Install Dependencies

pip install transformers trl peft bitsandbytes datasets accelerate

Load Base Model with 4-bit Quantization (QLoRA)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_name = "meta-llama/Llama-3.2-3B-Instruct"

# 4-bit quantization configuration for QLoRA
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"  # important for causal LM training

Configure LoRA

from peft import LoraConfig, prepare_model_for_kbit_training

# Prepare quantized model for training
model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,              # rank — higher = more capacity, more trainable params
    lora_alpha=32,     # scaling factor (rule of thumb: 2×r)
    target_modules=[   # apply LoRA to these weight matrices
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

Rank selection guide:

• r=8 — simple task adaptation, minimal memory overhead
• r=16 — good default for most instruction-following tasks
• r=64 — complex tasks, large vocabulary domains, 70B+ models

Training with SFTTrainer

from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load datasets
train_dataset = load_dataset("json", data_files={"train": "train.jsonl"})["train"]
eval_dataset  = load_dataset("json", data_files={"train": "val.jsonl"})["train"]

training_args = SFTConfig(
    output_dir="./fine-tuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,    # effective batch size = 8
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    save_steps=100,
    eval_strategy="steps",
    eval_steps=50,                    # evaluate on val set every 50 steps
    bf16=True,
    max_seq_length=2048,
    optim="paged_adamw_32bit",        # paged optimizer for QLoRA
)

trainer = SFTTrainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    peft_config=lora_config,
    args=training_args,
)

trainer.train()
trainer.save_model("./fine-tuned-model")
print("Training complete. Adapter saved.")

Merging LoRA Weights for Deployment

The LoRA adapter is a small file (50–200MB) that modifies the base model's behavior when loaded on top of it. For production inference, merge the adapter into the base model for slightly faster inference and simpler deployment.

from peft import PeftModel
from transformers import AutoModelForCausalLM
import torch

# Load base model in full precision for merging
base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Load and merge LoRA adapter
peft_model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("./merged-model")

tokenizer = AutoTokenizer.from_pretrained("./fine-tuned-model")
tokenizer.save_pretrained("./merged-model")
print("Merged model saved.")

Keep the adapter files. Merging is not reversible unless you have both the base model and adapter saved.

Running Inference

from transformers import pipeline

# Load merged model
pipe = pipeline(
    "text-generation",
    model="./merged-model",
    tokenizer=tokenizer,
    device_map="auto",
)

# Test a few examples from your use case
test_messages = [
    {"role": "system", "content": "You are a SQL expert."},
    {"role": "user",   "content": "Get the top 10 customers by total order value"},
]

output = pipe(
    test_messages,
    max_new_tokens=256,
    do_sample=False,  # greedy decoding for deterministic SQL
    temperature=None,
    top_p=None,
)
print(output[0]["generated_text"][-1]["content"])

Evaluation

Watch Validation Loss During Training

If validation loss starts increasing while training loss continues decreasing, you are overfitting. Stop training and use the checkpoint from before validation loss started rising.

Step  50: train_loss=1.82 eval_loss=1.85  ← good
Step 100: train_loss=1.54 eval_loss=1.58  ← good
Step 150: train_loss=1.31 eval_loss=1.55  ← overfitting starting
Step 200: train_loss=1.10 eval_loss=1.72  ← overfitting confirmed, use step 100 checkpoint

Manual Evaluation Set

Create 50–100 test examples that were never seen during training. For each, manually evaluate:

• Is the answer correct?
• Is the format what you specified?
• Are there any hallucinations or errors?

Track these as a percentage — your fine-tuned model should score significantly higher than the base model on your specific task.

Common Mistakes

Not applying the chat template correctly — Each base model has a specific chat template that adds special tokens around messages. Applying the wrong template (or none at all) causes the model to see malformed training data and produce garbage output at inference.

Training the model to reproduce the prompt — The loss should be computed only on the assistant's response tokens, not on the system and user turns. SFTTrainer handles this automatically with the messages format; verify that packing=False and the dataset format is correct.

Catastrophic forgetting — Fine-tuning on a narrow task can cause the model to forget general capabilities. Mitigation: use fewer epochs (1–2), use a lower rank (r=8 or r=16), and optionally mix in 5–10% general instruction data.

Too high a learning rate — QLoRA is more sensitive to learning rate than standard LoRA. If training loss spikes or oscillates, reduce from 2e-4 to 1e-4 or 5e-5.

Skipping gradient checkpointing — gradient_checkpointing=True reduces peak GPU memory usage by recomputing activations during the backward pass. Enable it when training large models with limited memory, at the cost of ~20% slower training.

Merging before validating — Run your evaluation on the LoRA adapter before merging. Once merged, reverting requires keeping both checkpoints.

Key Takeaways

• Fine-tuning is the last resort, not the first — exhaust prompt engineering and RAG before starting a fine-tuning project, as most problems are solvable without modifying model weights.
• LoRA reduces trainable parameters by 100–1000x vs full fine-tuning, with near-equivalent quality for task adaptation tasks like style, format, and domain terminology.
• QLoRA adds 4-bit base model quantization on top of LoRA, enabling a 7B model to fit in 10GB VRAM and a 13B model in 16GB — accessible on consumer hardware.
• Dataset quality is the primary driver of fine-tuning success: 200 carefully reviewed examples outperform 5,000 scraped examples with noisy labels consistently.
• Validation loss is the key training signal — if validation loss rises while training loss falls, you are overfitting and must use an earlier checkpoint.
• Always apply the base model's correct chat template using tokenizer.apply_chat_template() — using a wrong template or a custom format produces malformed training data and degraded outputs.
• Catastrophic forgetting is a real risk with full fine-tuning: the model can lose general capabilities when trained on narrow data with many epochs; LoRA is largely immune because base weights are frozen.
• Before deploying a fine-tuned model, merge the LoRA adapter into the base model for faster inference — but always preserve both the base model identifier and the adapter checkpoint first.

FAQ

How much data do I need to fine-tune an LLM?

For instruction-following tasks, 100–500 high-quality examples is a realistic minimum and often sufficient. Quality matters far more than quantity: 200 carefully curated and reviewed examples consistently outperforms 5,000 scraped examples with noisy labels. If you have fewer than 50 examples, use few-shot prompting instead — fine-tuning on this little data reliably produces poor results.

Can I fine-tune GPT-4o or Claude?

OpenAI offers fine-tuning for GPT-4o-mini and GPT-3.5-turbo via their API, but not GPT-4o as of early 2026. Anthropic does not offer fine-tuning for Claude through a public API. For frontier model customization, OpenAI's fine-tuning API is the most practical route; for full control, fine-tune an open-source model like Llama 3 or Mistral on your own infrastructure.

How long does fine-tuning take on a single GPU?

A QLoRA run on a 7B model with 500 examples takes roughly 15–30 minutes on an A100 (80GB). A 13B model with the same data takes about 45–60 minutes. Training time scales with dataset size, number of epochs, and model size — not linearly, but predictably. Cloud GPU rentals (Lambda, RunPod, Modal) typically cost $1–3 per training run for a small dataset.

How do I know if fine-tuning actually improved my model?

Track validation loss during training to detect overfitting, then evaluate on a held-out test set of 50–100 examples that were never seen during training. Score the base model and fine-tuned model on the same examples side by side and measure the delta on your task-specific metric (format compliance, factual accuracy, output length). Without a before/after comparison on a fixed test set, you cannot reliably assess whether fine-tuning helped.

What is the right learning rate for LoRA vs full fine-tuning?

LoRA adapters train from random initialization and tolerate higher learning rates, typically 2e-4. Full fine-tuning updates pre-trained weights directly and requires much lower rates — typically 5e-6 to 1e-5. Using a LoRA learning rate with full fine-tuning causes rapid overwriting of pre-trained knowledge. Using a full fine-tuning learning rate with LoRA results in severe underfitting.

When should I increase LoRA rank?

Start with r=16 for most tasks. If evaluation shows the model is clearly underfitting — validation loss plateaus high, generated outputs are low quality — increase to r=32 or r=64. For simple format or style adaptation, r=8 is often sufficient. For major domain shifts requiring new vocabulary and reasoning patterns, r=64 or higher provides more adapter capacity.

Is it possible to continuously update a fine-tuned model?

Yes, but with care. You can continue fine-tuning from an existing LoRA adapter checkpoint on new data. For knowledge that changes frequently, this is preferable to retraining from scratch each time. Monitor for catastrophic forgetting — adding new data can degrade performance on the original task if the new data is very different in distribution.

What to Learn Next

• QLoRA Explained: Efficient Fine-Tuning on Consumer Hardware
• LoRA Fine-Tuning Tutorial: Train Custom LLMs on a Single GPU
• Full Fine-Tuning vs LoRA: When to Use Each
• How to Build Fine-Tuning Datasets for LLMs
• Fine-Tuning LLMs with HuggingFace Transformers