RAG vs Fine-Tuning vs Long-Context: Choosing the Right Architecture

Most architecture debates about private LLMs collapse into a vendor preference. The retrieval team sells retrieval. The fine-tuning team sells fine-tuning. The model vendor with a 1M+ token window sells long context. In reality, the three patterns answer different questions, cost different amounts, and fail in different ways under audit. A CIO or CISO signing off on a private deployment needs a selection matrix, not a sales deck.

This piece puts RAG, parameter fine-tuning, and long-context prompting side by side on the dimensions a compliance committee actually asks about: latency, GPU memory, cost per query, auditability, and the update cycle. The goal is a defensible choice, including the hybrid choice, that holds up in front of HIPAA, GLBA, CMMC, or SOC 2 reviewers.

The Three Patterns, Stated Plainly

Retrieval-augmented generation keeps the model frozen and injects relevant chunks of your corpus into the prompt at query time. The original RAG paper framed it as a way to let a parametric model consult a non-parametric index. In practice it is a vector database, a retriever, a reranker, and a prompt template wrapped around a base model.

Fine-tuning modifies the weights. Supervised fine-tuning on labeled examples, LoRA/QLoRA adapters, or full-parameter continued pretraining all fall in this bucket. The model learns a behavior or domain rather than consulting documents at inference. OpenAI's supervised fine-tuning guide is a reasonable reference for the pattern even if the deployment target is open-weight.

Long-context prompting is the newest option. You load the relevant material directly into a very large window and skip retrieval entirely. Llama 4 Scout advertises 10M tokens on a single H100, Qwen3 and Gemini-class open weights sit in the 1M range, and Google documents long-context behavior for Vertex-hosted models in its long context guide. The window is a token budget, not a memory. The model forgets between calls.

These are not interchangeable. Which one belongs in your stack depends on how often the underlying knowledge changes, how precisely you need to cite it, and how much GPU you are willing to buy.

Where Each Pattern Actually Breaks

Long context has a quiet problem that marketing material tends to skip. The Lost in the Middle paper from Liu et al. showed that model accuracy on multi-document QA follows a U-shaped curve: high at the start of the window, high at the end, and significantly worse in the middle. Databricks reproduced the effect in production-style RAG. In their long context RAG benchmark, Llama 3.1 405B accuracy began to decrease after 32k tokens and GPT-4-0125-preview after 64k, well short of their advertised limits. A 10M-token window does not mean 10M useful tokens.

RAG breaks differently. The retriever is now part of your attack surface and your audit trail. A document that is not retrieved cannot be cited, and a document that is retrieved out of context can poison the generation. Chunking strategy, embedding model drift, and reranker quality all become operational concerns. The upside is that every answer is traceable to a specific chunk ID, which is exactly what compliance reviewers want to see.

Fine-tuning breaks on the update cycle. Weights encode a snapshot. When the policy manual, the drug formulary, or the trading rulebook changes, a fine-tuned model is wrong until you retrain. Retraining a 70B open-weight model with LoRA takes hours on a modest GPU cluster; a full continued pretraining run takes days and costs real money. For material that changes weekly, that cadence is the wrong tool.

The Compliance Committee View

Auditability is where the three patterns separate most clearly. RAG produces a per-answer provenance record: query, retrieved chunk IDs, reranker scores, final prompt, model output. That record maps directly to SOC 2 CC7 logging requirements and to the evidence a HIPAA auditor expects for PHI access. We cover the specifics in our write-up on HIPAA private LLM design.

Fine-tuning records live upstream. The audit artifact is the training set, the training run log, and the evaluation suite that was green before promotion. If a reviewer asks why the model said what it said, the answer is "it learned to," which is true but unsatisfying under GLBA or an EU AI Act conformity assessment. Pair fine-tuning with a retrieval layer whenever a specific citation is a regulatory requirement rather than a nice-to-have.

Long context sits in an awkward middle. The prompt itself is the provenance, which is auditable in principle but awkward to store at 1M+ tokens per request. KV-cache memory also grows roughly linearly with context length, which drives GPU budget decisions you may not have planned for. Our private GPU cluster cost breakdown walks through the arithmetic for a 200-user deployment.

Latency, Memory, and Cost Per Query

Rough operating envelopes, for a 70B-class open-weight model on H100-class hardware:

• RAG: 200ms to 2s added for retrieval and reranking, 4k to 16k tokens of context, predictable VRAM footprint, cheapest per query at scale.
• Fine-tuned base: No retrieval latency, same context budget as the base model, lowest cost per query, highest cost per update.
• Long context: Latency scales with the prefill. A 200k-token prompt on current open weights is seconds of time-to-first-token. KV cache can dominate VRAM at 1M+ tokens, which is why serving frameworks like vLLM often cap effective length well below the model card maximum.

The practical shape most regulated deployments land on is hybrid. Fine-tune a base model on tone, format, and domain vocabulary. Keep the knowledge in a vector store so it can be updated without a retraining cycle. Reserve long context for the specific workflows that genuinely need to see an entire contract or clinical record in one pass. Our note on hybrid LLM patterns for law firms is a concrete worked example.

A Selection Matrix You Can Defend

Walk the decision in this order. It is boring on purpose.

1. How often does the ground truth change? Daily or weekly points to RAG. Quarterly or slower makes fine-tuning viable.
2. Does every answer need a citation? If yes for regulatory reasons, retrieval is in the stack regardless of what else sits around it.
3. How large is a typical working set? If a single query realistically needs 300k+ tokens of context in one shot (long contracts, full case files, entire codebases), long context earns its place. Otherwise it is overkill.
4. What is the behavior gap? If the base model handles the knowledge but writes in the wrong register, cites the wrong statute, or refuses tasks it should accept, that is a fine-tuning problem, not a retrieval problem. The fine-tuning open weights write-up covers the specifics.
5. What is the GPU budget? Long context is the most VRAM-hungry pattern per request. Fine-tuning is the most VRAM-hungry per update. RAG distributes the cost across cheaper retrieval infrastructure.

Running this list in front of a committee produces a defensible answer and, more often than not, a hybrid. That is the right outcome. A private deployment that does one thing well and three things badly is harder to govern than one that uses each pattern for what it was designed for. For teams building the broader program around the model, our custom LLM deployment services page describes what the full stack looks like when it is installed on infrastructure you control.

What To Write Down Before You Build

Before any procurement, three documents should exist. A data classification map that says which corpora can be indexed, which can be used for training, and which cannot leave their system of record. An update cadence table that lists each knowledge source and how often it changes. A latency and cost budget per workflow, denominated in milliseconds and dollars per query, not in adjectives.

With those three in hand, the RAG vs fine-tuning vs long-context question answers itself for most workflows. Where it does not, the honest answer is to prototype two patterns against the same evaluation set and let the numbers decide. The architecture that survives contact with your own data and your own auditors is the one worth deploying.