Closing the Loop with Langfuse, RAG and Expert Feedback

Most private LLM programs instrument the model the day it ships and stop there. Traces land in a dashboard, a few engineers glance at latency, and the system never actually gets better. The retriever keeps returning the same stale chunks, the model keeps hedging on the same questions, and the subject-matter experts who could fix both are nowhere in the loop.

Closing that loop on an air-gapped stack is a solved problem, but it requires wiring. The pieces are Langfuse for traces and scores, a RAG pipeline you already own, a human review workflow for the experts, and a training runner that consumes the output as supervised or preference data. None of it has to call a hosted evaluator. All of it can run inside your VPC or on a rack in your building.

What follows is the concrete architecture, named components and all.

Why Observability Alone Does Not Improve a Model

Observability tells you what happened. It does not tell you what to do about it. Langfuse stores traces, observations and scores in ClickHouse, with Postgres for transactional config and S3 for raw ingestion events and multi-modal attachments. That gives you a queryable record of every retrieval call, every tool invocation and every generation. It does not, by itself, change the next answer.

The missing primitive is the score. In Langfuse, a score is a number attached to a trace, from either a human review or an automated evaluator. Scores are the hinge between passive logging and active improvement. A trace with a score of 0.2 on answer_faithfulness and a reviewer comment of "cites the wrong statute" is now a training signal, not a log line. A trace with no score is noise.

The second missing primitive is the reviewer. A general-purpose annotator cannot tell you whether a radiology summary missed a finding, or whether a loan memo misread a covenant. Experts can. The job of the architecture below is to get the right trace in front of the right expert, capture the correction, and feed it back into both the retriever and the model without any of it leaving your network.

The Reference Stack, Component by Component

Assume an on-prem or VPC deployment with the following services. Everything is open-source or self-hostable, and nothing phones home.

• Inference: vLLM or TGI serving an open-weight model (Llama 3.1 70B, Qwen2.5 72B, Mistral Large) on your own GPUs. Sizing guidance for this layer lives in a GPU requirements breakdown on this site.
• Retrieval: PostgreSQL with pgvector, or a dedicated vector store, fronting your document corpus. The secure RAG pipeline pattern covers the ingestion and auth model.
• Observability: self-hosted Langfuse. Self-hosted Langfuse is MIT-licensed and free to deploy on your own infrastructure using Docker or Kubernetes. The runtime is two application containers (web, worker) plus Postgres, ClickHouse, Redis and S3-compatible blob storage.
• Expert review: the Langfuse annotation queue, scoped per project, with reviewer accounts for named SMEs.
• Training: a separate GPU host or partition running Hugging Face TRL (SFTTrainer, DPOTrainer) against datasets exported from Langfuse.
• Model registry: an internal object store or MLflow instance holding checkpoints, with signed metadata tying each checkpoint to the Langfuse dataset version that produced it.

The ingestion path is asynchronous by design. Traces are received in batches by the Langfuse Web container and immediately written to S3, with a reference persisted in Redis for queueing; the Worker then pulls them into ClickHouse. For a regulated deployment this matters, because the raw payloads sit in storage you control under a retention policy you set.

Instrumenting the RAG Pipeline So Scores Have Something to Grip

Scores are only useful if they attach to the right span. A RAG trace with one giant generation observation and nothing else cannot tell you whether the model hallucinated or the retriever handed it garbage. Instrument the pipeline so each stage is its own observation under one trace:

• retrieve span: query, rewriter output, top-k document IDs, similarity scores, filters applied.
• rerank span (if present): input candidates, output order, model name.
• generate observation: full prompt, model name, token counts, latency, raw completion.
• tool_call spans for any function invocation, with arguments and return values.

Langfuse models this natively. Observations form a hierarchical, parent-child tree structure where each observation can have nested child observations, which lets a reviewer open one trace and see the retrieval, the rerank and the generation side by side. That is the view the SME needs. Without it, a score of "bad" is unactionable because there is no way to tell which stage failed.

Attach automated scores at ingestion time where cheap heuristics apply: retrieval hit rate against a known-answer set, citation presence, token-count anomalies, refusal detection. Reserve human review for the traces those heuristics flag as ambiguous or low-confidence. This is the same economics as code review: cheap linters first, expert eyes on the rest.

Routing Traces to the Right Expert

The Langfuse annotation queue is the piece most teams underuse. For high-stakes use cases, traces can be routed to human reviewers through the Langfuse annotation queue, where reviewers see the input, output and full trace context before assigning scores. In a regulated enterprise, "the right reviewer" is not a pool of contractors. It is a named attorney, a named clinician, a named compliance officer, each with authority to make the call for their domain.

Build the routing around tags. Tag each trace at ingestion with the originating workflow (contract-review, claims-triage, sox-controls), the risk tier, and the business unit. Create one annotation queue per domain, filtered on those tags. Reviewers see only what belongs to them, which keeps throughput honest and avoids the compliance headache of a cardiologist reading finance prompts.

Define a score schema per domain before anyone annotates. A thin schema is worse than useless because it produces thumbs-up data that no training job can learn from. A workable minimum, per trace:

• faithfulness (0 to 1): did the answer stay within the retrieved context?
• completeness (0 to 1): did it cover what the question required?
• safety (categorical): clean, borderline, refuse.
• preferred_rewrite (free text): the reviewer's version of what the answer should have been.

That last field is what makes the whole loop work. It is the raw material for SFT, and the "better" half of a DPO pair.

1

faithfulness

0 to 1 · Retriever + generation tuning

1

completeness

0 to 1 · SFT target selection

3

safety

3 categories · Refusal policy, DPO rejected pool

0

preferred_rewrite

free text · SFT target + DPO chosen

From Scored Traces to Training Data

Langfuse is open-data by design. For large-scale exports such as fine-tuning datasets, blob storage export can automatically sync data to S3, GCS or Azure on a schedule instead of paginating through the API. On an air-gapped stack this points at a MinIO bucket on the same subnet, and the training runner reads directly from it.

Build two datasets, nightly, from the same scored traces:

• SFT set: prompts whose preferred_rewrite is populated, paired with that rewrite as the target. These are the gold corrections and should be rare, high-signal, and treated as the ground truth for the next checkpoint.
• DPO set: for each prompt with both an original generation and a preferred_rewrite, construct a (prompt, chosen, rejected) triple. TRL's DPOTrainer accepts explicit-prompt preference examples with prompt, chosen and rejected fields, which maps one-to-one to what the reviewer produced.

Not every low-scored trace belongs in the DPO set. Filter on inter-rater agreement where you have more than one reviewer, drop traces where the retriever was the real failure (those feed the retriever job, not the model job), and cap the ratio of synthetic rewrites to real production traffic. A DPO run poisoned by one reviewer's idiosyncratic prose will degrade the model in ways the Langfuse dashboard will not catch until the next week's scores dip.

The retriever has its own feedback path. Low faithfulness scores correlated with low retrieval similarity are a chunking or embedding problem, not a generation problem. Feed those cases into a reranker fine-tune or an embedding adaptation job. More detail on where to spend training effort lives in the RAG vs fine-tuning decision guide.

Keeping the Loop Auditable

A self-improving system that cannot be audited is a liability under HIPAA, GLBA, SOC 2 and the EU AI Act. The architecture above produces a clean audit trail if you treat the artifacts as records:

• Every training run is pinned to a Langfuse dataset version (hash the export).
• Every reviewer action is attributable to a named user; Langfuse audit logs can be exported as CSV or JSON from the UI alongside datasets.
• Every deployed checkpoint carries metadata linking back to the dataset hash, the reviewer set, and the evaluation scores on a frozen holdout.
• Rollback is a model-registry operation, not a code change.

None of this requires a hosted evaluator, and that is the point. The same reasons you self-host the model, documented in the hidden risks of public AI APIs, apply doubly to the evaluator: the evaluator sees both the prompt and the reviewer's correction, which is often more sensitive than the prompt itself. Shipping that to a third party undoes the compliance posture you paid for.

The loop is not glamorous. It is a nightly export, a reviewer queue, a TRL job and a model-registry promotion. Done consistently, it moves a private LLM from a static artifact to a system that quietly gets better at the specific work your organization does, on infrastructure you own, with evidence a committee can read. That is the version of "self-improving" worth deploying in a regulated environment, and the version that does not require you to trust a vendor's dashboard with your experts' judgment.