Using Langfuse to Train Your Open-Weight LLM

Most teams reach for an observability tool to see what their LLM is doing wrong. The more interesting use is turning that same trace data into training examples. Every production call your model handles is a labelled or labellable instance of the job you actually want it to do, which is better signal than any public dataset. The gap is tooling: you need a place to score, triage, and export those traces without copying them into someone else's cloud.

Self-hosted Langfuse fills that gap. It captures traces, lets reviewers score them, groups them into datasets, and exports in formats a fine-tuning library can read. Run inside your VPC or air-gapped rack, it never phones home. What follows is a concrete walkthrough of using it to produce SFT and DPO datasets for an open-weight model like Llama 3.1 or Qwen 2.5, with the plumbing kept inside a regulated boundary.

Why Langfuse Earns a Place in the Training Loop

Langfuse is an open-source LLM engineering platform with an MIT-licensed core that runs under Docker, Docker Compose, or Kubernetes. For regulated buyers, the deployment story matters as much as the features. You can stand it up next to your inference stack, point it at an internal PostgreSQL and ClickHouse, and let traces stay on the same side of the firewall as the prompts that produced them.

The licensing picture also matters. In June 2025, Langfuse open-sourced all product features under MIT, including LLM-as-a-Judge evaluations, annotation queues, prompt experiments, and the playground. In January 2026, ClickHouse acquired Langfuse alongside a $400 million Series D led by Dragoneer, with the project staying open-source under the same license. The practical read: the self-hosted path is unlikely to disappear behind a paywall, which is the only reason to build a training pipeline around a vendor's tool in the first place.

The LLM observability for fine-tuning story only works if the three roles stay in one system. Developers emit traces, reviewers score them, trainers export them. If any of those lives in a separate SaaS, your audit log fragments and your compliance committee asks hard questions. Keeping all three inside one self-hosted instance is the point.

Instrumenting Traces With Training in Mind

A trace useful for debugging is not automatically useful for training. Three fields decide whether a row survives into a dataset: the input must be the complete prompt the model saw, including system message and retrieved context; the output must be the raw completion before any post-processing; and the metadata must carry enough context to filter on. Tag each trace with the application route, the model name and version, the retriever version if RAG is involved, and a redaction flag indicating whether PII scrubbing has run.

For a HIPAA-regulated workload, add two more fields: the data classification of the inputs, and the identity of the subsystem that produced them. These let you later exclude any class of trace from the training export without having to re-read every record. Running the Langfuse OTEL collector inside the same Kubernetes namespace as the inference service keeps raw traces on the same network segment as the inference logs.

Prompt and response schemas should be enforced at ingest, not at export. Langfuse datasets accept JSON Schemas for input and expected_output, and malformed items are rejected on creation with the exact validation error. Fix the schema before you have ten thousand bad rows.

Scoring Rubrics That Produce Training Signal

Scores are where observability turns into labels. Langfuse lets you define score configs that are numeric, categorical, or boolean, and attach them to a trace through three paths: user feedback from the application, LLM-as-a-Judge evaluators, and human annotation queues. A workable rubric for a regulated assistant uses four scores:

• Correctness — categorical: correct, partially correct, incorrect. Decides SFT inclusion.
• Groundedness — boolean: did the answer stay within retrieved context. Required for RAG-adjacent tuning.
• Policy — boolean: did the answer violate a defined content rule. A single failure disqualifies the trace.
• Preference — categorical: chosen or rejected when two responses are shown side by side. Feeds DPO.

LLM-as-a-Judge handles volume; humans resolve ambiguity. The published MT-Bench work found that strong judges like GPT-4 reach over 80% agreement with human preferences, roughly the same agreement humans show each other. For a regulated deployment, run the judge against a self-hosted open-weight model so no trace content leaves the boundary. The agreement rate drops with weaker judges, which is why routing contested items into an annotation queue for in-house reviewers remains part of the workflow rather than an afterthought.

Judge	vs. A-1	vs. A-2	vs. A-3	Average	vs. Majority
GPT-4	59.2%	60.8%	59.1%	59.7%	68.6%
Annotator A-1	—	58.1%	54.7%	57.3%	60.3%
Annotator A-2	58.1%	—	55.4%	58.1%	63.3%

From Traces to a Curated Dataset

Datasets in Langfuse are collections of items, each with an input, an optional expected_output, and metadata linking back to the source trace. The curation step is a filter plus a promotion: filter the traces table by score ("correctness = correct AND policy = pass AND groundedness = true"), then add the matching rows to a dataset. A field-mapping step controls how observation data becomes a dataset item, so you can rewrite the system prompt, strip retrieved context, or merge several fields into one before promotion.

For DPO pairs, you need two completions for the same input, one preferred over the other. The cleanest source is a prompt experiment: run a candidate prompt or model variant against production inputs, store both the baseline and candidate completions as linked traces, then score them head-to-head. The preference score becomes the "chosen" and "rejected" fields. For an alternative source, two concurrent models in A/B deployment produce the same pairs organically.

Langfuse supports export in CSV, JSON, or JSONL directly from the UI, with the JSONL file importable into fine-tuning tooling that follows the ChatML schema. For anything more structured than a flat SFT export, use the public API: endpoints under /api/public/traces and /api/public/observations let you pull filtered data into a nightly job that writes Parquet shards to your training bucket.

Fine-Tuning the Open-Weight Model

With a curated dataset, the training run is the smaller problem. QLoRA has become the default for teams that want to specialize an open-weight model on a single-GPU budget. The original paper showed a 65B LLaMA fine-tuned on a single 48GB GPU with full 16-bit task performance preserved. For Llama 3.1 8B, the practical footprint is roughly 10 to 14 GB of VRAM with 4-bit base weights and bfloat16 adapters, which fits on a workstation card.

The standard RLHF pipeline has three stages: supervised fine-tuning, preference sampling with reward model training, and reinforcement learning with PPO. Most private deployments stop at SFT or substitute DPO for the reward-model-plus-PPO stages, since DPO trains directly on preference pairs and avoids the instability of PPO. The Langfuse export maps cleanly onto both: SFT reads the "correct" subset; DPO reads the "chosen/rejected" pairs.

For the training itself, see fine-tuning your open-weights model for the practical steps, and GPU requirements for private LLMs for sizing. The private LLM fine-tuning workflow worth building is boring on purpose: a nightly job pulls the Langfuse export, runs validation, kicks off a QLoRA run on an internal cluster, evaluates against a held-out golden set, and promotes the adapter only if quality improves without regressions on policy scores.

Keeping the Whole Pipeline Inside the Boundary

The reason to build this at all is that your traces are sensitive. A pipeline that leaks them during curation is worse than one that never existed. Four controls hold the boundary:

• Network isolation. Langfuse, PostgreSQL, ClickHouse, the judge model, and the training cluster sit in one VPC with egress rules that drop traffic to any public model endpoint.
• Identity and audit. Reviewer logins federate through your SSO. Every annotation, score, and export is attributable, and the Langfuse database is backed up to object storage you control.
• Redaction at ingest. PII scrubbing runs before the trace is written. Scrubbed fields are stored alongside the original hash so that reversibility, if needed for incident response, requires a separate key.
• Export governance. Dataset exports are gated by a role separate from the one that creates datasets, and every export writes a record to your SIEM.

None of this is exotic. It is the same control surface you would apply to a secure RAG pipeline on PostgreSQL and pgvector, extended to cover annotation and export. For buyers evaluating whether to run this internally or hand it to a partner, our custom LLM deployment services cover the integration work, and the LLM appliance packages the training and inference stack as a shipped rack.

What a Working Loop Looks Like After Ninety Days

By the end of the first quarter, a team that has instrumented traces, defined four score configs, and run two SFT cycles typically sees three things. The judge agreement rate on policy scoring stabilizes once reviewers have corrected the first few hundred disagreements. The dataset size doubles between SFT runs because the production surface area is larger than any synthetic set. And the open-weight model, now specialized on your own traces, starts to beat the general-purpose baseline on the exact tasks your users care about, which is the only benchmark that matters.

Langfuse is not the training framework and does not pretend to be. It is the system of record for what your model did, who judged it, and which judgements became training data. Treated that way, the Langfuse dataset curation path is the shortest route from production traffic to a fine-tuned open-weight model that stays inside your walls.