Open Weights Training Options for Custom On-Prem LLMs

Most teams come to private LLM work expecting a single answer: fine-tune the model, deploy it, done. The actual menu is wider. Inside your own perimeter you can continue pretraining, run supervised fine-tuning, bolt on LoRA or QLoRA adapters, align with DPO or ORPO, distill a smaller model from a larger one, or skip training entirely and merge two existing checkpoints. Each has a different cost curve, a different hardware footprint, and a different audit profile.

The point of this piece is practical. If your data cannot leave the building — because it is PHI, privileged, CUI, or simply too sensitive for a hosted API — these are the techniques you can actually run on your own GPUs, and the trade-offs that decide which one you pick.

Continued Pretraining Buys Vocabulary, Not Behavior

Continued pretraining (sometimes called domain-adaptive pretraining) takes an open-weights base model and keeps training it on raw domain text: contracts, radiology reports, maintenance logs, SCADA alerts. It is the only step on this list that teaches the model new tokens and new jargon rather than new behavior.

It is also the most expensive step by orders of magnitude. For reference, the DeepSeek-V3 base model is a 671-billion-parameter MoE that activates 37 billion parameters per token and required 2.788 million H800 GPU hours, roughly $5.58 million in compute. A domain continued-pretrain on a 7B–70B dense model is cheaper, but still measured in days of multi-node H100 or H200 time. The Chinchilla-adjacent ratio of about 20 tokens per parameter is a reasonable floor for how much domain corpus you need before the loss curve means anything.

Pick continued pretraining when the model literally does not know your words: drug codes, part numbers, Bates stamps, internal product names. Skip it when the model already reads your documents fluently and you just want it to answer them differently. For most regulated enterprises, that distinction decides whether the project needs a multi-node GPU cluster or a single training box.

SFT Is Still the Workhorse

Supervised fine-tuning on instruction-response pairs is where most private LLM programs actually spend their training budget. It is well understood, the tooling is mature, and the output is easy to evaluate against held-out examples. Hugging Face's TRL library ships trainers for SFT, DPO, GRPO, and reward modeling on top of Transformers, which has become the de facto post-training stack for open-weights work.

Full-parameter SFT on a 70B model is memory-brutal: roughly 16 bytes per parameter when you count weights, gradients, and Adam optimizer state. That is why most on-prem programs do SFT through LoRA or QLoRA instead. Reserve full SFT for the cases where the behavior change is deep enough that adapters leave quality on the table — new output formats, new reasoning patterns, structured tool use that must hold across long contexts.

LoRA and QLoRA Are the Default

Low-Rank Adaptation freezes the base weights and trains small rank-decomposition matrices inserted into the attention and MLP layers. The trainable parameter count drops by two orders of magnitude, and the resulting adapter is a few hundred megabytes rather than hundreds of gigabytes.

QLoRA goes further. It stores the frozen base in 4-bit NormalFloat and trains the adapter in bf16 on top. The published result is that a 65B parameter model can be fine-tuned on a single 48 GB GPU while preserving 16-bit task performance. For planning purposes, a 70B QLoRA run leaves roughly 5 GB of headroom on a 48 GB card, a 32B run fits in about 21 GB, and an 8B run fits in about 6 GB.

Practical guidance for on-prem LLM fine-tuning:

• Default to QLoRA when the base model does not fit comfortably in VRAM at bf16. Expect 20–40% slower training throughput versus plain LoRA, in exchange for roughly 75% less memory.
• Apply LoRA to all linear layers, not just attention. The QLoRA authors flagged this as critical for matching 16-bit performance.
• Keep adapters versioned and signed. One base model plus signed adapter checkpoints is cleaner for audit than monolithic fine-tuned weights.

DPO and ORPO Replaced Most of the RLHF Stack

Preference training used to mean standing up a reward model and running PPO against it. Direct Preference Optimization collapsed that pipeline. DPO is RL-free: it treats the language model itself as an implicit reward model and optimizes directly against pairs of preferred and rejected completions. The NeurIPS paper reports it matches or exceeds PPO-based RLHF on alignment benchmarks while being markedly more stable to train.

ORPO goes one step further by folding the preference signal into the SFT loss itself, so you train once instead of twice. For on-prem programs, the operational win is the same: no reward model to maintain, no PPO instability, and a much shorter training loop that fits inside an auditable change window.

Use DPO or ORPO training when the problem is behavioral rather than factual: tone, refusal boundaries, citation discipline, format adherence. Collect preference pairs from your own domain experts — paralegals, underwriters, nurse reviewers — and keep that collection inside the perimeter. Preference data is often more sensitive than the training corpus, because it encodes judgment.

RLAIF And Distillation Scale The Teacher

When you cannot get enough human preference labels, Reinforcement Learning from AI Feedback uses a larger local model to generate the preference signal. On-prem, this usually means running a 70B or 120B teacher model to score outputs from a 7B or 13B student, then training the student with DPO against those scores. It is a reasonable compromise when the teacher is already deployed for other workloads and the GPUs are not the bottleneck.

Distillation is the compression story. A larger teacher produces soft targets or synthetic completions; a smaller student is trained to reproduce them. The point is not quality parity. It is latency and unit economics: a distilled 8B model running on a single L40S can serve a workload that would otherwise require an H100 for a 70B. For most edge AI deployments and branch-office inference, distillation is the step that makes the hardware math work.

Model Merging With mergekit And LandFuse

Model merging combines the weights of two or more models of the same architecture into one without any gradient step. The merge runs on CPU in minutes and needs only enough RAM to hold the weights. There is no training data involved, which is the part that matters for regulated environments: you can combine a general instruction-tuned checkpoint with a domain-fine-tuned checkpoint without ever bringing the training corpus near the operation.

mergekit is the de facto open-source toolkit. It implements the methods that actually get used in practice:

• SLERP for interpolating two models along the arc of the unit hypersphere. Simple, two-model only, usually the right first attempt.
• TIES for three or more models. Trims low-magnitude deltas, elects a sign per parameter, then merges only the parameters that agree.
• DARE-TIES for larger merges where redundant deltas are diluting the signal. DARE randomly drops and rescales each model's delta before TIES runs.
• Task Arithmetic when you want to add or subtract a capability expressed as the delta from a shared base.

LandFuse is a newer entrant in the LandFuse model merging family of tooling, aimed at the same workflow: combine open-weights checkpoints without retraining. Its appeal in on-prem contexts is procedural rather than algorithmic. Merges have no training data to log, no gradient updates to attribute, and no optimizer state to purge, which collapses several lines of the SOC 2 change record.

Merging is also where small Apache 2.0 licensed models like Mistral's Ministral and Small lines pay off. Licensing permits commercial self-hosting without royalty, and the small footprint makes experimental merges cheap enough to run as part of normal release engineering.

Keeping The Pipeline Auditable

Every method above has to produce an artifact your compliance team can sign off on. That means the training pipeline logs the base checkpoint hash, the dataset snapshot hash, the training recipe (hyperparameters, seed, library versions), and the resulting adapter or merged-model hash. Those four fields reproduce any run.

Two external frameworks are worth citing here. The NIST AI RMF is the de facto reference for internal AI risk governance in US regulated industries. And in the EU, the EU AI Act's GPAI rules, in force since August 2025, grant partial exemptions for models released under a free and open-source license — though the exemption disappears once a model crosses the 10^25 FLOP systemic-risk threshold. Private continued-pretrains almost never hit that threshold; it is still worth knowing where the line is.

How To Choose

Pick the smallest intervention that moves your eval. For most regulated enterprises the sequence is: start with retrieval, then QLoRA SFT on your domain examples, then DPO on preference pairs collected from your reviewers, then consider a merge to combine a tone-tuned adapter with a tool-use-tuned adapter. Continued pretraining and full RLAIF enter the picture only when the smaller interventions have been exhausted.

The infrastructure case is the same case made in fine-tuning your open-weights model and in the private LLM training stack discussion: own the base model, own the data, own the pipeline. The training method is a dial, not an identity. Picking the right one is a matter of matching the behavior you need against the audit record you can defend.