What Breaks in the First 90 Days of a Private LLM Rollout

Most private LLM programs do not fail at the model layer. They fail in the plumbing between weeks two and twelve, when the proof-of-concept meets real identity systems, real logs, real retrieval queries, and a real change-advisory board. By then the contract is signed, the GPUs are racked, and the finger-pointing has already started.

The pattern is consistent enough to plan for. An analysis of 847 AI agent implementations found that 76% experienced critical failures within the first 90 days, 43% were abandoned entirely by month six, and only 18% delivered on their original ROI promises. The failures were not exotic. They were the same six or seven items, in roughly the same order, hitting roughly the same teams. What follows is the shortlist, with the dollar ranges and the remediation steps, written for a buyer who still has leverage because the ink is wet.

Week One Breaks on the GPU Driver, Not the Model

Day one is almost never a model problem. It is a driver problem. The vendor delivers a validated image. Your platform team re-pins it against an internal Ubuntu AMI, a hardened kernel, or an older NVIDIA driver that security has standardized on. Inference starts. Throughput is a third of the benchmark number. Sometimes the kernels silently fall back to CPU and nobody notices for a week.

This is not hypothetical. CUDA driver drift between host, container, and GPU is a leading silent cause of training and inference instability in production ML clusters, and vLLM's paged attention kernels in particular are compiled at install time against a specific CUDA version. Install vLLM against CUDA 12.1 and run it on a host pinned to 11.8 and the engine will either throw a cryptic runtime error or quietly serve from CPU at a fraction of expected tokens per second.

Negotiate three things before signature. A documented driver and CUDA matrix the vendor warrants against. A loud-fail health check at container start that refuses to serve if compute capability or runtime version is wrong. And a 48-hour burn-in on your actual hardware image, not a reference one. Remediation after the fact runs one to three engineering weeks and typically a four- to six-figure delay on business value. It is cheaper to put it in the statement of work.

Embedding Swaps Invalidate the Vector Store

Week three or four, somebody wants to try a better embedding model. Maybe the original choice was 768-dimensional and the new one is 1,024. Maybe the tokenizer changed. Either way, every vector in the store is now mathematically meaningless against the new query encoder. Retrieval precision collapses overnight and the fix is a full re-embedding of the corpus, which on a mid-sized document set means hours of GPU time and a maintenance window on the production index.

Chunking choices compound this. NVIDIA benchmarked seven chunking strategies across five datasets and found page-level chunking won with 0.648 accuracy and the lowest standard deviation across tasks. If your initial deployment chose token-window chunking because it was the default, you are likely to re-chunk and re-embed at least once in the first ninety days. Budget for it. Our view on when that work pays for itself is covered in the business case for owning your vector database.

Two protections matter in the contract. First, version-pin the embedding model and the chunking policy as part of the acceptance criteria, so a change requires a documented re-indexing plan rather than a Slack message. Second, require the vendor to deliver the ingestion pipeline as code, so a re-embed is a job run, not a rebuild. A custom LLM deployment written this way can re-index overnight; one written against a managed console often cannot.

SSO and SCIM Gaps Surface at the First Real Pilot

The pilot runs on fifteen hand-provisioned accounts. On day thirty, the request comes to open it to two hundred users across four business units. That is when identity breaks. The model gateway supports OIDC but not SAML, or SAML but not SCIM provisioning, or SCIM but with no group-to-role mapping, so every new user lands in a default group that can see every index. Legal gets access to the finance corpus. The project pauses.

This is also where IBM's 2025 breach data stops being abstract: one in five organizations reported a breach tied to shadow AI, and high shadow AI levels added an average of $670,000 to breach costs. Users who cannot get into the sanctioned system paste sensitive content into a consumer one. LayerX's survey in the same roundup found 77% of employees paste data into GenAI tools and 82% of those pastes come from unmanaged personal accounts.

Three line items in the RFP close most of this gap:

• SCIM 2.0 with group-based entitlement, mapped to specific indices and tools, not to the whole corpus.
• Just-in-time provisioning with an identity provider the security team already owns.
• A documented break-glass path for revoking access that does not require a vendor ticket.

Our RFP requirements checklist covers the exact clauses.

SIEM Log Volume Will Surprise Finance

Week five or six, the SOC turns on verbose logging across the inference gateway, the retrieval service, the audit trail, and the agent orchestrator. The volume is larger than anyone modeled. A single moderately active agent workflow with tool calls can produce tens of thousands of structured events per user per day. Those events are now billable in the SIEM.

The math is not kind. Enterprise telemetry at 5 TB per day can translate to $3.6M to $7.3M per year on ingestion alone at $2 to $4 per GB. Microsoft Sentinel pay-as-you-go lists at $4.30 to $5.59 per GB in US regions. An LLM workload that adds even 50 GB per day of structured logs is a $75K to $100K annual line item nobody put in the project budget.

The fix is a tiered logging policy agreed with the SOC before go-live: high-cardinality operational traces to object storage, security-relevant events to the SIEM, and sampled prompt and response pairs to a separate evidence store with its own retention clock. Negotiate the vendor's logging schema and its sampling knobs as part of acceptance, not after.

Retrieval Precision Collapses Once Real Users Arrive

The pilot queries were written by the project team. They are clean, specific, and generous with keywords. Real users ask three-word questions, paste in a half-redacted email, and expect the system to know what matter number they mean. Recall holds. Precision does not. The model dutifully answers from the top-k chunks and invents the parts it cannot find.

Hallucination rates in production vary enormously by task shape. One review found extractive QA systems hallucinate on 3 to 8 percent of responses, open-ended generation on 15 to 25 percent, and multi-step agent workflows on 20 to 40 percent of tool-call chains. Long-context does not save you: a 172-billion-token study across 35 open-weight models found only 2 of 35 models stayed under a 5% fabrication rate at 32K context, and fabrication exceeded 10% for every model tested at 200K.

Three remediations show up repeatedly in the first ninety days: a reranker tuned on actual user queries, a confidence-gated refusal path, and an evaluation harness that runs nightly against a labeled set the business owns. The deeper problem, which we have written about in why accuracy is the wrong metric, is that nobody agreed on what "right" meant before the pilot started.

The GRC Committee Will Stall the Rollout

Week eight or nine, the project reaches the governance, risk, and compliance review. The committee meets monthly. The paperwork was not pre-staged. The data classification review wants evidence that PII is tokenized before it reaches the vector store. The model risk function wants a documented evaluation against the NIST AI Risk Management Framework. The vendor security review wants an updated SOC 2 Type II report. Any one of those items missing adds a month.

Large enterprises take nine months on average to scale AI agents into production, versus roughly 90 days for mid-market firms, and the delta is mostly governance and data readiness. The companies that hit the 90-day window do three things in parallel with the technical build: they commission an independent model and data-flow review, they draft the GRC package during week two, and they book committee time before they need it. An LLM security audit delivered as evidence rather than a promise shortens that cycle meaningfully.

What To Negotiate Before You Sign

None of these failures are new. They are the predictable consequences of treating a private LLM rollout as a software install rather than a platform program. The buyers who get through the first ninety days intact share a short list of pre-signature moves:

• A written driver, CUDA, and OS matrix with a 48-hour burn-in clause.
• Version-pinned embeddings and chunking, with the ingestion pipeline delivered as code.
• SCIM, group-based entitlement, and a documented revocation path.
• A tiered logging schema agreed with the SOC and priced against the SIEM contract.
• An evaluation harness and a labeled gold set owned by the business, not the vendor.
• A GRC evidence pack staged by week two, including an independent security review.

The appliance path exists in part to collapse several of these items into a single procurement. The LLM appliance approach ships the driver matrix, the gateway, and the logging schema pre-integrated, which trades flexibility for a shorter path through the GRC committee. Whether that trade is right for you is a separate conversation, and we work through it in appliance versus self-built GPU cluster. The point is to make the choice with the ninety-day failure modes already on the table, rather than discovering them in week seven.