Open-Weight LLM · Private & Custom AI
GLM-4.5-Air-FP8
Compact, multi-lingual agentic model (106B total / 12B active) designed for private deployment of reasoning, tool-calling, and code workflows without API dependency.
GLM-4.5-Air-FP8 is a mixture-of-experts reasoning model from Zhipu AI, optimized for inference efficiency via FP8 quantization and speculative decoding. For ops teams, it enables self-hosted agent loops—tool use, multi-step reasoning, code generation—while keeping data and model weights in your environment. The Air variant trades some raw capability for 4–8× lower compute than the full GLM-4.5.
Model facts
Private deployment
Run GLM-4.5-Air-FP8 in your own environment
Self-hosting requires 2× H100 or 1× H200 GPU (FP8 inference, ~55GB VRAM estimated per GPU). Deployment frameworks (vLLM, SGLang, transformers) are documented and widely compatible. Full 128K context requires 4× H100 / 2× H200. Fine-tuning on your data uses standard LoRA or SFT tooling (Llama Factory, Swift). Core win: all model weights, activations, and outputs stay in your infrastructure—no data leaves for API calls.
Operational AI use cases
Internal support & knowledge agent
Deploy GLM-4.5-Air-FP8 as a tool-calling agent over internal docs, tickets, and FAQs. Model reasons through multi-step support workflows (classify issue → search KB → draft response → escalate if needed) without sending customer queries to external APIs. Hybrid reasoning mode switches between fast response and deep reasoning on escalations.
Code review & ops automation
Use code-generation and reasoning capabilities to lint pull requests, validate infrastructure-as-code (Terraform, CloudFormation), and generate runbooks. Fine-tune on your repo patterns, then run private inference—ideal for security-sensitive deployments (fintech, healthcare) where code cannot leave premises.
Multi-language document automation
Process and route inbound documents (contracts, invoices, reports) in English and Mandarin. Model extracts entities, classifies by risk/urgency, and calls internal APIs to file or trigger approvals. Runs entirely on your hardware; no language data exposed to third parties.
Custom AI
As a base for custom AI
Strong foundation for custom agent products. FP8 quantization + speculative decoding make it fast enough for real-time API endpoints. Mixture-of-experts routing (12B active of 106B) leaves room for fine-tuning on domain tasks without retraining the full model. Reasoning and tool-parser layers integrate with vLLM/SGLang for production pipelines. Use it to build: customer-facing agents, internal ops bots, or vertical SaaS tools where model privacy or data residency is a requirement.
In the operating system
Where it fits
Sits at the **agent & workflow layer** of an AI ops stack. Plays the reasoning + tool-orchestration engine: processes user/system input, reasons about which tools to call (APIs, databases, search), executes them, and synthesizes responses. Feeds on structured knowledge (embeddings, APIs, local databases) and outputs structured actions (function calls, approvals, notifications). Complements a retrieval layer (RAG) and sits above execution/logging layers.
Data control & security
Private deployment means model inference, weights, and data remain in your VPC/data center—no telemetry to Zhipu or external APIs by default. Reasoning chain, intermediate outputs, and tool results stay local. **Caveats:** FP8-quantized weights are still proprietary Zhipu IP (MIT license permits use but not redistribution of modified weights); fine-tuning your custom data does not retroactively encrypt it. Compliance (GDPR, HIPAA) depends on your infrastructure controls, not the model. Audit logging, access controls, and data deletion are your responsibility.
Hardware footprint
**Estimate (FP8 inference, batch ≤8):** ~50–55 GB VRAM per H100 GPU. With tensor parallelism across 2× H100: inference+batch ops feasible. **Full context (128K):** requires 4× H100 or 2× H200. **Fine-tuning LoRA:** 4× H100 at batch 1. Activation memory and KV cache typically 20–30% of weight size; FP8 reduces both vs. BF16 (~90GB per H100 for BF16 variant).
Integration
Integrates via vLLM or SGLang APIs (OpenAI-compatible `/chat/completions`, `/tool_calls` endpoints). Wires into orchestration: LangChain, LlamaIndex, or custom Python. Tool-call parser (built-in GLM-4.5 mode) JSON-serializes function calls—connect to your internal APIs (Jira, Slack, databases, approval systems) via webhook or agent framework. MTP (Multi-Token Prediction) + speculative decoding lower latency for agent loops. Context window ~128K (model card specifies hybrid reasoning but not exact max context for FP8 variant—verify in docs).
When it's not the right fit
- —Real-time, ultra-low-latency inference required (<100ms p99): MTP + speculative decoding help, but 106B active params still slower than 7–13B models. Use smaller open models (Llama-3.1-8B, Phi-4) if sub-50ms is critical.
- —Serving thousands of concurrent users on modest GPU inventory: GLM-4.5-Air scales better than full GLM-4.5, but still requires 2+ H100s minimum. SaaS with 1–2 GPUs should consider smaller or quantized alternatives (GGUF, AWQ, Int4).
- —Fine-tuning budget <$50k: H100s are expensive; LoRA training on 4× H100 for domain adaptation costs ~$30–50k/week. Smaller models or API-based fine-tuning may be cheaper for small datasets.
- —No multi-modal: Text-only. If you need vision, code, or audio in one model, use GPT-4V, Claude 3, or Llama-3.2-Vision.
Alternatives to consider
Llama-3.1-405B (Meta)
Larger, higher-performance open model; no MoE (full 405B active). Better for complex reasoning, worse for constrained hardware. No built-in reasoning/tool-calling mode; requires custom integration. MIT license.
Mixtral-8x22B (Mistral)
Smaller MoE (8 experts, ~39B active). Faster inference, lower VRAM (~60GB for BF16), but weaker at reasoning/agentic tasks. Apache 2.0 license, strong on code.
Qwen2.5-72B (Alibaba)
Dense 72B, no MoE; multilingual (EN/ZH/others). Strong on code and instruction-following. Larger than GLM-Air but still fits 8× H100. Apache 2.0. No built-in reasoning mode; you manage tool calls yourself.
Related open models
FAQ
Can I fine-tune GLM-4.5-Air-FP8 on proprietary customer data and keep it private?
Yes. Download the model weights, run LoRA fine-tuning on your infrastructure using Llama Factory or Swift, and serve inference locally. Your training data and fine-tuned LoRA adapters stay in your VPC. The base model weights remain Zhipu IP (MIT license permits use but not redistribution); you own the LoRA layers.
Is commercial use allowed?
Yes. MIT license explicitly permits commercial use, modification, and distribution (with license/copyright notice). No royalties or API call fees. You only pay for compute (GPUs) and your own operational costs.
What is the context window for the FP8 variant?
Model card states GLM-4.5-Air supports ~128K context in full-featured mode. FP8 precision may impose stricter limits on KV cache; exact max context for FP8 is not specified in the card. Verify with `transformers` config or benchmark on your target hardware before production.
How does hybrid reasoning mode work, and when should I use it?
Two modes: thinking (slow, detailed reasoning chain exposed) and non-thinking (fast, direct response). Use thinking for complex multi-step tasks (agent planning, code review, high-stakes decisions); non-thinking for fast responses (support chat, classification). Model automatically chooses based on prompt complexity, or you can force via system flag.
Build a Private AI Ops Layer
GLM-4.5-Air-FP8 is ready to power internal agents, document automation, and code workflows on your hardware. Let's architect a self-hosted AI stack that keeps your data in-house. Start with LLM.co.