Open-Weight LLM · Private & Custom AI
SmolLM2-360M
Lightweight edge-deployable base model for custom ops automation, on-device private inference, and internal workflow agents where data residency and model control are non-negotiable.
SmolLM2-360M is a 361M-parameter transformer trained on 4 trillion tokens, engineered for CPU/GPU deployment with minimal resource footprint (~724 MB in bfloat16). For ops teams, it's a private-first foundation for automating support workflows, document processing, knowledge retrieval, and internal agentic tasks without cloud dependency or data egress.
Model facts
Private deployment
Run SmolLM2-360M in your own environment
Self-hosting is the design intent. The model fits on a single consumer GPU, runs on CPU, and scales to multi-GPU with `device_map='auto'`. A company controls the entire inference stack—no API calls, no third-party logs, no token tracking. Trade-off: you manage the hardware, inference servers (e.g., vLLM, Text Generation Inference), and operational burden; benefit is data never leaves your environment.
Operational AI use cases
Internal Support & Knowledge Agent
Instruct-tuned variant fine-tuned on function-calling datasets. Deploy as a private support chatbot: ingest company docs, runbooks, FAQ, and let it field Slack/Teams queries or ticket triage. All conversations stay internal. Instruction-following capability (IFEval 41.0) is weak vs. frontier, but sufficient for rule-based, structured Q&A.
Document Classification & Summarization
Use for invoice tagging, contract classification, email routing, or meeting notes summarization in compliance-sensitive orgs. Trained on text-rewriting datasets (Argilla). Runs in your data center—no external processing, no vendor lock-in, audit trail is yours.
On-Device Field/Remote Automation
Deploy on-device (laptop, tablet, edge server) for field teams: local form filling, field report generation, work-order routing. Model is lightweight enough for Raspberry Pi clusters or mobile edge. No latency from cloud round-trips; no connectivity dependency.
Custom AI
As a base for custom AI
Strong foundation for lightweight, proprietary ops tools. Use SmolLM2-360M as the backbone of a custom RAG system (private vector store + retriever), a fine-tuned workflow classifier, or an agentic task-decomposer. Its small size lets you run multiple instances (one per dept/workflow) and iterate on SFT/DPO cheaply. Apache 2.0 license permits commercial derivative products.
In the operating system
Where it fits
In an AI OS: the knowledge/reasoning layer (retrieval, summarization, doc understanding) and workflow-agent layer (task routing, instruction following). Not a multi-turn chat backbone (MT-Bench 3.66 is weak), but excellent for discrete, structured ops tasks chained into larger automation pipelines. Pair with a retrieval system and task schedulers.
Data control & security
Self-hosting ensures data residency: no inference logs to third parties, no model weights phoned home, no token counting via API. Company owns the deployment, can air-gap it, control GPU/CPU access, audit inference queries, and integrate with internal logging. Security is an architecture win, not a model property—still your responsibility to harden the inference server, networking, and access controls.
Hardware footprint
**Estimate (bfloat16):** ~724 MB model weights + ~150–200 MB activation/cache per inference instance. Total: ~900 MB–1.2 GB per concurrent inference. On RTX 3090 (24 GB): easily 20+ concurrent instances. On CPU (16 GB RAM): single instance, ~1–2 tokens/sec. For production: 1× A100 (40 GB) or 2× H100 covers 50–100 concurrent ops tasks.
Integration
Plug in via Hugging Face `transformers`, vLLM, or Text Generation Inference (model supports TGI). API wrappers (Flask/FastAPI) expose it as a local endpoint. Integrate with Zapier/Make for workflow automation, or call directly from Python/Node agents. For real-time ops: low latency on GPU (~50–100ms per token on H100-class hardware, ~200–500ms on commodity GPU). Batch/async for non-critical tasks. Token context is unknown—verify if your docs exceed assumed window before production.
When it's not the right fit
- —Multi-turn conversational reasoning (MT-Bench 3.66 is poor; use for single-turn instruction/task, not chat).
- —Complex math or code generation (GSM8K 5-shot = 7.43%; use Claude/GPT for numerical workflows).
- —Non-English content (trained primarily English; multilingual support unknown).
- —Real-time latency <50ms required (achievable on GPU, but CPU inference is slow; add 200–500ms).
Alternatives to consider
Qwen2.5-0.5B
Comparable size, better at math (GSM8K 5-shot = 26.8 vs. 7.43), lower instruction-following. Depends on commercial terms; verify license for self-hosting.
Llama 3.2-1B
Slightly larger (1B params), strong instruction tuning, multilingual. Llama Community License permits research/commercial self-hosting. Better all-rounder for ops workflows.
Phi-4 / Phi-3.5-mini
Microsoft's lightweight line, optimized for reasoning & instruction. MIT license. Similar footprint, stronger benchmarks on MMLU/commonsense. Good for knowledge-heavy ops.
Related open models
FAQ
Can I deploy this privately (keep data in-house)?
Yes, by design. Download weights, run via `transformers` or vLLM on your GPU/CPU. All inference stays in your environment—no cloud, no third-party logs. You manage the server, networking, and access.
Can I use this commercially or in a proprietary product?
Yes. Apache 2.0 permits commercial use, modification, and derivative works. You can fine-tune it, embed it, and sell applications. Retain the license attribution.
What's the context window / maximum input length?
Unknown from the model card. Review the paper (arxiv:2502.02737) or check Hugging Face config. Verify before deploying long-document workflows (RAG, summarization).
How does it compare to cloud APIs (OpenAI, Claude)?
Much slower (single-digit tokens/sec on CPU, 50–100ms/token on consumer GPU), less capable (lower MMLU, reasoning, instruction-following). Trade: data privacy, full control, no per-token cost, no API dependency. Use for ops tasks where accuracy/speed are acceptable and privacy is mandatory.
Ready to build a private AI operating system?
SmolLM2-360M is a proven foundation for edge deployment and internal automation. LLM.co helps you build custom AI workflows, fine-tune for your ops, and self-host with confidence. Explore how to integrate lightweight models into your operational stack.