Open-Weight LLM · Private & Custom AI
VLM2Vec-Full
Multimodal embedding model for private, in-house search, retrieval, and similarity tasks across images and text—built on Phi-3.5-V, deployable entirely on your infrastructure.
VLM2Vec is a vision-language embedding model trained on 36 multimodal datasets to convert images and text into comparable vector representations. It enables teams to build semantic search, content matching, and retrieval workflows without relying on external APIs. For ops, it's a foundation for automating document-image correlation, asset discovery, and knowledge retrieval—all running inside your network.
Model facts
Private deployment
Run VLM2Vec-Full in your own environment
Self-hosting is the intended architecture: the full model (~4.1B parameters) runs on consumer-grade GPUs (NVIDIA A100 or multi-GPU CPU clusters). Deploy via HuggingFace Transformers and custom MMEB inference code; data—images, text, embeddings—never leaves your environment. No cloud callbacks, no third-party logging. Trade-off: you own compute and latency tuning.
Operational AI use cases
Internal Document + Image Search
Index PDFs, manuals, internal wikis, and scanned docs alongside product photos or facility images. Employees query in natural language or upload an image to find matching documentation—e.g., 'Show me all repair guides with similar damage patterns.' Runs offline, zero external API calls.
Support Ticket Routing & Asset Matching
Embed customer support tickets (text) and attach screenshots. Automatically surface similar past tickets and linked knowledge articles by semantic similarity, not keyword search. Reduces triage time and improves first-contact resolution without third-party ticket intelligence services.
Inventory & Facilities Management
Embed inventory catalogs, purchase orders, and facility photos. Match incoming damage reports or maintenance photos to asset records and historical issues. Automate work-order tagging and prioritization based on semantic similarity across structured and unstructured data.
Custom AI
As a base for custom AI
Strong foundation for custom retrieval-augmented generation (RAG) workflows and multimodal search products. Plug embeddings into vector databases (Pinecone, Weaviate, Milvus) to build branded search experiences, internal AI assistants, or customer-facing image-search tools. The model is fine-tuned for cross-modal tasks, so retraining on proprietary image-text pairs is viable if you need domain-specific accuracy.
In the operating system
Where it fits
Operates in the Knowledge & Retrieval layer of an AI OS—converting raw documents and images into searchable embeddings. Feeds into Agent/Workflow layers (e.g., agents query embeddings to retrieve context before generating responses). Not a generative model; complements LLMs by providing semantic grounding for retrieval and ranking.
Data control & security
Private deployment architecture means all embeddings, source images, and queries remain in your environment—no data transmitted to external services. Suitable for regulated industries (healthcare, finance, defense) where data residency is mandatory. However, the model itself is open-weight; security depends on your infrastructure hardening (network isolation, access control, audit logging). No inherent encryption or compliance claims—treat it as a data-processing component subject to your governance policies.
Hardware footprint
~16–20 GB VRAM (bfloat16 precision on A100/H100); ~30 GB in float32. Estimate: single A100 (40GB) with headroom for batch inference; multi-GPU setups for production throughput. CPU inference possible but impractical for real-time ops. Exact memory validated only in controlled benchmarks—your mileage varies by batch size and num_crops settings.
Integration
Requires PyTorch + Transformers integration; custom inference code provided in the GitHub repo. Expose embeddings via REST API (FastAPI, Flask) or async job queues (Celery, Ray) for batch processing. Connect to vector DBs via Python SDKs. Supports ONNX export (Unknown if tested—verify with TIGER-Lab); batching and caching embeddings is critical for latency. Scaling requires GPU pooling or multi-node inference orchestration (Kubernetes recommended).
When it's not the right fit
- —You need real-time generative responses (no text generation—embedding-only model).
- —Your images are highly domain-specific and not well-represented in MMEB training data (e.g., specialized medical imaging, satellite data); fine-tuning effort required.
- —You lack GPU infrastructure or can't justify the compute cost for embedding inference at scale.
- —Context-length requirements exceed model capacity (exact max length Unknown—likely similar to Phi-3.5-V ~128K tokens, but verify for your text corpus).
Alternatives to consider
CLIP (OpenAI, via open clones: OpenCLIP, SigLIP)
Lightweight, well-established vision-text embeddings; smaller models available. Trade-off: less optimized for text-only or text-heavy tasks; fewer language-model capabilities.
LLaVA + Embedding Extraction (Meta/HF)
Generative VLM that can also produce embeddings via hidden states. More flexible for hybrid search + QA workflows, but heavier (7B–13B) and less embedding-task-optimized.
Qwen-VL or similar open multimodal models
Alternative VLM backbones with embedding-extraction capability; regional or non-Apache licensing may apply. Check license compatibility before production use.
Related open models
FAQ
Can I run VLM2Vec entirely on-premises without touching the internet after deployment?
Yes. Download the model weights once, containerize with your inference code, and deploy offline. No callbacks to HuggingFace or external services required. Ensure your system has adequate GPU compute and storage for model cache.
Is VLM2Vec licensed for commercial use?
Yes. Apache 2.0 license permits commercial deployment, modification, and integration. Verify no viral copyleft clauses apply to the base model (Phi-3.5-V is also Apache 2.0). Always review your deployment's compliance obligations; the license is permissive but does not indemnify liability.
What if I need embeddings for a language or domain VLM2Vec wasn't trained on?
The model trains on English multimodal data. Non-English support is Unknown. For domain specificity (e.g., medical images, manufacturing), fine-tuning on your proprietary data is feasible but requires GPU resources and expertise. Start with zero-shot performance and evaluate; TIGER-Lab provides training scripts on GitHub.
How does this differ from simply using an LLM's embedding layer?
VLM2Vec is purpose-built for cross-modal (image-text) similarity using contrastive learning and large-scale multimodal training. General LLM embeddings optimize for language; VLM2Vec optimizes for multi-modal relevance ranking, which is more aligned with search and retrieval tasks.
Build Private Multimodal Search into Your Operations
VLM2Vec empowers teams to embed images and documents, then search and retrieve at scale—all inside your infrastructure. LLM.co helps you integrate it into custom AI workflows, RAG systems, and operational automation. Let's design your private embedding layer.