Open-Weight LLM · Private & Custom AI
fg-clip-base
Fine-grained vision-language alignment model for private image classification, retrieval, and multimodal search without vendor lock-in.
FG-CLIP is an open-weight CLIP variant (150M parameters) trained on global and region-level caption-image pairs for precise visual-textual matching. Built for ops teams needing to classify, retrieve, or search images within a closed environment—product catalogs, support tickets, compliance documents—while keeping all data on-premise.
Model facts
Private deployment
Run fg-clip-base in your own environment
Self-hosting is the primary architecture: load weights via Hugging Face, deploy on internal GPU (RTX 4090 / A100), and run inference against your image corpus without data leaving your network. Requires PyTorch + transformers library; custom_code flag means you audit the model loading logic. No vendor API calls, no data telemetry.
Operational AI use cases
Support Ticket Image Classification
Automatically tag incoming support images (screenshots, hardware damage, installation errors) with fine-grained labels. Deploy privately to avoid exposing customer screenshots to external APIs. Route tickets to the right team based on visual content without manual review.
Product/Inventory Visual Search
Index your internal product catalog images and enable employees or internal systems to retrieve SKUs, variants, or suppliers by uploading a photo. Fine-grained alignment means matching 'white ceramic mug with handle' vs. 'clear glass mug' without retraining.
Compliance Document Scanning & Triage
Batch-process document images (forms, receipts, contracts) to extract region-level details (signature blocks, approval stamps, classification zones) for audit workflows. Dense feature visualization helps QA teams verify model is looking at the right parts of a page.
Custom AI
As a base for custom AI
Strong foundation for building a proprietary image-understanding product or internal tool. Leverage the two-stage training approach (global + region captions) as a reference to fine-tune on domain-specific data (medical imaging, parts inspection, real estate photos). Dense features enable interpretable heatmaps for regulated workflows. Can be extended with downstream classifiers or retrieval indices tailored to your ops.
In the operating system
Where it fits
Sits in the **knowledge & perception layer** of an AI operating system: grounds text queries and rules in visual evidence. Feeds into workflow agents (route support ticket by image content), knowledge bases (visual indexing), and compliance automation (region-level extraction for audit trails).
Data control & security
All inference happens in your VPC/on-premise infrastructure—images never leave your network. No telemetry or external API calls during deployment. Org can audit model code (custom_code flag requires review before load). Data residency and access control are determined by your infrastructure, not the model. Does not provide encryption or compliance certification; those are your responsibility via deployment architecture.
Hardware footprint
**Estimate (unverified).** ~150M parameters, typically 300–600 MB weights in FP32; ~150–300 MB in FP16. Inference VRAM on RTX 4090 / A100: ~4–8 GB for batch=1 (224×224 image + captions). Batch inference can use 12–16 GB. CPU inference possible but slow; GPU recommended for ops latency requirements.
Integration
Standard Hugging Face integration: use `AutoModelForCausalLM.from_pretrained()` + `AutoImageProcessor` + `AutoTokenizer`. Supports short (77-token) and long (248-token) captions; choose based on query specificity. Output is cosine-similarity scores and dense patch-level features (224×224 → 14×14 patches). Integrate via REST API wrapper (FastAPI), batch processing (Airflow), or real-time agent loops. Requires trust_remote_code=True; review custom loading.
When it's not the right fit
- —You need real-time image detection / localization (bounding boxes, segmentation): FG-CLIP does classification & similarity, not object detection.
- —Your images contain highly specialized domains (medical radiology, satellite imagery, industrial microscopy) without domain-specific fine-tuning—base model trained on web data.
- —Regulatory compliance requires model explainability guarantees: dense features help interpretability, but no certified XAI audit trail provided by the model itself.
- —You need non-English text understanding in captions: model trained on English; multilingual support is unknown.
Alternatives to consider
OpenAI CLIP (open weights via onnx-community)
Original CLIP architecture, broader adoption, simpler baseline. But FG-CLIP adds region-level alignment; use CLIP if you need broad generalization over fine-grained matching.
Salesforce ALBEF
Another vision-language model with dense features. More research-focused; FG-CLIP is newer (ICML 2025) and designed for fine-grained retrieval.
Meta DINOv2 (vision-only encoder) + LLM for grounding
Decoupled approach: use DINOv2 for image features, add an LLM for text reasoning. More modular but requires custom integration; FG-CLIP is end-to-end.
Related open models
FAQ
Can I run FG-CLIP entirely on-premise without calling an external API?
Yes. Download weights from Hugging Face once, load locally, and run inference inside your VPC. No cloud dependency after initial download. Ideal for compliance-sensitive workflows (healthcare, financial, government).
Is FG-CLIP licensed for commercial use?
Yes. Apache 2.0 license permits commercial use, modification, and distribution. You may build proprietary products on top. Verify you comply with any upstream dataset licenses (model card notes dependencies on original datasets; review as needed).
How do I fine-tune FG-CLIP on my own images?
Model card does not provide fine-tuning instructions. Refer to the GitHub repo (360CVGroup/FG-CLIP) for training code. You would likely need region-level captions for stage-2 refinement, or use transfer learning on top of base embeddings.
What's the difference between short (77-token) and long (248-token) caption mode?
Short mode is for concise queries ('a white cat'); long mode for detailed descriptions with region info. Use short for fast retrieval; long when descriptive accuracy matters. Choose via `walk_short_pos` flag.
Build a Private Image Understanding System
FG-CLIP is ready to deploy in your environment. Work with LLM.co to integrate it into your ops workflows—support triage, product search, compliance scanning—without vendor APIs or data leakage. Let's architect a custom AI system that stays yours.