Vision-Language Models
What you’ll learn. The two generations of vision-language models (VLMs): contrastive dual encoders (CLIP, SigLIP) and generative “LLM with eyes” models (Flamingo, BLIP-2, LLaVA, and successors); the anatomy of a LLaVA-style VLM (vision encoder, connector, LLM); training stages and data; how resolution is handled (AnyRes/tiling, native-resolution encoders); grounding, OCR and document understanding; and how to evaluate VLMs and recognize their failure modes. Background: Self-Supervised & Foundation Vision Models, Vision Transformers, and the LLM side in Attention and Transformers.
Two generations
Section titled “Two generations”- Contrastive dual encoders (CLIP, ALIGN, SigLIP): separate image and text towers, aligned embeddings, great for retrieval, zero-shot classification and as vision encoders. They cannot generate text or reason over multiple steps.
- Generative VLMs: an LLM consumes visual tokens plus text and generates text (captions, answers, structured output, even boxes as text). These are the “multimodal assistants” (GPT-4V-class, Gemini, Claude with vision, Qwen-VL, InternVL, LLaVA, PaliGemma, Pixtral, Llama-3.2/4 vision models, and others). The exact lineup changes every few months; the architecture pattern is stable.
CLIP recap
Section titled “CLIP recap”CLIP trains a ViT and a text transformer with a symmetric InfoNCE loss on hundreds of millions to billions of image-text pairs (lecture 4 has the loss). For VLMs, the point is that the vision tower already produces features aligned to language, so it is a natural plug-in encoder. SigLIP-family encoders (sigmoid loss) are a common modern choice, with some systems using multiple encoders (e.g., a semantic encoder plus a DINOv2 encoder for fine spatial detail, as in Cambrian-1 and several others).
The LLaVA-style architecture
Section titled “The LLaVA-style architecture”image ──> vision encoder (ViT: CLIP/SigLIP) ──> patch features (N x Dv) │ connector / projector ▼ visual tokens (M x Dllm) │ text prompt ──> tokenizer ──> text embeddings ──── concat ──> LLM (decoder-only) ──> answer tokensComponents:
- Vision encoder: pretrained ViT, often frozen at first. Patch tokens from a late-but-not-last layer (CLIP’s penultimate layer worked better in LLaVA) preserve more spatial information than the pooled embedding.
- Connector: maps -dimensional visual features to the LLM embedding dimension. Choices:
- Linear / 2-layer MLP projector (LLaVA-1.5): simple and surprisingly strong; keeps all patch tokens (576 for ViT-L/14 at 336px).
- Q-Former (BLIP-2): a small transformer with learned queries cross-attending to the image features, compressing to a fixed number (e.g., 32) of tokens.
- Perceiver resampler (Flamingo): similar latent-query compression.
- Cross-attention layers inside the LLM (Flamingo, Llama-3.2 Vision): visual features are injected through gated cross-attention rather than as input tokens.
- Pooling / pixel-shuffle token reduction: merge neighboring tokens (e.g., with a channel concat) to cut token count 4x (InternVL, Qwen-VL).
- LLM: a pretrained decoder (Vicuna → Llama, Qwen, Gemma, Mistral, …). Image tokens are placed at the position of an
<image>placeholder.
A minimal projector in PyTorch:
import torch, torch.nn as nn
class MLPConnector(nn.Module): def __init__(self, dv=1024, dllm=4096): super().__init__() self.net = nn.Sequential(nn.Linear(dv, dllm), nn.GELU(), nn.Linear(dllm, dllm)) def forward(self, feats): # (B, N, dv) -> (B, N, dllm) return self.net(feats)
def build_inputs(text_emb_before, vis_tokens, text_emb_after): return torch.cat([text_emb_before, vis_tokens, text_emb_after], dim=1) # fed to LLM as inputs_embedsCost. Visual tokens are expensive: with 576 tokens per image, each image consumes context and time like ~400 words of text, and the LLM’s attention/KV cache cost scales with them. Multi-image and video inputs multiply this, so token compression is a major design axis (Video, 3D & Spatial Vision).
Training recipe
Section titled “Training recipe”Typical multi-stage training:
- Alignment pretraining: freeze the encoder and LLM, train only the connector on image-caption pairs (hundreds of thousands to millions). Cheap; teaches “what do these embeddings mean in LLM space”.
- Instruction tuning / SFT: unfreeze the LLM (and optionally the encoder, or use LoRA) on visual instruction data: VQA, conversation, reasoning, OCR, charts, grounding. LLaVA bootstrapped this with GPT-4-generated conversations from captions and boxes; later work is more data-heavy and more diverse.
- Optional preference optimization / RL (DPO-style, RLHF/RLVR) to reduce hallucination and improve reasoning; “thinking” VLMs trained with reinforcement learning on verifiable visual-math and chart tasks emerged in 2025.
The loss is ordinary next-token cross-entropy on the answer tokens only (image and prompt tokens masked out). Key empirical lessons across LLaVA-1.5, Cambrian, MM1, Idefics, Molmo and similar open work: data quality and mixture dominate (especially OCR/document and high-quality dense captions), the LLM size matters substantially, a stronger/higher-resolution encoder helps, and the connector can be simple. Unfreezing the encoder helps for fine-grained tasks but risks destabilization.
Some models skip the separate encoder and train from scratch natively multimodal (Gemini-style, Chameleon-style early fusion, Fuyu/Emu-style direct patch input). Treat the “encoder + connector + LLM” recipe as the open, reproducible default.
Resolution handling
Section titled “Resolution handling”A fixed or input destroys small text and fine detail. Strategies:
- AnyRes / tiling (LLaVA-NeXT, InternVL, many others): choose a grid (e.g., ) matching the aspect ratio, split the image into tiles each encoded at the base resolution, plus a downscaled global thumbnail; concatenate all tile tokens. Token cost grows with the number of tiles.
- Native-resolution / dynamic-resolution encoders (Qwen2-VL and later, NaViT-style, SigLIP 2 NaFlex): the ViT accepts any size with 2D rotary position encodings; token count = (H/p)*(W/p) after merging. Aspect ratios are preserved, no tiling artifacts.
- Token compression: pixel-shuffle merging, resamplers, learned token pruning (drop background tokens) to keep contexts manageable.
Rule of thumb: tokens for patch size and merge factor . A 1000x1000 document at , is about 1,270 tokens. Choose the minimum resolution at which the required details (text height in pixels) remain legible: roughly 10+ px per character height for reliable OCR.
Grounding and referring
Section titled “Grounding and referring”Grounding links language to image regions: referring expression comprehension (box for “the red mug left of the laptop”), phrase grounding (boxes for noun phrases in a caption), and pointing (return coordinates). VLMs implement this either with special location tokens (PaliGemma, Florence-2 discretize coordinates into tokens like <loc_0123>), or by emitting coordinates as plain text in JSON, often normalized to 0-1000 (Qwen-VL family). Training on box-annotated data (RefCOCO, Visual Genome, Objects365-derived, Grounding DINO pseudo-labels, Molmo’s pointing data) enables detection-like behavior from a generative model. Specialized detectors are still much more accurate and faster for dense detection (Object Detection); VLM grounding shines on open-ended, relational queries. Grounded output can feed SAM for masks (Segmentation).
# HuggingFace-style inference sketch (model names change often; check the Hub)from transformers import AutoProcessor, AutoModelForVision2Seqfrom PIL import Imageimport torch
name = "Qwen/Qwen2.5-VL-3B-Instruct" # example checkpoint; substitute a current oneproc = AutoProcessor.from_pretrained(name)model = AutoModelForVision2Seq.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Return the bounding box of the red mug as JSON."}]}]prompt = proc.apply_chat_template(msgs, add_generation_prompt=True)inputs = proc(text=[prompt], images=[Image.open("desk.jpg")], return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=128)print(proc.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])OCR and document understanding
Section titled “OCR and document understanding”Reading text in images is both a core use case and a stress test.
- Pipeline approach (classic): layout detection → OCR engine (Tesseract, PaddleOCR, cloud OCR) → text + layout → LLM. Robust, explainable and cheap, but errors cascade and visual context (charts, figures) is lost.
- OCR-free end-to-end: Donut, Pix2Struct, Nougat (papers to Markdown), Florence-2, and general VLMs read pixels directly to text/JSON/Markdown. Modern VLMs (Qwen-VL, InternVL, PaliGemma, GPT/Gemini/Claude-class models) are strong at documents, forms, tables, charts and screenshots, driven by large OCR-heavy training mixes and high-resolution encoders. Specialized OCR VLMs (compact models trained to emit Markdown/HTML tables from page images) appeared in 2025.
- Layout-aware document models (LayoutLM family) use text + 2D position + image; still used for form extraction with structured outputs.
Practical considerations: render PDFs at 150-300 DPI; split long documents by page; ask for structured output (JSON schema) and validate it; verify numbers against the source because VLMs can hallucinate plausible digits; for high-stakes extraction use a pipeline with checks (checksums, field validation) or dual-run with a second method.
Evaluation
Section titled “Evaluation”| Capability | Typical benchmarks |
|---|---|
| General VQA | VQAv2, GQA, MMBench, SEED-Bench, MME |
| Multi-discipline reasoning | MMMU, MathVista, MMStar |
| OCR / documents / charts | TextVQA, DocVQA, ChartQA, InfoVQA, OCRBench, AI2D |
| Grounding | RefCOCO/+/g, Visual Genome, pointing benchmarks |
| Hallucination | POPE, HallusionBench |
| Video | Video-MME, MVBench, EgoSchema |
| Long-context / multi-image | MMLongBench-Doc and similar |
Caveats you should internalize:
- Benchmark saturation and contamination: public test sets leak into training data; leaderboards move quickly and many are near ceiling. Build a private evaluation set of your own images and questions.
- Language priors: many VQA questions can be answered without the image. Test with blank/shuffled images to quantify.
- Metrics: exact-match, ANLS (for DocVQA), relaxed accuracy (ChartQA) and LLM-as-judge each have biases; judge models can favor verbose answers.
- Failure modes (still, in 2026): counting, precise spatial relations (“left of” with flipped viewpoint), reading tiny/rotated text, small-object detail, chart value reading, object hallucination (describing things that aren’t there), and sycophancy toward misleading prompts. Latency scales with visual token count.
Choosing between a VLM and a task-specific model
Section titled “Choosing between a VLM and a task-specific model”- Fixed label set, high throughput, tight latency, edge deployment → train a small detector/classifier (Efficient & Edge Vision). Use the VLM to auto-label.
- Open-ended questions, rare categories, document reading, low volume, rapid prototyping → VLM.
- Hybrid: a VLM routes/handles the hard tail; a fast model handles the bulk.
Key takeaways
Section titled “Key takeaways”- Contrastive models (CLIP/SigLIP) align image and text embeddings; generative VLMs feed visual tokens into an LLM via a connector (MLP, Q-Former, resampler, or cross-attention).
- Train in stages: connector alignment, then instruction tuning, then optional preference/RL tuning; data quality and mixture matter more than connector complexity.
- Resolution handling (AnyRes tiles, native-resolution ViTs, token merging) trades accuracy for tokens and latency; text legibility drives the requirement.
- Grounding is done through location tokens or coordinate text; OCR-free reading is now strong but numbers need verification.
- Evaluate on your own data; public benchmarks are contaminated, saturating and subject to language priors.
Exercises
Section titled “Exercises”- Conceptual. A ViT-L/14 at yields 576 tokens. With AnyRes tiles plus a thumbnail and no compression, how many visual tokens? With pixel-shuffle merging?
- Conceptual. Why mask the loss on image and prompt tokens during SFT?
- Hands-on. Take 30 screenshots with text of varying font size. Run a small open VLM on them at two input resolutions and plot accuracy of transcribed text versus text height in pixels.
- Hands-on. Implement the MLP connector, freeze a ViT and a small LLM, and verify that only the connector’s parameters appear in
optimizer.param_groups. - Conceptual. Give two ways to test whether a VQA benchmark can be solved from language priors alone.
Solutions (sketch)
- Five images worth of tokens (4 tiles + thumbnail) = ; with merge, .
- The model should learn to produce answers, not to predict (copy) the prompt; image tokens have no discrete targets in this setup.
- Expect a sharp drop below roughly 8-10 px text height at low resolution; higher resolution shifts the curve left.
- Set
requires_grad_(False)on the other modules; passfilter(lambda p: p.requires_grad, ...). - Evaluate with a blank/noise image or text-only LLM; compare with the image-conditioned score. Also shuffle images across questions.
Further reading
Section titled “Further reading”- Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” (CLIP, 2021).
- Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” (2022).
- Li et al., “BLIP-2” (2023).
- Liu et al., “Visual Instruction Tuning” (LLaVA, 2023); “Improved Baselines with Visual Instruction Tuning” (LLaVA-1.5, 2023); LLaVA-NeXT blog posts.
- Tong et al., “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs” (2024).
- McKinzie et al., “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training” (2024).
- Wang et al., “Qwen2-VL” (2024); Chen et al., InternVL technical reports.
- Beyer et al., “PaliGemma” (2024); Xiao et al., “Florence-2” (2023).
- Kim et al., “OCR-free Document Understanding Transformer” (Donut, 2021).
- Yue et al., “MMMU” (2023); Li et al., “POPE” (2023).