Skip to content

Vision-Language Models

What you’ll learn. The two generations of vision-language models (VLMs): contrastive dual encoders (CLIP, SigLIP) and generative “LLM with eyes” models (Flamingo, BLIP-2, LLaVA, and successors); the anatomy of a LLaVA-style VLM (vision encoder, connector, LLM); training stages and data; how resolution is handled (AnyRes/tiling, native-resolution encoders); grounding, OCR and document understanding; and how to evaluate VLMs and recognize their failure modes. Background: Self-Supervised & Foundation Vision Models, Vision Transformers, and the LLM side in Attention and Transformers.

  1. Contrastive dual encoders (CLIP, ALIGN, SigLIP): separate image and text towers, aligned embeddings, great for retrieval, zero-shot classification and as vision encoders. They cannot generate text or reason over multiple steps.
  2. Generative VLMs: an LLM consumes visual tokens plus text and generates text (captions, answers, structured output, even boxes as text). These are the “multimodal assistants” (GPT-4V-class, Gemini, Claude with vision, Qwen-VL, InternVL, LLaVA, PaliGemma, Pixtral, Llama-3.2/4 vision models, and others). The exact lineup changes every few months; the architecture pattern is stable.

CLIP trains a ViT and a text transformer with a symmetric InfoNCE loss on hundreds of millions to billions of image-text pairs (lecture 4 has the loss). For VLMs, the point is that the vision tower already produces features aligned to language, so it is a natural plug-in encoder. SigLIP-family encoders (sigmoid loss) are a common modern choice, with some systems using multiple encoders (e.g., a semantic encoder plus a DINOv2 encoder for fine spatial detail, as in Cambrian-1 and several others).

image ──> vision encoder (ViT: CLIP/SigLIP) ──> patch features (N x Dv)
│
connector / projector
▼
visual tokens (M x Dllm)
│
text prompt ──> tokenizer ──> text embeddings ──── concat ──> LLM (decoder-only) ──> answer tokens

Components:

  • Vision encoder: pretrained ViT, often frozen at first. Patch tokens from a late-but-not-last layer (CLIP’s penultimate layer worked better in LLaVA) preserve more spatial information than the pooled embedding.
  • Connector: maps DvD_v-dimensional visual features to the LLM embedding dimension. Choices:
    • Linear / 2-layer MLP projector (LLaVA-1.5): simple and surprisingly strong; keeps all NN patch tokens (576 for ViT-L/14 at 336px).
    • Q-Former (BLIP-2): a small transformer with learned queries cross-attending to the image features, compressing to a fixed number (e.g., 32) of tokens.
    • Perceiver resampler (Flamingo): similar latent-query compression.
    • Cross-attention layers inside the LLM (Flamingo, Llama-3.2 Vision): visual features are injected through gated cross-attention rather than as input tokens.
    • Pooling / pixel-shuffle token reduction: merge neighboring tokens (e.g., 2×2→12\times2 \to 1 with a channel concat) to cut token count 4x (InternVL, Qwen-VL).
  • LLM: a pretrained decoder (Vicuna → Llama, Qwen, Gemma, Mistral, …). Image tokens are placed at the position of an <image> placeholder.

A minimal projector in PyTorch:

import torch, torch.nn as nn
class MLPConnector(nn.Module):
def __init__(self, dv=1024, dllm=4096):
super().__init__()
self.net = nn.Sequential(nn.Linear(dv, dllm), nn.GELU(), nn.Linear(dllm, dllm))
def forward(self, feats): # (B, N, dv) -> (B, N, dllm)
return self.net(feats)
def build_inputs(text_emb_before, vis_tokens, text_emb_after):
return torch.cat([text_emb_before, vis_tokens, text_emb_after], dim=1) # fed to LLM as inputs_embeds

Cost. Visual tokens are expensive: with 576 tokens per image, each image consumes context and time like ~400 words of text, and the LLM’s attention/KV cache cost scales with them. Multi-image and video inputs multiply this, so token compression is a major design axis (Video, 3D & Spatial Vision).

Typical multi-stage training:

  1. Alignment pretraining: freeze the encoder and LLM, train only the connector on image-caption pairs (hundreds of thousands to millions). Cheap; teaches “what do these embeddings mean in LLM space”.
  2. Instruction tuning / SFT: unfreeze the LLM (and optionally the encoder, or use LoRA) on visual instruction data: VQA, conversation, reasoning, OCR, charts, grounding. LLaVA bootstrapped this with GPT-4-generated conversations from captions and boxes; later work is more data-heavy and more diverse.
  3. Optional preference optimization / RL (DPO-style, RLHF/RLVR) to reduce hallucination and improve reasoning; “thinking” VLMs trained with reinforcement learning on verifiable visual-math and chart tasks emerged in 2025.

The loss is ordinary next-token cross-entropy on the answer tokens only (image and prompt tokens masked out). Key empirical lessons across LLaVA-1.5, Cambrian, MM1, Idefics, Molmo and similar open work: data quality and mixture dominate (especially OCR/document and high-quality dense captions), the LLM size matters substantially, a stronger/higher-resolution encoder helps, and the connector can be simple. Unfreezing the encoder helps for fine-grained tasks but risks destabilization.

Some models skip the separate encoder and train from scratch natively multimodal (Gemini-style, Chameleon-style early fusion, Fuyu/Emu-style direct patch input). Treat the “encoder + connector + LLM” recipe as the open, reproducible default.

A fixed 3362336^2 or 2242224^2 input destroys small text and fine detail. Strategies:

  • AnyRes / tiling (LLaVA-NeXT, InternVL, many others): choose a grid (e.g., 2×22\times2) matching the aspect ratio, split the image into tiles each encoded at the base resolution, plus a downscaled global thumbnail; concatenate all tile tokens. Token cost grows with the number of tiles.
  • Native-resolution / dynamic-resolution encoders (Qwen2-VL and later, NaViT-style, SigLIP 2 NaFlex): the ViT accepts any size with 2D rotary position encodings; token count = (H/p)*(W/p) after merging. Aspect ratios are preserved, no tiling artifacts.
  • Token compression: pixel-shuffle merging, resamplers, learned token pruning (drop background tokens) to keep contexts manageable.

Rule of thumb: tokens ≈HW(p s)2\approx \frac{H W}{(p\,s)^2} for patch size pp and merge factor ss. A 1000x1000 document at p=14p=14, s=2s=2 is about 1,270 tokens. Choose the minimum resolution at which the required details (text height in pixels) remain legible: roughly 10+ px per character height for reliable OCR.

Grounding links language to image regions: referring expression comprehension (box for “the red mug left of the laptop”), phrase grounding (boxes for noun phrases in a caption), and pointing (return coordinates). VLMs implement this either with special location tokens (PaliGemma, Florence-2 discretize coordinates into tokens like <loc_0123>), or by emitting coordinates as plain text in JSON, often normalized to 0-1000 (Qwen-VL family). Training on box-annotated data (RefCOCO, Visual Genome, Objects365-derived, Grounding DINO pseudo-labels, Molmo’s pointing data) enables detection-like behavior from a generative model. Specialized detectors are still much more accurate and faster for dense detection (Object Detection); VLM grounding shines on open-ended, relational queries. Grounded output can feed SAM for masks (Segmentation).

# HuggingFace-style inference sketch (model names change often; check the Hub)
from transformers import AutoProcessor, AutoModelForVision2Seq
from PIL import Image
import torch
name = "Qwen/Qwen2.5-VL-3B-Instruct" # example checkpoint; substitute a current one
proc = AutoProcessor.from_pretrained(name)
model = AutoModelForVision2Seq.from_pretrained(name, torch_dtype=torch.bfloat16, device_map="auto")
msgs = [{"role": "user", "content": [
{"type": "image"},
{"type": "text", "text": "Return the bounding box of the red mug as JSON."}]}]
prompt = proc.apply_chat_template(msgs, add_generation_prompt=True)
inputs = proc(text=[prompt], images=[Image.open("desk.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128)
print(proc.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Reading text in images is both a core use case and a stress test.

  • Pipeline approach (classic): layout detection → OCR engine (Tesseract, PaddleOCR, cloud OCR) → text + layout → LLM. Robust, explainable and cheap, but errors cascade and visual context (charts, figures) is lost.
  • OCR-free end-to-end: Donut, Pix2Struct, Nougat (papers to Markdown), Florence-2, and general VLMs read pixels directly to text/JSON/Markdown. Modern VLMs (Qwen-VL, InternVL, PaliGemma, GPT/Gemini/Claude-class models) are strong at documents, forms, tables, charts and screenshots, driven by large OCR-heavy training mixes and high-resolution encoders. Specialized OCR VLMs (compact models trained to emit Markdown/HTML tables from page images) appeared in 2025.
  • Layout-aware document models (LayoutLM family) use text + 2D position + image; still used for form extraction with structured outputs.

Practical considerations: render PDFs at 150-300 DPI; split long documents by page; ask for structured output (JSON schema) and validate it; verify numbers against the source because VLMs can hallucinate plausible digits; for high-stakes extraction use a pipeline with checks (checksums, field validation) or dual-run with a second method.

Capability Typical benchmarks
General VQA VQAv2, GQA, MMBench, SEED-Bench, MME
Multi-discipline reasoning MMMU, MathVista, MMStar
OCR / documents / charts TextVQA, DocVQA, ChartQA, InfoVQA, OCRBench, AI2D
Grounding RefCOCO/+/g, Visual Genome, pointing benchmarks
Hallucination POPE, HallusionBench
Video Video-MME, MVBench, EgoSchema
Long-context / multi-image MMLongBench-Doc and similar

Caveats you should internalize:

  • Benchmark saturation and contamination: public test sets leak into training data; leaderboards move quickly and many are near ceiling. Build a private evaluation set of your own images and questions.
  • Language priors: many VQA questions can be answered without the image. Test with blank/shuffled images to quantify.
  • Metrics: exact-match, ANLS (for DocVQA), relaxed accuracy (ChartQA) and LLM-as-judge each have biases; judge models can favor verbose answers.
  • Failure modes (still, in 2026): counting, precise spatial relations (“left of” with flipped viewpoint), reading tiny/rotated text, small-object detail, chart value reading, object hallucination (describing things that aren’t there), and sycophancy toward misleading prompts. Latency scales with visual token count.

Choosing between a VLM and a task-specific model

Section titled “Choosing between a VLM and a task-specific model”
  • Fixed label set, high throughput, tight latency, edge deployment → train a small detector/classifier (Efficient & Edge Vision). Use the VLM to auto-label.
  • Open-ended questions, rare categories, document reading, low volume, rapid prototyping → VLM.
  • Hybrid: a VLM routes/handles the hard tail; a fast model handles the bulk.
  • Contrastive models (CLIP/SigLIP) align image and text embeddings; generative VLMs feed visual tokens into an LLM via a connector (MLP, Q-Former, resampler, or cross-attention).
  • Train in stages: connector alignment, then instruction tuning, then optional preference/RL tuning; data quality and mixture matter more than connector complexity.
  • Resolution handling (AnyRes tiles, native-resolution ViTs, token merging) trades accuracy for tokens and latency; text legibility drives the requirement.
  • Grounding is done through location tokens or coordinate text; OCR-free reading is now strong but numbers need verification.
  • Evaluate on your own data; public benchmarks are contaminated, saturating and subject to language priors.
  1. Conceptual. A ViT-L/14 at 3362336^2 yields 576 tokens. With 2×22\times2 AnyRes tiles plus a thumbnail and no compression, how many visual tokens? With 2×22\times2 pixel-shuffle merging?
  2. Conceptual. Why mask the loss on image and prompt tokens during SFT?
  3. Hands-on. Take 30 screenshots with text of varying font size. Run a small open VLM on them at two input resolutions and plot accuracy of transcribed text versus text height in pixels.
  4. Hands-on. Implement the MLP connector, freeze a ViT and a small LLM, and verify that only the connector’s parameters appear in optimizer.param_groups.
  5. Conceptual. Give two ways to test whether a VQA benchmark can be solved from language priors alone.
Solutions (sketch)
  1. Five images worth of tokens (4 tiles + thumbnail) = 5×576=28805 \times 576 = 2880; with 2×22\times2 merge, 720720.
  2. The model should learn to produce answers, not to predict (copy) the prompt; image tokens have no discrete targets in this setup.
  3. Expect a sharp drop below roughly 8-10 px text height at low resolution; higher resolution shifts the curve left.
  4. Set requires_grad_(False) on the other modules; pass filter(lambda p: p.requires_grad, ...).
  5. Evaluate with a blank/noise image or text-only LLM; compare with the image-conditioned score. Also shuffle images across questions.
  • Radford et al., “Learning Transferable Visual Models From Natural Language Supervision” (CLIP, 2021).
  • Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning” (2022).
  • Li et al., “BLIP-2” (2023).
  • Liu et al., “Visual Instruction Tuning” (LLaVA, 2023); “Improved Baselines with Visual Instruction Tuning” (LLaVA-1.5, 2023); LLaVA-NeXT blog posts.
  • Tong et al., “Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs” (2024).
  • McKinzie et al., “MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training” (2024).
  • Wang et al., “Qwen2-VL” (2024); Chen et al., InternVL technical reports.
  • Beyer et al., “PaliGemma” (2024); Xiao et al., “Florence-2” (2023).
  • Kim et al., “OCR-free Document Understanding Transformer” (Donut, 2021).
  • Yue et al., “MMMU” (2023); Li et al., “POPE” (2023).