Visual Intelligence in Claude: Analyzing PDFs, Charts, and Documents

Claude can analyze images and PDFs directly through the Anthropic API. Send an image (PNG, JPEG, GIF, or WebP) as an image content block, or send a PDF as a document block, and Claude reads both the text and the visual content on each page, including charts, tables, and figures. With citations enabled, Claude also tells you exactly which pages each part of its answer came from.

Claude is especially useful for tasks that need interpretation and explanation: summarizing a research paper, explaining what a chart shows, or checking whether a document's figures support its claims. It's less suited to pure perception tasks like counting many small objects or real-time video analysis.

In this guide, we build a scientific paper analyzer that uploads a PDF, sends it to Claude, and returns a structured, page-cited analysis. Along the way, we cover how the API handles images and PDFs, how to design prompts for structured extraction, and when to use Claude for visual tasks.

How Claude Handles Visual Content

Claude reads images and documents in the same conversation as your text instructions, so it can connect what it sees with what you ask. When Claude looks at a chart, it doesn't just identify a "bar chart with five bars." It reads the axis labels, understands what's being measured, interprets the relationships between values, and explains what the chart means in context.

In practice, this shows up in three ways when building applications:

1. Contextual interpretation over isolated recognition. Claude connects visual elements to their surroundings. A figure, its caption, and the text that references it are understood as a unit. This is why Claude works well on academic papers, where figures support methodology described elsewhere in the document.
2. Explanation quality. Ask Claude what a diagram shows, and you'll get an explanation suitable for teaching someone the concept, not just a list of shapes and labels. This makes Claude valuable for educational applications, document analysis, and any task where people need to understand the output.
3. Structured reasoning about visual content. Claude can follow complex instructions for analyzing visual content: compare two charts, identify inconsistencies between the text and the figures, or extract only the statistical claims.

Under the hood, Claude processes images as a grid of 28×28-pixel patches, and it handles PDFs by converting each page into an image and pairing it with the page's extracted text. Claude analyzes both together, which is why it can answer questions about a chart and the paragraph that describes it in the same response.

These strengths come with tradeoffs. According to Anthropic's documentation, Claude's object counts are approximate, its spatial and coordinate outputs aren't exact, and it can make mistakes on low-quality, rotated, or very small images. It also processes discrete images and documents, not live video streams. Understanding these boundaries helps you choose the right tool for each task.

Getting Started with Claude

Claude's vision and PDF capabilities use the same Messages API as text-only requests. Install the SDK:

npm install @anthropic-ai/sdk

You'll need an API key from the Claude Console. Set your API key as an environment variable:

export ANTHROPIC_API_KEY=your-api-key-here

The SDK automatically reads this environment variable, so you don't need to pass the key explicitly in your code:

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();
// API key is read from ANTHROPIC_API_KEY environment variable

This pattern keeps credentials out of your source code, which matters for version control and deployment security.

Sending Images

Claude accepts images in four formats: PNG, JPEG, GIF, and WebP. You can provide them in three ways:

1. Base64 encoding embeds the image data directly in your request. This is the most common approach for server-side applications that process uploaded files.
2. URL references point to publicly accessible images, which keeps requests small.
3. File IDs reference images you've uploaded once with the Files API, which is useful when you'll ask about the same image more than once.

Here's the minimal structure for sending an image to Claude:

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

const response = await anthropic.messages.create({
  model: "claude-opus-5-5",
  max_tokens: 4096,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image",
          source: {
            type: "base64",
            media_type: "image/png", // or "image/jpeg", "image/gif", "image/webp"
            data: base64ImageData, // raw base64, without a data URL prefix
          },
        },
        {
          type: "text",
          text: "Describe what you see in this image.",
        },
      ],
    },
  ],
});

console.log(response.content[0].text);

Notice that the content field is an array, not a string. It can contain multiple images, documents, and text blocks. Anthropic recommends placing images and documents before your text instructions, so Claude has the full visual context before it reads your question.

Sending PDFs

You don't need to convert PDFs to images first. Send the file as a document block, and Claude processes every page:

const response = await anthropic.messages.create({
  model: "claude-opus-5-5",
  max_tokens: 4096,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "document",
          source: {
            type: "base64",
            media_type: "application/pdf",
            data: base64PdfData,
          },
        },
        {
          type: "text",
          text: "What are the key findings in this document?",
        },
      ],
    },
  ],
});

Like images, PDFs can also be sent by URL or by a Files API file_id.

Limits and Costs

These are the main limits to plan around, from Anthropic's vision and PDF support documentation:

Images are billed by visual tokens, where each 28×28-pixel patch is one token. Larger images are automatically downscaled before processing: Claude 4.7 and later models accept images up to 2,576 px on the long edge (up to about 4,800 tokens), while earlier models downscale to 1,568 px (about 1,600 tokens). PDF pages are billed for both their extracted text (typically 1,500 to 3,000 tokens per page) and their page image. Use Anthropic's token counting endpoint to estimate costs for your own documents.

Building our Scientific Paper Analyzer

Let's build a complete application that accepts PDF uploads, analyzes them with Claude, and returns a structured analysis with page citations. This demonstrates patterns you'll use in production document processing.

The paper analyzer has three layers:

1. Web interface: A drag-and-drop upload zone with progress feedback
2. Express server: Handles file uploads and orchestrates the analysis
3. Analysis pipeline: The Claude API call and citation formatting

Here's how a request flows through the system:

User uploads PDF → Server receives file in memory →
PDF sent to Claude as a document → Analysis with page citations returned →
Results displayed to user

Because Claude reads PDFs natively, we need just three dependencies:

{
  "dependencies": {
    "@anthropic-ai/sdk": "^1.2.0",
    "express": "^5.3.0",
    "multer": "^2.5.0"
  }
}

Each serves a specific purpose:

• @anthropic-ai/sdk: Official Claude API client with TypeScript support and automatic retries
• express: Web server framework for handling HTTP requests and serving static files
• multer: Middleware for handling multipart form data (file uploads)

Sending the Paper to Claude With Citations

The analysis function sends the whole PDF in a single request, so Claude can reason across the entire paper, connecting figures to their descriptions, methodology to results, and claims to evidence. We also turn on citations, so every finding points back to the page it came from.

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

async function analyzePaper(pdfBuffer, fileName) {
  const response = await anthropic.messages.create({
    model: "claude-opus-5-5",
    max_tokens: 8000,
    system: systemPrompt, // Defined in the next section
    messages: [
      {
        role: "user",
        content: [
          {
            type: "document",
            source: {
              type: "base64",
              media_type: "application/pdf",
              data: pdfBuffer.toString("base64"),
            },
            title: fileName,
            citations: { enabled: true },
          },
          {
            type: "text",
            text: "Please analyze this scientific paper comprehensively. Examine all pages, figures, tables, and sections to provide a thorough analysis following the structured format.",
          },
        ],
      },
    ],
  });

  return formatWithCitations(response.content);
}

Let's break down the key parts of this function:

• The document block carries the raw PDF. There's no conversion step, no rendering library, and no page cap beyond the API's limits, so the analyzer handles full-length papers.
• Document before instructions. Placing the PDF ahead of the text prompt gives Claude the complete paper before it reads your request. This lets you reference "the methodology section" or "Figure 3" in your prompt, and Claude will know what you mean.
• citations: { enabled: true } tells Claude to ground its answer in the document. The title field labels the source in each citation.
• The token budget. max_tokens: 8000 leaves room for a detailed analysis covering methodology, results, figures, and implications. If you're getting truncated responses, increase this value.

With citations enabled, Claude's response arrives as a series of text blocks, and the blocks that make a claim about the paper include a citations array. For PDFs, each citation identifies the page range it came from:

{
  "type": "page_location",
  "cited_text": "The exact text being cited",
  "document_index": 0,
  "document_title": "internvl3.pdf",
  "start_page_number": 5,
  "end_page_number": 6
}

Page numbers are 1-indexed, and end_page_number is exclusive, so the example above refers to page 5 only. The cited_text field doesn't count toward your output tokens. A small helper turns these blocks into readable text with page references:

function formatWithCitations(content) {
  return content
    .filter((block) => block.type === "text")
    .map((block) => {
      const pages = [
        ...new Set(
          (block.citations ?? [])
            .filter((citation) => citation.type === "page_location")
            .map((citation) => citation.start_page_number)
        ),
      ];
      return pages.length > 0
        ? `${block.text} [p. ${pages.join(", ")}]`
        : block.text;
    })
    .join("");
}

Two limitations are worth knowing. Citations point to text, not images, so a finding about a figure is cited to the text on the page around it, and scanned PDFs without extractable text can't be cited. Citations are also incompatible with structured outputs: if you need Claude to return strict JSON instead, turn citations off and use structured outputs.

When should you still convert pages to images? Rendering pages yourself is useful when you only want to send selected pages, crop a dense figure or table so it fills the full image resolution, or preprocess scans before analysis. For most document workflows, the native document block is simpler and gives Claude the page text as well as the page image.

Analyzing the same paper repeatedly? Upload it once with the Files API and reference it by file_id, and use prompt caching so follow-up questions don't reprocess the whole document.

How to Guide Visual Analysis Through the System Prompt

The system prompt is where you shape Claude's analytical approach. For document analysis, explicit structure produces much more consistent results than open-ended requests.

const systemPrompt = `You are an expert scientific paper analyst. Your role is to provide comprehensive, structured analysis of academic papers, including their text, figures, and tables.

Analyze the paper thoroughly and provide your response in the following structured format:

## Paper Overview
Provide title, authors, institution, and publication venue if visible.

## Abstract Summary
A concise plain-English summary of what the paper is about (2-3 sentences).

## Key Contributions
Bullet points of the paper's main contributions to the field.

## Methodology
Step-by-step breakdown of the research approach:
- Study design
- Data collection methods
- Analysis techniques
- Tools/frameworks used

## Key Findings
The main results with relevant statistics, organized as bullet points.

## Figures & Tables Analysis
For each significant figure or table:
- What it shows
- Key takeaways
- How it supports the paper's claims

## Technical Concepts Explained
Define and explain key technical terms and concepts from the paper in accessible language.

## Strengths
What the paper does well.

## Limitations
Potential weaknesses, gaps, or concerns with the methodology or conclusions.

## Practical Implications
How these findings might be applied in the real world.

## TL;DR
A single paragraph summary suitable for someone with general scientific literacy but not domain expertise.

Be thorough but concise. Use clear formatting with headers and bullet points. If a figure or section is unclear, say so rather than guessing at its contents.`;

This prompt incorporates several techniques that improve output quality:

• Role establishment ("expert scientific paper analyst") primes Claude's response style and vocabulary.
• Explicit section structure ensures a consistent output format across different papers. Every analysis has the same sections in the same order, making results predictable for downstream processing or display.
• Nested formatting guidance (the bullet points under Methodology) shows Claude how detailed you want each section. Without this, you might get single-sentence sections or sprawling paragraphs.
• Interpretation requests ("How it supports the paper's claims") push Claude beyond description into analysis. This engages its reasoning capabilities rather than treating it as an OCR tool.
• Uncertainty instructions ("say so rather than guessing") reduce the risk of Claude inventing details for figures it can't read clearly. Combined with page citations, they make the output much easier to verify.

The Analysis Endpoint

The server uses Express with multer to handle PDF uploads. Because Claude reads the PDF directly, multer can keep the upload in memory, so there are no temporary files to clean up. The /api/analyze endpoint passes the file to analyzePaper():

import express from "express";
import multer from "multer";

const app = express();
const upload = multer({
  storage: multer.memoryStorage(),
  limits: { fileSize: 20 * 1024 * 1024 }, // stay under the 32 MB request limit after base64 encoding
});

app.post("/api/analyze", upload.single("paper"), async (req, res) => {
  if (!req.file || req.file.mimetype !== "application/pdf") {
    return res.status(400).json({ error: "Please upload a PDF file" });
  }

  try {
    const analysis = await analyzePaper(req.file.buffer, req.file.originalname);
    res.json({ success: true, analysis });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

app.use(express.static("public"));
app.listen(3000, () => console.log("Paper analyzer running on http://localhost:3000"));

The flow is straightforward: receive the PDF, send it to Claude with citations enabled, and return the formatted analysis. The 20 MB upload limit leaves headroom for base64 encoding, which makes the request about a third larger than the file. For larger documents, upload them with the Files API instead.

The Paper Analyzer in Action

Let's see what this system produces when analyzing a real scientific paper: InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. It's a good test because it has a lot of technical detail, plus plenty of graphs and tables:

All we need to do is upload it to our analyzer, and the flow starts automatically:

The screenshots and output in this section are from our original run of the analyzer, which sent the paper's pages to Claude as images. The version above sends the PDF directly and adds page citations to each finding, but the analysis follows the same structure.

The output starts with the authors and affiliations:

But that isn't really what we're interested in here: it is straightforward text extraction. What we want to know is what Claude made of the figures in the paper, like this one:

Here's Claude's analysis of that figure:

Figure 2: OpenCompass Leaderboard Scatter Plot

What it shows: Parameters (x-axis) vs. OpenCompass Average Score (y-axis) for ~20 MLLMs
Key takeaways:
* InternVL3-78B reaches ~78 score, near Gemini-2.5-Pro
* Clear scaling law visible across model sizes
* InternVL3 series shows efficient scaling (high performance per parameter)
* Significant gap between InternVL3 and previous InternVL versions

Support for claims: Demonstrates competitive positioning against both open and closed-source models

These observations match what the figure shows, and they come from reading the chart itself. The overall analysis demonstrates several of Claude's visual reasoning strengths:

• Cross-reference understanding: Claude connects figures to their descriptions in the text, explaining not just what a chart shows but how it supports specific claims made elsewhere in the paper.
• Technical term extraction: Domain-specific terminology is identified and explained in accessible language, making the analysis useful for readers outside the paper's immediate field.
• Critical evaluation: The Strengths and Limitations sections show Claude's ability to assess methodology, not just describe it.
• Hierarchical structure: Information is organized from a high-level summary (TL;DR, Abstract Summary) to detailed examination (Figure Analysis, Technical Concepts), accommodating different reader needs.

Effective Prompt Engineering for Visual Tasks

Prompts for visual analysis differ from text-only prompts. The visual modality introduces ambiguity (what should Claude focus on?), scale challenges (images contain far more information than equivalent text), and format considerations (how should visual insights be expressed?).

Request Interpretation, Not Description

Claude's strength lies in understanding what visual content means, not just cataloging what it contains. Compare these two prompts:

• Weak: "List all the figures in this paper."
• Strong: "For each figure, explain what it demonstrates and how it supports the paper's central argument."

The first prompt treats Claude as an inventory system. The second engages its reasoning capabilities, producing output that's actually useful for understanding the paper.

This principle extends to all visual tasks:

Provide the Output Structure Upfront

Visual content is inherently less structured than text. A page might contain headers, body text, figures, captions, footnotes, and marginalia, all competing for attention. Giving Claude a clear output template produces more consistent results:

const structuredPrompt = `Analyze this financial chart and provide:

## Trend Summary
One sentence describing the overall direction.

## Key Data Points
- Highest value and when it occurred
- Lowest value and when it occurred
- Current value

## Notable Patterns
Any cycles, anomalies, or inflection points visible.

## Interpretation
What this trend suggests about the underlying subject.`;

Without structure, Claude might provide a wandering analysis that's hard to parse programmatically or display consistently. With structure, every response follows the same format, making downstream processing predictable. If your application needs machine-readable output, use structured outputs to enforce a JSON schema instead of a Markdown template.

Handle Uncertainty Explicitly

Documents often contain partially visible content, low-resolution figures, or ambiguous elements. Tell Claude how to handle these situations:

const uncertaintyAwarePrompt = `Analyze this technical diagram.

If any labels or components are unclear:
- Note what you can partially see
- Explain what typically appears in that position
- Indicate your confidence level

Do not guess at illegible text. Instead, describe its apparent purpose based on context.`;

This instruction reduces hallucination while still extracting value from imperfect inputs. Instead of confidently stating an incorrect value, Claude is more likely to say something like, "The axis label appears to show units of measurement, possibly milliseconds, but the text is not fully legible." It's not a guarantee, so verify important values against the source.

Balance Scope and Depth

For multi-page documents, decide whether you need comprehensive coverage or focused extraction. Comprehensive prompts produce longer, more expensive responses:

// Comprehensive: covers everything, higher cost
const comprehensivePrompt = `Analyze all aspects of this research paper:
methodology, results, figures, implications, and limitations.`;

// Focused: specific extraction, lower cost
const focusedPrompt = `Extract only the statistical claims from this paper.
For each claim, note the specific numbers, sample size, and p-values if provided.`;

Focused prompts are particularly useful when you're building pipelines that process large volumes of documents. You might run a low-cost, focused extraction first, then perform a comprehensive analysis only on documents that meet specific criteria. For high volumes, Anthropic's Message Batches API processes requests asynchronously at a lower cost.

When Should You Use Claude for Visual Tasks?

Claude is exceptionally good at some visual tasks and less suited to others. Understanding these boundaries helps you choose the right tool.

Strong Fits

• Academic papers and technical documentation: Claude excels at understanding relationships between sections. It can link a methodology description to its corresponding figure, identify when results don't support the stated conclusions, and explain technical concepts in accessible terms.
• Charts and graphs requiring interpretation: Beyond reading values, Claude explains trends, identifies anomalies, and puts data in context. A financial analyst might use it to generate first-draft interpretations of quarterly reports.
• Diagrams with labeled components: Flowcharts, architecture diagrams, process flows, and system schematics play to Claude's strength in sequential reasoning. It can walk through a diagram step by step, explaining how components interact.
• Educational content: Claude's explanation quality makes it valuable for analyzing textbooks, lecture slides, and instructional materials.
• Forms and structured documents: When context affects interpretation (which it usually does), Claude understands that a "Date" field in a medical form means something different than in a shipping label.

Weaker Fits

• Real-time video analysis: Claude processes discrete images and documents, not video streams. Applications requiring sub-second latency or continuous processing need specialized real-time video models.
• Fine-grained object detection and counting: Tasks like "count every person in this crowd photo" or "identify all instances of this component in a circuit board" are better handled by detection-focused models.
• Precise spatial reasoning: Claude can return coordinates and bounding boxes, but Anthropic notes they're approximate. Verify them before relying on them for tasks like measurement or precise localization.
• High-throughput image processing: When you need to process thousands of images per minute, the API latency and token costs of Claude may be prohibitive compared to specialized classification models.

For many applications, the best solution combines Claude with specialized tools:

• Use object detection to identify regions of interest, then Claude to analyze those regions
• Use a dedicated OCR engine for very high volumes of simple text extraction, then Claude to interpret the results
• Use Claude for initial understanding, then specialized models for specific extraction tasks

Frequently Asked Questions

1. Can Claude read PDFs directly?

Yes. Send the PDF as a document content block, encoded as base64, by URL, or by a Files API file_id. Claude converts each page into an image and pairs it with the page's extracted text, so it can analyze charts, tables, and figures as well as the text. A single request can include up to 600 pages (100 when the request's context window is under 1M tokens), within a 32 MB request size limit.

2. What's the best way to prompt Claude for layout-aware reasoning?

Provide explicit output structure in your prompt and request interpretation rather than description. For example, instead of "list the figures," ask "explain how each figure supports the paper's argument." Include section headers in your prompt template so Claude organizes its analysis to match the document's logical structure. Place images and documents before your text instructions so Claude has full context before responding.

3. When is Claude preferable to OpenAI or Gemini for structured tasks?

Claude is a strong choice when you need explanation quality and cross-reference understanding: academic papers, technical documentation, charts requiring interpretation, and any task where people need to understand the output. Its page-level citations for PDFs also make its answers easy to verify. Choose other models when you need real-time video processing, fine-grained object detection, or high-throughput image classification, where reasoning depth matters less than speed.

4. How do you extract structured data from charts or tables using Claude?

Send the image or PDF with a prompt that specifies both the data format you want and the interpretation you need. For example: "Extract the data from this table as JSON, then explain any trends or anomalies in the values." To guarantee valid JSON, use structured outputs with a schema. For dense tables, crop the table so it fills the image instead of increasing the resolution of the whole page, because Claude downscales any image larger than its maximum resolution.

5. How can citations and self-explanations improve validation and debugging?

Turn on citations for documents so each finding points to the page it came from, and ask Claude to explain its reasoning alongside its conclusions, for example, "indicate your confidence level for any partially visible content." This creates an audit trail: when the output appears incorrect, you can check the cited page and see whether Claude misread the content, drew an incorrect inference, or relied on incomplete information.

Use Claude for Visual Reasoning, Not Just Perception

Building effective document analyzers with Claude comes down to its distinctive strengths: contextual reasoning across visual and textual content, high-quality explanations, and controllable structured output. The paper analyzer demonstrates the core patterns you'll use in production: native PDF input, page-level citations, structured prompts, and robust error handling.

The key insight is to treat Claude as a reasoning system that accepts visual input. Design prompts that engage reasoning (interpret, explain, evaluate) rather than perception alone (list, describe, identify), and structure your output requests explicitly to get consistent, parseable results.

When you need a genuine understanding of what documents mean, not just the extraction of what they contain, Claude is a powerful tool. Combined with citations for verification, it enables document workflows that previously required a person to read every page.