Visual Intelligence in Claude: Interpreting Documents and Structured Content

Claude isn’t the model most users turn to when needing visual capabilities. Rather than optimizing primarily for object detection or scene description, Claude processes visual content through the same reasoning architecture it uses for text. This design choice has significant implications for developers: Claude excels at tasks requiring interpretation and explanation rather than pure perception.

Here, we show how this works in practice by working through an example: a scientific paper analyzer that leverages Claude’s strengths. This will show how visual reasoning works in Claude, how the Anthropic API handles images, how to design prompts for structured extraction, and how to address the practical challenges of document processing.

Our goal is to help developers understand not just the mechanics of Claude for visual tasks, but when and why to use it.

Understanding Claude's Visual Reasoning

Most vision models are trained with perception as the primary objective: identify objects, segment regions, classify scenes. Language capabilities are then layered on top to describe what the model perceives. This is how the true multimodal GPT and Gemini families of models work.

Claude reverses this priority. It is a language model with visual perception integrated into that framework. When Claude looks at a chart, it doesn't just identify "bar chart with 5 bars." It reads axis labels, understands what's being measured, interprets relationships between values, and explains the chart's meaning in context.

This architecture produces three practical differences you'll notice when building applications:

1. Contextual interpretation over isolated recognition. Claude connects visual elements to their surroundings. A figure caption, the surrounding text, and the figure itself are understood as a unit. This is why Claude performs well on academic papers where figures reference methodology described elsewhere in the document.
2. Explanation quality. Ask Claude what a diagram shows, and you'll get an explanation suitable for teaching someone the concept, not just a description of shapes and labels. This makes Claude particularly valuable for educational applications, document analysis, and any task where humans must understand the output.
3. Structured reasoning about visual content. Claude can follow complex instructions for analyzing visual content: compare these two charts, identify inconsistencies between the text and figures, and extract only the statistical claims. This controllability stems from its language-reasoning capabilities.

These strengths come with tradeoffs. Claude isn't optimized for tasks like counting large numbers of small objects, real-time video processing, or fine-grained spatial reasoning. Understanding these boundaries helps you choose the right tool for each task.

The clearest takeaway is that Claude is a reasoning-and-generation model with vision capabilities, rather than a defined multimodal representation of vision.

Getting Started with Claude

Claude is available through the same API as text-only Claude.

1
npm install @anthropic-ai/sdk

You'll need an API key from the Anthropic Console. Set your API key as an environment variable:

1
export ANTHROPIC_API_KEY=your-api-key-here

The SDK automatically reads this environment variable, so you don't need to pass the key explicitly in your code:

1
2
3
4
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();
// API key is read from ANTHROPIC_API_KEY environment variable

This pattern keeps credentials out of your source code, which matters for version control and deployment security.

Supported Image Formats

Claude accepts images in four formats: PNG, JPEG, GIF, and WebP. You can provide images in two ways:

Base64 encoding embeds the image data directly in your API request. This is the most common approach for server-side applications processing uploaded files:

1
2
3
4
5
6
7
8
{
  type: 'image',
  source: {
    type: 'base64',
    media_type: 'image/png',  // or 'image/jpeg', 'image/gif', 'image/webp'
    data: base64EncodedString   // the image data without data URL prefix
  }
}

URL references point to publicly accessible images. This reduces request size but requires the image to be hosted somewhere Claude can fetch it:

1
2
3
4
5
6
7
{
  type: 'image',
  source: {
    type: 'url',
    url: 'https://example.com/image.png'
  }
}

For document processing workflows, base64 is typically more practical. You're usually working with uploaded files or images generated from PDFs rather than pre-hosted content.

What about PDFs? Claude doesn't accept PDF files directly through the API. You'll need to convert each page to an image first. This adds a processing step but gives you control over resolution and page selection. We'll cover this conversion in the paper analyzer we build below.

Here's the minimal structure for sending an image to Claude:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
import Anthropic from '@anthropic-ai/sdk';

const anthropic = new Anthropic();

const response = await anthropic.messages.create({
  model: 'claude-opus-4-5-20251101',
  max_tokens: 4096,
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'image',
          source: {
            type: 'base64',
            media_type: 'image/png',
            data: base64ImageData
          }
        },
        {
          type: 'text',
          text: 'Describe what you see in this image.'
        }
      ]
    }
  ]
});

console.log(response.content[0].text);

Notice that the content field is an array, not a string. This array can contain multiple images and text blocks in any order. Claude processes them sequentially, building context as it goes. This is essential for multi-page document analysis where you want Claude to reason across an entire document.

The response structure mirrors other Claude API calls. The analysis appears in response.content[0].text for single text responses. For longer interactions, you might receive multiple content blocks.

Building our Scientific Paper Analyzer

Let's build a complete application that accepts PDF uploads, converts them to images, analyzes them with Claude, and returns structured insights. This demonstrates real-world patterns you'll use in production document processing.

The paper analyzer has three layers:

1. Web interface: A drag-and-drop upload zone with progress feedback
2. Express server: Handles file uploads, orchestrates processing, and manages cleanup
3. Analysis pipeline: PDF conversion, Claude API calls, response formatting

Here's how a request flows through the system:

User uploads PDF → Server saves file → PDF converted to images → 
Images sent to Claude → Analysis returned → File cleaned up → 
Results displayed to user

Because Claude is doing all the heavy lifting, we need just four main dependencies:

1
2
3
4
5
6
7
8
{
  "dependencies": {
    "@anthropic-ai/sdk": "^0.52.0",
    "express": "^4.18.2",
    "multer": "^1.4.5-lts.1",
    "pdf-to-img": "^4.2.0"
  }
}

Each serves a specific purpose:

• @anthropic-ai/sdk: Official Claude API client with TypeScript support and automatic retries
• express: Web server framework for handling HTTP requests and serving static files
• multer: Middleware for handling multipart form data (file uploads)
• pdf-to-img: Converts PDF pages to PNG images using the pdf.js library

Converting PDFs to Images

Since Claude accepts images but not PDFs directly, we need a conversion step. The pdf-to-img library wraps Mozilla's pdf.js to render each page as a PNG image.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import { pdf } from 'pdf-to-img';

async function convertPdfToImages(pdfPath, maxPages = 10) {
  const images = [];
  let pageCount = 0;

  // Initialize the PDF document with rendering options
  const document = await pdf(pdfPath, { scale: 2.0 });

  // Iterate through pages as an async generator
  for await (const image of document) {
    if (pageCount >= maxPages) break;

    // Convert the PNG buffer to base64 for the API
    const base64 = image.toString('base64');
    images.push(base64);
    pageCount++;
  }

  return images;
}

Let's break down the key parts of this function:

• The scale parameter controls rendering resolution; 2.0 works well for most documents, while dense tables might benefit from 3.0.
• The maxPages limit prevents runaway processing since each page consumes roughly 1,000-1,500 API tokens.
• Async iteration (for await...of) streams pages rather than loading everything into memory at once.
• Base64 encoding is required by the Claude API; note this is raw base64 data, not a data URL with the data:image/png;base64, prefix.

After converting pages to images, we assemble them into a single API request. This allows Claude to reason across the entire document, connecting figures to their descriptions, methodology to results, and claims to evidence.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
async function analyzePaper(images) {
  // Build an array of image content blocks
  const imageContent = images.map((base64) => ({
    type: 'image',
    source: {
      type: 'base64',
      media_type: 'image/png',
      data: base64
    }
  }));

  const response = await anthropic.messages.create({
    model: 'claude-opus-4-5-20251101',
    max_tokens: 8000,
    system: systemPrompt,  // Defined in next section
    messages: [
      {
        role: 'user',
        content: [
          ...imageContent,  // Spread all images first
          {
            type: 'text',
            text: 'Please analyze this scientific paper comprehensively. Examine all visible pages, figures, tables, and sections to provide a thorough analysis following the structured format.'
          }
        ]
      }
    ]
  });

  return response.content[0].text;
}

Claude processes content sequentially. By placing all images first, Claude has the complete visual context before it encounters your instructions. This lets you reference "the methodology section" or "Figure 3" in your prompt, and Claude will know what you mean.

A token budget is essential here. The max_tokens: 8000 setting enables detailed analysis. Scientific papers require substantial output to cover methodology, results, figures, and implications. If you're getting truncated responses, increase this value. The model's context window accommodates much larger values if needed.

How to Guide Visual Analysis Through the System Prompt

The system prompt is where you shape Claude's analytical approach. For document analysis, explicit structure produces dramatically better results than open-ended requests.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
const systemPrompt = `You are an expert scientific paper analyst. Your role is to provide comprehensive, structured analysis of academic papers using visual understanding of the document pages.

Analyze the paper thoroughly and provide your response in the following structured format:

## Paper Overview
Provide title, authors, institution, and publication venue if visible.

## Abstract Summary
A concise plain-English summary of what the paper is about (2-3 sentences).

## Key Contributions
Bullet points of the paper's main contributions to the field.

## Methodology
Step-by-step breakdown of the research approach:
- Study design
- Data collection methods
- Analysis techniques
- Tools/frameworks used

## Key Findings
The main results with relevant statistics, organized as bullet points.

## Figures & Tables Analysis
For each significant figure or table visible:
- What it shows
- Key takeaways
- How it supports the paper's claims

## Technical Concepts Explained
Define and explain key technical terms and concepts from the paper in accessible language.

## Strengths
What the paper does well.

## Limitations
Potential weaknesses, gaps, or concerns with the methodology or conclusions.

## Practical Implications
How these findings might be applied in the real world.

## TL;DR
A single paragraph summary suitable for someone with general scientific literacy but not domain expertise.

Be thorough but concise. Use clear formatting with headers and bullet points. If you cannot see certain sections clearly, note what information would typically be found there.`;

This prompt incorporates several techniques that improve output quality:

• Role establishment ("expert scientific paper analyst") primes Claude's response style and vocabulary. This isn't just flavor text; it measurably affects the depth and precision of analysis.
• Explicit section structure ensures consistent output format across different papers. Every analysis will have the same sections in the same order, making results predictable for downstream processing or display.
• Nested formatting guidance (the bullet points under Methodology) shows Claude precisely how detailed you want each section. Without this, you might get single-sentence sections or sprawling paragraphs.
• Interpretation requests ("How it supports the paper's claims") push Claude beyond description into analysis. This engages its reasoning capabilities rather than treating it as an OCR tool.
• Graceful degradation instructions ("If you cannot see certain sections clearly...") prevent hallucination. Claude won't invent content for blurry figures; it will note the gap and explain what typically appears there. This is crucial for maintaining trust in the output.

The Analysis Endpoint

The server uses Express with multer middleware to handle PDF uploads. When a user uploads a file, multer saves it to disk and provides the file path. The /api/analyze endpoint then orchestrates the conversion and analysis:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
app.post('/api/analyze', upload.single('paper'), async (req, res) => {
  const pdfPath = req.file?.path;

  if (!pdfPath) {
    return res.status(400).json({ error: 'No PDF file uploaded' });
  }

  try {
    // Step 1: Convert PDF pages to base64 images
    const images = await convertPdfToImages(pdfPath);

    if (images.length === 0) {
      throw new Error('Could not extract any pages from the PDF');
    }

    // Step 2: Send images to Claude for analysis
    const analysis = await analyzePaper(images);

    // Step 3: Clean up and return results
    await fs.unlink(pdfPath).catch(() => {});

    res.json({
      success: true,
      pageCount: images.length,
      analysis
    });

  } catch (error) {
    // Clean up even on failure
    if (pdfPath) {
      await fs.unlink(pdfPath).catch(() => {});
    }

    res.status(500).json({ error: error.message });
  }
});

The flow is straightforward: receive the PDF, convert each page to a base64-encoded PNG, pass the image array to analyzePaper(), which calls the Claude API shown above, and return the results. The temporary file is deleted after processing, regardless of success or failure.

The Paper Analyzer in Action

Let's see what this system produces when analyzing a real scientific paper. Here’s the paper we’ll analyze, InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. This is a good option as it has a lot of technical detail, as well as great graphs and tables:

All we need to do is upload it to our analyzer, and the flow starts automatically:

This is a substantial paper, so the analysis takes a few minutes. We’re passing the images to Claude together and allowing it to use its reasoning to go through the paper in detail.

After a few minutes, we get the output, starting with the authors and affiliations:

But that isn’t really what we’re interested in here: it is straightforward OCR. What we want to know is what Claude made of the images in the paper, like this one:

Here’s the reasoning for that image:

Figure 2: OpenCompass Leaderboard Scatter Plot

What it shows: Parameters (x-axis) vs. OpenCompass Average Score (y-axis) for ~20 MLLMs
Key takeaways:
* InternVL3-78B reaches ~78 score, near Gemini-2.5-Pro
* Clear scaling law visible across model sizes
* InternVL3 series shows efficient scaling (high performance per parameter)
* Significant gap between InternVL3 and previous InternVL versions

Support for claims: Demonstrates competitive positioning against both open and closed-source models

All correct observations were taken only from the image. The overall analysis demonstrates several of Claude's visual reasoning strengths:

• Cross-reference understanding: Claude connects figures to their descriptions in the text, explaining not just what a chart shows but how it supports specific claims made elsewhere in the paper.
• Technical term extraction: Domain-specific terminology is identified and explained in accessible language, making the analysis useful for readers outside the paper's immediate field.
• Critical evaluation: The Strengths and Limitations sections show Claude's ability to assess methodology, not just describe it. This goes beyond summarization into genuine analysis.
• Hierarchical structure: Information is organized from high-level summary (TL;DR, Abstract Summary) to detailed examination (Figure Analysis, Technical Concepts), accommodating different reader needs.

Effective Prompt Engineering for Visual Tasks

Prompts for visual analysis differ from text-only prompts. The visual modality introduces ambiguity (what should Claude focus on?), scale challenges (images contain far more information than equivalent text), and format considerations (how should visual insights be expressed?).

Request Interpretation, Not Description

Claude's strength lies in understanding what visual content means, not just cataloging what it contains. Compare these two prompts:

• Weak: "List all the figures in this paper."
• Strong: "For each figure, explain what it demonstrates and how it supports the paper's central argument."

The first prompt treats Claude as an inventory system. The second engages its reasoning capabilities, producing output that's actually useful for understanding the paper.

This principle extends to all visual tasks:

Provide the Output Structure Upfront

Visual content is inherently less structured than text. A page might contain headers, body text, figures, captions, footnotes, and marginalia, all competing for attention. Giving Claude a clear output template produces more consistent results:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
const structuredPrompt = `Analyze this financial chart and provide:

## Trend Summary
One sentence describing the overall direction.

## Key Data Points
- Highest value and when it occurred
- Lowest value and when it occurred  
- Current value

## Notable Patterns
Any cycles, anomalies, or inflection points visible.

## Interpretation
What this trend suggests about the underlying subject.`;

Without structure, Claude might provide a wandering analysis that's hard to parse programmatically or display consistently. With structure, every response follows the same format, making downstream processing predictable.

Handle Uncertainty Explicitly

Documents often contain partially visible content, low-resolution figures, or ambiguous elements. Tell Claude how to handle these situations:

1
2
3
4
5
6
7
8
const uncertaintyAwarePrompt = `Analyze this technical diagram.

If any labels or components are unclear:
- Note what you can partially see
- Explain what typically appears in that position
- Indicate your confidence level

Do not guess at illegible text. Instead, describe its apparent purpose based on context.`;

This instruction prevents hallucination while still extracting value from imperfect inputs. Claude will say, "The axis label appears to show units of measurement, possibly milliseconds, based on the context, but the text is not fully legible," rather than confidently stating an incorrect value.

Balance Scope and Depth

For multi-page documents, decide whether you need comprehensive coverage or focused extraction. Comprehensive prompts produce longer, more expensive responses:

1
2
3
4
5
6
7
// Comprehensive: covers everything, higher cost
const comprehensivePrompt = `Analyze all aspects of this research paper: 
methodology, results, figures, implications, and limitations.`;

// Focused: specific extraction, lower cost
const focusedPrompt = `Extract only the statistical claims from this paper. 
For each claim, note the specific numbers, sample size, and p-values if provided.`;

Focused prompts are particularly useful when you're building pipelines that process large volumes of documents. You might run a low-cost, focused extraction first, then perform a comprehensive analysis only on documents that meet specific criteria.

When Should You Use Claude for Visual Tasks?

Claude's reasoning-first architecture is exceptionally good at specific visual tasks but less suited to others. Understanding these boundaries helps you choose the right tool.

Strong Fits

• Academic papers and technical documentation: Claude excels at understanding relationships between sections. It can link a methodology description to its corresponding figure, identify when results do not support the stated conclusions, and explain technical concepts in accessible terms.
• Charts and graphs requiring interpretation: Beyond reading values, Claude explains trends, identifies anomalies, and contextualizes data within broader narratives. A financial analyst might use it to generate first-draft interpretations of quarterly reports.
• Diagrams with labeled components: Flowcharts, architecture diagrams, process flows, and system schematics play to Claude's strength in sequential reasoning. It can walk through a diagram step-by-step, explaining how components interact.
• Educational content: Claude's explanation quality makes it valuable for analyzing textbooks, lecture slides, and instructional materials. It can identify pedagogical structures and suggest improvements.
• Forms and structured documents: When context affects interpretation (which it usually does), Claude understands that a "Date" field in a medical form means something different than in a shipping label.

Weaker Fits

• Real-time video analysis: Claude processes discrete images, not video streams. Applications requiring sub-second latency or continuous processing need specialized video models.
• Fine-grained object detection and counting: Tasks like "count every person in this crowd photo" or "identify all instances of this component in a circuit board" are better handled by detection-focused models.
• Photorealistic scene understanding: Complex spatial reasoning about 3D scenes, occlusion relationships, or physical properties isn't Claude's specialty.
• High-throughput image processing: When you need to process thousands of images per minute, the API latency and token costs of Claude may be prohibitive compared to specialized classification models.

For many applications, the best solution combines Claude with specialized tools:

• Use OCR to extract text, then Claude to interpret meaning
• Use object detection to identify regions of interest, then Claude to analyze those regions
• Use Claude for initial understanding, then specialized models for specific extraction tasks

Frequently Asked Questions

1. How is Claude Vision optimized for document interpretation?

Claude processes visual content through its language reasoning architecture rather than treating vision as a separate system. This means it reads documents the way a human expert would: understanding how sections relate to one another, connecting figures to their descriptions in the text, and interpreting meaning rather than merely extracting text. The result is an analysis that explains what documents communicate, not just what they contain.

2. What's the best way to prompt Claude for layout-aware reasoning?

Provide explicit output structure in your prompt and request interpretation rather than description. For example, instead of "list the figures," ask "explain how each figure supports the paper's argument." Include section headers in your prompt template so Claude organizes its analysis to match the document's logical structure. When dealing with multi-page documents, place all images before your text instructions so Claude has full context before responding.

3. When is Claude preferable to OpenAI or Gemini for structured tasks?

Claude excels when you need explanation quality and cross-reference understanding: academic papers, technical documentation, charts requiring interpretation, and any task where output needs to be understood by humans. Choose other models when you need real-time processing, fine-grained object detection, or high-throughput image classification, where reasoning depth matters less than speed.

4. How do you extract structured data from charts or tables using Claude?

Send the image with a prompt that specifies both the data format you want and the interpretation you need. For example: "Extract the data from this table as JSON, then explain any trends or anomalies in the values." Claude can output structured formats (JSON, markdown tables, CSV) while also providing context about what the data means. For complex tables, increase image resolution (scale 2.5-3.0) to ensure text remains legible.

5. How can Claude's self-explanations improve validation and debugging?

Ask Claude to explain its reasoning alongside its conclusions. For document analysis, include prompts like "note which sections you drew each finding from" or "indicate your confidence level for any partially visible content." This creates an audit trail: when output appears incorrect, you can trace back to determine whether Claude misread the content, drew an incorrect inference, or relied on incomplete information. These explanations also help catch hallucinations, since Claude will flag uncertainty rather than invent content when properly instructed.

Use Claude for Visual Reasoning, Not Perception

Building effective document analyzers with Claude requires understanding its distinctive strengths: contextual reasoning across visual and textual content, high-quality explanations, and controllable structured output. The paper analyzer demonstrates core patterns you'll use in production: PDF conversion, multi-image requests, structured prompts, and robust error handling.

The key insight is to treat Claude as a reasoning system that accepts visual input, not a vision system with language bolted on. Design prompts that engage reasoning capabilities (interpret, explain, evaluate) rather than perception alone (list, describe, identify). Structure your output requests explicitly to get consistent, parseable results.

When you need a genuine understanding of what documents mean, not just the extraction of what they contain, Claude is a powerful tool. Combined with appropriate preprocessing and error handling, it enables applications that were previously impossible without human review.