How to Evaluate Exa Search

​

Overview

• Retrieval Quality: Accuracy and relevance of returned results
• Latency: Response time from query to results
• Freshness: Ability to retrieve up-to-date information
• Cost Efficiency: Value delivered per API call
• Agentic Suitability: Performance in multi-step reasoning workflows

• Deep Research: Multi-hop queries requiring comprehensive context and query expansion
• Agentic Workflows: Complex tasks involving multiple search iterations and reasoning steps
• Low-Latency QA: Fast factual question-answering for real-time applications
• Semantic Discovery: Finding conceptually related content beyond keyword matching

​

Best Practice: Start with Defaults

# Option 1: Use text with character limit (recommended for consistent comparisons)
exa.search_and_contents(
    query,
    type="fast",  # or "auto" (for `Deep`, see Option 2)
    num_results=10,
    text={"max_characters": 5000}
)

# Option 2: Use context string for RAG (single string with total max characters)
# Note: `Deep` search may require context=True to return results
# exa.search_and_contents(
#     query,
#     type="deep",
#     additional_queries=["variation 1", "variation 2"],  # Optional query variations
#     num_results=10,
#     context={"max_characters": 20000}
# )

# Option 3: Use full text (may result in very long content)
# exa.search_and_contents(
#     query,
#     type="fast",
#     num_results=10,
#     text=True
# )

​

Compare Within Latency Classes

• For Exa Fast (<500ms): Compare to other sub-1s APIs with similar latency
• For Exa Auto (~1s): Compare to mid-latency systems (800ms-1500ms)
• For Exa Deep (>2s): Compare to other multi-second agentic/research systems

​

Search Types: Understanding the Quality-Latency Spectrum

​

Fast Search

• Median latency: ~500ms (excluding network and optional features)
• Streamlined neural and reranking models
• Best for single-step factual queries

• Low-latency QA datasets (SimpleQA, WebWalkerQA)
• Real-time applications (voice agents, autocomplete)
• High-volume agentic workflows where latency accumulates

result = exa.search_and_contents(
    "latest AI breakthroughs in 2025",
    type="fast",
    num_results=10,
    text={"max_characters": 5000}
)

​

Auto Search (Default)

• Median latency: ~1000ms
• Intelligently combines multiple search methods
• Reranker model adapts to query type

• General-purpose search evaluations
• When query types vary significantly
• Production workloads requiring versatility

result = exa.search_and_contents(
    "companies building climate tech solutions",
    type="auto",  # or omit - auto is default
    num_results=10,
    text={"max_characters": 5000}
)

​

Deep Search

• Median latency: ~5000ms
• Automatic query expansion or custom query variations via additional_queries (Python) / additionalQueries (JavaScript)
• Rich contextual summaries for each result (requires context=True)
• Parallel search across multiple query formulations

• Agentic workflows (FRAMES, MultiLoKo, BrowseComp)
• Complex research tasks requiring multiple perspectives
• Scenarios where comprehensive coverage matters more than speed

result = exa.search_and_contents(
    "impact of quantum computing on cryptography",
    type="deep",
    additional_queries=[
        "quantum threats to encryption",
        "post-quantum cryptography research"
    ],
    num_results=10,
    text=True,
    context=True  # Required for `Deep` search summaries
)

​

Neural Search

• Embeddings-based next-link prediction
• Excels at thematic and conceptual relationships
• Incorporated into Fast and Auto search types

• Exploratory search tasks
• Finding semantically related content
• Long-form query matching

​

Evaluating Exa with Tool Calling

​

Why Tool Calling Matters for Evaluation

• Agents decide when to search: The LLM determines if/when to call Exa based on the task
• Dynamic parameter selection: Agents may choose search parameters (though we recommend minimal defaults)
• Multi-step workflows: Agents can make multiple Exa calls in sequence or parallel

​

Tool Calling Best Practices for Evaluation

1. Keep tool definitions minimal: Don’t expose too many parameters to the agent — this encourages over-filtering
2. Use consistent tool schemas: Standardize tool definitions across all evaluated systems
3. Monitor tool call patterns: Track how often and when agents invoke Exa vs competitors

​

Implementation Guides

• Anthropic Tool Calling - Using Claude with Exa search integration
• OpenAI Tool Calling - Integrating Exa with GPT models
• OpenAI Responses API - Recommended for new OpenAI projects

​

Evaluation Methodology

​

Core Principles for Fair Benchmarking

1. Use default settings: Start with minimal parameters (type, num_results, text). Avoid adding restrictive filters (date ranges, domains, text inclusion/exclusion) unless they’re core to your evaluation — these often cause over-optimization that artificially limits results without meaningful benefit.
2. Standardize queries: Use identical query sets across all systems
3. Control downstream processing: Use the same LLM for answer synthesis and grading
4. Disable prompt engineering: Evaluate base API performance without query optimization
5. Measure consistently: Track P50 latency, accuracy, and coverage using identical metrics
6. Document configurations: Record all parameter settings for reproducibility

​

Four-Phase Evaluation Workflow

​

Phase 1: Scope Definition

• What capabilities are you testing? (factual QA, research depth, freshness, etc.)
• What latency requirements matter for your use case?
• Are you evaluating single-step retrieval or multi-step agentic workflows?

​

Phase 2: Dataset Selection

• Low-latency factual QA: SimpleQA, WebWalkerQA
• Single-step retrieval: FRAMES (single-step slice), Seal0
• Agentic workflows: FRAMES (agentic slice), MultiLoKo, BrowseComp
• Complex reasoning: HLE (hard, long, emerging questions)
• Freshness: FreshQA, time-sensitive queries

​

Phase 3: Run Configurations

# 1. Retrieval step
results = exa.search_and_contents(
    query,
    type="fast",  # or "auto", "deep"
    num_results=10,
    text={"max_characters": 5000}
)

# 2. Answer synthesis (downstream LLM restricted to retrieved context)
context = "\n\n".join([r.text for r in results.results])
answer = llm.generate(
    f"Answer the question using only the provided context.\n\n"
    f"Context: {context}\n\n"
    f"Question: {query}\n\n"
    f"Answer:"
)

# 3. Grading (LLM-based correctness evaluation)
grade = grading_llm.evaluate(
    question=query,
    expected_answer=ground_truth,
    generated_answer=answer
)
# Returns: "correct", "partial", or "incorrect"

​

Phase 4: Results Analysis

• Accuracy: Percentage of correct answers
• Partial-credit accuracy: Weighted score (e.g., correct=1.0, partial=0.5, incorrect=0.0)
• Retrieval coverage: Percentage of queries where relevant information was retrieved
• P50 latency: Median response time across all queries
• Cost per query: Total API cost divided by number of queries

​

Optimal Exa Settings for Evaluation

​

Configuration Parameters

​

Recommended Configuration Templates

​

Fast-Baseline Configuration

result = exa.search_and_contents(
    query,
    type="fast",
    num_results=10,
    text={"max_characters": 5000}
)

​

Auto-Quality Configuration

result = exa.search_and_contents(
    query,
    type="auto",
    num_results=10,
    text={"max_characters": 5000}
)

​

Deep-Comprehensive Configuration

result = exa.search_and_contents(
    query,
    type="deep",
    additional_queries=[variation1, variation2],
    num_results=10,
    text=True,
    context=True,
    livecrawl="fallback"
)

​

Choosing Datasets for Evaluation

​

Benchmark-to-Search-Type Mapping

​

Dataset Characteristics

​

SimpleQA

• Purpose: Tests fast, factual question-answering
• Query style: “What is the capital of France?”, “Who invented the telephone?”
• Evaluation focus: Accuracy and latency for straightforward queries
• Exa configuration: Fast search with cached content

​

FRAMES

• Purpose: Evaluates both single-step and multi-step retrieval
• Two slices:
  • Single-step: Direct queries answerable from one search
  • Agentic: Complex queries requiring multiple search iterations
• Evaluation focus: Versatility across task complexity
• Exa configuration: Fast/Auto for single-step, Deep for agentic

​

MultiLoKo & BrowseComp

• Purpose: Multi-hop reasoning and deep comprehension
• Query style: Questions requiring synthesis across multiple sources
• Evaluation focus: Quality of context and reasoning support
• Exa configuration: Deep search with query expansion

​

Seal0

• Purpose: General search quality benchmark
• Query style: Diverse real-world queries
• Evaluation focus: Overall retrieval accuracy
• Exa configuration: All search types (compare performance)

​

HLE (Hard, Long, Emerging)

• Purpose: Stress-test with difficult queries
• Query style: Complex, lengthy queries about recent topics
• Evaluation focus: Handling edge cases and emerging information
• Exa configuration: Deep search with livecrawling

​

Benchmark Results

​

Low-Latency Search Engines

• Exa Fast achieves 94% accuracy on SimpleQA with median latency <500ms
• Strong performance across multiple benchmarks while maintaining speed advantage
• Ideal for real-time applications and high-volume agent workflows

​

Agentic Search APIs

• Exa Deep leads on FRAMES (96%) and MultiLoKo (89%) benchmarks
• Query expansion and rich context enable superior agentic performance
• Higher latency justified by comprehensive, high-quality results

​

Quality-Latency Tradeoffs

​

Understanding the Spectrum

​

Interpreting Tradeoffs

1. Don’t compare across latency classes: Fast search at 500ms vs Deep search at 5000ms serve different purposes. Always find the closest competitor in terms of latency for meaningful comparisons — compare systems with similar P50 latency ranges.
2. Benchmark within peer groups:
   • Compare Exa Fast (<500ms) to other sub-1s APIs
   • Compare Exa Auto (~1s) to similar mid-latency systems
   • Compare Exa Deep (>2s) to other agentic/research-oriented systems
3. Consider total workflow time: For multi-step agents, Fast search may complete the entire workflow faster than Deep search on a single query
4. Account for quality requirements: If accuracy >90% is required, accept higher latency; if <1s is required, accept some quality tradeoff

​

Factors That Impact Latency

​

Running Production-Grade Evaluations

​

Example: SimpleQA Evaluation Script

from exa_py import Exa
import json
from datetime import datetime

exa = Exa(api_key="YOUR_API_KEY")

def evaluate_simpleqa(dataset_path, config):
    """
    Run SimpleQA evaluation with specified configuration.

    Args:
        dataset_path: Path to SimpleQA JSON file
        config: Dict with keys: type, num_results, text, livecrawl
    """
    with open(dataset_path) as f:
        questions = json.load(f)

    results = []
    latencies = []

    for item in questions:
        query = item['question']
        ground_truth = item['answer']

        # Retrieval
        start = datetime.now()
        search_result = exa.search_and_contents(
            query,
            type=config['type'],
            num_results=config['num_results'],
            text=config['text'],
            livecrawl=config['livecrawl']
        )
        latency = (datetime.now() - start).total_seconds() * 1000
        latencies.append(latency)

        # Synthesis (using your LLM)
        context = "\n\n".join([r.text for r in search_result.results])
        answer = your_llm.generate(
            f"Answer concisely using only the context.\n\n"
            f"Context: {context}\n\n"
            f"Question: {query}\n\n"
            f"Answer:"
        )

        # Grading (using your grading LLM)
        grade = grading_llm.evaluate(
            question=query,
            expected=ground_truth,
            generated=answer
        )

        results.append({
            'query': query,
            'grade': grade,
            'latency_ms': latency
        })

    # Calculate metrics
    accuracy = sum(1 for r in results if r['grade'] == 'correct') / len(results)
    p50_latency = sorted(latencies)[len(latencies) // 2]

    return {
        'accuracy': accuracy,
        'p50_latency_ms': p50_latency,
        'total_queries': len(results),
        'config': config
    }

# Run evaluation
config = {
    'type': 'fast',
    'num_results': 10,
    'text': {'max_characters': 5000}
}

results = evaluate_simpleqa('simpleqa.json', config)
print(f"Accuracy: {results['accuracy']:.2%}")
print(f"P50 Latency: {results['p50_latency_ms']:.0f}ms")

​

Multi-Configuration Comparison

configs = [
    {'name': 'Fast', 'type': 'fast'},
    {'name': 'Auto', 'type': 'auto'},
    {'name': 'Deep', 'type': 'deep'},
]

for config in configs:
    results = evaluate_simpleqa('simpleqa.json', config)
    print(f"{config['name']}: {results['accuracy']:.1%} @ {results['p50_latency_ms']:.0f}ms")

`Fast`: 94.2% @ 450ms
`Auto`: 95.8% @ 1050ms
`Deep`: 97.2% @ 4950ms

​

Recommendations

​

For Low-Latency QA Benchmarks

• Use type="fast" or type="auto"
• Fix num_results=10
• Use text={"max_characters": 5000} for consistent context length

• Accuracy: 90-95% on factual queries
• Latency: 400-600ms (Fast), 900-1200ms (Auto)

​

For Agentic Workflow Benchmarks

• Use type="deep"
• Provide 2-3 query variations via additional_queries (Python) / additionalQueries (JavaScript) for best results
• Enable context=True for rich summaries
• Set livecrawl="fallback" for freshness

• Accuracy: 85-96% on complex multi-hop queries
• Latency: 4000-6000ms
• Higher comprehensive coverage vs single-query search

​

For Freshness Benchmarks

• Use any search type based on latency requirements
• Set livecrawl="preferred" or livecrawl="fallback"
• Include recent date filters if needed

• Freshness: Up-to-date information from recent sources
• Latency: +500-2000ms vs cached content

​

For Production Deployment

1. Run comparative benchmarks across Fast, Auto, and Deep to understand your quality-latency frontier
2. Match search type to use case:
   • Real-time user-facing: Fast
   • General chatbot/assistant: Auto
   • Deep research/agent workflows: Deep
3. Monitor in production: Track accuracy, latency, and cost metrics continuously
4. Optimize parameters: Adjust livecrawl, num_results, and content options based on actual usage patterns
5. Document your evaluation: Record configurations, datasets, and results for reproducibility

​

For Meaningful Cross-System Comparisons

1. Standardize everything:
   • Identical query sets
   • Same downstream LLM for synthesis
   • Same grading model/rubric
   • Fixed num_results across systems
2. Compare within latency classes — find the closest competitor in terms of P50 latency:
   • For Exa Fast (<500ms): Compare to other sub-1s APIs with similar latency
   • For Exa Auto (~1s): Compare to mid-latency systems (800ms-1500ms)
   • For Exa Deep (>2s): Compare to other multi-second agentic/research systems
3. Account for feature differences:
   • Some systems don’t offer content retrieval
   • Some don’t support livecrawling
   • Some have different context limits
4. Measure what matters for your use case:
   • If latency <500ms is required, only benchmark Fast-class systems
   • If accuracy >95% is required, accept higher latency configurations

​

Additional Resources

• How Exa Search Works - Deep dive into neural search and search types
• Exa’s Capabilities Explained - Feature overview and use cases
• Livecrawling Contents - When and how to use livecrawling
• API Reference: Search - Complete parameter documentation