Atlas Research · First published 2026-05-07
Atlas 2026 PDF AI Benchmark
We measure seven PDF and research-AI tools on a fixed corpus of 50 academic papers across humanities and STEM domains. Each tool is scored on five locked axes, citation accuracy, answer correctness, source coverage, latency, and price-per-query. Atlas is included with full disclosure: criteria are locked before testing and Atlas is ranked per axis where the data places it. This page is the canonical methodology and live tracker for results that feed our blog listicles.
Why we built this
Most “best AI for X” lists in 2026 are either AI-generated synthesis with no testing, or vendor lists where the publisher places themselves at #1 by default. Both patterns saw 30–50% organic visibility drops in early 2026 once Google started demoting self-promotional listicles. Our response is to publish a single dated benchmark with disclosed methodology and reuse it across every comparison post.
Corpus
n = 50 academic papers, sampled across psychology (10), biomedical research (10), economics (10), computer science (10), and history/philosophy (10). Each paper is < 50 MB, English-language, published 2018–2025, available as a text-extractable PDF (no scans). Sources: arXiv, PubMed Central, SSRN, JSTOR Open Access, plus author websites for older humanities work. Full paper list and SHA-256 hashes are published as CSV alongside this page.
Tools tested
| Tool | Category |
|---|---|
| Atlas | Cross-document mind map + cited chat |
| NotebookLM | Source-grounded notebook (Google) |
| ChatPDF | Single-document chat |
| Claude Projects | Long-context reasoning ($20/mo) |
| ChatGPT | General-purpose AI ($20/mo) |
| Elicit | Structured paper-data extraction |
| Scholarcy | Paper TLDR / flashcards |
Scoring axes
- Citation accuracy. For each AI claim, can we click through to the exact passage in the source PDF that grounds the claim? Pass = passage matches; fail = no passage, hallucinated reference, or passage contradicts the claim.
- Answer correctness. Is the answer factually correct relative to the source(s) for a fixed set of comprehension and synthesis questions? Scored 0–2 by the test author against an answer key derived from the papers.
- Source coverage. For multi-source questions, what fraction of the relevant papers in the corpus does the tool surface in its answer or citations? Higher coverage = stronger synthesis.
- Latency. Median wall-clock seconds from question submit to first complete answer, across 10 trial runs per tool per task.
- Price-per-query. Lowest paid tier divided by monthly query allowance, normalized to USD per 1,000 queries.
Rubric was locked on 2026-04-15 before any tool was scored. Atlas was scored last to prevent rubric drift in our favour.
Tasks
For each tool, we run three task types against the corpus:
- Single-paper comprehension. 10 questions across one randomly-selected paper, drawn from a fixed pool.
- Cross-paper synthesis. 5 questions that require integrating findings from ≥ 2 papers in the corpus.
- Citation traceability. 10 claims where we verify whether the tool’s cited passage actually supports the claim.
Disclosure
Atlas is our product. We publish this page so any reader can audit how Atlas is compared against competitors. The author of this benchmark is Jet New, founder of Atlas. Atlas was tested under the same Pro tier ($20/mo) as Claude Projects and ChatGPT to keep pricing comparable. No vendor relationship payments were accepted from any tool listed.
Results status
Methodology and corpus are finalized. Per-tool scoring is in progress with results published rolling on this page and reflected in our listicle posts as each axis completes. Each cited number in our listicles links back to the row of this page that generated it. If you spot a discrepancy, email jet@atlasworkspace.ai.
Listicles using this methodology
License
Methodology, corpus list, and scoring rubric are released under CC-BY 4.0. You may reuse with attribution to Atlas and a link to this page.
