DEEP RESEARCH AGENT · HEAD-TO-HEAD EVALUATION
How it compares, case by case
Twelve real queries across the agent's four modes, run on the DeepSeek backend and compared against the tools people would otherwise reach for: Tavily (a paid single-shot search API) on the eight question cases, and Claude on the four article cases. Every external tool saw the exact same input. The distinguishing passages are highlighted so the verdict is legible at a glance, not asserted.
Questions — vs Tavily
The headline finding: on all eight questions, Tavily returned "answer": null — five raw links and no synthesized answer. So the real contrast is a grounded, cited answer versus a pile of links the reader has to reconcile. Tap a case for the side-by-side.
Returned five real benchmark leaderboards for other models (GPT-5.6, Gemini 3.1 Pro, Claude Opus 4.8) — with nothing signalling that Zephyr-Q9 does not exist. A reader skimming them could misattribute stray numbers to the fake model.
Could not find any evidence of a model called 'Zephyr-Q9'. Noted the real Zephyr models (7B-beta, 7B-Gemma) but none by that name, and asked for a link rather than inventing scores.
The clearest grounding win: refuse to hallucinate, vs authoritative-looking but irrelevant data.
No synthesized answer — five pricing links. The numbers exist across the pages, but the reader must gather and compare them by hand.
Classified COMPLEX, then answered: 'Yes … DeepSeek V4 Flash is $0.14/M input and $0.28/M output versus GPT-4o at $2.50 / $10.00,' and worked the RAG-specific math — input tokens dominate; a cache hit at ~$0.0028/M cuts effective cost ~98%. Its own derived percentages were flagged by the grounding check.
Scattered prices turned into a decision, with the arithmetic shown and self-policed.
No synthesized answer — good source links that name players and drivers, but leave the 'why' unassembled.
A structured report with a thesis: 'inference accounts for ~two-thirds of all AI compute demand, up from one-third in 2023,' then profiled the players and the landmark deals — IPOs, chip-backed loans, multi-gigawatt supply agreements — that show it is a real transition, not R&D.
Cross-source synthesis of a trend is exactly what a single search cannot do.
No synthesized answer — strong links, but the reader must extract the timeline and the open-weight carve-outs and relate them.
Answered both sub-parts — a dated timeline table ('Aug 2, 2025 — GPAI obligations apply — the most relevant date for open-weight models') and the concrete effects the deadline has already had on releases.
A two-part question answered in two parts, not just the half a search surfaces.
No synthesized answer, but five genuinely on-topic links, including a break-even calculator (crossover near ~264M tokens/day).
Returned an empty report. It burned all 8 iterations and 55 fetches on the hardest question and hit the iteration cap without ever writing a conclusion — worse than the baseline here, which at least returned relevant links.
The honest failure of the run. Ask-mode has no early-stop and no forced conclusion at the cap — the top fix. (On the Sarvam backend, this same case concluded.)
Links covered the verb 'grok' and xAI's Grok model, but nothing on Groq, the inference-hardware company — the meaning 'inference' most points to.
Enumerated the verb and xAI's Grok and picked Grok given 'inference' — but also missed Groq, the inference company, arguably the intended reading.
A genuinely hard near-homophone. Both missed it, so it is not just an agent bug — but a cheap prompt fix.
Links converge cleanly: 128K for Llama 3.3 70B, 131,072 for gpt-oss-120b. This is what single-shot search is for.
Same correct answers — a 128,000-token context window and 131,072 tokens — and even flagged an unsupported source as unverified. But it spent 8 iterations / 28 sources on a one-fact lookup.
The answers tie; the agent over-researches trivial questions — slower for the same result.
Articles — vs Claude
Tavily is not a comparator here — it cannot fact-check a draft or write. So the article modes were run against Claude on identical prompts. Note the deliverable difference: the agent produces publication-ready Hindi; Claude produced English.
'The incentive amount is wrong … Rs 2,500 per kWh, capped at Rs 5,000'. 'It does not cover all … above Rs 1.5 lakh are ineligible'. 'It does not run indefinitely … extended until July 31, 2026'. Also softened the unsupported 'cheaper than petrol outright' claim to total cost of ownership.
Not a flat Rs 15,000 — under PM E-DRIVE it is Rs 2,500 per kWh, capped at Rs 5,000; the Rs 15,000 figure matches the defunct FAME-II scheme. Not 'all' — a Rs 1.5 lakh price cap applies. Not indefinite — fund-limited, extended only to 31 Jul 2026.
Proof the tests also cover faulty-information fixing: both caught every planted lie and traced the wrong figure to its real source.
Strong, accurate coverage of scale, pricing, providers, and the parallel G42/Cerebras supercomputer, plus a fair implementation-risk caveat — but not the budget-underspend signal.
Found the same scale facts (Rs 10,371.92 cr outlay; 34,000 to 38,000 GPUs; 100k target) and the angle Claude missed: the disbursement lag — the 2024-25 allocation cut 69% after underspend, and 2026-27's annual allocation halved year-on-year.
Both excellent; the agent adds one genuinely non-obvious analytical angle.
Polished, well-structured, accurate. Broader on some stakeholders — the Internet Freedom Foundation / Editors Guild press-freedom critique, and Nasscom's cross-border objection on DPDP — but wrote in English, so an editor still needs it translated.
DPDP: added angles Claude lacked — GDPR's 4%-of-turnover fines vs India's fixed-rupee penalties; the Esya survey (57% of AI firms expect >10%-of-turnover compliance cost); ITI lobbying to delay the children's-data rules. Heat: the mortality-undercount thesis — the government's 500-1,500/yr vs a Frontiers estimate of ~3,400 excess deaths per extreme-heat day.
Near-parity research (complementary strengths); the agent wins the deliverable outright — a finished, grounded Hindi article, not an English draft to translate.
What it proved, and what it didn't
On complex, interpretive questions and on faulty-information cases, the agent decisively beats single-shot search — because Tavily returns links, not answers, and the agent grounds and synthesizes. On article work it matches or edges Claude on research and wins the deliverable outright with a finished, grounded Hindi article. The two honest weaknesses the run surfaced are in the cards above, not buried: an empty report on the hardest question when the loop hit its cap, and over-researching trivial lookups — both fixable with the early-stop / forced-conclusion discipline that concise mode already applies (it stopped the ambiguous case at 3 iterations). The one miss shared with Tavily — Groq the company, on the “grok” query — is a near-homophone worth a small prompt nudge.
The backend axis — DeepSeek vs Sarvam on this same suite — has its own page: Sarvam vs DeepSeek, measured.