TECHDRISHTI · EVALUATION REPORT · 2026-07-19
Measuring what the pipeline actually gets wrong
Three hand-labeled golden sets, judge-validated where a metric can't be re-run directly, checked against the pipeline's real output, not anecdotes. This is the evidence behind the numbers in the case study, held to the same honesty standard as every field report: the one regression here is reported as-is, not tuned away.
1,505 hand-labeled rows across 3 golden sets
2 verified prompt iterations
1,107 claims fact-checked for the faithfulness pass
Deciding what's worth writing about
149 hand-labeled items (news, job postings, listicles, Show HN posts) run through Stage 1's skip/publish decision twice, once per prompt version, against the identical golden set, so the table below is a direct, apples-to-apples before/after on the same 149 items.
Only rejected job postings, marketplace/Show HN listings, blog/tutorial pages, and stories with no identifiable news event. Never checked whether the topic itself was actually about technology: a company name anywhere in the piece was enough to pass.
Added an explicit off-topic category covering 6 concrete shapes (retail promos, non-tech business/legal news, entertainment news, unrelated pure-science research, multi-topic newsletter digests, personal-reaction gadget posts), each built from a real misclassified example below, not a guess. Also added a rule protecting genuine new GitHub tool releases from being wrongly skipped as “not a news event.”
| Metric | Before | After | Δ |
|---|---|---|---|
| Overall accuracy | 73.8% | 81.9% | +8.1pp |
| Skip precision | 75.6% | 86.7% | +11.1pp |
| Skip recall | 51.7% | 65.0% | +13.3pp |
| off_topic recall | 12.5% (3/24) | 50.0% (12/24) | +37.5pp |
| too_thin recall | 33.3% (1/3) | 0% (0/3) | −33.3pp |
The one regression (n=3) is reported as-is, not re-tuned away. Same standard applied throughout this eval work.
Getting the right names, typed correctly
Stage 1's entity/query output is genuinely run-to-run inconsistent (re-running the identical input can change the answer), so it can't be graded with a single before/after run the way triage was. Two separate things are measured here; don't read them as one before/after.
① Is the judge trustworthy? A sarvam-105b judge was graded against 293 hand-labeled entities and 56 hand-labeled queries: how often does the judge's grade match a human's, on the exact same fixed output? This is a one-time calibration number, not a before/after.
| Judged set | Rows | Judge vs. human agreement |
|---|---|---|
| Entity extraction | 293 | 74.7% |
| Search-query quality | 56 | 66.1% |
Listed the 12 valid entity types, but never said a type had to be exactly one of them. Nothing stopped the model from reusing the article's own topic label, inventing a new category, or listing a bare place name as an “entity.”
Added an explicit rule: type must be one of the 12 words, never the article's own topic label, never an invented category; excluded bare place names and spec numbers from extraction entirely.
Measured differently from the calibration table above: this is a live re-extraction on 97 fresh entities from real articles, not the frozen 293-row golden set (which can't be re-run against a newer prompt), comparing the resulting type-validity rate directly. No judge opinion involved.
| Metric | Before (v1, 293) | After (v2, 97) | Δ |
|---|---|---|---|
| Entity-type schema violations | 22.2% (65/293) | 3.1% (3/97) | −19.1pp |
“Hal Abelson (general)” · “Y Combinator (general)”
The model was reusing the article's own TYPE classification (general, model_release), a label for the whole article, as if it were an individual entity's type. Should have been (person) and (organization).
“Hal Abelson (person)” · “Y Combinator (organization)”
With the rule that a type must be exactly one of the 12 valid words and never the article's own topic label, the same two entities are typed correctly, and this whole error class drops from 14.0% (41/293) of v1 entities to 0 in the v2 re-extraction.
Does the Hindi article actually say what the source said?
1,107 claim-level sentences from 40 published articles, checked against each article's real scraped source text. A free deterministic layer catches claims stating a number absent from the source; an LLM judge (validated at 69.4% agreement on 49 human-graded claims) handles the rest.
Faithfulness had never been measured on this pipeline before. There is no earlier prompt version or prior number to compare against. 26.3% is a first baseline, not a delta. The next faithfulness prompt/pipeline change will be the “before” this number gets compared to.
Unsupported-claim rate by article category. acquisition is a single article (n=26 claims), too small to trust on its own.
“Grok 4.5 is mid-pack in coding evals, scoring lower than GPT-4o and Claude 3 Opus on MMLU-Pro.”
Judge: the source states real benchmark metrics, but never this specific comparative ranking: a fabricated competitive claim, not a vague embellishment.
“…a significant milestone for AI developers.”
Judge: the source README describes the architectural rewrite; it never frames it this way. Editorial overclaiming, the same pattern behind repo_analysis's high rate above.
“Failing to fix [the vulnerability] would erode user trust and hinder adoption of Figma's new platform.”
Judge: the source reports the security issue itself; this consequence is the model's own unhedged speculation, stated as fact.
Limitations, stated plainly
Single-annotator golden sets. One person's judgment, not inter-rater agreement. Real disagreement was found even within the same labeler on structurally similar cases.
26.3% is very likely an overestimate, concentrated in repo_analysis. Stage 3 also uses search-derived entity_context, which this eval never checked. Confirmed directly on one article where “unsupported” numbers were real, just sourced from search results this eval couldn't see.
Faithfulness judge validated on 49 claims, not the full 1,107 (a disclosed tradeoff), and that sample happened to contain zero hedged_opinion human labels.
Fluency and writing quality are not measured at all. Every number here is about factual/editorial correctness.
Stage 1's judgment is documented as run-to-run inconsistent on borderline cases: a known, accepted property, not something these evals eliminate.