NewsScope: Schema-Grounded News Claim Extraction

The problem

When you're building anything that has to verify news at scale — a fact-checker, a summarizer, a moderation pipeline — you need a way to pull structured claims out of unstructured articles. The claim isn't just "this article talks about X." It's "Actor A did Action B to Target C at Location D on Date E, according to Source F." That structure is what makes downstream verification possible.

Two problems in the existing landscape: closed-frontier models like GPT-4 are accurate but expensive at scale and unavailable offline. Open-weight models generalize badly across domains — a news claim extractor trained on politics falls apart on health articles.

The constraint

I wanted an open-weight model that ran on commodity hardware, cost under $20 to train, was reproducible, and matched the accuracy of a paid frontier model on domains that mattered. Everything about that constraint said: "don't use a big model, use a small one and fine-tune it well."

What I killed: prompt-engineering GPT-4o-mini as the production path

The easy path was: prompt-engineer GPT-4o-mini into extracting claims, call it done. It works and it's accurate. But it's closed. Every inference costs money. Every user needs internet. Every deployment ties you to OpenAI's API surface + pricing. For a research contribution AND for civic-tech applications where offline deployment matters, this was a no-go.

What I shipped

Dataset: 455 articles across politics, health, science/environment, and business. 395 in-domain articles plus 60 out-of-source articles specifically for testing generalization. Human-annotated with 94.6% inter-annotator agreement on SUPPORTED judgments.

Model: LLaMA 3.1 8B, fine-tuned with LoRA on 315 training examples. LoRA-only adapter weights so the base model stays untouched and the adapter is ~50MB to distribute.

Evaluation: Held-out 80-article in-domain test set + 60 out-of-source articles. Human evaluation on 400 claims, blind. Additional numeric-grounding filter that improves accuracy from 89.4% → 91.6%, narrowing the gap with GPT-4o-mini to 2.1 percentage points.

Where you can see it

Paper: arXiv 2601.08852 — full methodology, ablation, and reproducibility instructions.

Model: nidhipandya/NewsScope-lora on Hugging Face — LoRA adapter weights, ready to load with `peft`.

Demo: Hugging Face Space (VERA) — public demo, live inference.

Code + dataset: github.com/nidhip1611/NewsScope — repro-ready.

What I'd measure

The bet was: a small open-weight model, well-fine-tuned, can match closed frontier models on a specific domain for a fraction of the cost. The measurement that would tell me I was wrong: if the p-value between NewsScope and GPT-4o-mini's accuracy were < 0.05 (statistically significant gap) rather than the p=0.07 we observed. It wasn't. The bet held.

What I'd do differently

I'd expand the training set past 315 examples. The compute cost is minor but I bet the last 4 percentage points of accuracy vs GPT-4o-mini could close with 500–1,000 more high-quality examples. Also: I'd add a contradictory-claims test set — the current benchmark only tests SUPPORTED claims. A "REFUTED" split would test the model's ability to catch misinformation, which is the real downstream use case.

Related writing: This case study is expanded from Headlines on a Leash (Medium) . The paper has the full methodology; the Medium article has the shipping story.

Read next: Professor Digital Twin — how the same LoRA + retrieval techniques get applied with a calibrated abstention layer for trustworthy AI.

← Back to all work