GroundedGeo: A Citation-Grounded Geographic QA Benchmark
TL;DR
- Situation: LLMs answer geographic questions confidently, but you can't tell whether the answer came from real reference material or from the model's guess. Existing geo-QA benchmarks only grade the answer, not the source.
- My decision: Publish a benchmark where "correct" requires both a right answer AND a verifiable citation to a real source. Systems that answer without citing, or cite fabricated sources, fail — even when the answer happens to be right.
- Outcome: Open dataset published on Hugging Face and Zenodo (DOI
10.5281/zenodo.18142378). Paper under submission. 200+ downloads on Hugging Face and counting.
The problem
Geographic QA — questions like "which congressional district contains this address," "what's the school district for this ZIP," "which precinct covers this polling place" — is a domain where a wrong answer that looks right is worse than no answer at all. Users act on these answers. A voter shows up to the wrong precinct; a parent enrolls in the wrong school district. The cost of a confident hallucination is real.
Existing geo-QA evaluations only ask "did the model get the right answer?" They don't ask "did the model get the right answer for the right reason, from a source we can check?" That's the gap.
The constraint
To be useful the benchmark had to be: reproducible without private data, small enough to run on a laptop, licensed for open re-use, and constructed so that "citation quality" and "answer quality" are separately scorable. That separation is the point. If they're conflated, you can't tell whether a system failed because it didn't know the answer or because it knew but cited badly.
What I killed: answer-only scoring
The simplest benchmark design would have been binary: answer, then exact-match score. Fast, clean, boring. And exactly what already exists. I killed that and made citation a first-class requirement, because in this domain "the model happened to be right without a source" is the worst kind of right — you can't trust it and you can't audit it.
What I shipped
Dataset: A set of geographic questions where each item carries the correct answer and a canonical reference for that answer (an authoritative dataset, a government API, a public shapefile). Published openly on Hugging Face and archived on Zenodo with a permanent DOI.
Evaluation methodology: Two-axis scoring. Axis one: is the answer correct? Axis two: did the model cite a verifiable source, and is the cited source real? A system can score high on answer accuracy and still fail the benchmark by citing sources that don't exist.
Companion benchmark: BoundaryBench — 13,000 queries across all 50 U.S. states for the specific sub-problem of "does this point fall inside this polygon." GroundedGeo covers the broader question of "can the model source its geographic claims."
Where you can see it
Dataset: huggingface.co/datasets/nidhipandya/GroundedGeo
Permanent archive: Zenodo DOI 10.5281/zenodo.18142378
Companion: BoundaryBench case study
What I'd measure
Whether the benchmark actually surfaces the failure mode it was built for. The bet is that at least one popular frontier model will score high on answer accuracy and low on citation validity — proving that answer-only scoring hides a real problem. If every model scores similarly on both axes, the benchmark is redundant with what already exists.
What I'd do differently
I'd add a "temporal grounding" split — questions where the correct source has been superseded by a newer authoritative version. Geographic data changes: districts get redrawn, ZIP codes shift, school catchments update. A benchmark that only tests "cite a real source" doesn't catch models that cite outdated real sources. A temporal split would.
Read next: BoundaryBench — the companion 13,000-query benchmark on GPS boundary containment across all 50 states.