BoundaryBench: A Geospatial LLM Benchmark Across All 50 U.S. States

The problem

Every civic and geographic product runs into the same underlying operation: "what boundary contains this point?" It's the query that powers "which representative is this address's?", "what school district covers this home?", "what precinct is this polling place in?" On paper it's a point-in-polygon lookup. In practice, GPS is off by tens of meters, addresses geocode to slightly wrong points, and administrative boundaries follow historical lines that don't always match modern streets.

Newer product designs plug LLMs into this pipeline — either as the geocoder, the resolver, or the explanation layer — without any public standard for measuring how well the model does the geospatial reasoning. BoundaryBench exists to give teams that standard.

The constraint

To be useful the benchmark had to be: large enough to be statistically meaningful, geographically diverse enough to generalize (not just one region), reproducible without private data, licensed for open re-use, and shipped with tool-augmented baselines so teams could immediately see how today's approaches perform. It also had to be small enough to run on a laptop.

What I killed: method-only publication

The tempting shortcut was to publish a method paper — "here's a tool-augmented technique for boundary containment" — and let readers bring their own data. That's how most academic benchmarks fail: nobody assembles the input, so nobody runs the comparison. I killed method-only in favor of dataset-first. The 13,000-query set with ground-truth boundaries is the artifact that lets other teams answer "how does my system do?" without spending a week preparing input.

What I shipped

Dataset — 13,000 queries across all 50 U.S. states. Coordinate/boundary pairs with ground-truth containment. Broad geographic coverage so results generalize past any single region. Published openly on Hugging Face.

Evaluation harness. Reproducible Python eval that scores LLM outputs on the resolve-to-boundary task and reports per-state, urban vs rural, and edge-case (boundary-adjacent points) breakdowns. Full code on GitHub.

Tool-augmented baselines. Comparisons between naive LLM prompting, retrieval-augmented prompting, and a PostGIS-tool-call baseline, so readers can immediately see what today's approaches actually achieve on the task.

Companion benchmark: GroundedGeo — a citation-grounded geographic QA benchmark that sits alongside BoundaryBench and addresses the sibling problem of "does the model source its geographic claims to real references?"

Where you can see it

Dataset: huggingface.co/datasets/nidhipandya/boundarybench

Code + eval harness: github.com/nidhip1611/boundarybench

Companion benchmark: GroundedGeo case study

What I'd measure

Adoption. The benchmark is only useful if teams run it. I track Hugging Face download counts as a floor for "someone bothered to load the dataset," and citations / cross-references from external teams as the ceiling for "someone bothered to use it in their own writeup." The bet fails if downloads plateau and no external comparison shows up.

What I'd do differently

I'd add a hard "boundary-adjacent" split — points within 50 meters of a boundary edge — as a first-class evaluation category rather than a breakdown. That's the case where naive point-in-polygon fails most often and where downstream products (assigning a resident to the wrong district) do the most damage. Making it a first-class split would push method authors to optimize for the case that actually matters.

Related writing: From "1 Pace Plaza" to a Point (Medium) — the multi-tier geocoding pipeline that led into BoundaryBench.

Read next: GroundedGeo — the companion benchmark on citation-grounded geographic QA.

← Back to all work