Batch Geocoding with geopy Without Rate-Limit Errors

Twelve thousand addresses pushed through geolocator.geocode in a plain .apply() will collect an HTTP 429 inside the first second and a blocked IP inside the first minute, and the run will look like it simply found nothing. This guide is for anyone who has to geocode a whole column and needs the job to finish, finish politely, and survive being killed halfway through; it sits under Geocoding & Address Pipelines in Geospatial Data Ingestion & Processing Workflows.

Why This Approach / What Goes Wrong

geopy is a client library, not a scheduler. It sends a request the moment you call it, so a pandas .apply() over a column fires as fast as the interpreter and the socket allow — typically twenty to fifty calls a second. The public OpenStreetMap endpoint permits one. The mismatch is not marginal, it is two orders of magnitude, and the server's answer is a 429 followed by a block keyed to your IP address and user agent together.

The obvious patch — time.sleep(1) at the bottom of a loop — is wrong in two directions at once. It adds a second on top of however long the request took, so a run that should take 1.0 s per address takes 1.6 s, and it does nothing at all about failures. RateLimiter fixes the first half properly: it records the clock time at which the previous call started and sleeps only the remainder of min_delay_seconds, making the delay a floor on the interval rather than an addition to it. A call that itself took 0.7 s waits 0.4 s more, not 1.1 s more.

The second half is failure handling, and this is where the defaults betray you. RateLimiter is constructed with swallow_exceptions=True, which means any GeocoderServiceError that outlives max_retries is caught and turned into return_value_on_exception, whose default is None — the exact value the geocoder returns when it honestly found no match. A run that hit a wall of 429s and a run against a column of nonsense addresses produce byte-identical output. Neither the retry counter nor the server's Retry-After header is anywhere in that output.

The third failure is operational rather than protocol-level. A paced run over twelve thousand distinct addresses takes about three and a half hours. Anything that lasts that long will eventually be interrupted — a laptop lid, an OOM killer, a dropped VPN — and an in-memory result list evaporates with it. The fix is not a bigger try/except but a durable store written as the run proceeds, so that the second attempt starts where the first stopped instead of at zero.

Prerequisites

• geopy>=2.4,<3 — RateLimiter lives in geopy.extra.rate_limiter, and this line exposes GeocoderRateLimited with its retry_after attribute
• geopandas>=1.0 — builds the point layer and carries the CRS
• shapely>=2.0 — the geometry engine behind points_from_xy
• pandas>=2.2 — the address column and the merge back
• pyproj>=3.6 — estimate_utm_crs() and the projection step

python -m pip install "geopy>=2.4,<3" "geopandas>=1.0" "shapely>=2.0" \
                      "pandas>=2.2" "pyproj>=3.6"

The on-disk store below uses sqlite3 from the standard library, so there is nothing further to install and no server to run. One environment detail is worth knowing before you debug a timeout: geopy selects geopy.adapters.RequestsAdapter when requests is importable and falls back to URLLibAdapter when it is not, so connection reuse, proxy resolution and TLS behaviour can differ between two machines running identical geopy versions. Pin requests explicitly if your pacing measurements need to be reproducible.

Step-by-Step Implementation

1. Build one geocoder with a real identity. The rate limit is counted against the pair of your IP address and your user agent, so the user agent is not decoration — it is the thing being throttled. Name the application and give a contact address; the library default is answered with 403, not with data.

from geopy.geocoders import Nominatim

geolocator = Nominatim(
    user_agent="permit-audit/0.4 (data-team@cityworks.example)",
    timeout=10,          # seconds before geopy raises GeocoderTimedOut
)

2. Wrap it in RateLimiter and switch off the dangerous defaults. Every argument here is a deliberate override, not boilerplate.

from geopy.extra.rate_limiter import RateLimiter

geocode = RateLimiter(
    geolocator.geocode,
    min_delay_seconds=1.1,     # floor on the interval between call STARTS
    max_retries=2,             # 2 retries = 3 attempts, and only for
                               # GeocoderServiceError subclasses
    error_wait_seconds=5.0,    # flat pause between attempts — not exponential
    swallow_exceptions=False,  # a failure must raise, never masquerade as None
)

swallow_exceptions=False is the single most important line on this page. With it, an exhausted retry chain raises and your own handler decides what the row means; without it, the same chain returns None and the row is filed as "address not found". Note also that error_wait_seconds is flat — three attempts against an overloaded service are spaced 5 s, 5 s, which is exactly the pattern a server under load least wants, and it is blind to the Retry-After header the server actually sent.

3. Give every failure mode a different name. The exception hierarchy is precise, and the ordering of the except clauses matters because GeocoderRateLimited is a subclass of GeocoderQuotaExceeded — catch the parent first and you will never see retry_after.

from geopy.exc import (
    GeocoderAuthenticationFailure,
    GeocoderQuotaExceeded,
    GeocoderRateLimited,
    GeocoderTimedOut,
    GeocoderUnavailable,
)

def resolve(query: str):
    """Return (status, location, seconds_to_wait_before_trying_again)."""
    try:
        location = geocode(query, exactly_one=True, addressdetails=True)
    except GeocoderRateLimited as exc:            # HTTP 429 — must precede the parent
        return "throttled", None, float(getattr(exc, "retry_after", None) or 30.0)
    except GeocoderQuotaExceeded:                 # daily/plan allowance spent
        return "quota", None, 300.0
    except (GeocoderTimedOut, GeocoderUnavailable):   # no reply, or 502/503
        return "transient", None, 5.0
    except GeocoderAuthenticationFailure:         # 401/403 — retrying cannot help
        raise
    if location is None:
        return "empty", None, 0.0                 # the provider genuinely has no match
    return "ok", location, 0.0

The distinction that pays for itself is empty versus everything else. An empty is a fact about the world — the index has no such address — and caching it permanently saves the same fruitless request on every future run. A throttled or transient result is a fact about the moment, and caching it as final would silently discard a row that would have resolved five seconds later.

4. Make the store the checkpoint. A SQLite file keyed on the normalised address does three jobs at once: it deduplicates, it makes reruns free, and it is the resume marker — there is no separate offset to keep in sync with the data.

import json, sqlite3, unicodedata
from pathlib import Path

def open_store(path=Path("permit_geocode.sqlite")):
    con = sqlite3.connect(path, timeout=30)
    con.execute("PRAGMA journal_mode=WAL")
    con.execute("""
        CREATE TABLE IF NOT EXISTS resolved (
            address_key TEXT PRIMARY KEY,
            status      TEXT NOT NULL,     -- 'ok' | 'empty' | 'error'
            lat         REAL,
            lon         REAL,
            raw         TEXT
        )
    """)
    con.commit()
    return con

def address_key(text: str) -> str:
    """Same address in, same key out — forever. Change this and you invalidate the store."""
    folded = unicodedata.normalize("NFKC", str(text)).casefold()
    return " ".join(folded.replace(",", " ").split())

def pending(con, keys):
    """Whatever is not already settled. Order preserved, duplicates collapsed."""
    settled = {row[0] for row in con.execute(
        "SELECT address_key FROM resolved WHERE status IN ('ok', 'empty')")}
    return [k for k in dict.fromkeys(keys) if k not in settled]

dict.fromkeys deduplicates while preserving order, which matters more than it looks: a customer table of twelve thousand rows routinely holds only eight or nine thousand distinct addresses, and that reduction is pure saved quota. Rows written with status = 'error' are deliberately not in the settled set, so a transport failure is recorded for auditing yet still retried on the next pass.

5. Run the batch so that a crash costs nothing.

import time

def run_batch(con, keys, commit_every=25):
    todo = pending(con, keys)
    print(f"{len(keys)} rows, {len(todo)} still to resolve")
    started = time.perf_counter()
    for n, key in enumerate(todo, start=1):
        status, location, wait = resolve(key)
        if status in {"throttled", "quota", "transient"}:
            time.sleep(wait)                      # honour the server's own number
            status, location, wait = resolve(key) # one deliberate second pass
        row = (key, status, None, None, None)
        if status == "ok":
            row = (key, "ok", location.latitude, location.longitude,
                   json.dumps(location.raw))
        elif status != "empty":
            row = (key, "error", None, None, None)
        con.execute(
            "INSERT OR REPLACE INTO resolved (address_key, status, lat, lon, raw)"
            " VALUES (?, ?, ?, ?, ?)", row)
        if n % commit_every == 0:
            con.commit()
    con.commit()
    return len(todo), time.perf_counter() - started

Storing location.raw verbatim rather than only the coordinate pair is what lets you re-grade results later without spending another request; the payload is a few hundred bytes and the request costs a second. Committing every twenty-five rows bounds the loss from a hard kill to under thirty seconds of work, and because the primary key is the address, replaying those rows is idempotent rather than duplicative.

6. Attach the coordinates in EPSG:4326, then project before measuring anything. Every mainstream geocoder answers in WGS 84, and geopy hands the pair back latitude-first while points_from_xy wants longitude first — the swap is silent because both numbers are plausible.

import geopandas as gpd
import pandas as pd
from shapely.geometry import Point

store = open_store()
permit_sites = pd.read_csv("permit_applications.csv", encoding="utf-8-sig")
permit_sites["address_key"] = permit_sites["site_address"].map(address_key)
fetched, elapsed = run_batch(store, permit_sites["address_key"].tolist())

resolved = pd.read_sql_query(
    "SELECT address_key, status, lat, lon FROM resolved WHERE status = 'ok'", store)
permit_sites = permit_sites.merge(resolved, on="address_key", how="left")

located = permit_sites[permit_sites["lat"].notna()].copy()
sites = gpd.GeoDataFrame(
    located,
    geometry=gpd.points_from_xy(located["lon"], located["lat"]),  # x = lon, y = lat
    crs="EPSG:4326",                                              # declare it, never infer
)

Keep that frame in EPSG:4326 for storage and for anything a web map will consume. The moment a distance appears — how far each permit site is from the works depot, how tightly a group of results has stacked on one street centreline — reproject first, because a degree of longitude is 111 km at the equator and 82 km in Detroit. estimate_utm_crs() reads the data's own extent and returns the correct zone; never substitute EPSG:3857, whose scale factor grows with latitude and turns a fixed threshold into a moving one. The mechanics of the GeoDataFrame and the projection machinery in Coordinate Systems with PyProj both apply unchanged here.

metric_crs = sites.estimate_utm_crs()             # e.g. EPSG:32617 for Michigan
sites_m = sites.to_crs(metric_crs)
depot = (gpd.GeoSeries([Point(-83.045, 42.331)], crs="EPSG:4326")
           .to_crs(metric_crs).iloc[0])

sites["depot_distance_m"] = sites_m.geometry.distance(depot).to_numpy()
far = sites[sites["depot_distance_m"] > 40_000]   # metres, because the CRS is metric

Verification

Three properties define a correct run: nothing is outstanding, no coordinate is nonsense, and the pacing actually held.

import numpy as np

# 1. The run is complete — a second pass has nothing left to fetch.
keys = permit_sites["address_key"].tolist()
assert pending(store, keys) == [], "resume set is non-empty — call run_batch again"

# 2. Every stored coordinate is a plausible WGS 84 pair, and not lon/lat swapped.
ok = pd.read_sql_query("SELECT lat, lon FROM resolved WHERE status = 'ok'", store)
assert ok["lat"].between(-90, 90).all(), "latitude out of range — arguments swapped"
assert ok["lon"].between(-180, 180).all()
assert not (ok["lat"].abs().lt(1e-9) & ok["lon"].abs().lt(1e-9)).any(), "Null Island"

# 3. Pacing held: seconds of wall clock per network call must clear min_delay_seconds.
per_call = elapsed / max(fetched, 1)
print(f"{fetched} fetched in {elapsed:.0f}s -> {per_call:.2f} s/call")
assert fetched == 0 or per_call >= 1.0
# 3588 fetched in 3954s -> 1.10 s/call

print(pd.read_sql_query(
    "SELECT status, count(*) AS n FROM resolved GROUP BY status", store
).to_string(index=False))
# status     n
#  empty   541
#  error    57
#     ok  8319

The s/call figure is the honest audit of the policy claim: if it drops below min_delay_seconds the pacing was bypassed somewhere — usually a second RateLimiter instance, or a worker pool. The error count should be a rounding error; a few hundred means the endpoint was struggling and those rows deserve another pass rather than a manual review.

Edge Cases & Debugging

• Every result is None and nothing raised. swallow_exceptions is at its default of True, so failures were converted into return_value_on_exception. Set it to False and let your own handler classify the error.
• 429s despite min_delay_seconds=1.1. RateLimiter paces one instance in one process. Two notebooks, a worker pool, or a colleague on the same office IP each pace independently and the server sees the sum. Serialise resolution into a single process, or self-host — the trade-offs are laid out in Nominatim vs Pelias for self-hosted geocoding.
• retry_after is always None. The except GeocoderQuotaExceeded clause is catching the 429 first, because GeocoderRateLimited inherits from it. Move the specific clause above the general one.
• sqlite3.OperationalError: database is locked. The store is on a synced folder or a network share, or two processes are writing. Keep it on local disk with journal_mode=WAL and exactly one writer.
• A rerun still issues thousands of requests. address_key is not stable — an unpinned normalisation step, a changed separator, or trailing whitespace produces new keys for identical addresses. Compare len(set(keys)) against SELECT count(*) FROM resolved before blaming the store.
• Distances are ten times too large or too small. The measurement ran in degrees, or in EPSG:3857. Call estimate_utm_crs() and to_crs() on both operands first, and check sites.crs.is_projected before trusting any number in metres.

Frequently Asked Questions

Does min_delay_seconds=1 guarantee I stay inside a one-request-per-second policy? Not on its own. The value is a floor on the interval between call starts within a single RateLimiter instance, measured against a monotonic clock, so nothing prevents a second instance, a second process, or a thread pool from tripling your real rate. Use 1.1 rather than 1.0 so that clock granularity and DNS jitter cannot push two calls into the same second, keep resolution single-threaded against any shared endpoint, and treat a 429 as a signal to slow the whole run down rather than to retry harder.

Should I use RateLimiter's retries or write my own? Both, at different layers. Leave max_retries at one or two to absorb the momentary blips — a dropped connection, a single 503 — because that path is cheap and already written. Handle 429 yourself, because RateLimiter's wait is a flat error_wait_seconds that ignores the Retry-After value the server just sent you, and honouring that number is the difference between being throttled and being blocked. If you want the library out of the retry business entirely, set max_retries=0 and keep it purely as a pacer.

Can I speed this up with threads or asyncio? Not against a shared public endpoint — concurrency there is the fastest route to a ban, and geopy ships AsyncRateLimiter for providers whose terms actually permit parallel streams. The real throughput levers are ordered differently: deduplicate the column first (typically a 25–35% reduction), let the store absorb every rerun, and once a batch exceeds roughly ten thousand distinct addresses, run your own instance, where the only limit is your hardware.

What should happen to addresses that come back empty? Store them as empty and keep the row in the frame with a null geometry. A negative result is expensive information — it cost a full request — and caching it stops every future run re-attacking the provider with the same unmatchable strings. Give those entries a review date rather than treating them as permanent: provider indexes improve, and re-running only the empty rows after a data refresh is a one-line WHERE clause. Dropping the rows instead corrupts every count downstream.