Geocoding & Address Data Pipelines in Python

An address is a human convention; a coordinate is a measurement. Geocoding is the probabilistic bridge between them, and treating it as a deterministic lookup is how pipelines end up with parcels in the wrong county and delivery points in the sea. This stage of Geospatial Data Ingestion & Processing Workflows covers the full address-to-geometry path in Python: normalising raw strings before they ever hit a provider, choosing between Nominatim, Photon, Pelias and commercial services, staying inside usage policy with pacing and backoff, caching so a rerun costs nothing, grading match quality instead of trusting every hit, and attaching the result to a GeoPandas DataFrame in the right coordinate reference system. Downstream it feeds directly into Spatial Joins & Merging and the projection discipline set out in Coordinate Reference System Transformations.

Architecture & Data Structures

A geocoding pipeline has four distinct stages, and conflating them is the root of most production trouble. Normalisation is deterministic, local, and free — it turns messy user input into a canonical query string. Resolution is a network call against a provider's index, slow and rate-limited. Scoring decides whether the answer is good enough for the use case. Attachment binds the accepted coordinate to a row and a CRS. Only the second stage costs money or quota, which is exactly why the other three deserve as much engineering attention as they do.

geopy is the common client layer. It normalises a dozen provider APIs behind three objects:

from geopy.geocoders import Nominatim

# One adapter for the whole run. The user agent must identify YOUR application
# and give a contact — a generic or missing UA is rejected with HTTP 403.
geolocator = Nominatim(user_agent="parcel-audit/1.2 (gis@example.org)", timeout=10)

hit = geolocator.geocode("Piazza del Colosseo 1, Roma", addressdetails=True)
print(hit.latitude, hit.longitude)   # 41.89… 12.49…  → note: LAT first from geopy
print(hit.raw["addresstype"], hit.raw["address"].get("postcode"))

The provider landscape splits three ways. Nominatim is the reference OpenStreetMap geocoder: excellent coverage of named features, weaker on house numbers outside well-mapped regions, and governed by a usage policy the public instance enforces. Photon wraps the same OSM data in Elasticsearch, so it tolerates typos and partial input and is the right choice for interactive search boxes, with location_bias to prefer results near a known point. Pelias composes several open datasets (OpenAddresses, OSM, Who's on First, GeoNames), returns a confidence score, a match_type and a layer on every result, and accepts genuinely structured queries — which is why it is the usual target for a self-hosted deployment; the trade-offs are laid out in Nominatim vs Pelias for self-hosted geocoding. Commercial APIs buy rooftop-level precision, an SLA and support, at the cost of per-request pricing and licence terms that frequently restrict how long you may store the returned coordinates.

Address to GeoDataFrame ingestion path A raw address column flows through six stages: normalise, cache lookup, provider call, quality gate, and finally a GeoDataFrame. A green shortcut runs from the cache lookup straight to the GeoDataFrame, skipping the network on a cache hit. Three red failure branches drop out of the main line: unparseable input goes to a manual-fix list, transport failures such as 429 or timeout go through exponential backoff and end as an error status, and low-grade matches are kept and flagged with no geometry attached. No branch discards the row. Address to GeoDataFrame: the ingestion path cache hit — no network call Raw address one CSV column Normalise casefold + expand Cache lookup sha256 key Provider call paced + backoff Quality gate grade the match GeoDataFrame EPSG:4326 Unparseable input no number, no locality never sent to a provider held in a manual-fix list Transport failure 429, timeout, 502, 503 exponential backoff first then status = error Low-grade match locality or centroid only row kept and flagged no geometry attached Every branch keeps the row — a geocoder that silently drops input destroys the audit trail.
The cache sits before the network call, so a rerun of the same batch issues zero requests; every failure mode has a named landing place rather than a silent drop.

Environment Configuration & Dependency Resolution

python -m pip install "geopy>=2.4,<3" "geopandas>=1.0" "shapely>=2.0" \
                      "pandas>=2.2" "pyproj>=3.6" "usaddress>=0.5.10"

geopy is pure Python with no compiled dependencies — it uses the standard library's urllib by default, so it does not inherit requests' proxy or session behaviour and does not participate in requests_cache. That single fact drives the design below: the cache has to be yours. Pin geopy>=2.4 because that release added GeocoderRateLimited, which surfaces the server's Retry-After value instead of collapsing every 429 into a generic quota error.

Provider configuration belongs in the environment, never in the source file. geopy.geocoders.options holds process-wide defaults that every adapter inherits, which is the cleanest place to set a timeout, a proxy, or a custom SSL context for a corporate TLS-inspecting proxy:

import os
from geopy.geocoders import options, Nominatim, Photon, Pelias

options.default_timeout = 10          # seconds, applies to every adapter
options.default_user_agent = "parcel-audit/1.2 (gis@example.org)"
if os.environ.get("HTTPS_PROXY"):
    options.default_proxies = {"https": os.environ["HTTPS_PROXY"]}

def build_geocoder(name: str):
    """Swap providers by configuration, not by editing the pipeline."""
    if name == "nominatim_public":
        return Nominatim(domain="nominatim.openstreetmap.org")
    if name == "nominatim_self":
        return Nominatim(domain=os.environ["NOMINATIM_HOST"], scheme="http")
    if name == "photon":
        return Photon()
    if name == "pelias":
        return Pelias(domain=os.environ["PELIAS_HOST"], scheme="http")
    raise ValueError(f"unknown provider: {name}")

Three configuration rules are worth stating plainly. First, the user agent is not optional on the public Nominatim instance — it must identify the application and provide a contact, and a default or absent value is answered with HTTP 403, not a friendly error. Second, the usage policy is a hard limit, not a suggestion: a maximum of one request per second, no parallel request streams, no bulk geocoding of large datasets on the shared instance, results must be cached rather than re-queried, and OSM-derived output carries ODbL attribution obligations. Third, the provider name belongs in the cache key, because a coordinate produced by Photon and one produced by Pelias are different measurements that must never be mixed in the same column without a provenance flag.

Vectorized Operations & Core Workflow

There is no vectorized geocode. The work is I/O bound and policy-bound, so the throughput levers are entirely different from the array-oriented ones used elsewhere in this section: collapse duplicates, cache aggressively, and pace deliberately. In practice, deduplication alone is the single biggest win — a customer table of 50,000 rows typically contains 12,000 distinct normalised addresses, so the network cost drops by three quarters before a single request is sent.

Normalisation is where pandas is vectorized, and it should do everything that does not require the network:

import pandas as pd

# Order matters: strip noise first, then expand abbreviations on word boundaries.
ABBREVIATIONS = {
    r"\bst\b": "street", r"\bave?\b": "avenue", r"\brd\b": "road",
    r"\bdr\b": "drive",  r"\bblvd\b": "boulevard", r"\bhwy\b": "highway",
    r"\bn\b": "north",   r"\bs\b": "south", r"\be\b": "east", r"\bw\b": "west",
}
UNIT_PATTERN = r"\b(?:apt|apartment|unit|suite|ste|fl|floor)\.?\s*[\w\-]+"

def normalise_addresses(raw: pd.Series) -> pd.DataFrame:
    """Deterministic pre-pass — no network, runs on the whole column at once."""
    text = (raw.fillna("")
               .str.normalize("NFKC")          # collapse compatibility characters
               .str.lower()
               .str.replace(r"[^\w\s,\-/]", " ", regex=True)
               .str.replace(r"\s+", " ", regex=True)
               .str.strip())
    # Units and floor numbers confuse every geocoder — keep them, don't send them.
    unit = text.str.extract(f"({UNIT_PATTERN})", expand=False)
    text = text.str.replace(UNIT_PATTERN, " ", regex=True)
    for pattern, expansion in ABBREVIATIONS.items():
        text = text.str.replace(pattern, expansion, regex=True)
    query = text.str.replace(r"\s+", " ", regex=True).str.strip()
    return pd.DataFrame({"query_norm": query, "unit": unit})

deliveries = pd.read_csv("deliveries.csv", encoding="utf-8-sig")
deliveries = deliveries.join(normalise_addresses(deliveries["address_raw"]))
print(len(deliveries), deliveries["query_norm"].nunique())   # 50000 12143

Note what normalisation deliberately does not do: it keeps accented characters (stripping them breaks European and Latin American matching far more often than it helps), and it keeps the unit designator in a separate column rather than discarding it, because the shipping label still needs it.

The resolution stage is a cache-first loop over the distinct queries. A SQLite file is the right cache: it is transactional, it survives a killed process, it needs no server, and it makes the run idempotent — rerun the same batch and every query is a local read.

import hashlib, json, random, sqlite3, time
from geopy.exc import (GeocoderQuotaExceeded, GeocoderServiceError,
                       GeocoderTimedOut, GeocoderUnavailable)

SCHEMA = """
CREATE TABLE IF NOT EXISTS geocode_cache (
    cache_key  TEXT PRIMARY KEY,
    provider   TEXT NOT NULL,
    query      TEXT NOT NULL,
    status     TEXT NOT NULL,      -- 'ok' | 'empty' | 'error'
    payload    TEXT,               -- the provider's raw JSON, verbatim
    fetched_at REAL NOT NULL
);
"""

def open_cache(path="geocode_cache.sqlite"):
    con = sqlite3.connect(path, timeout=30)
    con.execute("PRAGMA journal_mode=WAL")     # concurrent readers while writing
    con.executescript(SCHEMA)
    return con

def cache_key(provider: str, query: str, params_version: str = "v1") -> str:
    """Bump params_version whenever the query construction changes."""
    return hashlib.sha256(f"{provider}|{params_version}|{query}".encode()).hexdigest()

def call_with_backoff(geocode, query, *, attempts=5, base=1.0, cap=60.0, **kwargs):
    """Retry only the transient failures; let permanent ones propagate."""
    for attempt in range(attempts):
        try:
            return geocode(query, **kwargs)
        except GeocoderQuotaExceeded as exc:
            # geopy >= 2.4 exposes Retry-After on GeocoderRateLimited
            wait = float(getattr(exc, "retry_after", 0) or 0) or min(cap, base * 2 ** attempt)
        except (GeocoderTimedOut, GeocoderUnavailable):
            wait = min(cap, base * 2 ** attempt)
        except GeocoderServiceError:
            raise                                  # auth / bad query: retrying cannot help
        time.sleep(wait + random.uniform(0, 0.25 * wait))   # jitter breaks lockstep
    raise TimeoutError(f"gave up after {attempts} attempts: {query!r}")

RateLimiter handles the pacing half — the guaranteed minimum gap between calls — while call_with_backoff handles the failure half. Keeping them separate is deliberate: pacing must apply to every call including successful ones, whereas backoff applies only after an error, and conflating the two produces a client that either crawls or floods.

from geopy.extra.rate_limiter import RateLimiter

PROVIDER = "nominatim_public"
geolocator = build_geocoder(PROVIDER)
# min_delay_seconds honours the 1 req/s policy with headroom; retries are ours.
paced_geocode = RateLimiter(geolocator.geocode, min_delay_seconds=1.1,
                            max_retries=0, swallow_exceptions=False)

def fill_cache(con, queries, provider=PROVIDER):
    """Resolve only what is missing. Safe to interrupt and rerun."""
    for query in sorted(set(queries)):
        key = cache_key(provider, query)
        if con.execute("SELECT 1 FROM geocode_cache WHERE cache_key = ?", (key,)).fetchone():
            continue                                    # already resolved — skip
        try:
            location = call_with_backoff(paced_geocode, query,
                                         exactly_one=True, addressdetails=True)
            status = "ok" if location else "empty"
            payload = json.dumps(location.raw) if location else None
        except Exception as exc:                        # exhausted or permanent
            status, payload = "error", json.dumps({"error": repr(exc)})
        con.execute("INSERT OR REPLACE INTO geocode_cache VALUES (?,?,?,?,?,?)",
                    (key, provider, query, status, payload, time.time()))
        con.commit()                                    # commit per row: crash-safe

cache = open_cache()
fill_cache(cache, deliveries["query_norm"].dropna())

Committing after every row looks wasteful and is not — the transaction is microseconds next to a one-second network pause, and it means a Ctrl-C at request 8,412 loses nothing. Caching empty results matters as much as caching hits: without a negative cache, every rerun re-attacks the provider with the same unmatched addresses. Give negative entries a review date rather than treating them as permanent; provider indexes improve, and re-running only the empty rows after a data refresh is a one-line WHERE clause. The pacing and retry mechanics have their own worked walkthrough in batch geocoding with geopy without rate-limit errors.

State of one geocoding request under rate limiting A single request starts in the Ready state, waits at least 1.1 seconds for its pacing slot, then goes in flight with a ten second timeout. Three outcomes follow. A 200 OK writes the cache row and advances to the next query. A 429 Too Many Requests leads to a sleep for the duration the server dictates in the Retry-After header, then a retry. A timeout or 502 or 503 leads to an exponential backoff sleep of base times two to the attempt number plus jitter, capped at sixty seconds, then a retry. When the attempt counter reaches its maximum the request gives up and writes an error status, keeping the row with a null geometry. One request, four possible states Ready wait 1.1 s Request in flight timeout = 10 s 200 OK a result, or an empty body 429 Too Many Requests Retry-After header Timeout / 502 / 503 transport failure Write the cache row advance to next query Sleep Retry-After server-dictated pause Sleep base x 2^n + jitter capped at 60 s retry while attempt < 5 retry while attempt < 5 attempts exhausted Give up — status = error row survives, geometry stays null
A 429 is answered with the server's own Retry-After; a timeout is answered with exponential backoff and jitter. Only the exhausted path writes an error, and even then the row survives.

Address Matching & Result Quality Details

Free-text queries make the provider guess at structure. When you already know which token is the house number and which is the postcode, say so — a structured query removes an entire class of ambiguity and typically lifts the rooftop-match rate by double digits. usaddress parses US-style addresses with a trained model; libpostal (via the postal Python bindings, which need the compiled C library) handles international formats.

import usaddress

tagged, address_type = usaddress.tag("1600 Amphitheatre Pkwy, Mountain View, CA 94043")
# tagged  -> OrderedDict([('AddressNumber', '1600'), ('StreetName', 'Amphitheatre'),
#                         ('StreetNamePostType', 'Pkwy'), ('PlaceName', 'Mountain View'),
#                         ('StateName', 'CA'), ('ZipCode', '94043')])
# address_type -> 'Street Address'

def to_structured(tagged: dict) -> dict:
    """Nominatim accepts a dict query — one field per address component."""
    street = " ".join(filter(None, [tagged.get("AddressNumber"),
                                    tagged.get("StreetName"),
                                    tagged.get("StreetNamePostType")]))
    return {"street": street or None,
            "city": tagged.get("PlaceName"),
            "state": tagged.get("StateName"),
            "postalcode": tagged.get("ZipCode"),
            "country": "us"}

# geolocator.geocode({"street": "1600 amphitheatre parkway", "city": "mountain view", ...})

usaddress.tag raises RepeatedLabelError when the same label appears twice (two house numbers, typically a range or a mangled paste). Catch it and fall back to the free-text query rather than letting one bad row kill the batch.

Once a result comes back, the question is not did it match but how well. Every provider answers that question in its own vocabulary, and the pipeline's job is to collapse those vocabularies into one ordered grade. Nominatim exposes addresstype, class/type, and an address sub-dictionary when addressdetails=True; Pelias returns confidence (0–1), match_type (exact, interpolated, fallback) and layer (address, street, locality, …); Google-style APIs return location_type values such as ROOFTOP and GEOMETRIC_CENTER.

def grade_nominatim(raw: dict, wanted_number: str | None) -> str:
    """Collapse Nominatim's signals into one ordered quality grade."""
    kind = raw.get("addresstype") or raw.get("type") or ""
    got_number = (raw.get("address") or {}).get("house_number")
    if got_number and wanted_number and got_number == wanted_number:
        return "rooftop"
    if kind in {"house", "building"} and got_number:
        return "rooftop"
    if kind in {"road", "residential", "pedestrian"}:
        return "street"
    if kind in {"postcode", "suburb", "neighbourhood", "quarter"}:
        return "locality"
    return "coarse"

GRADE_ORDER = {"rooftop": 0, "street": 1, "locality": 2, "coarse": 3, "unmatched": 4}
ACCEPT_AT_OR_ABOVE = "street"        # policy lives in ONE place, as data
Match-quality grades mapped to provider signals and actions Four grades in descending quality. Rooftop: OSM reports addresstype house with a house number, Pelias reports match_type exact on the address layer, and the action is to accept and attach the point. Interpolated: OSM returns an address range on a way, Pelias reports match_type interpolated, and the action is to accept for aggregation but not for dispatch. Street or centroid: OSM reports addresstype road, Pelias reports the street layer, and the action is to send the row to a review queue. Locality or worse: OSM reports a city or postcode addresstype, Pelias reports the locality layer with confidence under 0.5, and the action is to reject the point while keeping the flagged row. What the provider said, and what to do about it Grade OSM (Nominatim) Pelias Pipeline action Rooftop addresstype = house house_number present match_type = exact layer = address Accept — attach the point safe for dispatch and routing Interpolated osm_type = way address range on a line match_type = interpolated Accept for aggregation tens of metres of error Street or centroid addresstype = road no house_number layer = street confidence ~ 0.6 Review queue usable at street level only Locality or worse addresstype = city or postcode layer = locality confidence < 0.5 Reject the point keep the row, flag it, move on
One ordered grade per row, derived from whichever signals the provider happens to publish — the rest of the pipeline reads the grade, never the provider-specific field.

Partial matches are the interesting case, and the wrong instinct is to discard them. A locality-level hit still tells you the city; for a choropleth of complaints per district it is perfectly adequate, while for dispatching a technician it is useless. Keep the coordinate and the grade in the frame, and let each downstream consumer filter on the grade it needs. The one thing never to do is write a locality centroid into a column called location with no provenance — that is how a hundred customers end up stacked on a town hall.

Reverse geocoding runs the same machinery backwards: geolocator.reverse((lat, lon)) takes a latitude-first tuple, which is the opposite order from every Shapely constructor you will use in the next paragraph. For anything beyond a few thousand points, do not call a provider at all — load the administrative boundaries once and run a point-in-polygon join locally, which is faster by orders of magnitude and has no quota; that path is worked through in reverse geocoding points to administrative boundaries.

CRS Alignment & Projection Pipeline

Every mainstream geocoder returns geographic coordinates on WGS 84 — EPSG:4326, degrees of longitude and latitude. That is the input contract, and it is also a trap, because two adjacent APIs disagree about ordering. geopy hands back (latitude, longitude); Shapely's Point(x, y) and geopandas.points_from_xy(x, y) want longitude first. Getting this backwards moves a point in Rome to the Somali coast, and because both values are plausible numbers, nothing raises.

import geopandas as gpd
import pandas as pd

# results: DataFrame with query_norm, lat, lon, match_grade — read back from the cache
deliveries = deliveries.merge(results, on="query_norm", how="left")

located = deliveries[deliveries["lat"].notna()].copy()
points = gpd.GeoDataFrame(
    located,
    geometry=gpd.points_from_xy(located["lon"], located["lat"]),  # x = lon, y = lat
    crs="EPSG:4326",                                              # declare, never guess
)

# Cheap sanity gate: anything at exactly (0, 0) is a failed parse, not a location.
null_island = points.geometry.x.abs().lt(1e-6) & points.geometry.y.abs().lt(1e-6)
assert not null_island.any(), points.loc[null_island, "query_norm"].tolist()

Keep the frame in EPSG:4326 for storage and for handing to a web map, and project the moment any distance enters the picture. The canonical geocoding quality check is exactly such a measurement: run two providers over the same addresses and measure how far apart their answers are. Degrees make that number meaningless — a degree of longitude is 111 km at the equator and 71 km in Paris — so reproject to a metric CRS first. estimate_utm_crs() picks the correct UTM zone from the data's own extent, the automated version of the logic in choosing the right UTM zone automatically.

# Provider disagreement as a quality metric — metres, not degrees.
metric_crs = points.estimate_utm_crs()          # e.g. EPSG:32633 for central Italy
osm_pts = points.to_crs(metric_crs)
pelias_pts = pelias_points.to_crs(metric_crs)   # same index, second provider

gap_m = osm_pts.geometry.distance(pelias_pts.geometry, align=True)
points["provider_gap_m"] = gap_m.to_numpy()

# Two independent geocoders landing 500 m apart means neither is trustworthy here.
suspect = points[points["provider_gap_m"] > 500]

Never reach for EPSG:3857 to make this measurement "metric". Web Mercator's scale factor grows with latitude, so a 500 m threshold silently becomes 700 m in Stockholm and 1,000 m in Reykjavík. It is a display projection and nothing else. Equally, if the accepted points are about to be joined to census tracts or delivery zones, reproject both layers to the same projected CRS before the join — mismatched CRSs are the standard cause of a spatial join that returns zero rows, and the alignment rules live in Coordinate Reference System Transformations.

Production Export & Integration

The production checklist for a geocoding job differs from an ordinary ETL step because the expensive resource is someone else's service:

Choosing a geocoding backend by volume and licence A decision tree with three outcomes. Public OSM endpoints such as Nominatim and Photon suit fewer than ten thousand addresses, capped at one request per second with no bulk runs and ODbL attribution, and fit prototypes and small joins. A self-hosted Nominatim or Pelias stack in Docker has no rate cap, supports structured queries and confidence scores, and fits millions of rows. A commercial API buys rooftop precision, support and per-request pricing, but its results are licence-bound and may not be storable. Choosing the backend volume, then licence, then accuracy Size the batch before picking a provider not the other way round Public OSM endpoints Nominatim, Photon 1 req/s, no bulk runs ODbL attribution under 10k addresses Self-hosted stack Nominatim or Pelias no rate cap, your own SLA structured queries millions of rows Commercial API rooftop precision, support per-request pricing read the storage clause licence-bound results
Volume decides the backend before accuracy does: the public endpoints are a prototyping tool, self-hosting is the answer to bulk, and a commercial contract buys precision with strings attached.

A pragmatic production pattern is the fallback ladder: query the cheap self-hosted provider first, and escalate to a paid one only for the rows that graded below street. Because the cache is keyed by provider, both attempts persist independently, and the escalation set shrinks on every run as the cheap index improves.

Windows / Platform Edge Cases & Debugging

Frequently Asked Questions

Can I geocode half a million addresses with the public Nominatim instance? No. The usage policy explicitly forbids bulk geocoding on the shared instance and caps you at one request per second, which is roughly six days of continuous querying for that volume — and you would be blocked long before finishing. Deduplicate first, then self-host Nominatim or Pelias in Docker, or buy a commercial batch endpoint. Self-hosting also removes the pacing constraint entirely, so the same job finishes in minutes.

Should I cache the coordinates or the whole provider response? The whole response, serialised verbatim. Coordinates are the least interesting part of the payload — the match type, the returned address components, the confidence score and the bounding box are what let you re-grade results later without spending quota again. Storage is trivial (a few hundred bytes per row) next to the cost of re-querying, and it makes the grading logic freely revisable.

Two providers put the same address 60 metres apart. Which one is right? Probably neither is wrong. Sixty metres is the typical gap between a rooftop point, an address-range interpolation along a street centreline, and a parcel centroid — three different definitions of "where the address is". Decide which definition your use case needs, record the grade, and treat the disagreement distance as a quality signal rather than an error. Only a gap of hundreds of metres implies one of them matched the wrong feature entirely.

Is RateLimiter on its own enough to stay inside the rules? Only for pacing inside a single process. It guarantees a minimum delay between calls but knows nothing about other workers, other machines, or the server's own Retry-After instruction. Pair it with the backoff wrapper shown above, keep resolution single-threaded against shared endpoints, and treat any 429 as a signal to slow the whole run down rather than to retry harder.

Do I have to reproject before joining geocoded points to boundary polygons? Both layers must share one CRS, and if the join involves any distance — a nearest-match, a tolerance, a buffer — that shared CRS must be projected and metric. A plain point-in-polygon containment test is valid in EPSG:4326 as long as both sides are genuinely in EPSG:4326, but a UTM or equal-area CRS is the safer default because it keeps the door open for distance work. The predicate mechanics are covered in Spatial Joins & Merging.

What should happen to addresses that never match anything? They stay in the frame with match_grade = "unmatched" and a null geometry. Dropping them silently corrupts every count downstream — a report that says "1,240 incidents" when 300 addresses failed to geocode is simply wrong. Export the unmatched set as its own file for manual review; in practice most of them share a handful of fixable normalisation problems, and repairing those lifts the match rate for the whole batch.

Related