GeoPandas vs Fiona for Large Files

A vector file that comfortably fits in RAM is a job for GeoPandas; one that does not is a job for Fiona's streaming iterator — and knowing which regime you are in is the difference between a clean run and a MemoryError. This guide is for anyone whose gpd.read_file() call dies on a national parcels dump, a country-scale building footprint layer, or any file whose size rivals available memory. It sits under GeoPandas DataFrames Explained in Mastering Core Geospatial Python Libraries, and pairs with the memory-focused workflow in reprojecting large datasets without memory errors.

Why This Approach / What Goes Wrong

gpd.read_file() materializes every feature, geometry, and attribute of a layer into a single in-memory GeoDataFrame. That is exactly what you want for vectorized analysis — the whole point of GeoPandas is that operations like .area, .buffer(), and sjoin run as batched C calls over a column of geometries — but it is fatal when the file is larger than RAM. The read never gets a chance to finish: GDAL/OGR builds the feature array, the process crosses the memory ceiling, and Python raises MemoryError (or the OOM killer terminates the interpreter with no traceback at all).

Fiona is the lower-level door into the same GDAL/OGR drivers that GeoPandas reads through. Instead of returning a table, fiona.open() returns a lazy iterator: each for feature in src pull decodes exactly one feature into a plain Python dict, and the previous one is free to be garbage-collected. Memory stays flat regardless of whether the file holds ten thousand features or two hundred million. The trade is that you have left vectorization behind — every geometry is a Python object you touch one at a time, and per-feature Python is inherently slower than a batched C loop.

The right pattern for large files is rarely "all Fiona" or "all GeoPandas." It is a funnel: stream-filter with Fiona down to the subset you actually care about, then hand that subset to GeoPandas for the vectorized work. The classic mistake is loading an entire national file just to keep 2% of it — the commercial parcels, the features in one county, the buildings above a height threshold. Fiona lets you make that cut before anything hits the DataFrame machinery.

Modern GeoPandas narrows the gap from the other side. Since 0.14 the default I/O engine is pyogrio, which vectorizes reads and — crucially — can push a spatial or attribute filter down into GDAL so the features you do not want are never decoded into Python at all. A bbox= or mask= argument filters by geometry during the read; a where= argument applies a SQL-style attribute predicate in the driver. When your subset is a small slice of a large file, a pushdown read is both simpler and faster than a manual Fiona loop, and it returns a ready-to-use GeoDataFrame. Reach for the raw Fiona stream when even the matching subset is too large to hold, or when your filter logic is more complex than a bounding box or a single SQL clause. The geometry objects both paths yield are Shapely geometries once inside GeoPandas, and the file format matters too: a spatially indexed cloud-native container like FlatGeobuf or GeoParquet makes bounding-box pushdown genuinely cheap, whereas a plain Shapefile or GeoJSON forces a near-full scan no matter which engine you use.

Prerequisites

• geopandas>=0.14 — ships with pyogrio as the default engine, enabling bbox/where read-time pushdowns
• fiona>=1.9 — the streaming feature iterator over GDAL/OGR
• shapely>=2.0 — geometry objects and the shape() / mapping() bridge between dicts and geometries
• pyogrio>=0.7 — the vectorized read/write engine GeoPandas delegates to

conda install -c conda-forge "geopandas=0.14.*" "fiona=1.9.*" "shapely=2.0.*" "pyogrio=0.7.*"

Install from conda-forge rather than mixing pip wheels — GeoPandas, Fiona, and pyogrio each bind the GDAL C library, and letting conda resolve one consistent GDAL build avoids the ABI and PROJ-path mismatches that produce silent CRS failures.

Step-by-Step Implementation

1. The convenient path (subset fits in memory): GeoPandas with a pushed-down filter.

When the features you want are a small slice of a large file, let pyogrio do the filtering during the read. Only the matching features are ever decoded, so peak memory tracks the result size, not the file size.

import geopandas as gpd

# Read only features intersecting a bbox — pyogrio filters during the read.
# bbox is (xmin, ymin, xmax, ymax) in the FILE's CRS, not necessarily lon/lat.
aoi = (7.6, 45.0, 7.8, 45.1)
city_parcels = gpd.read_file("national_parcels.fgb", bbox=aoi)
print(len(city_parcels), "features loaded")

For an attribute cut, push a SQL where clause into the driver instead of filtering in Python after the fact:

# Filter by attribute at read time — GDAL evaluates the predicate, not pandas
commercial = gpd.read_file("national_parcels.fgb", where="use = 'commercial'")

2. The streaming path (subset still exceeds memory): a Fiona iterator with a flat profile.

When even the matching subset is too large to load, or the filter logic is richer than a bbox or single SQL clause, drop to Fiona and iterate. Memory stays flat because only one feature dict is live at a time.

import fiona

# Keep only commercial parcels from a file too large to load whole.
# Read src.crs BEFORE leaving the context so it survives past the `with`.
kept = []
with fiona.open("national_parcels.fgb") as src:
    src_crs = src.crs
    total = len(src)
    for feature in src:                     # one feature decoded at a time
        props = feature["properties"]
        if props.get("use") == "commercial" and props.get("area_ha", 0) > 0.5:
            kept.append(feature)

print(f"Filtered {len(kept)} of {total} features")

3. Hand the filtered survivors to GeoPandas for vectorized analysis.

Once the subset is small enough to hold, rebuild a GeoDataFrame and switch back to fast, batched operations. Reproject to a metric CRS before any area or distance work — never measure in geographic degrees or in Web Mercator (EPSG:3857). estimate_utm_crs() picks the right projected zone from the data's extent; the mechanics of that choice are covered in choosing a UTM zone automatically in Python.

import geopandas as gpd

commercial = gpd.GeoDataFrame.from_features(kept, crs=src_crs)   # carry the CRS!
commercial = commercial.to_crs(commercial.estimate_utm_crs())    # metric grid
commercial["area_m2"] = commercial.geometry.area                 # vectorized, fast

4. When the survivors themselves are huge: stream straight to disk, never accumulate.

If your filter still keeps tens of millions of features, do not build the kept list at all — appending full feature dicts reintroduces the memory problem you were avoiding. Instead, open an output with Fiona and write each match as it streams past, so both input and output stay one feature deep.

import fiona

with fiona.open("national_parcels.fgb") as src:
    profile = src.profile                    # schema + crs + driver, reused verbatim
    profile["driver"] = "FlatGeobuf"         # spatially indexed output
    with fiona.open("commercial_parcels.fgb", "w", **profile) as dst:
        for feature in src:
            if feature["properties"].get("use") == "commercial":
                dst.write(feature)           # constant memory, any input size

The result is a smaller, spatially indexed file that a later GeoPandas read_file(..., bbox=...) can slice cheaply — you have converted an unmanageable one-shot read into a repeatable, memory-bounded pipeline.

Verification

Confirm the streaming filter held memory flat and produced the same result a full load would. A quick way to prove correctness is to recount the matches by streaming again and assert the number equals the GeoDataFrame you built.

import fiona

# Count matches by streaming (no full load) and compare to the built subset
with fiona.open("national_parcels.fgb") as src:
    streamed = sum(1 for f in src if f["properties"].get("use") == "commercial")

print("Streamed match count:", streamed)        # Streamed match count: 18254
assert streamed == len(commercial), "Filter mismatch between Fiona and GeoDataFrame"
assert commercial.crs is not None and commercial.crs.is_projected

To prove the memory claim rather than just trust it, wrap the streaming read in a peak-memory probe. A flat peak on a multi-gigabyte file is the whole point of the exercise.

import tracemalloc, fiona

tracemalloc.start()
with fiona.open("national_parcels.fgb") as src:
    n = sum(1 for f in src if f["properties"].get("use") == "commercial")
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print(f"Matched {n} features, peak {peak / 1e6:.1f} MB")   # peak stays low + flat

Edge Cases & Debugging

• MemoryError on read_file. The file exceeds RAM. Switch to a bbox=/where= pushdown read if your subset is small, or to the Fiona stream (step 2) if it is not.
• CRS lost via from_features. GeoDataFrame.from_features() does not read the source projection. Capture src.crs inside the with block and pass it explicitly as crs= — otherwise every downstream to_crs() is undefined.
• bbox= returns everything. The bounding box is interpreted in the file's own CRS. Passing lon/lat degrees against a file stored in a projected metric CRS matches nothing or matches all — convert your AOI to the file's CRS first, and confirm the engine is pyogrio (the legacy Fiona engine ignores some pushdowns).
• Streaming is slow. Per-feature Python iteration cannot compete with vectorized C. Filter as aggressively as possible in the loop, keep only the fields you need, then vectorize the survivors — do not stream work that a where= clause could push into GDAL.
• Memory still climbs while streaming. You are accumulating full feature dicts in a list. Keep only the attributes you need, or write matches straight to disk (step 4) so nothing accumulates.
• The subset is still too big to analyse in GeoPandas. At that scale, move the query off pandas entirely — query GeoParquet with DuckDB Spatial pushes both the filter and the geometry math into a columnar engine that never loads the whole layer.