Resampling GPS Traces to a Fixed Interval in pandas

This page answers a narrow question: given one vehicle’s cleaned GPS fixes at an irregular interval, how do you produce a fixed-interval trace in pandas without inventing movement? It extends the trace resampling and densification topic with a single self-contained implementation you can drop into a preprocessing job.

The short version: do not use resample(). Use reindex() with a nearest-neighbour tolerance to move real fixes onto grid slots, then interpolate the remaining holes under an explicit limit, then label every row with where its coordinates came from. resample().mean() averages positions and resample().interpolate() extrapolates past the ends of the trace; both produce coordinates that look like measurements and are not.

Compatibility and Configuration Requirements

The grid interval and the interpolation limit are the two parameters that matter, and both should be derived from the device’s observed rate rather than from its configured one. A device advertising 30-second reporting whose 90th-percentile interval is four minutes is not a 30-second device.

The Resampler

from __future__ import annotations

import numpy as np
import pandas as pd


def resample_trace(
    fixes: pd.DataFrame,
    interval: str = "30s",
    tolerance: str = "10s",
    gap_limit: int = 2,
    coord_cols: tuple[str, str] = ("lat", "lon"),
) -> pd.DataFrame:
    """Resample one vehicle's fixes onto a fixed grid.

    Parameters
    ----------
    fixes
        Cleaned fixes for a single vehicle. Must contain a tz-aware ``ts_utc``
        column plus the two coordinate columns.
    interval
        Grid spacing. Never set this finer than the device's median interval —
        upsampling adds rows, not information.
    tolerance
        How far a real fix may be moved to land on a slot. Keep it at or below
        one third of ``interval`` so no two slots compete for the same fix.
    gap_limit
        Maximum number of consecutive empty slots that may be interpolated.
        Anything longer stays null and is reported as missing.

    Returns
    -------
    DataFrame indexed by grid slot with the coordinate columns plus a
    ``provenance`` column taking the values ``observed``, ``interpolated``
    or ``missing``.
    """
    lat_col, lon_col = coord_cols

    src = (
        fixes.loc[:, ["ts_utc", lat_col, lon_col]]
        .dropna(subset=["ts_utc"])
        .drop_duplicates(subset="ts_utc", keep="last")
        .sort_values("ts_utc")
        .set_index("ts_utc")
    )
    if src.empty:
        return src.assign(provenance=pd.Series(dtype="string"))

    grid = pd.date_range(
        src.index.min().floor(interval),
        src.index.max().ceil(interval),
        freq=interval,
        tz="UTC",
    )

    # 1. Alignment only: every value here is a real measurement, possibly
    #    moved by up to `tolerance`. No coordinates are created.
    aligned = src.reindex(grid, method="nearest", tolerance=pd.Timedelta(tolerance))
    observed = aligned[lat_col].notna()

    # 2. Gap-limited interpolation. `limit_area="inside"` refuses to
    #    extrapolate beyond the first and last real fix.
    filled = aligned.copy()
    for col in (lat_col, lon_col):
        filled[col] = aligned[col].interpolate(
            method="time", limit=gap_limit, limit_area="inside"
        )

    # 3. Provenance for every emitted row.
    provenance = np.where(
        observed,
        "observed",
        np.where(filled[lat_col].notna(), "interpolated", "missing"),
    )
    filled["provenance"] = pd.Series(provenance, index=grid, dtype="string")
    filled.index.name = "slot_utc"
    return filled

Three details carry most of the correctness.

method="nearest" with a tolerance, not resample(). Every coordinate that comes out of step one was measured. A slot with two candidate fixes takes the nearer one and discards the other rather than averaging them into a position the vehicle never occupied.

limit=gap_limit counts slots, not seconds. Two slots on a 30-second grid is a minute; the same two slots on a 5-second grid is ten seconds. Expressing the limit in slots means the parameter travels correctly when the grid changes, which it will.

limit_area="inside". Without it, pandas extrapolates before the first fix and after the last one. That silently extends a trace beyond the period anything was observed, which is how a vehicle ends up apparently parked at a customer site for the twenty minutes before it arrived.

Execution and Tuning Guidelines

Run it per vehicle-day and concatenate; the function is deliberately single-vehicle so that the grouping stays visible in the caller.

out = (
    fixes.groupby(["vehicle_id", "operating_date"], group_keys=True)
    .apply(lambda g: resample_trace(g, interval="30s", gap_limit=2))
    .reset_index()
)

interval is the parameter people get wrong first. Setting it finer than the device’s median interval produces a table that is mostly interpolation, and every downstream consumer that counts rows — sample counts, variance windows, point-in-polygon tests — will be reading a number driven by the grid rather than by the vehicle. Start at the median observed interval, rounded up to something tidy.

tolerance trades reproducibility against coverage. At one third of the interval, a fix belongs unambiguously to one slot. At half the interval, two slots can both claim it and the winner depends on iteration order, which changes between pandas releases. Raising the tolerance also silently smears the timing of turns, because a fix moved by fourteen seconds at 50 km/h has been moved by nearly two hundred metres along the path.

gap_limit is the safety-critical one. Raising it from 2 to 6 on a 30-second grid means the resampler will now draw a straight line across three minutes of unobserved driving. The effect on totals is not symmetric: filling across a coverage hole tends to under-report distance, while filling across a stop adds distance that never happened.

A useful habit is to emit the provenance distribution as a metric on every run:

share = out.groupby("vehicle_id")["provenance"].value_counts(normalize=True).unstack(fill_value=0)
assert (share.get("interpolated", 0) < 0.25).all(), "device family needs a coarser grid"

An assertion like this one has to be able to fail to be worth anything. Point it at a device family that reports every four minutes and it will fire immediately, which is the correct response — that family should not be on a 30-second grid at all.

Common Pitfalls Specific to This Technique

Interpolating latitude and longitude independently across a curve. Linear interpolation of the two coordinates cuts the corner, so a filled point on a bend sits inside the turn rather than on it. For a single slot at urban speeds the error is a few metres and tolerable; across three slots on a motorway slip road it can be sixty. This is one of the reasons the gap limit is small.

Forgetting that method="time" needs a DatetimeIndex. With a plain integer index, interpolate silently falls back to treating rows as equally spaced, which is exactly wrong on a grid that contains nulls. The resampler sets the index before interpolating for this reason.

Reindexing a non-monotonic index. A trace that has been concatenated from two files, or that contains a device clock stepping backwards, will raise ValueError: index must be monotonic increasing. Sorting is not enough if duplicate timestamps remain — deduplicate first, keeping the last observation, which is what the implementation above does.

Validating the Output Before It Leaves the Job

A resampler is easy to get subtly wrong and hard to notice, because every failure mode produces a well-formed table. Four assertions catch essentially all of them, and all four are cheap enough to run on every partition rather than in a nightly audit.

Row count against elapsed time. The number of emitted slots must equal the span of the trace divided by the interval, plus one. A mismatch means the grid was built from the wrong bounds — usually a timezone-aware and timezone-naive comparison silently succeeding.

Observed count against input count. Every input fix should land on exactly one slot or be dropped for being outside the tolerance. If the observed count exceeds the input count, two slots have claimed the same fix, which means the tolerance is too generous relative to the interval.

Monotonic index. The output index must be strictly increasing with a constant step. A duplicated slot is the signature of a grid built across a daylight-saving fold in local time rather than in UTC.

Provenance distribution within bounds. Assert an upper bound on the interpolated share and alert rather than fail when it is exceeded, because the correct response is usually a configuration change rather than a rejected batch.

def validate(out: pd.DataFrame, src: pd.DataFrame, interval: str) -> None:
    step = pd.Timedelta(interval)
    expected = int((out.index.max() - out.index.min()) / step) + 1
    assert len(out) == expected, f"{len(out)} rows for {expected} slots — check the grid bounds"
    assert out.index.is_monotonic_increasing and out.index.is_unique
    n_obs = int((out["provenance"] == "observed").sum())
    assert n_obs <= len(src), f"{n_obs} observed rows from {len(src)} input fixes — tolerance too wide"

The third assertion is the one that has to be able to fail. Feed the resampler a trace whose grid is built in Europe/Amsterdam rather than UTC and run it across the October clock change: the hour between 02:00 and 03:00 occurs twice, the index gains duplicates, and the assertion fires. A check that has never rejected anything is not evidence that the code is correct.

Related

• Trace resampling and densification — the parent topic covering grid choice, densification and simplification
• Snapping irregular fixes to a common time grid with Polars — the same job at fleet scale
• Choosing an interpolation limit for fleet GPS gaps — how to derive gap_limit from your own traces
• Interpolating GPS gaps during tunnel signal loss — what to do with the gaps this resampler refuses
• GPS Data Preprocessing & Cleaning Fundamentals — parent section