Compressing Fleet Trajectory Archives Without Losing Fidelity

After the encoding work in writing matched trajectories to GeoParquet, the geometry column is four fifths of what remains. The tempting next step is to simplify the lines, and that is the one thing not to do — a simplified line reports a shorter distance, and an archive whose distances are wrong is not an archive of anything.

There are three techniques that shrink the geometry column substantially while keeping every vertex and every digit anybody can measure. Together they typically remove seventy percent of an archive, and none of them changes an answer.

Compatibility and Configuration Requirements

That last row is the real cost. A delta-encoded geometry column is no longer readable by a generic geospatial tool, so this is a technique for the deep archive rather than for the tier analysts query directly. Keeping the last two years in standard WKB and delta-encoding everything older is a sensible split.

Three Techniques

Drop what can be recomputed

length_m is ST_Length(geom). mean_speed_kmh is length over duration. Both are convenient and both are derivable, and in a columnar archive each is eight bytes per row that compresses poorly because the values are effectively random.

Removing derivable columns is the cheapest saving available and the one most often overlooked, because in a row store those columns are genuinely worth having. In an archive read by a query engine that can compute them on the fly, they are not.

Quantise to the accuracy you actually have

A double-precision coordinate carries about fifteen significant digits. A consumer-grade GNSS fix has between three and eight metres of real error. Storing the twelfth decimal place of something accurate to five metres is storing noise, and noise is the least compressible thing there is.

import numpy as np

GRID_M = 0.01          # 1 cm — two orders of magnitude finer than the real error

def quantise(xy: np.ndarray, grid_m: float = GRID_M) -> np.ndarray:
    """Round projected coordinates onto a fixed grid and return integers."""
    return np.rint(xy / grid_m).astype(np.int64)

A centimetre grid is far finer than any fleet receiver justifies and still turns a double into an integer that delta-encodes well. The judgement here is to pick a grid you can defend — ten to a hundred times finer than the measurement error — and then never revisit it, because changing the grid means rewriting the archive.

Delta-encode along the line

Consecutive vertices of a matched trajectory are metres apart. Their absolute coordinates are seven-digit numbers with no relationship a compressor can exploit; their differences are two- or three-digit numbers that zstd compresses to almost nothing.

def encode_line(xy: np.ndarray, grid_m: float = GRID_M) -> tuple[np.ndarray, np.ndarray]:
    """Absolute first vertex, integer deltas thereafter."""
    q = quantise(xy, grid_m)
    origin = q[0]
    deltas = np.diff(q, axis=0).astype(np.int32)
    return origin, deltas


def decode_line(origin: np.ndarray, deltas: np.ndarray, grid_m: float = GRID_M) -> np.ndarray:
    q = np.vstack([origin, origin + np.cumsum(deltas, axis=0)])
    return q * grid_m

decode_line(*encode_line(xy)) returns the original coordinates to within the grid, which is the whole claim. Assert it on a sample every time the archive is written — a decoder that drifts is the one failure mode this technique has, and it is silent.

What It Adds Up To

Applied together to a fleet month, the three techniques compound because each makes the next more effective — dropping columns leaves a higher proportion of compressible geometry, and quantisation is what makes the deltas small enough to encode in 32 bits.

Verifying That Nothing Was Lost

A compression scheme that is claimed to be faithful has to be able to prove it, and the proof belongs in the pipeline rather than in a document.

def assert_roundtrip(sample: list[np.ndarray], grid_m: float = GRID_M) -> None:
    """Every vertex must survive encode/decode to within half a grid cell."""
    for xy in sample:
        back = decode_line(*encode_line(xy, grid_m), grid_m=grid_m)
        assert back.shape == xy.shape, "vertex count changed"
        worst = float(np.abs(back - xy).max())
        assert worst <= grid_m / 2 + 1e-9, f"coordinate moved {worst:.4f} m"


def assert_metrics_stable(before, after, tol_m: float = 0.05) -> None:
    """Derived metrics must be unchanged beyond the quantisation floor."""
    d = abs(before.length - after.length)
    assert d < tol_m, f"length changed by {d:.3f} m"

Both assertions must be able to fail. Point the first at a grid of fifty metres and it fires immediately; point the second at a simplified geometry and it fires too. That is the difference between this technique and simplification, stated as a test rather than as a claim.

Run them on a sample of every archive write, not once during development. The failure mode that matters — a decoder change that drifts by one grid cell per vertex — only appears at scale, and only the cumulative-sum reconstruction can produce it.

Common Pitfalls Specific to This Technique

Quantising in degrees. A grid of 1e-7 degrees is 1.1 cm of latitude and between 1.1 and 0.4 cm of longitude depending on where you are, so the effective precision varies across the fleet. Quantise in a projected CRS, always.

Choosing the grid from the storage saving rather than from the accuracy. A metre grid saves a little more than a centimetre grid and starts being visible in stop centroids and geofence tests. Derive it from the receiver, then leave it alone.

Delta-encoding across line boundaries. Deltas must reset at the start of every linestring. Continuing across a boundary produces one enormous delta and, worse, makes every subsequent reconstruction depend on the previous row’s decode succeeding.

Applying this to the tier analysts query directly. Standard WKB is what makes a file readable by every geospatial tool. Keep the recent tier standard and compress only the deep archive, where the reader is your own pipeline.

Deciding Which Tier to Compress

These techniques trade interoperability for size, so the decision is really about which readers a tier has to serve.

The recent tier serves everybody. Analysts with a notebook, a BI tool, an occasional external consumer. It must be standard GeoParquet with WKB geometry, because the cost of a file that only your pipeline can read is paid by every person who tries to use it.

The deep archive serves one reader. By the time data is three years old, essentially the only thing that opens it is a reprocessing job you control. That tier can afford a custom encoding, because there is exactly one decoder to maintain and it lives beside the writer.

A workable boundary is the retention window of the interactive tools. Whatever period the dashboards and ad-hoc queries actually reach back into stays standard; everything older gets compressed. For most fleets that boundary sits somewhere between eighteen months and three years.

Migrating between tiers

Compression should be a scheduled job over whole partitions, never a per-row operation. Read a month, encode it, write it under a new prefix, verify the round trip, then delete the original. Verifying before deleting is the entire safety property, and it is easy to lose when the job is rewritten for speed.

Keep the decoder version in the file metadata. A delta-encoded archive whose decoder has changed since it was written is a data-loss incident waiting for someone to notice, and a version string in the metadata turns it into a compatibility check the reader can make on open.

GEO_META["columns"]["geom"]["encoding"] = "delta-int32-cm"
GEO_META["columns"]["geom"]["decoder_version"] = "2"

The other thing worth recording is the grid. A file compressed at a centimetre grid and one compressed at a decimetre grid are visually identical and differ in what they can support downstream, and the only way to tell them apart later is to have written it down.

Related

• Storing and querying matched trajectories — parent topic and the tiering argument
• Writing matched trajectories to GeoParquet with Polars — the encoding work this page builds on
• Douglas-Peucker simplification for fleet trace storage — the lossy alternative, and when it is appropriate
• Storing matched traces in PostGIS for fleet analytics — the tier that stays uncompressed
• Trajectory Analysis & Map Matching Techniques — parent section