RouteMatching

Running Valhalla in Kubernetes for Fleet Workloads

Valhalla’s tile architecture makes it unusually pleasant to operate at fleet scale: the data is a directory of files, the server is stateless, and the memory profile is dominated by page cache rather than heap. That makes it a natural fit for Kubernetes — and it also makes it easy to deploy in a way that appears healthy and serves slow, wrong or failing requests.

The three things that go wrong are always the same. Tiles are treated as mutable state rather than a versioned artefact, memory limits are set against the process rather than the mapped working set, and readiness probes report success before the pod can actually match anything. This page covers all three, extending Valhalla and Meili map matching for telematics.

Compatibility and Configuration Requirements

Requirement Value Notes
Valhalla 3.4+ /status with verbose reports tile availability
Tile distribution Image layer or read-only volume Never a writable volume shared between pods
Memory limit Heap plus the mapped working set The mapped portion is the larger term
Readiness probe A real /trace_attributes request /status returns before tiles are warm
Rollout strategy New dataset, new deployment Mutating tiles in place cannot be rolled back

Tiles Are an Artefact, Not State

The single most consequential decision is treating the tile directory as immutable and versioned. A deployment whose pods mount a shared writable volume that a build job updates in place has three problems at once: no rollback, a window where pods serve half-written tiles, and no way to say which dataset produced a given match.

Two distribution patterns work.

Baked into the image. The tile build produces an image tagged with the dataset version, and the deployment references that tag. Rollout and rollback are ordinary deployment operations, the version is visible in kubectl describe, and the pod is entirely self-contained. Practical up to a few gigabytes.

Read-only volume per version. The tiles live on a volume named for the dataset version, mounted read-only. The deployment references the volume by name, so switching datasets is a deployment change rather than a data change. Necessary once the dataset is large enough that image pulls dominate rollout time.

spec:
  containers:
    - name: valhalla
      image: registry.internal/valhalla:3.4.0-tiles-2026-08-03
      env:
        - name: VALHALLA_DATASET_VERSION
          value: "2026-08-03"
      volumeMounts:
        - name: tiles
          mountPath: /data/valhalla
          readOnly: true
  volumes:
    - name: tiles
      persistentVolumeClaim:
        claimName: valhalla-tiles-2026-08-03
        readOnly: true

The readOnly: true is doing real work. It makes it impossible for a misconfigured build job to write into a volume that running pods are reading, which is the failure that produces intermittent, unreproducible matching errors across a subset of pods.

Mutable tiles against versioned tiles With one shared writable volume, a build job overwrites tiles while pods are reading them, and there is no previous version to return to. With one read-only volume per dataset version, pods reference a version by name, a new dataset is a new volume, and rollback is a deployment change. Where the rollback comes from mutable shared volume build job writes /data/valhalla pod A pod B no previous version exists versioned read-only volumes tiles-2026-07-27 tiles-2026-08-03 pods rollback = change one name The upper arrangement also produces intermittent errors while the write is in progress, on some pods only. Those are the hardest incidents to diagnose, because the same request succeeds when retried.

Sizing Memory Against the Mapped Working Set

Valhalla’s resident heap is small — typically a few hundred megabytes. Its tiles are memory-mapped, which means the pages the traffic touches count against the container’s memory limit even though the process never allocated them.

The practical consequence is that a limit set from ps output is far too low, and the pod is killed by the OOM reaper under load with no obvious cause in the application logs. Size the limit as heap plus the working set of tiles the traffic actually touches, which for a metropolitan dataset is typically two to four gigabytes and for a continental one considerably more.

resources:
  requests:
    memory: "3Gi"
    cpu: "1"
  limits:
    memory: "5Gi"        # heap + mapped working set + headroom
    cpu: "4"

The working set is not the dataset size. A fleet operating in one metropolitan area touches a small fraction of a national tile set, and the pages for everywhere else are never faulted in. Measuring it is more reliable than estimating: run representative traffic and watch container_memory_working_set_bytes until it plateaus.

Set requests at the plateau and limits with headroom. Setting them equal is tempting for guaranteed quality of service and leaves no room for the page cache to absorb a traffic pattern that touches a new area.

Heap is not the number to size against Container working set over ten minutes of representative traffic. Resident heap stays flat at around 320 megabytes. The working set climbs as tiles are faulted in and plateaus near 2.6 gigabytes. A limit set from the heap figure would kill the pod within the first minute. Working set against heap, first 10 minutes of traffic 4 GB 2 GB 0 working set — plateaus at 2.6 GB resident heap — 320 MB 0 min 10 min A 512 Mi limit derived from the heap kills the pod inside a minute, with nothing in the application log. Measure the plateau under representative traffic rather than estimating from the dataset size.

Readiness That Means Something

Valhalla’s /status endpoint answers before the tiles are usable. A pod that passes readiness on it joins the load balancer and serves the first few requests slowly or with errors, which shows up as a latency spike on every rollout.

Probe with a real match instead — a short, fixed trace over a road that exists in every dataset:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - >
        curl -sf --max-time 5
        "http://localhost:8002/trace_attributes?json=$(cat /probe/trace.json)"
        | grep -q '"edges"'
  initialDelaySeconds: 10
  periodSeconds: 5
  failureThreshold: 12

Two properties matter. It exercises the same code path as production traffic, so a pod that cannot match cannot pass. And it warms the page cache for the probe’s region, which removes part of the cold start from user-visible traffic.

Extend that idea into an explicit warm-up if rollouts show a latency spike: run a few dozen representative matches during the readiness window, covering the depots that generate most traffic. It costs a few seconds per pod and removes the first-request penalty entirely.

Keep the liveness probe separate and simple. Liveness should restart a wedged process, not a slow one, and pointing it at a match request means a temporary slowdown under load turns into a restart storm.

Rolling a New Dataset

Because tiles are versioned, a dataset change is a deployment change:

  1. Build tiles, tag the artefact with the dataset version.
  2. Deploy a second deployment referencing the new version, scaled to a fraction of capacity.
  3. Shift a small share of traffic and compare match rate and latency against the incumbent.
  4. Scale up, scale the old one down, keep it available for a rollback window.

That is the same fitness-gate pattern used in scheduling OSM extract refreshes, expressed as pods rather than as symlinks. The comparison in step three is the part not to skip: it is the only stage at which a bad dataset is caught by production traffic before it serves all of it.

What a warm-up removes from a rollout Median request latency across a dataset rollout. Without a warm-up, latency spikes to 1.9 seconds for the first two minutes as new pods fault tiles into page cache. With a warm-up run during the readiness window, the spike does not reach user traffic at all. Median latency across a dataset rollout 2.0 s 1.0 s 0 rollout starts no warm-up with warm-up in readiness −2 min +8 min The spike is page-cache cold start, not the engine — the tiles are on disk but not yet resident. A few dozen representative matches during readiness moves the cost off the user-visible path. Warm the depots that generate most traffic; warming everything wastes the readiness window.

Common Pitfalls Specific to This Technique

Setting memory limits from the process’s resident size. The mapped tiles are the larger term, and the failure is an OOM kill with a clean application log.

A ReadWriteMany volume shared with the build job. Produces intermittent, pod-specific failures during writes that no retry reproduces.

Liveness probing with a match request. Turns a load-induced slowdown into rolling restarts, which makes the load worse.

Forgetting to record the dataset version in the response path. Without it, matched output cannot be attributed to a dataset, which breaks the reproducibility described in pinning map versions.

Related

Up to section Valhalla Meili Map Matching for Telematics Related Configuring Valhalla trace_attributes for Fleet Routing