Don't Deserialize: Serving 1.5M RPKI ROA Records from a Memory-Mapped Trie

· Mingwei Zhang
release rpki

How wayback-rpki went from 1.30 GB of resident memory to about 500 MB — without changing a single query result.

Wayback RPKI is a BGPKIT service that crawls every daily RPKI Route Origin Authorization (ROA) dump from RIPE since 2011 and serves a REST API for historical lookups. The full dataset is substantial: 1,108,337 unique prefixes carrying 1,531,339 ROA records, each annotated with date ranges spanning up to 15 years of history.

For years, the service ran fine. After its bootstrap and initial update, its glibc allocator retained a 1.30 GB resident-memory high-water mark, and it took 2.3 seconds to start. This post explains how the v2 design cuts steady-state RSS to around 500 MB and cold-start-to-first-query to 23 ms: rkyv serves the trie from a memory-mapped archive, while jemalloc returns the bootstrap and update allocations that are no longer live to the OS.

The technique is not specific to RPKI data. Any Rust service that loads a large, read-heavy data structure at startup — a routing table, a geolocation database, a blocklist — can apply the same pattern.

The problem: deserialize-everything-at-startup

The v1 architecture was straightforward:

RIPE daily dumps → parse_roas_csv → ipnet-trie<HashMap<...>> → bincode → .bin.gz

At startup, the service read the .bin.gz archive and deserialized the entire trie into heap memory. Every HashMap, every HashSet<i64> of dates, every VecDeque of compressed ranges — all reconstructed into native Rust collections before the first query could be served.

On the production dataset, this meant:

Metricv1 (bincode + heap)
Startup + first query2.3 s
Steady-state RSS after bootstrap/update1.30 GB
On-disk archive (gzipped)12 MB

The 1.30 GB figure is real resident memory, not virtual. The trie’s node structure, combined with per-record HashSet and VecDeque allocations, created a massive heap working set. The data was inherently read-only after load — but the runtime had no way to know that. It dutifully rebuilt every collection.

The fix was obvious in retrospect: don’t deserialize at all.

rkyv in one paragraph

rkyv is a Rust serialization framework that produces zero-copy archives. Instead of writing data in a transport format that must be parsed back into native types, rkyv lays out the bytes in a shape that can be read directly as native Rust types — no allocation, no copy, no parsing. You derive Archive on a struct; rkyv generates a corresponding ArchivedFoo type whose fields are byte-level-compatible with the on-disk representation. A pointer into the mmap’d file is a valid &ArchivedFoo.

The trade-off: rkyv archives are not portable across architectures (endianness, pointer width). That is acceptable — even desirable — for a local serving archive that is generated and consumed on the same platform class. The rkyv support in prefix-trie was introduced in the v0.10.0 release (also published as 0.10.1 on crates.io), which is what made this migration possible.

v1 vs v2 at a glance

flowchart LR
  subgraph v1["v1: bincode + heap"]
      direction TB
      A1[".bin.gz (12 MB)"] -->|bincode deserialize| B1["Heap trie
1.30 GB RSS
2.3s startup"]
  end
  subgraph v2["v2: rkyv + mmap"]
      direction TB
      A2[".rkyv (128 MB raw)"] -->|mmap + jemalloc| B2["Page cache
about 500 MB RSS
23ms startup"]
  end
  v1 -.->|"100× faster startup
about 2.5× lower RSS"| v2

The migration

Step 1: prefix-trie with rkyv support

We replaced ipnet-trie with prefix-trie, which ships rkyv support via its rkyv feature. The core data type changed from:

// v1: IpnetTrie keyed on HashMap — not serializable as a zero-copy structure
IpnetTrie<HashMap<(u8, u32), RoasTrieEntry>>

to:

// v2: JointPrefixMap is rkyv-derivable
JointPrefixMap<IpNet, Vec<RoaRecord>>

JointPrefixMap is a combined IPv4/IPv6 radix trie that natively implements Archive/Serialize/Deserialize. It also provides get_spm() (shortest prefix match) and TrieView for subtree iteration — the exact operations needed for ROA validation and inclusive prefix search.

Step 2: rkyv-derivable record types

The per-prefix ROA record changed from a heap-heavy struct using HashSet + VecDeque:

// v1: dual storage, both heap-allocated collections
struct RoasTrieEntry {
    max_len: u8,
    origin: u32,
    dates: HashSet<i64>,              // uncompressed, during bootstrap
    dates_compressed: VecDeque<(i64, i64)>,  // merged ranges
}

to a lean, rkyv-friendly struct:

#[derive(Archive, Serialize, Deserialize)]
pub struct RoaRecord {
    pub max_len: u8,
    pub origin: u32,
    pub dates: Vec<(i64, i64)>,   // compressed (start, end) ranges, always
}

Date compression still happens — but at build time, not query time. During bootstrap, the builder (RoasTrieMut) collects dates into a HashSet, then calls full_compress() to merge consecutive days into (start, end) ranges before serialization. The archive on disk only ever contains the compressed ranges. No HashSet, no VecDeque — just a flat Vec<(i64, i64)>.

Step 3: the archive root

The top-level archive type carries header metadata alongside the trie:

#[derive(Archive, Serialize, Deserialize)]
pub struct RoasTrieData {
    pub format_version: u32,
    pub latest_date: i64,
    pub ipv4_count: u64,
    pub ipv6_count: u64,
    pub trie: JointPrefixMap<IpNet, Vec<RoaRecord>>,
}

ipv4_count and ipv6_count are pre-computed at build time. In v1, /health had to iterate the entire trie to produce per-family counts — an O(n) scan on every health check. Now it reads two integers from the header. latest_date is similarly cached, eliminating another full scan.

Step 4: mmap and query without deserializing

The read-side RoasTrie struct holds either a memory map or owned bytes, and provides queries that run directly against the archived bytes:

pub struct RoasTrie {
    bytes: TrieBytes,
}

enum TrieBytes {
    Mmap(memmap2::Mmap),
    Owned(Vec<u8>),
}

Opening the archive is a single mmap call plus a one-time validation pass:

pub fn open(path: &str) -> Result<Self> {
    let file = std::fs::File::open(path)?;
    let mmap = unsafe { memmap2::Mmap::map(&file)? };
    let trie = RoasTrie { bytes: TrieBytes::Mmap(mmap) };
    trie.validate_bytes()?;
    Ok(trie)
}

fn validate_bytes(&self) -> Result<()> {
    let data: &ArchivedRoasTrieData =
        rkyv::access::<_, rkyv::rancor::Error>(self.bytes.as_slice())?;
    // ... version check ...
    Ok(())
}

After validation, every subsequent access skips the check:

fn data(&self) -> &ArchivedRoasTrieData {
    // SAFETY: validated at open(), immutable for lifetime of this struct
    unsafe { rkyv::access_unchecked::<ArchivedRoasTrieData>(self.bytes.as_slice()) }
}

Queries operate on ArchivedRoaRecord references — no .to_native() deserialization into owned types until the final conversion to the API response:

pub fn validate(&self, prefix: &IpNet, origin: u32, date_ts: i64) -> RpkiValidation {
    for (_p, records) in self.match_records(prefix) {
        for r in records.iter() {
            let r_origin = r.origin.to_native();     // cheap integer fixup
            let r_max_len = r.max_len;                // already native-width
            if r_origin == origin && r_max_len >= prefix.prefix_len()
                && record_contains_date(r, date_ts)
            {
                return RpkiValidation::Valid;
            }
        }
        // ... Invalid / Unknown logic ...
    }
}

r.origin is an Archived<u32>. On a little-endian host (which is what we deploy on), to_native() is a no-op at the hardware level — the bytes in the mmap’d file are already in native byte order. The “deserialization” is just a type-system formality.

Step 5: safe hot-reload via atomic rename

The service updates its trie every 8 hours. With the entire data structure living in a mmap’d file, we cannot update it in place. Instead:

  1. The background update thread loads the archive into a mutable RoasTrieMut, applies new ROA data, and calls dump() — which writes to a .tmp file and then renames it atomically over the live path.
  2. Readers holding the old Mmap handle continue querying the old bytes (the kernel keeps the inode alive until all mappings close).
  3. The next reload opens the new file and re-mmaps.
pub fn dump(&mut self, path: &str) -> Result<()> {
    // ... compress, build RoasTrieData, rkyv::to_bytes ...
    let tmp_path = format!("{}.tmp", path);
    std::fs::write(&tmp_path, &bytes)?;
    std::fs::rename(&tmp_path, path)?;
    Ok(())
}

No locks, no torn reads, no query downtime.

sequenceDiagram
  participant API as API server (mmap, inode A)
  participant FS as Filesystem
  participant BG as Background updater

  API->>FS: mmap roas_trie.rkyv (inode A)
  BG->>BG: load + update RoasTrieMut
  BG->>FS: write roas_trie.rkyv.tmp
  BG->>FS: atomic rename → roas_trie.rkyv (inode B)
  Note over API,FS: readers on inode A unaffected
  API->>FS: reopen + mmap roas_trie.rkyv (inode B)
  Note over FS: kernel evicts inode A after last reader closes

Step 6: an allocator that gives memory back

The mmap eliminates the heap-resident trie, but the process still allocates heavily at two points: bootstrap, where the JSONL import builds a mutable trie that peaks at roughly 200 MB, and every 8-hour update, which loads, mutates, and re-serializes the archive. With glibc’s malloc, those freed allocations were never returned — freed arenas are retained for future reuse, and the process sat permanently at a 1.28 GB high-water mark even though its live working set was far smaller.

The fix is a two-line allocator swap (tikv-jemallocator):

#[global_allocator]
static ALLOC: Jemalloc = Jemalloc;

jemalloc’s decay-based purging returns dirty pages to the operating system after they sit unused, rather than retaining them indefinitely. After a bootstrap or update completes, the process settles back to around 500 MB instead of ratcheting up to the glibc high-water mark. Between the two changes, the serving heap stays small: the dominant resident share is the mapped archive’s file-backed pages, which the kernel can evict under pressure and re-fault on demand.

Results

On the full production dataset (1,108,337 prefixes, 1,531,339 ROA records):

Metricv1 (bincode + heap)v2 (rkyv + mmap)Improvement
Startup + first query2.3 s23 ms100×
Steady-state RSS1.30 GBaround 500 MBabout 2.5×
Full-scan API query (1.5M records, paginated)unmeasured (full in-memory materialization)244 ms—
On-disk archive (raw)—128 MB—
On-disk archive (gzipped, transport)12 MB13 MB≈ same
/health per-family countsO(n) scanO(1) header read—

The 128 MB raw archive is larger than the 12 MB gzipped v1 blob — expected, since rkyv archives are uncompressed (mmap requires raw bytes). For transport and backup, the service uses the platform-agnostic .jsonl.gz format. Bootstrap streams that transport into a mutable builder, which writes the raw local archive that is then mmap’d. The active serving archive is always raw.

The service settles around 500 MB of RSS in production; the exact value varies with the archive-backed working set and recent update activity. The memory map avoids rebuilding the full trie as heap objects, while jemalloc releases bootstrap and update allocations that glibc retained. Importing the JSONL transport builds a mutable trie that peaks at roughly 200 MB; after an update, jemalloc purges those freed allocations instead of leaving the process at glibc’s 1.28 GB high-water mark. The zero-copy query path ends up looking like this:

flowchart TB
  REQ["GET /search?prefix=1.1.1.0/24"] --> ROUTER["Axum handler"]
  ROUTER -->|zero-copy| ACCESS["access_unchecked(bytes)"]
  ACCESS --> TRIE["ArchivedRoasTrieData.trie"]
  TRIE -->|get_spm| NODE["multi-bit node walk
(≤7 hops for IPv4)"]
  NODE --> REC["ArchivedRoaRecord
.max_len, .origin, .dates"]
  REC -->|to_native| RESP["JSON response"]
  style TRIE fill:none,stroke-dasharray: 5 5
  style NODE fill:none,stroke-dasharray: 5 5
  style REC fill:none,stroke-dasharray: 5 5

Correctness verification

The migration was verified by a differential audit against the v1 production archive. After converting roas_trie.bin.gz → roas_trie.rkyv, every query (search, validate, lookup_prefix) returns byte-identical result sets. The only behavioral difference: v1 returned per-prefix records in nondeterministic HashMap order; v2 sorts by (origin, max_len) for deterministic output — a fix, not a regression.

Practical lessons

rkyv archived types are not transparent

ArchivedRoasTrieData is a distinct type from RoasTrieData. Its fields are Archived<T> wrappers: ArchivedVec<ArchivedRoaRecord> instead of Vec<RoaRecord>, Archived<u32> instead of u32. You cannot pattern-match archived tuples directly — ArchivedTuple2 is not destructurable, so you access fields with .0 and .1. These are paper cuts, not blockers, but they require discipline: the query layer must consistently work with archived types rather than reaching for .to_native() prematurely.

prefix-trie’s archived joint map exposes internal family fields

The archived JointPrefixMap exposes t1 (IPv4) and t2 (IPv6) as separate ArchivedPrefixMap fields. Subtree iteration — needed for the “include supernets and subnets” search mode — goes through AsView::view_at + TrieView::iter, which yields (IpNet, &ArchivedVec<...>) by value. This is slightly awkward but workable:

fn match_records(&self, prefix: &IpNet)
    -> Vec<(IpNet, &ArchivedVec<ArchivedRoaRecord>)>
{
    let data = self.data();
    let spm = match data.trie.get_spm(prefix) {
        Some((spm, _)) => spm,
        None => return vec![],
    };
    match spm {
        IpNet::V4(spm4) => match data.trie.t1.view_at(&spm4) {
            Some(view) => view.iter()
                .map(|(p, recs)| (IpNet::V4(p), recs))
                .collect(),
            None => vec![],
        },
        IpNet::V6(spm6) => match data.trie.t2.view_at(&spm6) {
            // ... same for IPv6 ...
        },
    }
}

Backward compatibility costs one enum, not an architecture

During the transition, the API layer uses a TrieBackend enum that routes to either the v2 mmap backend or the v1 legacy in-memory backend based on the file suffix:

pub enum TrieBackend {
    V2(RoasTrie),       // .rkyv → mmap
    V1(LegacyRoasTrie), // .bin / .bin.gz → heap
}

Both implement the same query surface (search, validate, counts, latest_date_ts). The /health endpoint reports format_version: 1 | 2. A missing .rkyv file is automatically generated from a sibling .bin.gz before any remote download is attempted. This costs ~600 lines of legacy shim code, but means zero downtime migration: cronjobs can keep producing .bin files while live APIs serve from .rkyv.

When to use this pattern

This approach is a good fit when:

  • The dataset is large and read-only at query time. If you need frequent mutations, rkyv’s immutable archive forces a full re-serialize on every update.
  • Startup latency matters. If your service restarts frequently (autoscaling, serverless, crash recovery), 23 ms vs 2.3 s is the difference between “invisible” and “noticeable outage.”
  • Memory is constrained or shared. Multiple processes can mmap the same archive file; the kernel deduplicates the pages. The trie effectively becomes a shared read-only resource backed by the OS page cache.
  • You control the archive generation. Since rkyv archives are platform-specific, you generate them on (or for) the target architecture.

It is not a good fit for:

  • Cross-platform data exchange (use serde + a portable format).
  • Write-heavy workloads (use a real database or in-memory mutable structure).
  • Datasets smaller than ~10 MB where bincode deserialization is already negligible.

Try it against the live API

The service now backs the RPKI ROA endpoint in the public BGPKIT API: GET /v3/roas/search. The API gateway forwards the same search parameters (prefix, asn, max_len, date, current, exact, page, and page_size) to wayback-rpki, so a caller can retrieve historical ROAs alongside the other api.bgpkit.com services. The date filter selects ROAs active on that day; each matching record still returns its complete observed date-range history.

For example, a historical lookup for Cloudflare’s 1.1.1.0/24 (AS13335) returns one ROA record:

curl "https://api.bgpkit.com/v3/roas/search?prefix=1.1.1.0/24&date=2020-06-01"
{
  "total": 1,
  "error": null,
  "data": [
    {
      "prefix": "1.1.1.0/24",
      "max_len": 24,
      "asn": 13335,
      "date_ranges": [
        ["2018-04-05", "2026-07-28"]
      ],
      "current": true
    }
  ],
  "meta": {
    "latest_date": "2026-07-28",
    "format_version": 2,
    "backend": "railway"
  },
  "page": 0,
  "page_size": 100
}

To run the service yourself, the one-command bootstrap downloads a pre-built archive, builds the local .rkyv file, and memory-maps it:

cargo install wayback-rpki
wayback-rpki serve --bootstrap

Wrapping up

The full implementation lives in wayback-rpki PR #12 and the roas_trie.rs module. The production service has been serving from the rkyv+mmap backend since July 2026.

The core insight is simple: if your data is read-only at query time, don’t pay for deserialization. Memory-map the file, derive Archive, and let the kernel manage your working set. Pair it with an allocator that gives freed bootstrap memory back: 1.30 GB becomes about 500 MB, startup becomes unnoticeable, and the query code barely changes.

BGPKIT builds open-source BGP data analysis tools in Rust. Wayback RPKI provides historical RPKI ROA data through the free public api.bgpkit.com/v3/roas/search API.