BGPKIT Blog

Anatomy of the Virtualizor BGP Hijack: Reading the Attack in Raw BGP Bytes

Mingwei Zhang — Mon, 31 Aug 2026 00:00:00 GMT

On August 31, 2026, Softaculous disclosed that a BGP hijack against their infrastructure two days earlier was not just a routing outage: during the diversion, a malicious Virtualizor update package was served to hosting providers, compromising hypervisors at the root level [1]. This post reconstructs the routing attack from primary data, the BGP UPDATE messages recorded by public route collectors, and follows the failure chain from RPKI origin validation, through certificate issuance, to an unsigned software-update channel.

Everything below is reproducible from public archives: RIPE RIS and Route Views store every message, and the tools we use to slice them (bgpkit-parser and BGPKIT Broker) are open source. A PDF version of this analysis is available: Full incident report (PDF).

The one-paragraph summary

Between 2026-08-28 20:57:30 UTC and 2026-08-30 ~06:10 UTC, AS62390 (NexonHost) announced 162.55.80.0/24, a slice of Hetzner's 162.55.0.0/16 that hosts Softaculous' update and billing servers, through transit provider AS6204 (ZET.NET). The forged announcement was more specific than Hetzner's legitimate /16, so it won route selection everywhere it reached. Crucially, the operator of AS62390 kept the real origin, AS24940, at the end of the AS path, which made the hijacked route validate cleanly against Hetzner's RPKI ROA. Traffic for the update servers flowed to the impostor, who used the diversion to pass Let's Encrypt's domain validation and mint a legitimate TLS certificate for 26 Softaculous domains, cloned the update endpoint, and served a backdoored update to installations that happened to check during the window.

The hijack path

The path below reads observer → origin, the direction an MRT record stores. Every vantage that received the forged route saw some variant of:

<observer> <upstream...> 6204 62390 24940

Segment by segment, from the origin end:

24940: Hetzner, the real origin. The attacker kept it on the path as camouflage, so RPKI origin validation saw the authorized origin and passed the route.
62390: NexonHost, the hijacking AS. It forged the announcement and inserted itself one hop before the origin.
6204: ZET.NET, the transit provider that accepted and propagated the forged route to the rest of the Internet.
<upstream...>: however many additional ASNs the route accumulated between AS6204 and the recording collector, different at every vantage.

The attack topology in one picture:

graph LR
  LEG["162.55.0.0/16 | RPKI ROA: AS24940, maxlen 24"]
  H["AS24940 Hetzner
legitimate /16"] -- "announces /16" --> W["The Internet
(longest prefix match)"]
  N["AS62390 NexonHost
hijacking AS / apparent source"] -- "announces 162.55.80.0/24
AS path: ... 6204 62390 24940" --> Z["AS6204 ZET.NET
transit that propagated the forged route"]
  Z -- "more specific wins" --> W
  W -- "diverted traffic" --> IMP["impostor server
at 162.55.80.8"]
  style N fill:#fee2e2,stroke:#dc2626
  style Z fill:#ffedd5,stroke:#ea580c
  style IMP fill:#fee2e2,stroke:#dc2626
  style H fill:#dcfce7,stroke:#16a34a

Two properties did all the work:

More-specific beats origin authority. BGP route selection prefers the longer prefix regardless of who announces it. Hetzner's /16 could not compete with a /24 covering the same space.
The real origin stayed on the path. This is what separates this incident from a naive origin hijack, and it is why RPKI origin validation, deployed anywhere along the path, rejected nothing. More on that below.

Reading the first hijacked message, byte by byte

The first collector sighting of the forged route is a single MRT record from RIPE RIS rrc00, timestamped 2026-08-28 20:57:30 UTC. Here it is as a BGP4MP-wrapped UPDATE:

BGP4MP|1787950650|A|185.193.84.191|29504|162.55.80.0/24|29504 15935 6204 62390 24940|IGP|...

You can open this exact record, dissected field-by-field down to every byte, in wirescope, our Wireshark-style BGP message dissector. Click any tree row to pin the corresponding bytes. Three fields are worth your attention:

AS_PATH (4002 16 02 05 ...): an AS_SEQUENCE of five ASNs, 29504 15935 6204 62390 24940. The last two ASNs are the entire attack: 62390 appeared immediately before the real origin; routing data alone cannot say who operated it.
NLRI (18 a2 37 50): prefix length 24, then 162.55.80.0: the more-specific that wins longest-prefix match.
Communities: 6204:1000, 6204:1002 stamped inside AS6204, the transit that carried the forged route, 15935:100 15935:103 and 31230:0 31230:2 picked up along the way. Communities accumulate hop by hop, which is how a third party can later reconstruct which path the route took.

We extracted those 144 bytes verbatim with bgpkit-parser's RawMrtRecord::raw_bytes() API, so what you see in wirescope is the unaltered wire image from the collector, not a reconstruction.

Timeline, verified against raw MRT where possible

Virtualizor's disclosure includes a minute-by-minute table reconstructed from RIPEstat. We re-verified the key moments directly against raw update files from two independent collector projects; that is the level of evidence routing claims deserve. Every timestamp below is from the primary data. Throughout the post we separate three evidence classes: verified directly from MRT (routes, paths, propagation, withdrawals); verified from other public data (ROA, ASPA corpus); and reported by Virtualizor or operators (malicious package, impostor host behavior, malware IoCs).

Moment (UTC)	What happened	Evidence
08-28 20:57:30	First forged /24 seen; `29504 15935 6204 62390 24940` at rrc00, same second at rrc01 via a different transit chain (`6830 6204 62390 24940`)	RIS rrc00/rrc01 `updates.20260828.2055`
08-28 21:01:04	First Route Views sighting, `57866 5511 6204 62390 24940`	route-views2 `updates.20260828.2100`
08-29 08:45-08:50	Hetzner countermeasures: begins announcing the /24 itself (e.g. `852 3320 3320 3320 24940`); diversion drops to zero in minutes	RIS rrc00 `updates.20260829.0850`
08-29 ~14:30	Hetzner withdraws the countermeasure /24; Internet falls back to the /16	RIPEstat bgp-state table [1]
08-29 19:55:25	Second wave begins, same path signature, minutes after the clean /24 disappeared	RIS rrc00 `updates.20260829.1955`
08-30 05:45:16	Last forged-route record at rrc00; zero 62390 sightings after 05:50	RIS rrc00 `updates.20260830.0545`

The two-wave structure is worth pausing on. The forged route disappeared as Hetzner's countermeasure took effect, then reappeared several hours after the clean /24 was withdrawn. The routing data alone cannot establish whether that timing was deliberate, but the second wave reopened the same diversion path.

Phase	Window (UTC)	Duration	Diversion
Wave 1	08-28 20:57 → 08-29 08:50	~12 h	high, flapping (peak ~72% of RIS peers)
Suppressed	08-29 08:50 → 19:55	~11 h	~zero (clean /24, then /16 fallback)
Wave 2	08-29 19:55 → 08-30 05:45	~10 h	high again, same path signature

Through the waves, the route was exceptionally unstable: roughly 10,600 withdrawals inside the ~33-hour window (per the disclosure's event reconstruction), with transit route dampening repeatedly suppressing it. Peak diversion reached ~72% of RIS's 368 collector peers (median 266), and every single RIS peer carried the forged route at some point. Those percentages are routing-visibility measures, not traffic volume; the flapping means any individual network's exposure was intermittent.

How long the hijack "worked" end to end

The question that matters for a supply-chain attack is not when routes flapped but how long the impostor could actually serve forged content with a valid certificate. Summing only the phases where the forged route was propagated:

Aggregate attack-wave duration: ~22 of the 33 hours, adding wave 1 (about 12 hours) and wave 2 (about 10 hours). Actual exposure at any individual network was intermittent because of the severe flapping and dampening quantified below.
ROV exposure: the entire window. No RPKI mechanism would have rejected the route; it validated as Valid from the first announce to the last withdrawal. The only effective countermeasure was Hetzner answering the forged /24 with its own legitimate /24, and it took ~12 hours from onset to take effect (08-28 20:57 → 08-29 08:50), reportedly after escalation by the victim rather than proactive detection.
Certificate validity outlived the hijack. The fraudulently issued certificate was technically valid the moment Let's Encrypt's domain validation completed over the diverted path. Once issued, it kept working for any on-path position the impostor could regain, which is exactly what wave 2 was. Revocation of the misissued cert is the only thing that truly retires it; the disclosure does not state a revocation time, so treat the certificate as live through at least the end of the incident window (08-30 06:10), a ~31-hour worst case from onset; the attack waves themselves span ~22 of those hours, with intermittent exposure as quantified above.

The gap between "route visible" and "operator response" is the real story: public collectors saw the forged route at 20:57:30, the effective countermeasure arrived roughly 12 hours later, and throughout that interval the forged announcement remained RPKI Valid: ROV provided no basis for rejecting it.

What the entire public collector fleet saw

The disclosure's diversion numbers cover RIPE RIS's 368 peers. We wanted the whole public picture, so we streamed every update file from both public collector projects (RIPE RIS and Route Views, 79 collectors, 22,613 files, 94,117 route events) through a Rust replay built on bgpkit-broker and bgpkit-parser, classifying every collector peer's best route for the /24 minute by minute: hijacked (path through AS62390) or on Hetzner's clean /24. The result:

Reading the curve: at onset the forged route reaches 584 peers within its first minute and peaks at 615 peers simultaneously diverted (about three hours into wave 1). When Hetzner's countermeasure /24 appears at 08:50, the mix flips almost instantly to green (clean /24 held by 787+ peers), and the forged route stays near-zero through the quiet window. Wave 2 snaps ~609 peers back onto the hijacked path within minutes of the clean /24 disappearing, and the final withdrawal at 05:45 collapses it in one step. One detail the aggregate view exposes that per-collector checks miss: a single collector peer still held the hijacked route for about an hour after everything else had converged (the thin red tail past 05:45), a classic BGP zombie. Raw minute-bucket data: hijack-visibility-buckets.tsv.

The gaps in the curve are real routing states, not missing data. In wave 1 the forged route flapped so violently that transit route dampening repeatedly suppressed it to a single peer for hours at a stretch (the disclosure's table shows the same near-zero rows). And the blue-shaded mid-day window is genuinely empty: after Hetzner withdrew its countermeasure /24 around 14:30 and before the impostor returned at 19:55, no /24 route existed anywhere in the public view; every network had fallen back to the legitimate /16. Minute-by-minute visibility therefore swung between "hijack everywhere", "suppressed to almost nothing", and "no /24 at all"; an operator watching only reachability of the /24 could reasonably have believed the attack was over three separate times.

Where the flapping came from

We also traced the flapping itself. For every wave-1 announcement we extracted the AS immediately upstream of AS6204 (the carrier that relayed the forged route into the transit) and measured the announce/withdraw rhythm per collector peer. Two facts pin the churn to the source rather than to any transit re-computing:

All carriers cycled together. Sixteen distinct upstream ASes relayed the route into AS6204 (AS5511 with 2,370 observations, AS3257 with 1,065, plus fourteen more). Every one of them shows the same announce/withdraw cycling. The synchronized churn across otherwise independent carrier paths localizes the instability to the common segment at or below AS6204, most plausibly AS62390's export or the AS62390-AS6204 session, rather than to independent recomputation farther upstream.
Peers around the world alternate almost one-to-one. Collectors in Singapore, Korea, Brazil, South Africa, the UAE, and the US all saw announce/withdraw ratios near one, a complete up/down cycle roughly every nine minutes on average. That pattern is what a single source switching on and off, with every network faithfully repeating it, looks like; independent local routing decisions would not stay in sync.

The mechanism, end to end: the source export (on the AS62390-AS6204 segment) cycled, each drop triggered a withdrawal avalanche (the largest was 49 withdrawals in a single second, 75 seconds after onset), each recovery re-propagated the route, and the accumulated flapping pushed route-dampening networks along the path into suppressing it for 30-60 minutes at a stretch. Sloppy infrastructure or deliberate on/off hijacking: routing data cannot distinguish intent, but the synchronized churn localizes the instability to the AS62390-AS6204 segment, and the intermittent result was still enough to capture update checks and a certificate validation.

The counts here are collector-peer route states (all RIS + Route Views peers that carried a /24 route), a broader denominator than the disclosure's 368 RIS peers, so absolute numbers differ; the shape and the instants match the verified timeline exactly.

Why RPKI didn't stop it

The attacker did not forge the origin; the forged information was the relationship immediately before the origin. A ROA answers "is AS24940 allowed to originate this prefix?" and the answer was genuinely yes. The attack hinged on a different question: "is AS62390 allowed to claim connectivity to AS24940 this way?" Origin validation has no way to ask it. Everything in this section follows from that distinction.

Hetzner's ROA for the covering block is:

162.55.0.0/16 | origin AS24940 | maxLength 24

(Verifiable at any RPKI viewer; we confirmed the ROA held continuously through the incident window via BGPKIT's ROA history service, which stores daily snapshots back to 2011.)

Now validate the hijacked route the way a ROV-enforcing router would:

flowchart TD
  A["Route: 162.55.80.0/24
AS path ... 6204 62390 24940"] --> B{"Covering ROA?"}
  B -- "yes: 162.55.0.0/16" --> C{"Origin AS in ROA?
announced origin = 24940" }
  C -- "yes" --> D{"Prefix length ≤ maxLength?
24 ≤ 24" }
  D -- "yes" --> E["RPKI: VALID ✅
route accepted, hijack invisible to ROV"]
  style E fill:#fef9c3,stroke:#ca8a04

Because the forged path appended the real origin instead of replacing it, the announcement's origin ASN is exactly what the ROA authorizes, at exactly the maximum allowed length. ROV classified the forged route as Valid; even universal ROV deployment, by itself, would not have rejected this announcement.

A further wrinkle: had Hetzner signed the ROA for exactly the /16 without maxLength 24 (the minimal-ROA practice recommended by RFC 9319), the hijacked /24 would have been RPKI Invalid and dropped by every validating network; the over-broad maxLength is what kept it Valid. Bryton Herdes made exactly this point when the disclosure hit the community.

This forged-origin pattern (real origin retained, unauthorized AS inserted before it) is exactly what RFC 9319 warns about (see also the BGPKIT use-case note on ASPA path verification), and what ASPA (draft-ietf-sidrops-aspa-verification) is designed to catch.

What would have stopped the route from propagating? Unexciting, decades-old prefix filtering at the transit: a customer-prefix filter at AS6204 that restricted AS62390 to prefixes it was authorized to originate would have rejected this announcement. Its acceptance and re-export turned a single suspicious announcement into a global diversion. Customer prefix filtering at transit remains the control that would have confined this incident to a single hop.

How ASPA would have changed the outcome

ROV authorizes origins; it says nothing about who may sit immediately upstream of the origin, which is exactly the degree of freedom this attack used. ASPA closes that gap, and its effect here is unusually concrete, so we replayed the incident's actual paths through the ASPA verification algorithm (draft §5.2-5.6) under three ASPA states. The "today" state is not hypothetical: we checked the live global ASPA corpus (rpki-client export, 2,820 objects): no AS on these paths has published one, and no object anywhere lists AS62390 as a provider.

ASPA state	Hijack /24	Legit /16	Hetzner countermeasure /24
Today (no ASPA on path)	Unknown	Unknown	Unknown
ASPA(24940) published	Invalid	Unknown	Unknown / Valid†
ASPA(24940) + ASPA(6204)	Invalid	Unknown	Unknown / Valid†

† upstream (customer/peer-received) / downstream (provider-received) verdict; the recommended mitigation policy rejects Invalid only.

Why one object suffices: Hetzner's ASPA lists its real providers, which by definition exclude AS62390, so the hop 24940 → 62390 becomes provably "Not Provider+". The up-ramp from the origin collapses to zero hops and the algorithm halts at Invalid, in both the upstream and downstream variants. No cooperation from AS62390, AS6204, or anyone else on the path is needed.

Why there is no collateral damage: Hetzner's legitimate routes stay Unknown (not Invalid) because their un-attested hops are "No Attestation" (a missing assertion) rather than a contradicted one. Under the reject-Invalid-only policy, publishing the ASPA costs Hetzner nothing and takes the entire forged-upstream attack class off the table for its space.

You can watch this counterfactual play out live: Cloudflare Radar's ASPA validation tool checks any AS path against the ASPA records currently published, hop by hop. Paste the hijack path (or follow this pre-filled link) and the verdict today is Unknown. The day Hetzner publishes its ASPA, the same link turns Invalid for everyone who opens it.

And the deployment asymmetry is the punchline: defenders opt in one object at a time, each hardening their own origin regardless of what anyone else does. Had ASPA(24940) existed and verification been enabled at even the major transits that carried the forged route (AS3257, AS5511, AS6830, AS3320 were all on observed paths), the 20:57:30 announcement could have been classified Invalid and filtered at every validating network, without requiring AS62390 or AS6204 to publish ASPA of their own: no sustained diversion, no certificate minted through the hijacked path, no poisoned update channel.

Why TLS didn't stop it

With traffic for files.virtualizor.com (162.55.80.8 during the incident) flowing to the impostor, the certificate layer was subverted in a familiar way: the impostor requested certificates from Let's Encrypt, and the CA's automated domain-control validation, an HTTP request to the hijacked address, completed over the diverted path. The result was a technically valid cert covering virtualizor.com, api.virtualizor.com, files.virtualizor.com, softaculous.com, and 22 more names (full list in the disclosure's Appendix A [1]). No client saw a warning. The impostor host even carried reverse DNS server.softaculous.com.

For the update channel itself, the final blow was that Virtualizor's update client did not cryptographically verify package signatures: a valid-looking HTTPS response from a lookalike server was sufficient to deliver the payload. The fake update advertised itself as 3.2.9.8; affected hosts reported the "update succeeded" while remaining on 3.2.9.7; the version mismatch in the logs is itself a detectable IoC [3].

The full chain, then:

flowchart LR
  BGP["BGP hijack
RPKI-valid /24"] --> CERT["Let's Encrypt DCV
via hijacked path"]
  CERT --> TLS["valid cert ×26 domains
no client warnings"]
  BGP --> CLONE["clone of update server
at 162.55.80.8"]
  TLS --> PKG["backdoored update pkg
(unsigned client accepts it)"]
  CLONE --> PKG
  PKG --> ROOT["hypervisor root
systemd persistence + SSH key"]
  style BGP fill:#fee2e2,stroke:#dc2626
  style ROOT fill:#fee2e2,stroke:#dc2626

Known indicators from the merged YARA rules [2]: a java-jre-update.service systemd unit, a /tmp/widdow.jar-class Java payload (SHA-256 b81a4e1f...a7870c7), a planted ed25519 SSH key, and C2 endpoints including 31.77.220.138:2025. Virtualizor could not produce a victim list (malicious responses never touched their logs), so every installation that ran an update check inside the window must be treated as potentially affected.

Detecting this class of hijack with BGPKIT tooling

The forged route was visible in public BGP data at 20:57:30 on the second it was first announced. The supply-chain consequences were only community-confirmed 2.5 days later. The information was there; the workflow was missing. So here is the workflow, concretely, with the tools we used on this very incident.

Stage 1: notice, a new more-specific inside a valid covering block

The detection predicate for this incident is one sentence: a /24 that had never been announced before appeared inside an RPKI-valid /16, and the AS right before the origin was one nobody had ever seen adjacent to that origin. RPKI origin validation checks out (that is the attack), so the discriminator is the penultimate hop, not the origin. We verified the "never announced before" half directly: zero 162.55.80.0/24 records in rrc00's update file immediately preceding onset. Daily RIB snapshots give you the baseline side for free.

Continuously running that predicate is a pipeline job, not a CLI query: discover each new update file as it lands (BGPKIT Broker new_file event stream), stream it through bgpkit-parser, and diff what you see against a per-prefix baseline built from the previous day's RIBs. That is exactly the architecture of our spectrum pipeline: Broker for discovery, parser for the streaming pass, a small state machine for the "new more-specific + unseen penultimate hop" rule.

Once the alert fires, monocle is the tool for the investigation that follows. It resolves the exact MRT files (via the Broker) and slices them by prefix, origin, or AS path:

# baseline: has this /24 ever existed? (monocle over yesterday's RIBs)
monocle search -t <yesterday> -D rib -p "162.55.80.0/24"

# live: new more-specific appears; who is the origin-adjacent AS?
monocle search -t 1787950650 -d 60s -c rrc00 -p "162.55.80.0/24" \
  -a "62390" -f timestamp,as_path,peer_asn
# 1787950650 | 162.55.80.0/24 | 29504 15935 6204 62390 24940 | 29504
# 1787950651 | 162.55.80.0/24 | 59919 9002 3257 6204 62390 24940 | 59919
# ...

Stage 2: corroborate, independent vantage, multiple transit chains

One collector can be a policy artifact. Two projects, different continents, different transit chains (RIS saw it via 15935 6204 and 6830 6204; Route Views via 5511 6204) make the finding robust. The The BGPKIT Broker search API is how each of these commands finds its exact input file: one API call, timestamp + collector + data type, no guessing archive URLs.

# route-views2 saw the same forged origin 3.5 minutes later
curl -sO https://archive.routeviews.org/route-views2/bgpdata/2026.08/UPDATES/updates.20260828.2100.bz2
monocle parse updates.20260828.2100.bz2 -p "162.55.80.0/24"

Stage 3: attribute the trust failure, ROA history

The claim "RPKI-valid by design" is checkable against a ROA history service, not just today's snapshot: the ROA must have been in force at incident time. The BGPKIT ROA history API answers this with one call:

curl "https://api.bgpkit.com/v3/roas/search?prefix=162.55.0.0/16"
# -> 162.55.0.0/16 | AS24940 | max_len 24 | date_ranges through 2026-08-31

Stage 4: pin the evidence, bytes, not summaries

When an incident escalates, "we saw a weird path" is not evidence. The 144-byte MRT record is. The bgpkit-parser RawMrtRecord API exists exactly for this: parse the archive, match the record, keep the unmodified wire bytes.

for raw in parser.into_raw_record_iter() {
    let wire = raw.raw_bytes();          // freeze the evidence
    let rec = raw.parse()?;              // then interpret it
    // match: prefix == 162.55.80.0/24 && path contains 62390
}

The frozen bytes feed the wirescope deep link above, so anyone can audit the claim against the raw message, no BGPKIT infrastructure required.

Stage 5: report it

The difference between a claim and evidence is a source list: which MRT file, which timestamp, which exact bytes. Every command in stages 1-4 is cheap to re-run; write them down as the report's replay recipe, and anyone can audit the conclusions against public archives.

The uncomfortable summary of this incident for the "just deploy RPKI" era: origin validation is necessary, but it could not detect this attack. The hijack lived in the gap RPKI explicitly does not cover, namely who is allowed to be your upstream. That is the gap ASPA targets, while AS62390's own transit accepting an out-of-scope prefix is the gap filtering discipline targets. And once routing folds, only update-client package signing separates a routing incident from a supply-chain compromise. In this incident it was absent, so nothing stood at the last line of defense.

Reproducing this analysis

Every number and timestamp above comes from public data you can pull yourself:

Locate the exact MRT files with the Broker search API (GET /v3/broker/search with ts_start, ts_end, project, collector_id, data_type=updates), then slice the hijack out of the update stream:

monocle search -t 1787950650 -d 86400 -c rrc00 -p "162.55.80.0/24" -a "62390"

The 144-byte record dissected above is also built into wirescope as a sample ("UPDATE · hijack, real MRT record, RIS rrc00"), and the deep link above reproduces it from the URL alone. The extraction used the bgpkit-parser RawMrtRecord API: parse the archive, match the record, dump raw_bytes(), paste into wirescope. A fuller case-study writeup, including the multi-collector timeline verification and ROA history, is available on request.

Sources

Virtualizor / Softaculous, Security Incident: BGP Hijacking, 2026-08-31. https://www.virtualizor.com/blog/security-incident-bgp-hijacking/
Neo23x0/signature-base PR #417, YARA rules and IoCs for the Virtualizor compromise, 2026-08-31. https://github.com/Neo23x0/signature-base/pull/417
LowEndTalk, URGENT: Virtualizor Compromised (31st AUG): operator reports from affected hosts. https://lowendtalk.com/discussion/220625/urgent-virtualizor-compromised-31st-aug/p1
RIPE RIS raw MRT archives (rrc00, rrc01) and Route Views (route-views2), 2026-08-28 through 08-30.
PeeringDB records for AS62390 (NEXONHOST) and AS6204 (ZET.NET), retrieved 2026-08-31.
BGPKIT ROA history API, daily ROA snapshots 2011-present.
rpki-client global ASPA export (NTT), 2026-09-01 build. https://rpki.gin.ntt.net/api/export.json
Bryton Herdes on the minimal-ROA counterfactual for this incident. https://x.com/next_hopself/status/2094553871087509799

Don't Deserialize: Serving 1.5M RPKI ROA Records from a Memory-Mapped Trie

Mingwei Zhang — Tue, 28 Jul 2026 00:00:00 GMT

How wayback-rpki went from 1.30 GB of resident memory to about 500 MB — without changing a single query result.

Wayback RPKI is a BGPKIT service that crawls every daily RPKI Route Origin Authorization (ROA) dump from RIPE since 2011 and serves a REST API for historical lookups. The full dataset is substantial: 1,108,337 unique prefixes carrying 1,531,339 ROA records, each annotated with date ranges spanning up to 15 years of history.

For years, the service ran fine. After its bootstrap and initial update, its glibc allocator retained a 1.30 GB resident-memory high-water mark, and it took 2.3 seconds to start. This post explains how the v2 design cuts steady-state RSS to around 500 MB and cold-start-to-first-query to 23 ms: rkyv serves the trie from a memory-mapped archive, while jemalloc returns the bootstrap and update allocations that are no longer live to the OS.

The technique is not specific to RPKI data. Any Rust service that loads a large, read-heavy data structure at startup — a routing table, a geolocation database, a blocklist — can apply the same pattern.

The problem: deserialize-everything-at-startup

The v1 architecture was straightforward:

RIPE daily dumps → parse_roas_csv → ipnet-trie<HashMap<...>> → bincode → .bin.gz

At startup, the service read the .bin.gz archive and deserialized the entire trie into heap memory. Every HashMap, every HashSet<i64> of dates, every VecDeque of compressed ranges — all reconstructed into native Rust collections before the first query could be served.

On the production dataset, this meant:

Metric	v1 (bincode + heap)
Startup + first query	2.3 s
Steady-state RSS after bootstrap/update	1.30 GB
On-disk archive (gzipped)	12 MB

The 1.30 GB figure is real resident memory, not virtual. The trie's node structure, combined with per-record HashSet and VecDeque allocations, created a massive heap working set. The data was inherently read-only after load — but the runtime had no way to know that. It dutifully rebuilt every collection.

The fix was obvious in retrospect: don't deserialize at all.

rkyv in one paragraph

rkyv is a Rust serialization framework that produces zero-copy archives. Instead of writing data in a transport format that must be parsed back into native types, rkyv lays out the bytes in a shape that can be read directly as native Rust types — no allocation, no copy, no parsing. You derive Archive on a struct; rkyv generates a corresponding ArchivedFoo type whose fields are byte-level-compatible with the on-disk representation. A pointer into the mmap'd file is a valid &ArchivedFoo.

The trade-off: rkyv archives are not portable across architectures (endianness, pointer width). That is acceptable — even desirable — for a local serving archive that is generated and consumed on the same platform class. The rkyv support in prefix-trie was introduced in the v0.10.0 release (also published as 0.10.1 on crates.io), which is what made this migration possible.

v1 vs v2 at a glance

{`flowchart LR
    subgraph v1["v1: bincode + heap"]
        direction TB
        A1[".bin.gz (12 MB)"] -->|bincode deserialize| B1["Heap trie\n1.30 GB RSS\n2.3s startup"]
    end
    subgraph v2["v2: rkyv + mmap"]
        direction TB
        A2[".rkyv (128 MB raw)"] -->|mmap + jemalloc| B2["Page cache\nabout 500 MB RSS\n23ms startup"]
    end
    v1 -.->|"100× faster startup\nabout 2.5× lower RSS"| v2`}

The migration

Step 1: prefix-trie with rkyv support

We replaced ipnet-trie with prefix-trie, which ships rkyv support via its rkyv feature. The core data type changed from:

// v1: IpnetTrie keyed on HashMap — not serializable as a zero-copy structure
IpnetTrie<HashMap<(u8, u32), RoasTrieEntry>>

to:

// v2: JointPrefixMap is rkyv-derivable
JointPrefixMap<IpNet, Vec<RoaRecord>>

JointPrefixMap is a combined IPv4/IPv6 radix trie that natively implements Archive/Serialize/Deserialize. It also provides get_spm() (shortest prefix match) and TrieView for subtree iteration — the exact operations needed for ROA validation and inclusive prefix search.

Step 2: rkyv-derivable record types

The per-prefix ROA record changed from a heap-heavy struct using HashSet + VecDeque:

// v1: dual storage, both heap-allocated collections
struct RoasTrieEntry {
    max_len: u8,
    origin: u32,
    dates: HashSet<i64>,              // uncompressed, during bootstrap
    dates_compressed: VecDeque<(i64, i64)>,  // merged ranges
}

to a lean, rkyv-friendly struct:

#[derive(Archive, Serialize, Deserialize)]
pub struct RoaRecord {
    pub max_len: u8,
    pub origin: u32,
    pub dates: Vec<(i64, i64)>,   // compressed (start, end) ranges, always
}

Date compression still happens — but at build time, not query time. During bootstrap, the builder (RoasTrieMut) collects dates into a HashSet, then calls full_compress() to merge consecutive days into (start, end) ranges before serialization. The archive on disk only ever contains the compressed ranges. No HashSet, no VecDeque — just a flat Vec<(i64, i64)>.

Step 3: the archive root

The top-level archive type carries header metadata alongside the trie:

#[derive(Archive, Serialize, Deserialize)]
pub struct RoasTrieData {
    pub format_version: u32,
    pub latest_date: i64,
    pub ipv4_count: u64,
    pub ipv6_count: u64,
    pub trie: JointPrefixMap<IpNet, Vec<RoaRecord>>,
}

ipv4_count and ipv6_count are pre-computed at build time. In v1, /health had to iterate the entire trie to produce per-family counts — an O(n) scan on every health check. Now it reads two integers from the header. latest_date is similarly cached, eliminating another full scan.

Step 4: mmap and query without deserializing

The read-side RoasTrie struct holds either a memory map or owned bytes, and provides queries that run directly against the archived bytes:

pub struct RoasTrie {
    bytes: TrieBytes,
}

enum TrieBytes {
    Mmap(memmap2::Mmap),
    Owned(Vec<u8>),
}

Opening the archive is a single mmap call plus a one-time validation pass:

pub fn open(path: &str) -> Result<Self> {
    let file = std::fs::File::open(path)?;
    let mmap = unsafe { memmap2::Mmap::map(&file)? };
    let trie = RoasTrie { bytes: TrieBytes::Mmap(mmap) };
    trie.validate_bytes()?;
    Ok(trie)
}

fn validate_bytes(&self) -> Result<()> {
    let data: &ArchivedRoasTrieData =
        rkyv::access::<_, rkyv::rancor::Error>(self.bytes.as_slice())?;
    // ... version check ...
    Ok(())
}

After validation, every subsequent access skips the check:

fn data(&self) -> &ArchivedRoasTrieData {
    // SAFETY: validated at open(), immutable for lifetime of this struct
    unsafe { rkyv::access_unchecked::<ArchivedRoasTrieData>(self.bytes.as_slice()) }
}

Queries operate on ArchivedRoaRecord references — no .to_native() deserialization into owned types until the final conversion to the API response:

pub fn validate(&self, prefix: &IpNet, origin: u32, date_ts: i64) -> RpkiValidation {
    for (_p, records) in self.match_records(prefix) {
        for r in records.iter() {
            let r_origin = r.origin.to_native();     // cheap integer fixup
            let r_max_len = r.max_len;                // already native-width
            if r_origin == origin && r_max_len >= prefix.prefix_len()
                && record_contains_date(r, date_ts)
            {
                return RpkiValidation::Valid;
            }
        }
        // ... Invalid / Unknown logic ...
    }
}

r.origin is an Archived<u32>. On a little-endian host (which is what we deploy on), to_native() is a no-op at the hardware level — the bytes in the mmap'd file are already in native byte order. The "deserialization" is just a type-system formality.

Step 5: safe hot-reload via atomic rename

The service updates its trie every 8 hours. With the entire data structure living in a mmap'd file, we cannot update it in place. Instead:

The background update thread loads the archive into a mutable RoasTrieMut, applies new ROA data, and calls dump() — which writes to a .tmp file and then renames it atomically over the live path.
Readers holding the old Mmap handle continue querying the old bytes (the kernel keeps the inode alive until all mappings close).
The next reload opens the new file and re-mmaps.

pub fn dump(&mut self, path: &str) -> Result<()> {
    // ... compress, build RoasTrieData, rkyv::to_bytes ...
    let tmp_path = format!("{}.tmp", path);
    std::fs::write(&tmp_path, &bytes)?;
    std::fs::rename(&tmp_path, path)?;
    Ok(())
}

No locks, no torn reads, no query downtime.

{`sequenceDiagram
    participant API as API server (mmap, inode A)
    participant FS as Filesystem
    participant BG as Background updater

    API->>FS: mmap roas_trie.rkyv (inode A)
    BG->>BG: load + update RoasTrieMut
    BG->>FS: write roas_trie.rkyv.tmp
    BG->>FS: atomic rename → roas_trie.rkyv (inode B)
    Note over API,FS: readers on inode A unaffected
    API->>FS: reopen + mmap roas_trie.rkyv (inode B)
    Note over FS: kernel evicts inode A after last reader closes`}

Step 6: an allocator that gives memory back

The mmap eliminates the heap-resident trie, but the process still allocates heavily at two points: bootstrap, where the JSONL import builds a mutable trie that peaks at roughly 200 MB, and every 8-hour update, which loads, mutates, and re-serializes the archive. With glibc's malloc, those freed allocations were never returned — freed arenas are retained for future reuse, and the process sat permanently at a 1.28 GB high-water mark even though its live working set was far smaller.

The fix is a two-line allocator swap (tikv-jemallocator):

#[global_allocator]
static ALLOC: Jemalloc = Jemalloc;

jemalloc's decay-based purging returns dirty pages to the operating system after they sit unused, rather than retaining them indefinitely. After a bootstrap or update completes, the process settles back to around 500 MB instead of ratcheting up to the glibc high-water mark. Between the two changes, the serving heap stays small: the dominant resident share is the mapped archive's file-backed pages, which the kernel can evict under pressure and re-fault on demand.

Results

On the full production dataset (1,108,337 prefixes, 1,531,339 ROA records):

Metric	v1 (bincode + heap)	v2 (rkyv + mmap)	Improvement
Startup + first query	2.3 s	23 ms	100×
Steady-state RSS	1.30 GB	around 500 MB	about 2.5×
Full-scan API query (1.5M records, paginated)	unmeasured (full in-memory materialization)	244 ms	—
On-disk archive (raw)	—	128 MB	—
On-disk archive (gzipped, transport)	12 MB	13 MB	≈ same
`/health` per-family counts	O(n) scan	O(1) header read	—

The 128 MB raw archive is larger than the 12 MB gzipped v1 blob — expected, since rkyv archives are uncompressed (mmap requires raw bytes). For transport and backup, the service uses the platform-agnostic .jsonl.gz format. Bootstrap streams that transport into a mutable builder, which writes the raw local archive that is then mmap'd. The active serving archive is always raw.

The service settles around 500 MB of RSS in production; the exact value varies with the archive-backed working set and recent update activity. The memory map avoids rebuilding the full trie as heap objects, while jemalloc releases bootstrap and update allocations that glibc retained. Importing the JSONL transport builds a mutable trie that peaks at roughly 200 MB; after an update, jemalloc purges those freed allocations instead of leaving the process at glibc's 1.28 GB high-water mark. The zero-copy query path ends up looking like this:

{`flowchart TB
    REQ["GET /search?prefix=1.1.1.0/24"] --> ROUTER["Axum handler"]
    ROUTER -->|zero-copy| ACCESS["access_unchecked(bytes)"]
    ACCESS --> TRIE["ArchivedRoasTrieData.trie"]
    TRIE -->|get_spm| NODE["multi-bit node walk\n(≤7 hops for IPv4)"]
    NODE --> REC["ArchivedRoaRecord\n.max_len, .origin, .dates"]
    REC -->|to_native| RESP["JSON response"]
    style TRIE fill:none,stroke-dasharray: 5 5
    style NODE fill:none,stroke-dasharray: 5 5
    style REC fill:none,stroke-dasharray: 5 5`}

Correctness verification

The migration was verified by a differential audit against the v1 production archive. After converting roas_trie.bin.gz → roas_trie.rkyv, every query (search, validate, lookup_prefix) returns byte-identical result sets. The only behavioral difference: v1 returned per-prefix records in nondeterministic HashMap order; v2 sorts by (origin, max_len) for deterministic output — a fix, not a regression.

Practical lessons

rkyv archived types are not transparent

ArchivedRoasTrieData is a distinct type from RoasTrieData. Its fields are Archived<T> wrappers: ArchivedVec<ArchivedRoaRecord> instead of Vec<RoaRecord>, Archived<u32> instead of u32. You cannot pattern-match archived tuples directly — ArchivedTuple2 is not destructurable, so you access fields with .0 and .1. These are paper cuts, not blockers, but they require discipline: the query layer must consistently work with archived types rather than reaching for .to_native() prematurely.

`prefix-trie`'s archived joint map exposes internal family fields

The archived JointPrefixMap exposes t1 (IPv4) and t2 (IPv6) as separate ArchivedPrefixMap fields. Subtree iteration — needed for the "include supernets and subnets" search mode — goes through AsView::view_at + TrieView::iter, which yields (IpNet, &ArchivedVec<...>) by value. This is slightly awkward but workable:

fn match_records(&self, prefix: &IpNet)
    -> Vec<(IpNet, &ArchivedVec<ArchivedRoaRecord>)>
{
    let data = self.data();
    let spm = match data.trie.get_spm(prefix) {
        Some((spm, _)) => spm,
        None => return vec![],
    };
    match spm {
        IpNet::V4(spm4) => match data.trie.t1.view_at(&spm4) {
            Some(view) => view.iter()
                .map(|(p, recs)| (IpNet::V4(p), recs))
                .collect(),
            None => vec![],
        },
        IpNet::V6(spm6) => match data.trie.t2.view_at(&spm6) {
            // ... same for IPv6 ...
        },
    }
}

Backward compatibility costs one enum, not an architecture

During the transition, the API layer uses a TrieBackend enum that routes to either the v2 mmap backend or the v1 legacy in-memory backend based on the file suffix:

pub enum TrieBackend {
    V2(RoasTrie),       // .rkyv → mmap
    V1(LegacyRoasTrie), // .bin / .bin.gz → heap
}

Both implement the same query surface (search, validate, counts, latest_date_ts). The /health endpoint reports format_version: 1 | 2. A missing .rkyv file is automatically generated from a sibling .bin.gz before any remote download is attempted. This costs ~600 lines of legacy shim code, but means zero downtime migration: cronjobs can keep producing .bin files while live APIs serve from .rkyv.

When to use this pattern

This approach is a good fit when:

The dataset is large and read-only at query time. If you need frequent mutations, rkyv's immutable archive forces a full re-serialize on every update.
Startup latency matters. If your service restarts frequently (autoscaling, serverless, crash recovery), 23 ms vs 2.3 s is the difference between "invisible" and "noticeable outage."
Memory is constrained or shared. Multiple processes can mmap the same archive file; the kernel deduplicates the pages. The trie effectively becomes a shared read-only resource backed by the OS page cache.
You control the archive generation. Since rkyv archives are platform-specific, you generate them on (or for) the target architecture.

It is not a good fit for:

Cross-platform data exchange (use serde + a portable format).
Write-heavy workloads (use a real database or in-memory mutable structure).
Datasets smaller than ~10 MB where bincode deserialization is already negligible.

Try it against the live API

The service now backs the RPKI ROA endpoint in the public BGPKIT API: GET /v3/roas/search. The API gateway forwards the same search parameters (prefix, asn, max_len, date, current, exact, page, and page_size) to wayback-rpki, so a caller can retrieve historical ROAs alongside the other api.bgpkit.com services. The date filter selects ROAs active on that day; each matching record still returns its complete observed date-range history.

For example, a historical lookup for Cloudflare's 1.1.1.0/24 (AS13335) returns one ROA record:

curl "https://api.bgpkit.com/v3/roas/search?prefix=1.1.1.0/24&date=2020-06-01"

{
  "total": 1,
  "error": null,
  "data": [
    {
      "prefix": "1.1.1.0/24",
      "max_len": 24,
      "asn": 13335,
      "date_ranges": [
        ["2018-04-05", "2026-07-28"]
      ],
      "current": true
    }
  ],
  "meta": {
    "latest_date": "2026-07-28",
    "format_version": 2,
    "backend": "railway"
  },
  "page": 0,
  "page_size": 100
}

To run the service yourself, the one-command bootstrap downloads a pre-built archive, builds the local .rkyv file, and memory-maps it:

cargo install wayback-rpki
wayback-rpki serve --bootstrap

Wrapping up

The full implementation lives in wayback-rpki PR #12 and the roas_trie.rs module. The production service has been serving from the rkyv+mmap backend since July 2026.

The core insight is simple: if your data is read-only at query time, don't pay for deserialization. Memory-map the file, derive Archive, and let the kernel manage your working set. Pair it with an allocator that gives freed bootstrap memory back: 1.30 GB becomes about 500 MB, startup becomes unnoticeable, and the query code barely changes.

BGPKIT builds open-source BGP data analysis tools in Rust. Wayback RPKI provides historical RPKI ROA data through the free public api.bgpkit.com/v3/roas/search API.

monocle v1.4.0: Remote BGP Search, PCH Archive Parsing, and RIB Correctness

Mingwei Zhang — Tue, 21 Jul 2026 00:00:00 GMT

Public BGP analysis is frequently constrained by both the source data and the machine doing the work. A single investigation may need to parse many MRT files, ingest Packet Clearing House's Cisco-style route dumps, and compare an older route observation with the RIB and RPKI state from the same period. Doing all of that on an analyst's working machine is impractical when it lacks the CPU, memory, disk, or persistent cache required for the data volume.

monocle v1.4.0 connects those requirements into one workflow. Its HTTP/SSE service moves downloads and parsing to a dedicated server, cloud VM, or container deployment while streaming matching results back to the working machine. The same release adds PCH text-dump parsing, reusable file-based filters, and RPKISPOOL-backed historical RPKI lookups, and corrects RIB and RIB-search behavior that could omit stable routes whose per-route timestamps predate the snapshot.

Install and run

The release can be installed with Cargo or Homebrew:

cargo install monocle --version 1.4.0
# or, on macOS and Linux with Homebrew
brew install monocle

monocle --version
# monocle 1.4.0

A multi-architecture bgpkit/monocle Docker image is available on Docker Hub. It provides a containerized deployment path for a dedicated search host or an orchestrated environment such as Kubernetes, where the service can retain its own database and MRT-file cache instead of relying on the workstation that submits a query.

docker run --rm \
  -p 8080:8080 \
  -v monocle-data:/data \
  bgpkit/monocle:1.4.0 \
  server --address 0.0.0.0 --port 8080

The persistent /data volume holds monocle's SQLite-backed data. Add a separate cache mount and configure cache_dir when repeatedly searching historical MRT files. With the binary or container in place, the next decision is whether the working machine should execute the search itself or submit it to a dedicated service.

Run searches where the data and compute are

For a one-off query, local monocle search remains the shortest path. Longer investigations, repeated queries, and applications benefit from separating the working machine from the search host: the latter can keep the MRT cache and provide the CPU, memory, disk, and network capacity required to retrieve and parse public archives.

v1.4.0 replaces the previous WebSocket service with HTTP. Search results arrive as Server-Sent Events (SSE); health checks, metadata, lookups, and database management use ordinary REST endpoints. This lets the working machine submit a narrow query, while the remote service does the download and parsing work and streams matching elements and progress back as they become available.

# On the search host
monocle server --address 0.0.0.0 --port 8080

# A health check remains unauthenticated, even when API authentication is enabled.
curl http://127.0.0.1:8080/health
# OK

The service exposes GET /health, GET /api/v1/system/info, and POST /api/v1/search/stream, as well as endpoints for time, country, IP, RPKI, prefix-to-ASN, AS relationships, inspection, and database operations. Database-backed endpoints return 503 NOT_INITIALIZED until their data has been explicitly refreshed; a request does not trigger an implicit refresh.

The SSE endpoint accepts the same kinds of filters as the CLI. For example, this asks RIPE RIS collector rrc00 for updates concerning Cloudflare's 1.1.1.0/24 in a ten-minute window:

curl -N \
  -H 'Accept: text/event-stream' \
  -H 'Content-Type: application/json' \
  -d '{
    "filters": {
      "prefix": ["1.1.1.0/24"],
      "start_ts": "2025-06-01T00:00:00Z",
      "end_ts": "2025-06-01T00:10:00Z",
      "collector": "rrc00",
      "project": "riperis",
      "dump_type": "updates"
    },
    "batch_size": 100
  }' \
  http://127.0.0.1:8080/api/v1/search/stream

A stream begins with started, may emit progress and elements, and terminates with exactly one completed, cancelled, or error event. Its final event includes matched-element and source-file counts, compressed-byte metadata, rate, and the collectors and files that actually produced matches. Disconnecting cancels the work.

The server also makes its operating limits explicit. server_max_concurrent_searches defaults to 3 and excess requests receive HTTP 429. A bounded channel protects element batches from being dropped under backpressure; progress updates may be coalesced. server_max_search_batch_size, server_max_search_results, server_search_timeout_secs, and search_concurrency can be set in monocle.toml, through MONOCLE_* environment variables, or with server flags. Token authentication protects /api/v1/* while leaving /health available for container health checks.

The CLI can submit the same search to that remote service. This keeps scripts and output formatting local while moving downloads, parsing, and cached data to the server:

monocle search \
  -p 1.1.1.0/24 \
  -t 2025-06-01T00:00:00Z \
  -T 2025-06-01T00:10:00Z \
  -c rrc00 -P riperis \
  --remote-url https://monocle.example.net/api/v1/search/stream \
  --remote-token "$MONOCLE_REMOTE_TOKEN" \
  --format json-line

--remote-url consumes the SSE stream and uses monocle's existing formatters. Invalid requests and pre-stream validation failures return HTTP errors; a stream that ends in error or cancelled, or drops without completed, makes the CLI exit non-zero. Remote execution addresses the compute constraint, but a useful search service must also accept the archive formats that researchers and operators actually have available.

Parse PCH's Cisco-style BGP-table archives

MRT is the usual exchange format for public BGP data, but Packet Clearing House's daily BGP routing archive publishes its route dumps as fixed-width Cisco-style text. The Cisco parser in v1.4.0 was added specifically so PCH data can enter the same monocle workflow without a separate conversion tool.

monocle parse detects the dump preamble and reads the prefix, AS path, next hop, metric, local preference, and origin. It also handles multiple paths for a prefix and layouts where an IPv6 prefix wraps onto its own line.

The following 2025-06-01 Accra collector snapshot is a 50 KB gzip-compressed example. monocle reads the remote archive directly:

monocle parse \
  https://downloads.pch.net/files/BGP_Routing_Data/BGP_ROUTES_V4/2025/06/route-collector.acc.pch.net/route-collector.acc.pch.net-ipv4_bgp_routes.2025.06.01.gz \
  --fields prefix,as_path,next_hop,local_pref,med \
  --format psv | head -5

The command produces rows such as:

1.179.112.0/20|15169 396982|196.201.2.118||0
2.16.77.0/24|35091 30997 20940|196.201.2.6||0
2.16.77.0/24|20940 20940|196.201.2.126||0
2.16.77.0/24|20940|196.201.2.160||0
2.16.77.0/24|37613 3741 37350 30997 20940 20940|196.201.2.32||0

The same input can be written as a TABLE_DUMP_V2 RIB for tools that expect MRT:

monocle parse router-show-ip-bgp.txt \
  --mrt-path router-snapshot.mrt \
  --mrt-type rib

Native support for the PCH text archives is the immediate goal of this release. Broader PCH archive support can follow as the archive formats and workflows are expanded. Once these archives can be queried alongside MRT data, the next practical problem is retaining the query definition as its scope grows.

Keep large filters out of shell arguments

An investigation that starts with one prefix often grows to hundreds or thousands. v1.4.0 adds --prefix-file and --filter-file to both parse and search, so that expanded query can be retained in a file instead of being reconstructed from a long shell command.

--prefix-file is newline-delimited text: blank lines and comments beginning with # are ignored. --filter-file is JSON and can hold prefixes, origin or peer ASNs, communities, an AS-path regular expression, element type, and time bounds.

{
  "prefixes": ["1.1.1.0/24", "2606:4700::/32"],
  "origin_asns": ["13335"],
  "peer_asns": ["174", "6939"],
  "include_sub": true,
  "start_ts": "2025-06-01T00:00:00Z",
  "end_ts": "2025-06-01T01:00:00Z"
}

monocle search \
  --filter-file cloudflare-investigation.json \
  -c rrc00 -P riperis \
  --format json-line

Values from a file merge with command-line values. Values within the same dimension are unioned; different dimensions remain ANDed. This supports versioned, reviewable query definitions without relying on oversized shell arguments. Those definitions can identify the messages around an event; answering what routing state existed at a particular instant requires reconstructing a RIB instead.

Reconstruct the RIB that existed at the requested time

A RIB dump is not a collection of routes learned only at the dump timestamp. Its per-route timestamps can be days or weeks earlier. Filtering a base RIB by the requested interval therefore discards stable routes and returns an incomplete state.

v1.4.0 fixes this in monocle rib and in monocle search --dump-type rib. The base RIB retains those stable entries, and a target timestamp exactly equal to a dump time selects that dump instead of an unnecessarily earlier snapshot. This avoids both incorrect output and needless update replay.

monocle rib 2025-06-01T00:00:00Z \
  -c rrc00 -P riperis \
  -p 1.1.1.0/24 \
  --format json-pretty

The command selects the latest RIB at or before the target timestamp, then replays updates through that exact time. The result answers a state question, "what routes were present then?", rather than returning only messages whose timestamps fall in the selected interval. monocle rib also gains --use-cache, --cache-dir, and --fields in this release, which are useful for repeated or narrowed reconstruction runs. A reconstructed route still does not establish whether its origin was authorized at that time, so the final part of the workflow is a contemporaneous RPKI lookup.

Pair reconstructed routes with historical RPKI data

Route-origin analysis needs the authorization state from the same period. Applying current ROAs to an old routing event can produce the wrong conclusion. v1.4.0 makes RPKISPOOL the default historical source for monocle rpki roas and monocle rpki aspas; the format stores CCR snapshots in .tar.zst archives. The ripe and rpkiviews sources remain available. RPKIViews provides the historical RPKI data used by the rpkiviews source.

monocle rpki roas 1.1.1.0/24 \
  --date 2025-06-01 \
  --source rpkispools \
  --format json-pretty

This makes it possible to pair a reconstructed RIB or an update search with a contemporaneous ROA or ASPA view, rather than treating present-day authorization data as historical evidence. Together, remote execution, PCH parsing, reusable filters, corrected RIB reconstruction, and historical RPKI lookups support an investigation from source discovery through to a time-aligned routing and authorization result.

The release also adds --concurrency for local and server searches, improves bulk database refresh behavior, and updates to bgpkit-parser 0.18.0. TableDumpV2 RIB elements now expose a peer BGP Identifier when it is present, and known-but-unsupported path attributes remain available in the parser's existing raw-attribute fields instead of being silently discarded. See the full v1.4.0 release on GitHub for the complete change list and release assets.

bgpkit-parser 0.18.0 Release

Mingwei Zhang — Thu, 02 Jul 2026 00:00:00 GMT

bgpkit-parser 0.18.0 preserves more of the original BGP wire data. Earlier versions parsed many common attributes into typed Rust structures, but element conversion could drop known attributes that did not yet have a semantic parser. That worked for many analysis workflows, but it was not enough for users who need to inspect, serialize, or re-encode less common BGP path attributes.

This release changes that behavior: the parser now retains known-but-unsupported, malformed, deprecated, and unknown BGP path attributes in AttrRaw-backed variants instead of silently dropping them.

Install

cargo add bgpkit-parser@0.18.0

For the WebAssembly package:

npm install @bgpkit/parser@0.18.0

RIS Live raw message parsing

RIS Live's JSON messages expose a convenient projection of UPDATE data, but that projection does not include every BGP path attribute. For example, attributes such as Only to Customer (OTC, code 35), BGP Prefix-SID, BFD Discriminator, and other less common path attributes may appear in the raw UPDATE while remaining absent from the projected JSON fields. Version 0.18.0 adds helpers that parse RIS Live messages containing the original raw BGP UPDATE bytes.

To request raw BGP messages from RIS Live, set socketOptions.includeRaw in the subscription message. The crate now re-exports RisSubscribe, RisSubscribeType, and RisLiveClientMessage from the crate root, so examples and downstream applications can build subscription messages directly.

use bgpkit_parser::{RisLiveClientMessage, RisSubscribe};

let subscribe = RisSubscribe::new()
    .with_host("rrc00")
    .with_include_raw(true);

let msg = RisLiveClientMessage::RisSubscribe(subscribe).to_json_string().unwrap();
// send `msg` over a WebSocket connection to RIS Live

When a RIS Live message arrives, parse the full ris_message envelope with the raw parser:

use bgpkit_parser::parse_ris_live_message_raw;

let elems = parse_ris_live_message_raw(message_text).expect("parse RIS Live message");
for elem in elems {
    println!("{} {} {:?}", elem.elem_type, elem.prefix, elem.as_path);
}

Users who prefer RIS Live's JSON-projected fields can call parse_ris_live_message_json() for the explicit JSON-path parser, or keep using the older parse_ris_live_message() function for the same JSON-projected path.

Raw BGP attribute retention

The main parser now keeps raw bytes for attributes it cannot or should not decode semantically. The new AttributeValue::Raw(AttrRaw) variant stores the wire attribute code and value bytes for known-but-unsupported attributes:

use bgpkit_parser::models::bgp::attributes::AttributeValue;

for attr in attributes.inner.iter() {
    match attr {
        AttributeValue::Raw(raw) => {
            println!("raw attribute code={} len={}", raw.code, raw.bytes.len());
        }
        other => {
            println!("typed attribute code={}", other.attr_code());
        }
    }
}

This matters for round-trip workflows. If a route contains a known attribute that bgpkit-parser does not yet decode into a typed model, 0.18.0 preserves and re-encodes the original attribute bytes.

Malformed attributes that target typed parsers now fall back to Raw after recording a validation warning, rather than disappearing from the parsed attribute list.

Deprecated attribute code handling

The IANA registry lists attribute code 13 as deprecated: RCID_PATH / CLUSTER_ID. Version 0.18.0 removes AttrType::CLUSTER_ID and retains code 13 as deprecated raw data instead.

If downstream code exhaustively matches AttrType or AttributeValue, this is one of the release's breaking changes. Use AttributeValue::attr_code() when the raw code point matters, and AttrRaw::attr_type() when you want to map the raw wire code back to an AttrType value.

New typed BGP path attribute parsers

This release adds typed parsers and round-trip encoding support for several RFC-defined path attributes:

RFC 7311 — AIGP, attribute code 26
RFC 9015 — SFP, attribute code 37
RFC 9026 — BFD Discriminator, attribute code 38
RFC 8669 — BGP Prefix-SID, attribute code 40
RFC 9793 — BIER, attribute code 41

Each parser preserves unknown TLVs, so applications can inspect known fields without losing the rest of the attribute payload.

For example, AIGP exposes an accumulated_metric() helper:

use bgpkit_parser::models::bgp::attributes::AttributeValue;

for attr in attributes.inner.iter() {
    if let AttributeValue::Aigp(aigp) = attr {
        if let Some(metric) = aigp.accumulated_metric() {
            println!("AIGP accumulated metric: {metric}");
        }
    }
}

API changes to note

The main breaking changes affect raw attribute representation and exhaustive matches:

AttrRaw now stores code: u8 instead of attr_type: AttrType.
AttrRaw.bytes now uses bytes::Bytes instead of Vec<u8>.
AigpTlv.value now uses Bytes.
AttributeValue has new variants: Raw, BfdDiscriminator, BgpPrefixSid, Bier, and Sfp.
The release removes AttrType::CLUSTER_ID.

For most users who iterate over BgpElem, the visible behavior should stay the same except that the parser preserves more attributes internally. Code that destructures attributes or performs exhaustive matching will need updates.

WebAssembly package

The @bgpkit/parser npm package also ships 0.18.0. The WASM package continues to expose parsers for BGP UPDATE, BMP/OpenBMP, and MRT data in Node.js, bundler, browser, and Worker environments.

import { parseBgpUpdate } from '@bgpkit/parser';

const elems = parseBgpUpdate(updateBytes);
for (const elem of elems) {
  console.log(elem.type, elem.prefix, elem.as_path);
}

The Rust crate includes RIS Live parsing in this release. We track WASM-level RIS Live parser exports and WebSocket helpers separately.

Finding Well-Known BGP Communities with monocle

Mingwei Zhang — Thu, 25 Jun 2026 00:00:00 GMT

BGP communities are easy to parse and hard to interpret. A route can carry a list of 32-bit values, but the meaning of each value depends on a mix of RFCs, IANA registries, operator conventions, and local routing policy.

This post walks through a small ad-hoc monocle analysis for turning raw MRT updates into a protocol-aware summary:

search public BGP updates for communities in the 65535:* range,
summarize where each value appears,
join observed values with the IANA BGP Well-known Communities registry,
check specific values, such as BLACKHOLE and NOPEER, against the relevant RFC text.

The goal is not to infer operator intent from route collector data. This is a one-off research note, not production automation. The goal is narrower: show how BGPKIT tooling can make this kind of evidence gathering easier to repeat and audit.

Why `65535:*`?

RFC 1997 introduced the BGP COMMUNITIES path attribute. In the "Well-known Communities" part of the document, it defines several globally significant values and says:

The following communities have global significance and their operations shall be implemented in any community-attribute-aware BGP speaker.

The current IANA registry is the BGP Well-known Communities registry.

The registry divides the community space into ranges, including:

Range	Registration rule
`0x00000000-0x0000FFFF`	Reserved
`0x00010000-0xFFFEFFFF`	Private Use
`0xFFFF0000-0xFFFF8000`	First Come First Served
`0xFFFF8001-0xFFFFFFFF`	Standards Action

That makes values with high-order 16 bits set to 65535 interesting: they sit in the part of the 32-bit community space where many globally significant values live.

Setup

On macOS, install monocle with Homebrew:

brew install monocle

If you prefer building from Rust crates, or are not using Homebrew:

cargo install monocle

For reproducibility, record the version before running the scan:

monocle --version

This run used:

monocle 1.3.0

This analysis used the public MRT index available to monocle at run time. To make the collector/file set auditable, first write the matching MRT file list:

START=1782324000   # 2026-06-24T18:00:00Z
END=$((START+900)) # 15 minutes
OUT=updates-20260624T1800Z-15m-allcollectors-wk

monocle search \
  --dump-type updates \
  --start-ts $START \
  --end-ts $END \
  --community '65535:*' \
  --broker-files \
  > "$OUT-files.txt"

wc -l "$OUT-files.txt"

For this run, monocle found 125 matching MRT update files.

The exact file count can vary if the public collector index changes or if late files appear. If you need a fixed rerun, save this file list with the analysis artifacts and use the same monocle version.

Searching MRT updates with `monocle`

For this run, I scanned the same 15-minute update window:

START=1782324000   # 2026-06-24T18:00:00Z
END=$((START+900)) # 15 minutes
OUT=updates-20260624T1800Z-15m-allcollectors-wk

monocle search \
  --dump-type updates \
  --start-ts $START \
  --end-ts $END \
  --community '65535:*' \
  --fields timestamp,collector,prefix,peer_asn,origin_asns,as_path,communities \
  --format json-line \
  > "$OUT.jsonl"

A representative JSONL row looks like this:

{
  "timestamp": 1782324014.12,
  "collector": "rrc07",
  "prefix": "83.171.244.0/24",
  "peer_asn": 3399,
  "origin_asns": ["197761"],
  "as_path": "3399 5511 6204 51202 51202 197761",
  "communities": ["3399:100", "3399:107", "5511:500", "6204:1000", "65535:65284"]
}

The important part is the community filter:

--community '65535:*'

monocle handles the collector/file discovery and MRT parsing. The output is JSON Lines, so the next step can be a short post-processing script rather than a custom MRT parser.

For this post, I used small Python scripts rather than building a reusable tool. The main script, summarize_existing_well_known.py, groups rows by observed 65535:* value and counts rows, distinct prefixes, collectors, peer ASNs, origin ASNs, address families, and prefix lengths. It also normalizes symbolic community names that monocle may emit, such as no-export, back to their numeric well-known-community values before doing the 65535:* grouping.

python3 summarize_existing_well_known.py "$OUT.jsonl" "$OUT"

The second script, enrich_iana_status.py, joins the per-community CSV against a local transcription of the IANA BGP Well-known Communities registry, so each observed value can be labeled as BLACKHOLE, NOPEER, GRACEFUL_SHUTDOWN, Unassigned, and so on. The registry source is the IANA BGP Well-known Communities registry, checked for this write-up on 2026-06-25.

python3 enrich_iana_status.py \
  "$OUT-by-community.csv" \
  "$OUT-by-community-iana.csv"

What the scan found

In this 15-minute window:

Metric	Value
Matching update rows	769,537
Distinct numeric `65535:*` values	109
Values not assigned in the IANA registry snapshot	102

The highest-volume values were:

Community	IANA status	Rows	Distinct prefixes	Collectors
`65535:6939`	Unassigned	553,352	20	27
`65535:1200`	Unassigned	63,194	10,455	2
`65535:34`	Unassigned	41,584	6,022	58
`65535:30`	Unassigned	37,436	2,896	54
`65535:65284`	NOPEER, RFC 3765	27,766	1,033	51
`65535:9957`	Unassigned	13,245	1,125	1
`65535:0`	GRACEFUL_SHUTDOWN, RFC 8326	2,176	210	51
`65535:666`	BLACKHOLE, RFC 7999	1,270	499	18

This table is a good example of why registry-aware scripting helps. A raw list of communities is just numbers. Once joined with IANA, the same data separates into named RFC-defined values and values that need local/operator-specific interpretation.

Importantly, "unassigned" does not mean "bad". Public BGP data alone does not tell us whether a value is a local convention, a route-server policy signal, a legacy convention, an implementation artifact, or something else. It only tells us where the value appeared and gives us concrete rows to inspect next.

Checking BLACKHOLE behavior against RFC 7999

BLACKHOLE is assigned as 65535:666 by RFC 7999.

Two parts of the RFC are useful for a mechanical check.

RFC 7999 Section 3.2 says:

A BGP speaker receiving an announcement tagged with the BLACKHOLE community SHOULD add the NO_ADVERTISE or NO_EXPORT community as defined in RFC 1997, or a similar community, to prevent propagation of the prefix outside the local AS.

RFC 7999 Section 3.3 says:

However, blackhole prefix length should be as long as possible in order to limit the impact of discarding traffic for adjacent IP space that is not under DDoS duress. The blackhole prefix length is typically as specific as possible, /32 for IPv4 or /128 for IPv6.

Those sentences translate into simple checks:

is the prefix /32 for IPv4 or /128 for IPv6?
does the route also carry NO_EXPORT or NO_ADVERTISE?

In the 15-minute scan, 65535:666 appeared in 1,270 rows:

Observed property	Rows with property	Rows without property
`/32` or `/128`	474 (37.3%)	796 (62.7%)
Carries `NO_EXPORT`	460 (36.2%)	810 (63.8%)
Carries `NO_ADVERTISE`	24 (1.9%)	1,246 (98.1%)

Address-family split: 463 IPv4 rows (36.5%) and 807 IPv6 rows (63.5%).

The resulting rows include examples that match the RFC 7999 profile:

{
  "collector": "decix.fra",
  "peer_asn": 6695,
  "prefix": "193.56.63.113/32",
  "origin_asns": ["215514"],
  "as_path": "201053 50607 215514",
  "communities": ["65535:666", "no-export"]
}

It also found examples that lack those observed properties. This is not a compliance judgment: RFC 7999 Section 3.2 describes behavior for a receiving BGP speaker, and collector data may show the route before or after other policy actions. Still, rows like this are useful starting points for follow-up:

{
  "collector": "iix.cgk",
  "peer_asn": 7597,
  "prefix": "2001:4:112::/48",
  "origin_asns": ["112"],
  "as_path": "136106 35280 50628 44097 112",
  "communities": ["65535:666", "7597:1000:1"]
}

That is as far as the data should be taken without more context. The collector peer may be a route server. The community may have been attached earlier in the AS path. The value may reflect a local convention. The useful part is that this ad-hoc monocle analysis can separate these cases quickly and produce concrete rows for follow-up.

Checking NOPEER visibility against RFC 3765

NOPEER is assigned as 65535:65284 by RFC 3765.

RFC 3765 Section 2 describes it as:

an advisory qualification to readvertisement of a route prefix, permitting an AS not to readvertise the route prefix to all external bilateral peer neighbour AS's.

In the same 15-minute update window, NOPEER was broadly visible:

Metric	Value
Rows	27,766
Distinct prefixes	1,033
Collectors	51
Peer ASNs	103
Origin ASNs	146
IPv4 rows	25,387
IPv6 rows	2,379

Example row:

{
  "collector": "rrc07",
  "peer_asn": 3399,
  "prefix": "83.171.244.0/24",
  "origin_asns": ["197761"],
  "as_path": "3399 5511 6204 51202 51202 197761",
  "communities": [
    "3399:100",
    "3399:107",
    "5511:500",
    "6204:1000",
    "65535:65284"
  ]
}

Route data alone does not prove whether any downstream AS enforces the advisory signal. But it does show that the value is operationally present across many collectors, peer ASNs, and origin ASNs in this window.

A note on policy behavior

It is worth being careful when interpreting well-known communities. RFC 8642 exists because well-known community handling has not always been consistent across implementations and policies. Its introduction says:

In hindsight, RFC 1997 did not prescribe as fully as it should have how well-known communities may be manipulated by policies applied by operators. Currently, implementations differ in this regard, and these differences can result in inconsistent behaviors that operators find difficult to identify and resolve.

That is another reason to keep the analysis evidence-focused. The data can say which routes carried which values at which collectors; it should not assign intent by itself.

Takeaways

A small ad-hoc monocle analysis can turn public MRT updates into useful, RFC-aware evidence. The first step is the MRT query; the second step is ordinary JSONL processing that groups rows, counts distinct routing dimensions, and annotates observed values with registry metadata.

The interesting parts are not hidden inside the MRT files anymore:

which 65535:* values are visible,
whether those values are assigned in the IANA registry,
where the values appear,
whether a route matches a simple RFC-derived profile,
which rows are worth deeper operator-policy investigation.

For this particular window, the main observations were:

many observed 65535:* values were not assigned in the current IANA well-known-community registry snapshot;
BLACKHOLE appeared both on highly specific prefixes with NO_EXPORT and on broader prefixes without NO_EXPORT/NO_ADVERTISE;
NOPEER was broadly visible across collectors, peers, origins, IPv4, and IPv6.

The broader lesson is the monocle research pattern: query a narrow slice of MRT data, summarize it with simple scripts, join with source-of-truth registries, then inspect specific rows with the RFC text next to the data.

The scripts used for this note are available here:

summarize_existing_well_known.py — summarize an existing JSONL file;
enrich_iana_status.py — add IANA status labels to the per-community CSV;
filter_well_known.py — filter generic JSONL output to rows carrying well-known communities;
summarize_well_known.py — stream-oriented variant that reads JSONL from stdin.

Investigating the India–Telegram BGP Hijack with monocle

Mingwei Zhang — Sat, 20 Jun 2026 00:00:00 GMT

On June 16, 2026, Indian carrier Reliance Communications (AS18101) began originating Telegram's IP prefixes in BGP, disrupting access for users inside and briefly outside India. The event has been covered from a policy angle (Anurag Bhatia's analysis, the IPv6 blackholing follow-up) and a traffic-impact angle (Kentik's post). This post does something different: it shows how to investigate the forensic evidence yourself using monocle and the BGPKIT API - open-source tools, real commands, real output. No screenshots of a vendor portal, no proprietary telemetry. Everything here you can run on your own machine.

The goal is forensics methodology, not opinion. We won't speculate about intent or mitigation policy. We'll establish ground truth, find the hijack in the data, measure how it propagated, and check the RPKI timeline - the same steps you'd follow for any BGP origin-conflict incident.

Prerequisites

brew install monocle  # monocle 1.3.0 used in this post
# cargo install monocle          # also works, but takes longer to build from source

All timestamps below are UTC. monocle accepts RFC 3339 strings and converts them internally:

$ monocle time "2026-06-16T16:14:19Z" -s
2026-06-16T16:14:19+00:00

Step 1: Establish ground truth - who should originate this prefix?

Before touching any MRT data, establish the expected state of the prefix. This is free and instant, and it gives you the baseline to compare against.

Telegram operates five ASNs: AS62041, AS62014, AS59930, AS44907, and AS211157. A single monocle inspect call pulls all of them:

monocle inspect 62041 62014 59930 44907 211157

ASN | Name | Country | Org
--- | --- | --- | ---
AS62041 | Telegram Messenger EUR | VG | Telegram Messenger Inc
AS62014 | Telegram Messenger APAC | VG | Telegram Messenger Inc
AS59930 | Telegram Messenger NA  | VG | Telegram Messenger Inc
AS44907 | Telegram Messenger Inc | VG | Telegram Messenger Inc
AS211157| Telegram Messenger Inc | VG | Telegram Messenger Inc

Now check who currently announces the prefix that was reported hijacked, 91.105.192.0/23:

monocle pfx2as 91.105.192.0/23 --show-name

The legitimate origin is AS211157 (Telegram). The suspect, AS18101 (Reliance Communications), should not appear here in a clean table.

The single most important check: RPKI validation

$ monocle rpki validate 91.105.192.0/23 18101 --format json

{
  "prefix": "91.105.192.0/23",
  "asn": 18101,
  "state": "invalid",
  "reason": "ASN 18101 not authorized; authorized ASNs: 211157",
  "covering_roas": [
    {"prefix": "91.105.192.0/23", "max_length": 23, "origin_asn": 211157, "ta": "RIPENCC"}
  ]
}

The prefix is covered by a ROA authorizing AS211157 only. AS18101 originating it is RPKI-invalid by definition. Any network enforcing Route Origin Validation (ROV) should have rejected this announcement. Confirm the legitimate origin validates clean:

$ monocle rpki validate 91.105.192.0/23 211157 --format json
{"prefix":"91.105.192.0/23","asn":211157,"state":"valid",...}

You now know: the prefix is signed, the legitimate origin is valid, and the suspect origin is invalid - before downloading a single MRT file.

List Telegram's signed prefixes

To understand what address space was protected by RPKI, pull every Telegram ROA across all five ASNs:

for asn in 62041 62014 59930 44907 211157; do
  monocle rpki roas $asn --format json-line
done | grep '^{' \
  | jq -r '"\(.prefix) AS\(.origin_asn)"' \
  | sort -u

This returns 55 ROAs - Telegram's signed prefix set, from 91.105.192.0/23 up through 149.154.160.0/22 and the 91.108.x.0/22 blocks. Every one of these prefixes, if originated by AS18101, would be RPKI-invalid.

Understand the suspect's connectivity

monocle as2rel 18101 --is-downstream --show-name  # upstreams
monocle as2rel 18101 15412 --format json | jq '.results[0]'  # relationship with FLAG

Reliance's primary upstreams are AS15412 (FLAG Telecom) and AS4755 (Tata Communications). Keep these in mind - they're the transits that would carry (or filter) any announcement Reliance originates.

Step 2: Find the hijack in the data

Now look for the actual BGP announcements. The reported time is 16:14:19 UTC on June 16. Start with a narrow 5-minute window and a single small collector.

Why a small collector?

rrc01 (London) and route-views2 are large collectors with hundreds of peers. Searching them downloads and parses a lot of data. For a first confirmatory search, a small collector is much faster - and if the event is real, a small collector that has a peer seeing the path will confirm it just as well.

Use the BGPKIT API to find collector peer counts:

$ curl -s "https://api.bgpkit.com/v3/broker/peers?collector=rrc14" | jq '.data | length'
21

rrc14 (Palo Alto, 21 peers) is small. Let's search there:

monocle search -p 91.105.192.0/23 -s -S -o 18101 \
  -t 1781626459 -d 5m -c rrc14 \
  --cache-dir ~/.cache/monocle \
  --format json-line \
  | jq -c '.timestamp |= todateiso8601'

{"timestamp":"2026-06-16T16:14:49Z","type":"ANNOUNCE","prefix":"91.105.192.0/23",
 "as_path":"19151 15412 18101","peer_asn":19151,"collector":"rrc14",...}

Confirmed. At 16:14:49 UTC, rrc14's peer AS19151 received 91.105.192.0/23 originated by AS18101, via the path 19151 15412 18101 - meaning FLAG (AS15412) was the transit carrying the hijack to the global table.

Why `-s -S`?

The -p prefix filter is exact-only by default. Hijacks routinely involve more-specifics - the victim announces /24s to fight back, the hijacker follows with even more specific routes. A bare -p 91.105.192.0/23 misses both the covering aggregate and any more-specifics. -s (include-super) and -S (include-sub) expand the match. Get in the habit of adding both to any hijack search.

The prefix arms race

Widen the search slightly (still one collector, still cached) to see the full set of prefixes Reliance originated in the escalation wave:

monocle search -p "91.108.4.0/22,149.154.160.0/22,91.105.192.0/23,95.161.64.0/20" \
  -s -S -o 18101 -t 1781594400 -T 1781630400 -c rrc14 \
  --cache-dir ~/.cache/monocle --format json-line \
  | jq -r '"\(.timestamp | todateiso8601) \(.prefix)"' | sort -u

2026-06-16T16:14:49Z  149.154.160.0/24
2026-06-16T16:14:49Z  149.154.161.0/24
2026-06-16T16:14:49Z  149.154.162.0/24
2026-06-16T16:14:49Z  149.154.163.0/24
2026-06-16T16:14:49Z  91.105.192.0/23
2026-06-16T16:14:49Z  91.108.4.0/23
2026-06-16T16:14:49Z  91.108.6.0/23
2026-06-16T16:14:49Z  95.161.64.0/21
2026-06-16T16:14:49Z  95.161.72.0/21

Nine prefixes, all announced simultaneously at 16:14:49. Note the specificity: Reliance originated /23s under Telegram's legitimate /22s, and /24s under the legitimate /23s. In BGP, a more-specific route wins. This is the prefix arms race Kentik described visually - and here it is in the raw update stream, with exact prefixes and timestamps.

Verify every hijacked prefix was RPKI-invalid

Validate each against AS18101:

for pf in 91.108.4.0/23 149.154.160.0/24 91.105.192.0/23; do
  monocle rpki validate $pf 18101 --format json-line 2>/dev/null \
    | grep '^{' \
    | jq -r --arg pf "$pf" '"\($pf) → \(.state): \(.reason)"'
done

91.108.4.0/23 → invalid: ASN 18101 not authorized; authorized ASNs: 62041
149.154.160.0/24 → invalid: ASN 18101 not authorized; authorized ASNs: 62041
91.105.192.0/23 → invalid: ASN 18101 not authorized; authorized ASNs: 211157

Every hijacked prefix was RPKI-invalid. This is machine-verified evidence, not an inference. Any transit that carried these was, by definition, not enforcing ROV.

Step 3: Was the hijacked route actually selected in the routing table?

Announcements in BGP updates tell you what was advertised. To see what was actually installed in a router's RIB at a given moment, use monocle rib. This is more expensive than search - always pass a collector filter.

$ monocle rib 1781627400 -p "91.105.192.0/23,91.108.4.0/22,149.154.160.0/22" \
    -s -S -c rrc14 --format json-line \
    | jq -r '"\(.prefix) origin=AS\(.origin_asns[0]) via \(.as_path)"' | sort -u

149.154.160.0/24 origin=AS18101 via 19151 15412 18101
149.154.161.0/24 origin=AS18101 via 19151 15412 18101
149.154.162.0/24 origin=AS18101 via 19151 15412 18101
149.154.163.0/24 origin=AS18101 via 19151 15412 18101
91.105.192.0/23  origin=AS18101 via 19151 15412 18101
91.108.4.0/23   origin=AS18101 via 19151 15412 18101
91.108.6.0/23   origin=AS18101 via 19151 15412 18101

At 16:30 UTC - sixteen minutes after the hijack began - rrc14's peer AS19151 had AS18101 installed as the origin for all seven prefixes, all via FLAG. The hijacked route wasn't just announced; it was the selected route in the RIB.

When did it end?

Check a later RIB snapshot:

$ monocle rib 1781683200 -p "91.105.192.0/23,91.108.4.0/22,149.154.160.0/22" \
    -s -S -c rrc14 --format json-line \
    | jq -r '"\(.prefix) origin=AS\(.origin_asns[0]) via \(.as_path)"' | sort -u

91.105.192.0/23 origin=AS211157 via 19151 9002 211157 211157

By 2026-06-17 08:00 UTC, the prefix is back to origin AS211157 (Telegram) - and notice the path changed from ... 15412 18101 to ... 9002 211157. The prefix is now reaching this vantage point via AS9002, not FLAG. The hijack was withdrawn sometime between 20:00 UTC on June 16 and 08:00 UTC on June 17. (Anurag's update noted the prefix fading behind FLAG around 01:47 IST / 20:17 UTC on June 16.)

Step 4: Who propagated the invalid routes?

From the RIB and update data, the propagation path at rrc14 is consistently 19151 15412 18101 - FLAG (AS15412) was the transit. Confirm the relationship:

$ monocle as2rel 18101 15412 --format json | jq '.results[0]'
{"asn1":18101,"asn2":15412,"connected":"86.5%","peer":"61.6%","as2_upstream":"24.9%"}

FLAG is both a peer and an upstream of Reliance at high visibility. Since the announcement was RPKI-invalid (Step 1) yet FLAG carried it into the global table, FLAG was not enforcing ROV on this session. The same logic applies to Tata (AS4755), Reliance's other transit, which appeared in paths at other collectors.

Hurricane Electric (AS6939) also appeared briefly in observed AS paths. In the UPDATE stream, HE propagated the invalid route for about 40 seconds and then withdrew it, consistent with an RPKI-invalid rejection path that was slower than the initial propagation. That is useful timing evidence, but it should not be read as a statement about traffic volume or broad customer impact.

Step 5: The RPKI timeline - did Telegram have ROAs before the incident?

A common pattern in BGP hijack history is reactive ROA creation: Twitter created ROAs for its routes after the 2021 Myanmar hijack, and that protection limited damage during the 2022 Russia incident. Was Telegram in the same reactive position, or did it already have ROAs?

The BGPKIT Alpha API provides historical ROA data with date_ranges arrays showing when each ROA was active:

$ curl -s "https://alpha.api.bgpkit.com/roas?asn=211157&date=2026-06-16" \
    | jq '.data[] | {prefix, date_ranges}'

{
  "prefix": "91.105.192.0/23",
  "date_ranges": [
    ["2023-09-23", "2025-05-04"],
    ["2025-05-06", "2026-06-12"],
    ["2026-06-14", "2026-06-20"]
  ]
}

Telegram's ROA for 91.105.192.0/23 dates back to September 2023 - nearly three years before this incident. Unlike Twitter's reactive adoption, Telegram had protection in place well ahead of time. This is why RPKI-RoV was able to limit the hijack's global impact (Kentik reported only ~1.6% of their BGP sources saw the hijacked route). The protection was already there; the question was whether transits enforced it.

Step 6: What about the IPv6 blackholing story?

Anurag's second post describes a separate phenomenon: Indian ISPs (TTSL AS45820, Lightstorm AS152144/AS135709) originating Telegram's IPv6 prefix 2a0a:f280::/48 and its aggregate 2a0a:f280::/32, which he hypothesizes is a misapplied remote-triggered blackhole configuration.

For this post, the important question is not whether the RFC 7999 BLACKHOLE community (65535:666) appears. A standards-compliant blackhole route should be tightly scoped: section 3.2 says receivers should add NO_ADVERTISE or NO_EXPORT, and section 3.3 says the neighboring network must be authorized to advertise the covering prefix. TTSL and Lightstorm are not authorized for Telegram's IPv6 space, so the observable BGP fact remains the same either way: these are unauthorized origins.

Bryton Herdes' False immunity: long prefixes that bypass ROV talk is useful background on how TE and RTBH exceptions can create loose handling for long prefixes. But the public BGP evidence here cannot prove the operators' internal mechanism; it can only show the invalid origins.

What is observable is that these ISPs are originating prefixes they do not own - RPKI-invalid for those origins. This is the same routing anomaly pattern as the Reliance Communications hijack, regardless of the mechanism. You can verify it the same way:

monocle pfx2as 2a0a:f280::/48 --show-name          # who announces it?
monocle rpki validate 2a0a:f280::/48 45820 --format json  # is TTSL authorized?

A secondary puzzle remains: if this is not standard RFC 7999 RTBH, what mechanism caused these ISPs to originate Telegram's IPv6 prefixes? That question is worth investigating, but it is not something BGP UPDATE data alone can answer - it requires off-band information about the operators' configuration practices.

Summary

Here is the complete evidence chain, reproducible from the commands above:

Ground truth: 91.105.192.0/23 is RPKI-signed for AS211157 (Telegram). AS18101 (Reliance Communications) originating it is RPKI-invalid. (monocle rpki validate)
The hijack: At 16:14:49 UTC on June 16, Reliance originated 9 Telegram prefixes (/23s and /24s under the legitimate aggregates), all via FLAG (AS15412). (monocle search -s -S -c rrc14)
RIB confirmation: At 16:30 UTC, the hijacked routes were installed in rrc14's RIB with AS18101 as origin. By 08:00 UTC on June 17, the prefix was back to AS211157. (monocle rib -c rrc14)
ROV failure: FLAG (AS15412) and Tata (AS4755) carried RPKI-invalid announcements, so neither enforced ROV on those sessions. (monocle as2rel + propagation data)
ROA timeline: Telegram's ROAs date to September 2023 - protection was proactive, not reactive. (BGPKIT Alpha API)
IPv6 anomaly: TTSL and Lightstorm originated Telegram's IPv6 prefixes (2a0a:f280::/48 and aggregate 2a0a:f280::/32), RPKI-invalid for those origins. RFC 7999 section 3.3 describes blackhole prefixes as "typically as specific as possible, /32 for IPv4 or /128 for IPv6," and section 3.3's MUST conditions require the origin to be authorized - neither fits aggregates originated by an unauthorized AS.

Every command in this post has been run against live public data. That's the point: BGP forensics doesn't require a proprietary platform. monocle, the BGPKIT API, and public MRT archives are sufficient to investigate - and verify - any routing incident.

References

monocle v1.3.0: Reconstruct BGP RIB State at Any Timestamp

Mingwei Zhang — Wed, 27 May 2026 00:00:00 GMT

We are happy to announce the release of monocle v1.3.0, now available on GitHub and crates.io, also available on Homebrew via brew install monocle.

This release introduces a major new command, monocle rib, which reconstructs BGP Routing Information Base (RIB) state at arbitrary timestamps.

New Feature: `monocle rib`

Most BGP tools show you what happened during an update file. monocle rib shows you the routing table as it existed at any moment in time.

Given a target timestamp, the command:

Finds the latest RIB dump at or before that time
Downloads and replays all overlapping BGP update files up to the exact target timestamp
Materializes the final route state per peer and prefix

The result is the reconstructed RIB — not raw BGP messages, but the actual route state.

Basic Usage

Reconstruct the RIB for a single timestamp and print to stdout:

monocle rib 2025-09-01T12:00:00Z -c route-views6

Write multiple snapshots to a single SQLite file:

monocle rib \
  2025-09-01T08:00:00Z \
  2025-09-01T12:00:00Z \
  --sqlite-path /tmp/rib-snapshots.sqlite3 \
  -c route-views6

Filtering

Apply filters at reconstruction time to narrow the result set:

# Only routes originated by AS13335 (Cloudflare)
monocle rib 2025-09-01T12:00:00Z -c route-views6 -o 13335

# Routes for a specific prefix, including more-specifics
monocle rib 2025-09-01T12:00:00Z -c route-views6 -p 2606:4700::/32 --include-sub

# Routes from full-feed peers only
monocle rib 2025-09-01T12:00:00Z -c route-views6 --full-feed-only

# Routes where AS path contains a specific ASN
monocle rib 2025-09-01T12:00:00Z -c route-views6 -a "13335"

Performance Notes

The rib command keeps the working RIB state in memory during reconstruction, avoiding SQLite lookups and writes on the hot path. The updates query window was tightened from +2 hours lookahead to the exact target timestamp, reducing update file downloads by roughly one third for typical requests.

Installation

cargo install monocle

Or download pre-built binaries for Linux (x86_64, aarch64) and macOS (universal) from the GitHub release page.

monocle v1.1.0

Mingwei Zhang — Sat, 14 Feb 2026 00:00:00 GMT

monocle v1.1.0 focuses on interface consistency and day-to-day usability. This release simplifies feature gates, standardizes data refresh APIs, and adds quality-of-life improvements across parsing, search, and configuration workflows.

TL;DR

Feature flags are now simplified to lib, server, and cli.
Config and update flows are more consistent (config update, config backup, config sources, and --no-update).
Data refresh APIs are standardized across ASInfo, AS2Rel, RPKI, and Pfx2as.
Cache TTL defaults are unified at 7 days, with clearer staleness reporting.
Parse/search workflows gain multi-value filters, negation filters, field selection, ordering, timestamp format control, and local cache support.

What's New

Simpler feature flags

The crate feature model is now reduced to three options:

lib: complete library functionality (database + lenses + display)
server: WebSocket server support (implies lib)
cli: full command-line binary (implies lib and server)

This replaces the previous multi-tier setup and makes dependency selection easier for downstream users.

Standardized data refresh APIs

Database refresh behavior is now more uniform across ASInfo, AS2Rel, RPKI, and Pfx2as:

Consistent needs_*_refresh(ttl) checks
A shared RefreshResult shape with source and load details
Standardized naming (refresh_*) with compatibility aliases where needed
URL and local-path loading paths available across repositories

This update reduces API drift and makes maintenance code paths more predictable.

Config command updates

Configuration and maintenance commands now use clearer naming:

monocle config db-refresh -> monocle config update
monocle config db-backup -> monocle config backup
monocle config db-sources -> monocle config sources

The global no-refresh toggle was also renamed for consistency:

--no-refresh -> --no-update

The following command shows the active configuration, cache TTL settings, database status, and server defaults:

monocle config

Example output:

monocle Configuration
=====================

General:
  Config file:    /home/user/.monocle/monocle.toml
  Data dir:       /home/user/.monocle/

Cache TTL:
  ASInfo:         7 days
  AS2Rel:         7 days
  RPKI:           7 days
  Pfx2as:         7 days

Database:
  Path:           /home/user/.monocle/monocle-data.sqlite3
  Status:         exists
  Size:           512.47 MB
  Schema:         initialized (v3)
  ASInfo:         120953 records (updated: 2026-02-02 19:54:01 UTC)
  AS2Rel:         877937 records (updated: 2026-02-02 14:25:34 UTC)
  RPKI:           796899 ROAs, 962 ASPAs (updated: 2026-02-10 19:53:50 UTC)
  Pfx2as:         1580626 records (updated: 2026-02-02 20:02:10 UTC)

Better cache control defaults

All major data sources now support configurable cache TTL with a 7-day default. This applies to ASInfo, AS2Rel, RPKI, and Pfx2as.

monocle config sources now reports staleness based on TTL, so it is easier to see what needs updating.

The following command shows per-source status, staleness, and last update recency:

monocle config sources

Example output:

Data Sources:

  Name         Status          Stale      Last Updated
  ------------------------------------------------------------
  asinfo       120953 records  yes        a week ago
  as2rel       877937 records  yes        a week ago
  rpki         797861 records  no         2 hours ago
  pfx2as       1580626 records yes        a week ago

Configuration:
  ASInfo cache TTL: 7 days
  AS2Rel cache TTL: 7 days
  RPKI cache TTL:   7 days
  Pfx2as cache TTL: 7 days

RPKI improvements

monocle now supports fetching ROAs via RTR (RPKI-to-Router), including endpoint override support and fallback behavior.

The following command refreshes only RPKI data and uses the provided RTR endpoint for this run instead of the default configured source.

monocle config update --rpki --rtr-endpoint rtr.rpki.cloudflare.com:8282

Parse and search enhancements

parse and search gained several output and filtering improvements:

Multi-value filters with OR semantics
Negation filters using !
Validation for ASN/prefix filter inputs
--fields for column selection
--order-by and --order for sorted output
--time-format for unix or RFC3339 display
search --cache-dir local file + broker query caching

The following command searches one hour of updates starting at 2024-01-01, filters for prefix 1.1.1.0/24, and caches downloaded MRT files plus broker query results under /tmp/mrt-cache for faster repeat runs.

monocle search -t 2024-01-01 -d 1h -p 1.1.1.0/24 --cache-dir /tmp/mrt-cache

The following command uses multi-value filters with negation to exclude two origin ASNs while also matching either of two peer ASNs:

monocle search -t 2024-01-01 -d 1h -o '!13335,!15169' -J 174,2914

Negation and positive values cannot be mixed within the same filter field.

Breaking Changes and Migration Notes

1) Feature flag migration

If you previously used feature tiers like database, lens-core, lens-bgpkit, lens-full, or display, switch to:

lib for library use
server for WebSocket API use
cli for full command-line use

2) CLI and subcommand renames

Update scripts and automation:

--no-refresh -> --no-update
config db-refresh -> config update
config db-backup -> config backup
config db-sources -> config sources

3) Parse/search filter type updates (library API)

ParseFilters moved from scalar optional fields to vector-based values for multi-value and negation support. Library consumers should update filter construction accordingly.

Additional Improvements

AS name rendering now prefers PeeringDB naming fields before falling back to AS2Org/core names
Data refresh logging now shows specific reasons (empty vs outdated)
Example layout was reorganized to one example per lens

Full Change List

See the v1.1.0 section in CHANGELOG.md for the complete list of changes.

BGPKIT Parser v0.14.0 Release and v0.13.0 Highlights

Mingwei Zhang — Thu, 25 Dec 2025 00:00:00 GMT

We are pleased to announce the release of BGPKIT Parser v0.14.0. This update introduces support for negative filters and the RPKI-to-Router (RTR) protocol. We also want to highlight key features from the recent v0.13.0 release, including enhanced debugging tools.

v0.14.0 Features

Negative Filter Support

A frequent request has been the ability to filter out specific data points. We have added support for negative filters across most filter types, allowing exclusion of specific origins, prefixes, peers, or communities.

In the CLI, use the != operator. For example, to process all records except those originating from AS 13335:

bgpkit-parser https://spaces.bgpkit.org/parser/update-example.gz --filter "origin_asn!=13335"

In Rust code, use the ! prefix:

let parser = BgpkitParser::new("...")
    .add_filter("!origin_asn", "13335")
    .add_filter("!peer_ip", "192.168.1.1");

Supported negative filters include !origin_asn, !prefix, !peer_ip, !peer_asn, !type, !as_path, !community, and !ip_version.

RPKI RTR Protocol Support

We have added support for the RPKI-to-Router (RTR) protocol, covering both version 0 (RFC 6810) and version 1 (RFC 8210).

The new models::rpki::rtr and parser::rpki::rtr modules allow developers to build custom RTR clients or servers.

Here is how you can use the library to connect to an RTR server and request data:

use bgpkit_parser::models::rpki::rtr::*;
use bgpkit_parser::parser::rpki::rtr::{read_rtr_pdu, RtrEncode, RtrError};
use std::net::TcpStream;
use std::io::Write;

// 1. Connect to the RTR server
let mut stream = TcpStream::connect("rtr.rpki.cloudflare.com:8282")?;

// 2. Send a Reset Query to request the full database
let reset_query = RtrResetQuery::new_v1();
stream.write_all(&reset_query.encode())?;

// 3. Read the response PDUs
loop {
    match read_rtr_pdu(&mut stream)? {
        RtrPdu::IPv4Prefix(p) => {
            println!("Received IPv4 ROA: {}/{}-{} -> AS{}", 
                p.prefix, p.prefix_length, p.max_length, p.asn);
        }
        RtrPdu::EndOfData(_) => break,
        _ => {}
    }
}

We have included a fully functional RTR client example that connects to a server, fetches ROAs, and performs route validation.

You can run the example yourself:

cargo run --example rtr_client -- rtr.rpki.cloudflare.com 8282

In Case You Missed It: v0.13.0

The v0.13.0 release introduced several improvements for debugging and analyzing MRT data.

Record-Level Output

The CLI supports inspecting individual MRT records rather than just parsed BGP elements. This aids in debugging parser issues or analyzing raw MRT files.

Switch to record-level output with --level records and choose a format (e.g., JSON):

bgpkit-parser https://spaces.bgpkit.org/parser/update-example.gz --level records --format json

Raw Bytes Access

For developers, RawMrtRecord now includes a header_bytes field, and the raw_bytes field has been renamed to message_bytes. This provides access to the exact bytes of the MRT header and the message body as they appeared on the wire, enabling byte-for-byte export and debugging without re-encoding.

Other Improvements

Testing & Fuzzing: Added a cargo-fuzz harness and initial fuzz targets.
Performance: Continued optimizations for faster processing.

Check out the full CHANGELOG for more details.

Happy Parsing!

BGPKIT Broker v0.9.0: Better Pagination and New Collectors

Mingwei Zhang — Sat, 01 Nov 2025 00:00:00 GMT

We are excited to announce the release of BGPKIT Broker v0.9.0. This release introduces total count support for efficient pagination, making it easier to build data exploration interfaces and manage large result sets. We also expand our collector coverage with two new Internet Exchange points.

Previously on BGPKIT Broker

Since the major v0.7 release that unified the architecture into a single SQLite-backed CLI application, BGPKIT Broker has been serving the community with stable uptime and continuous data coverage. The v0.8 series brought several enhancements including automated backup systems, configuration validation, and convenient query shortcuts.

However, one common challenge remained for developers building interfaces on top of Broker: pagination. When querying large time ranges, users needed to fetch all results to know the total count, making it inefficient to build proper pagination controls or display result statistics.

V0.9: Efficient Pagination Support

Version 0.9.0 addresses pagination needs with total count support at both the SDK and API levels. This allows applications to fetch result counts independently from the data itself, enabling responsive user interfaces without over-fetching data.

SDK: `query_total_count()` Method

The SDK now provides a dedicated query_total_count() method that returns the total number of matching items without retrieving the actual data. This is useful when you need to know how many results exist before deciding how to paginate through them.

use bgpkit_broker::BgpkitBroker;

#[tokio::main]
async fn main() {
    let broker = BgpkitBroker::new()
        .ts_start("2024-10-01")
        .ts_end("2024-10-02")
        .project("route-views");

    // Get total count without fetching items
    let total = broker.query_total_count().await.unwrap();
    println!("Total matching items: {}", total);

    // Now fetch paginated results
    let result = broker
        .page_size(100)
        .page(1)
        .query()
        .await
        .unwrap();

    println!("Retrieved {} items out of {} total", 
             result.data.len(), 
             result.total.unwrap_or(0));
}

This method executes a COUNT(*) query against the database with the same filters as your main query, returning just the number rather than all the data. For large time ranges with thousands of results, this can save significant bandwidth and processing time.

API: Total Field in Search Results

The /v3/search endpoint now includes a total field in every response, providing the total count of matching items alongside the paginated results. This enhancement makes it straightforward to build pagination controls in web interfaces or CLI tools.

curl "https://api.bgpkit.com/v3/broker/search?ts_start=2024-10-01&ts_end=2024-10-02&page_size=10&page=1"

Response:

{
  "total": 1234,
  "data": [
    {
      "ts_start": 1727740800,
      "ts_end": 1727740800,
      "collector": "route-views2",
      "project": "route-views",
      "url": "...",
      ...
    }
  ]
}

With the total field, client applications can:

Display "showing X of Y results" to users
Calculate the number of pages needed for pagination
Decide whether to fetch all results or paginate
Provide accurate progress indicators for data processing

New RouteViews Collectors

This release adds support for two new RouteViews collectors:

hkix.hkg: Hong Kong Internet Exchange (HKIX) collector in Hong Kong
ix-br.gru: IX.br (PTT.br) in São Paulo

These additions enhance BGPKIT Broker's global coverage, providing more diverse vantage points into internet routing behavior. Users who update to v0.9.0 can run bgpkit-broker update to automatically bootstrap data for these new collectors.

Behind the Scenes Improvements

Beyond the user-facing features, we also made several code improvements:

Refactored bootstrap download logging to centralize progress reporting and eliminate redundant code paths
Updated the oneio dependency to v0.20.0 with optimized feature flags for better handling rustls providers
Enhanced test coverage overall

Upgrading to v0.9.0

For SDK Users

Update your Cargo.toml to use the latest v0.9 release:

[dependencies]
bgpkit-broker = "0.9"

Then run:

cargo update

The new query_total_count() method is immediately available for use. The total field in BrokerQueryResult is backward compatible as an optional field.

For Self-Hosted Instances

If you run your own BGPKIT Broker instance, update to the latest version:

cargo install --force bgpkit-broker --version 0.9.0 --features cli

Or pull the latest Docker image:

docker pull bgpkit/bgpkit-broker:latest

After upgrading, run the update command to add the new collectors:

bgpkit-broker update --db-path your-database.sqlite3

The update process will automatically detect and bootstrap historical data for the two new collectors.

Looking Forward

BGPKIT Broker continues to serve as a fundamental component for BGP data processing pipelines. With v0.9.0, we focused on making the developer experience better for building applications that need pagination and result statistics. We continue to maintain the public Broker instance at https://api.bgpkit.com/v3/broker with 99.996% uptime, and encourage users to deploy on-premise instances for production pipelines to reduce external dependencies.

For the full v0.9.0 release notes, please check out our GitHub release page. If you have any comments, please drop us a message on Twitter, Mastodon, Bluesky or email.

Get Started

Ready to try out the new features? Here are some resources:

GitHub: github.com/bgpkit/bgpkit-broker
Examples: Browse example code demonstrating common usage patterns
Public API: Try queries at https://api.bgpkit.com

Have questions or feedback? Open an issue on GitHub or join discussions in our Discord Channel.

💖

If you find our libraries and services useful, we would highly appreciate if you consider sponsoring us on GitHub.

Run BGPKIT on Cloudflare Containers

Mingwei Zhang — Mon, 30 Jun 2025 00:00:00 GMT

For the longest time, I’ve been using Cloudflare exclusively for web/API hosting and relatively lightweight tasks. For the most of my work on BGP, there is really not much that I can accomplish with just JavaScript/TypeScript (except maybe working with RIS Live WebSocket). The computationally intensive nature of the most BGP data processing doesn't naturally fit within the typical Cloudflare developer platform.

This changes with the recent announcement of Cloudflare Containers. In short, it allows developers to build and run custom containers on the Cloudflare’s platform, enabling the heavy workload to mix with other primitives and unify the deployment.

In this blog post, I will show you how to build a BGP data search API with BGPKIT and deploy on Cloudflare Containers. The source code is available on GitHub.

BGP Data Search API

For this example, I will show you how to build a very straightforward HTTP API to allow accepting search parameters and let BGPKIT to fetch and parse BGP archives and return the parsed messages.

The API accepts four parameters: collector, prefix, ts_start, and ts_end, to filter and parse BGP archives efficiently.

collector: the BGP route collector ID to use (e.g. rrc00 or route-views2). We want to limit the search to a single collector.
prefix: the prefix of the BGP relevant updates. Open-ended search will burn up resources quick, but you can opt-out this requirement.
ts_start and ts_end: the starting and ending timestamps. The goal is to limit the search to end up parsing a very small amount of MRT files (e.g. give them the same value will do). We should probably leave large-scale data crunching to a environment with more CPU powers.

The parameters are defined in a struct to be able to passed in to a axum “GET” route:

#[derive(Deserialize, Serialize)]
struct Params {
    collector: String,
    prefix: String,
    ts_start: String,
    ts_end: String,
}

With the given parameters, we will first find the relevant MRT files by setting the filters of timestamps and collector to BgpkitBroker instance:

        let files = match bgpkit_broker::BgpkitBroker::new()
            .ts_end(ts_end.clone())
            .ts_start(ts_start.clone())
            .collector_id(collector.clone())
            .query(){
            Ok(items) => items,
            Err(e) => {
                return Json(Result {
                    error: Some(e.to_string()),
                    data: vec![],
                    meta: None,
                });
            }
        };

For each file, we will parse the whole MRT file and collect BGP updates that is relevant to the target prefix:

        for file in files {
            let mut parser = match bgpkit_parser::BgpkitParser::new(file.url.as_str()){
                Ok(parser) => parser,
                Err(e) => {
                    return Json(Result {
                        error: Some(e.to_string()),
                        data: vec![],
                        meta: None,
                    });
                }
            };

            parser = match parser.add_filter("prefix", prefix.as_str()){
                Ok(parser) => parser,
                Err(e) => {
                    return Json(Result {
                        error: Some(e.to_string()),
                        data: vec![],
                        meta: None,
                    });
                }
            };
            items.extend(parser.into_elem_iter());
        }

Because the BGPKIT parser and broker code are implemented in the sync environment, we will need to wrap the code above in a blocking thread in order to use it in async web frameworks like axum.

let result = tokio::task::spawn_blocking(move || {
   // THE BLOCKING CODE PIECES
}).await.unwrap();

Please see the full source code here for more details:

https://github.com/bgpkit/bgpkit-cf-containers/blob/main/container-src/src/main.rs

Cloudflare Containers Deployment

Now that we have an Rust-based API code working, we will need to 1. containerize the code, 2. put a Cloudflare Container wrapper around it for deployment.

The container definition is a typical two-stage build definition, with a builder stage to build the binary application, and a minimum deployment stage to call the executable. The two-stage build is almost necessary as Cloudflare Containers has limits on the size of each image and the total storage per account. The smaller the image the better.

# ---- Build Stage ----
FROM rust:1.86 AS builder

WORKDIR /app

# Install build dependencies
RUN apt-get update && apt-get install -y pkg-config libssl-dev

# Build application
COPY container-src/Cargo.lock container-src/Cargo.toml ./
COPY container-src/src ./src
RUN cargo build --release

# ---- Runtime Stage ----
FROM debian:bookworm-slim

# Install minimal runtime dependencies
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*

WORKDIR /app

# Copy the statically linked binary from the builder
COPY --from=builder /app/target/release/bgpkit-cf-container /app/bgpkit-cf-container

EXPOSE 3000

CMD ["/app/bgpkit-cf-container"]

The rest of the task is to build a API app for Workers and configure Containers. The following config is pretty much all it needs to configure the Workers script to build and push the container image, and create a Durable Object to coordinate and run Containers.

	"containers": [
		{
			"class_name": "BgpkitContainer",
			"image": "./Dockerfile",
			"max_instances": 5
		}
	],
	"durable_objects": {
		"bindings": [
			{
				"class_name": "BgpkitContainer",
				"name": "BGPKIT_CONTAINER"
			}
		]
	},
	"migrations": [
		{
			"new_sqlite_classes": [
				"BgpkitContainer"
			],
			"tag": "v1"
		}
	]

The main Workers script is only 22 lines long:

import { Container, getContainer } from '@cloudflare/containers';
import { Hono } from "hono";

export class BgpkitContainer extends Container {
	defaultPort = 3000;
	sleepAfter = '5m';
}

// Create Hono app with proper typing for Cloudflare Workers
const app = new Hono<{
	Bindings: { BGPKIT_CONTAINER: DurableObjectNamespace<BgpkitContainer> };
}>();

app.get("/search", async (c) => {
	if (!c.req.query('collector') || !c.req.query('prefix') || !c.req.query('ts_start') || !c.req.query('ts_end')) {
		return c.json({ error: "Missing required query parameters: collector, prefix, ts_start, ts_end" }, 400);
	}
	const container = getContainer(c.env.BGPKIT_CONTAINER);
	return await container.fetch(c.req.raw);
});

export default app;

The important pieces are:

class BgpkitContainer extends Container block defines the port to use and configures how long the container should run after the last query. In this example, containers will be killed after 5 minutes of inactivity. It is crucial to realize that Cloudflare Containers are not a drop-in replacement of some other container deployment platforms like fly.io or Railway, as the workload for Containers are intended to be short-lived (ping me if this changes) and scales horizontally depending on the requests amount.
the getContainer function here tries to reach a container. If the intended container is overloaded, it may create a new container on demand. You may choose to use the getRandom function to round-robin containers. See the docs for more.
the container.fetch(c.req.raw) forwards the query to the container, including the query parameters, which will then be handled by the running container.

Example Queries

The following example will reach the Workers script (handled by Hono), and then reach the container to run the actual BGP data crunching task. (This URL won’t actually work as we don’t have budget to provide such service openly. Feel free to deploy it on your own account to try it out.)
https://EXAMPLE.bgpkit.workers.dev/search?collector=rrc00&prefix=1.1.1.0/24&ts_start=1751231488&ts_end=1751231488

BGPKIT Broker 0.7 Release

Mingwei Zhang — Sat, 22 Jun 2024 00:00:00 GMT

BGPKIT Broker is a fundamental component to our design of a all-purpose BGP data processing pipeline. In short, it is a BGP data file meta information "broker" that tells the data consumers what MRT files from RouteViews and RIPE RIS are available for any given time range in question. It commonly serves as a data input entry point for data pipelines.

For instance, here is a simple diagram for a system that creates a semi-real-time BGP data stream with BGPKIT Broker and Parser (a very common use case for these two libraries).

BGPKIT Broker periodically crawls the websites of RIPE RIS and RouteViews MRT data pages of their collectors and index meta information into a database. Downstream consumers can ask and retrieve new files and process the files into BGP messages.

Previously on BGPKIT Broker

In the BGPKIT Broker version 0.1 to 0.6, a working broker instance consists of three individual components: a crawler, a Postgres database, and an API.

Each of the three component runs independently, and requires independent configuration, cronjobs, deployment, and all these goodies. For example, to run BGPKIT Broker v0.6, a user will need to configure and run

a PostgreSQL database with proper credentials and schema set up;
a cronjob instance that periodically crawls the data sources, with optional locks to prevent overlapping executions in case some crawl became slow;
a API application likely sitting behind a configured reverse proxy like Caddy to serve the data.

It's fun and exciting to set up all these for the first time, but quickly became tiring and too complex for repeated set up or bootstrapping for new users.

V0.7: one CLI app that does everything

We completely revamped the architecture for BGPKIT Broker in V0.7 to merge every functionality needed for a running Broker instance into one single command-line application: bgpkit-broker. V0.7 provides a single application to configure, run, debug, query everything on BGPKIT Broker.

To achieve this redesign, we made some major changes to our architecture.

SQLite instead of PostgreSQL

There are two major topics to concern when choosing a backend database for BGPKIT Broker: performance and portability.

SQLite is more than fast enough

BGPKIT Broker indexes metadata for all collectors from RouteViews and RIPE RIS, which includes time, URL, type, size of every RIB dump and updates MRT files from these two public archives. Dating all the way back to 1999, we have indexed roughly 48 million MRT files' metadata.

With a single index on the timestamp of files, we are able to search data files in less than 0.5s for any queries, which is more than fast enough for our use cases. We admit that we have spent our time for "early optimization" and in the end, the simple schema out-weighs the small performance gains.

Backup and bootstrap with just one file

Now in terms of portability, we can appreciate enough the beauty of single-file database like SQLite. In our current production setup, we periodically backup the database, and it literally involves just copying a single file to another directory (well, we also upload it to Cloudflare R2 for safekeeping).

Portability also means users can move their instance anywhere they want with ease. This is definitely the case for V0.7 where new users can bootstrap by simply download a SQLite file (our CLI provides all that functionality), and move to new locations by scp it to anywhere they desire.

Here is a video demonstrating bootstrapping a local BGPKIT Broker SQLite database with the new bgpkit-broker bootstrap command.

%[https://youtu.be/SsCyfQ5q0G0]

New file notification via NATS

Before V0.7, pipelines that needs to continuously processing new MRT files will need to "pull" data from BGPKIT Broker instance periodically and keep track of the latest files processed. We consider this a hassle that developers should not be dealing with and thus introduced a new NATS-based message channel allowing data consumers to subscribe to the public/private NATS channel where a Broker instance may publish new file notification to.

We dedicated nats.broker.bgpkit.com as the public endpoint for any NATS consumers to connect to. Whenever a new file becomes available in Broker, it will publish a new file notification with all metadata as in the database entry to the public channel. Consumers (e.g. data pipelines) can use the NatsNotifier::new(None).start_subscription() to start waiting for new files. The following snippet below shows how a simple pipeline can use this feature in a loop.

let mut notifier = match NatsNotifier::new(url).await {
    Ok(n) => n,
    Err(e) => {
        error!("{}", e);
        return;
    }
};
if let Err(e) = notifier.start_subscription(subject).await {
    error!("{}", e);
    return;
}
while let Some(item) = notifier.next().await {
    if pretty {
        println!("{}", serde_json::to_string_pretty(&item).unwrap());
    } else {
        println!("{}", item);
    }
}

We also implemented a simple new file watcher in the app as bgpkit-broker live subcommand. It will start a subscription to the public BGPKIT NATS endpoint and print out new file data as they come to the channel.

One command to serve and update

As mentioned previously, the new bgpkit-broker application includes everything one needs to start a instance. Once one bootstrapped the database to a local sqlite file (via bgpkit-broker bootstrap <FILENAME> command), all they need to start a auto-updating API is to run bgpkit-broker serve <FILENAME> .

bgpkit-broker serve --help
Serve the Broker content via RESTful API

Usage: bgpkit-broker serve [OPTIONS] <DB_PATH>

Arguments:
  <DB_PATH>  broker db file location

Options:
  -i, --update-interval <UPDATE_INTERVAL>  update interval in seconds [default: 300]
      --no-log                             disable logging
  -b, --bootstrap                          bootstrap the database if it does not exist
      --env <ENV>                          
  -s, --silent                             disable bootstrap progress bar
  -h, --host <HOST>                        host address [default: 0.0.0.0]
  -p, --port <PORT>                        port number [default: 40064]
  -r, --root <ROOT>                        root path, useful for configuring docs UI [default: /]
      --no-update                          disable updater service
      --no-api                             disable API service
  -h, --help                               Print help
  -V, --version                            Print version

The serve subcommand will also start a thread that periodically crawl and update the SQLite database to make sure the API always serve the up-to-date data.

Noticed that error message? It's by design as it tries to connect to notification channel for new files as a default behavior for a service, but not NATS URL is configured. We use the BGPKIT_BROKER_NATS_URL environment variable to configure the NATS channel to use.

We also allow users to optionally configure a heartbeat URL to monitor the data updating status. After every success data crawling run, Broker will try to execute a HTTP GET to a URL if BGPKIT_BROKER_HEARTBEAT_URL is set in the environment. This is useful to monitor the running status of the Broker instance without the need of setting up a cronjob.

We use Better Stack's Uptime monitoring service for page and heartbeat monitoring, and the public Broker instance is running V0.7 with the heartbeat URL set to this service. All status information can be found at https://status.bgpkit.com/

Production-ready, on-prem deployment

Although BGPKIT Broker has not yet reached V1.0, we consider it to be feature-complete and production-ready. Ever since V0.2, we have made our better efforts on not introducing any breaking changes and the service has been serving the community with a stable uptime ever since. We believe all libraries running in production should at least be 1.0, and thus we will release V1.0 soon this summer.

We also made significant efforts in V0.7 release to make sure BGPKIT Broker is as portable as possible. New users can spin up a fully functioning Broker instance with just two commands: bgpkit-broker bootstrap and bgpkit-broker serve, all within 5 minutes. With V0.7 released, we encourage all data pipeline designers to deploy a Broker instance on-premise, ensuring data pipelines are self-containing and reduce external dependencies as much as possible. We also will continue maintain our public instance to our best efforts (we are currently at 99.996% uptime). Thanks to our sponsors, we are able to keep the services up as we do, and we plan to continue serving the community the same way in the foreseeable future.

For the full V0.7 release notes, please check out our GitHub release page. If you have any comments, please drop us a message at Twitter, Mastodon, or email.

💖

If you find our libraries and services useful, we would highly appreciate if you consider sponsor us on GitHub.

Command-line Routing Stats with monocle and Cloudflare Radar API

Mingwei Zhang — Sun, 21 Apr 2024 00:00:00 GMT

BGPKIT monocle is a command-line utility program that helps users quickly pull Internet routing-related information from publicly available sources.

https://github.com/bgpkit/monocle

In BGPKIT monocle version V0.5, we add support for querying Cloudflare Radar's new BGP routing statistics and prefix-to-origin mapping APIs, the same APIs that power the Cloudflare Radar routing section. monocle users can now quickly glance overview of routing stats for any given ASN, country, or the whole Internet. Users can also quickly look up prefix origins and examine their RPKI validation status as well as prefix visibility on the global routing tables.

Using `monocle radar`

We added a new monocle radar command group in V0.5, which contains the following to subcommands:

monocle radar stats [QUERY]: get routing stats (like prefix count, rpki invalid count) for a given country or ASN.
monocle radar pfx2as [QUERY] [--rpki-status valid|invalid|unknown]: get prefix to origin mapping for a given prefix or ASN

mingwei@terrier ~ % monocle radar
Cloudflare Radar API lookup (set CF_API_TOKEN to enable)

Usage: monocle radar <COMMAND>

Commands:
  stats   get routing stats
  pfx2as  look up prefix to origin mapping on the most recent global routing table snapshot
  help    Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

Cloudflare API token needed

Since the monocle radar command relies on querying data using Cloudflare Radar public API, we also need to specify a user API token as the environment variable CF_API_TOKEN. Obtaining an API token is free and only needs a Cloudflare account. Interested users can follow their official tutorial to obtain a token. The environment variable can be set in a .env file in the current directory, or set in ~/.bashrc or ~/.profile etc.

`monocle radar stats`

Users can query the routing statistics for a given country or ASN. For example, monocle radar stats us returns the routing stats for the United States, while monocle radar stats 174 returns the stats for Cogent (AS174).

The displayed table is further divided into three rows, one for overall counting, and one for IPv4 and IPv6-specific counting. For each row, we show the following fields:

origins: the number of origins ASes registered in the given country
prefixes: the number of prefixes originated by the given ASN or ASes registered in the given country
rpki_valid/invalid/unknown: the number of RPKI valid/invalid/unknown prefix routes (prefix-origin mapping) on the global routing table and their percentage of the overall routes.

`monocle radar pfx2as`

Users can query the prefix-to-origin API to get the mapping of origin ASes and their originated prefixes on the global routing table.

In the following example, monocle radar pfx2as 174 --rpki-status invalid, we ask for all the prefixes originated by AS174 with the RPKI validation status to be invalid. This command returns us the list of RPKI invalid prefixes originated by AS174 at the time of generating the dataset.

Questions it can answer now (more in the future)

Here is a selected list of questions that monocle radar command can answer you:

How many ASes are there on the Internet that announce at least one prefix? (81,770)
How many of these ASes announce only IPv6 prefixes? (6,853)
How many prefixes are there on the global routing table? (1,205,218)
How many prefixes do AS400644 announce? (1)
Which AS(es) originates 1.1.1.0/24? (AS13335)
How many prefixes originated by AS174 are NOT covered by some RPKI ROA? (a lot, 94%+)
How about the RPKI valid ratio for the Philippines? (77%, nice!)

Powered by Cloudflare Radar free API

Cloudflare Radar is a hub that showcases global Internet traffic, attack, and technology trends and insights.

What Cloudflare Radar shines is its data openness. Everything you see on the Cloudflare Radar website is powered by their free publicly available APIs. It's a treasure trove there, and all users need is a free API token to access everything.

At BGPKIT, we think we can further improve the usability of the API by exposing them as a proper Rust SDK: radar-rs. This is our (unofficial) effort on bringing the Cloudflare Radar's rich data to Rust developers. For example, monocle radar is powered by this SDK.

ASPA Path Verification Explained

Mingwei Zhang — Sun, 25 Jun 2023 00:00:00 GMT

Originally published on Hashnode.

Autonomous System Provider Authorization (ASPA) is a recently proposed solution designed to detect and mitigate route leaks and improbable AS paths. ASPA utilizes the existing RPKI infrastructure to allow storing and verifying cryptographically protected AS-level relationship information. This post focuses on the AS path verification process used by ASPA.

Note: This post is based on this IETF draft from June 2023. The draft text, including address-family handling, may differ from the final RFC.

Route leak

In overview, the definition of route leak considered in ASPA is based on RFC 7908: Problem Definition and Classification of BGP Route Leaks. The route leaks defined in this RFC is mostly based on the "valley-free" principle.

A previous Cloudflare Radar post on route-leak detection summarized four common route leak types as follows:

Routes from	To provider	To peer	To customer
From provider	Type 1	Type 3	Normal
From peer	Type 4	Type 2	Normal
From customer	Normal	Normal	Normal

In essence, route leaks defined in RFC 7908 can be summarized into one principle: routes obtained from a non-customer (peer, customer, route-server) AS can only be propagated to customers.

In ASPA, the authors inherited the definition of BGP roles (e.g. customer, provider) from RFC 9234. See the RFC9234 post for more detail on RFC9234 and the OTC BGP attribute.

Local AS Role	Remote AS Role
Provider	Customer
Customer	Provider
RS	RS-Client
RS-Client	RS
Peer	Peer

Table: allowed pairs of BGP role capabilities.

ASPA TL;DR

At a high level, the ASPA mechanism has the following properties:

ASPA stores customer-to-provider information in RPKI
- One customer AS (CAS) associated with a set of provider ASes (SPAS)
ASPA is useful for verifying BGP AS_PATH attribute content
- AS_PATH that violates the "valley-free" routing principle can be detected with ASPA objects information
ASPA is incrementally deployable, benefiting early adopters

ASPA Verification Explained

Before describing how ASPA detects route leaks, it is useful to review the shape of a typical valley-free route.

A typical long BGP route that travels from one edge network to a different edge network would look like this:

The route travels three different phases:

upward: a consecutive customer-to-provider segments
peering: a single peer-to-peer segment
downward: a consecutive provider-to-customer segments

Note that in many cases, the routes may only have one or two phases, but they must be in this order. For example, one legit route could have an upward then downward phase, without a peering phase. A route leak route could be any route with reverse order of the phases, e.g. downward then upward.

In ASPA, the phases are defined into two categories:

upstream: path received from a customer or a peer or a route server
downstream: path received from a provider or sibling

ASPA verification uses different procedures for upstream and downstream paths. Each category is described below.

Upstream paths

ASPA verification for upstream paths (received from a customer or peer or route server) is relatively simple. The key principle is that a normal upstream path should only contain customer-to-provider relationships for the consecutive AS path hops.

A path is

Valid: if all hops in the AS paths are verifiable customer-to-provider, i.e. every hop on the path registers their providers on ASPA and the providers match the next hop on the path;
Unknown: one of the hops is missing ASPA information;
Invalid: one of AS has a set of provider AS (SPAS) but the next hop is not in it.

Downstream paths

The downstream paths are considerably more complex as it may have legit paths segments that travel all three phases (up, peer, down) before reaching the receiving AS. Interested readers should refer to the original IETF draft document's section on downstream path verification for a more definitive definition. The summary below simplifies the wording to give a high-level view of the verification process.

Based on the original documentation, a downstream path is Invalid if there exist hops indexed as u and v such that

u<=v
hop(AS(u-1), AS(u))="Not Provider+"
hop(AS(v+1),AS(v))="Not Provider+"

In practical terms, this condition can be read as follows:

two hops at the location u and v and u is before v
hop u is verified to be NOT a customer-to-provider hop, i.e. peer-to-peer or provider-to-customer, i.e. a peering or downward phase
hop v is verified to be NOT a provider-to-customer hop, i.e. peer-to-peer or customer-to-provider, i.e. a peering or upward phase

In summary, the path would be ASPA Invalid, if it contains a downward or peering then later a upward or peering. The AS relationship would look like the following figure, where u and v could be the same ASN. As long as this pattern appears somewhere in the path, it violates the "valley-free" routing and thus is considered as route leak.

graph LR
  UProvider["u-1"] -->|"provider-to-customer"| U["u"]
  UPeer["u-1"] -->|"peer-to-peer"| U
  U -.-> V["v"]
  V -->|"customer-to-provider"| VProvider["v+1"]
  V -->|"peer-to-peer"| VPeer["v+1"]

The Unknown and Valid definitions are then straightforward:

Unknown: the path is NOT Invalid and there are hops that miss ASPA information
Valid: the path is NOT Invalid and NOT Unknown

Note: The ASPA document allows different verification algorithms, as long as they produce results equivalent to the specified process.

ASPA vs RFC9234

ASPA and RFC 9234 both aim to prevent or mitigate route leaks, but they use different mechanisms. The table below summarizes the main differences.

	ASPA	RFC9234
By whom	Customer AS (owner) only	BGP sessions, providers
Content security	Cryptographically signed	Secured by BGP session; OTC attributes can be forged
Where to detect/filter	Any AS	Neighbors or ASes that receive OTC attribute
Prevent forged path?	Yes	No
Deployment timing	Initial objects had already appeared; general deployment was expected in 2025 or later (Job Snijders / MANRS)	Initial deployment had appeared; general deployment remained unclear

How to view ASPA objects on RPKI today?

Although still in the draft phase, ASPA support has already been added to popular routing software like routinator and OpenBGPd. BGPKIT monocle also supports reading ASPA objects from RPKI repositories through the monocle rpki read-aspa command. The following steps show how to inspect currently published ASPA objects.

Install BGPKIT monocle

BGPKIT monocle is a Rust-based BGP data processing command-line tool. Install the Rust toolchain using the instructions at https://rustup.rs/.

Then install monocle:

cargo install monocle

Download the recent RPKI archive

The RPKI repository archive used here comes from Job Snijders' RPKIViews project. The example below downloads one archive from josephine.sobornost.net (about 260 MB).

cd /tmp
wget http://josephine.sobornost.net/josephine.sobornost.net/rpkidata/2023/06/25/rpki-20230625T173535Z.tgz
tar xzf rpki-20230625T173535Z.tgz

Find and view ASPA objects

The ASPA objects are stored as files with the suffix .asa in the RPKI repository. You can use find command to search across the untarred repo file from the previous step:

➜  rpki-20230625T173535Z find . -name "*.asa"
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148808.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148809.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148810.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148804.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148805.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148807.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148806.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148802.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148803.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148801.asa
./data/rpki.sub.apnic.net/repository/A914BC7A0000/0/AS148800.asa
./data/rpki.cc/repo/MythicalKitten/9/AS60310.asa
./data/rpki.cc/repo/MythicalKitten/0/AS203635.asa
...

Use monocle rpki read-aspa to inspect an ASPA object. For example:

➜  rpki-20230625T173535Z monocle rpki read-aspa ./data/rpki.cc/repo/MythicalKitten/9/AS60310.asa

| asn   | afi_limit | allowed_upstream |
|-------|-----------|------------------|
| 60310 | none      | 924              |
|       |           | 6939             |
|       |           | 48581            |
|       |           | 50224            |
|       |           | 52210            |
|       |           | 204857           |
➜  rpki-20230625T173535Z monocle rpki read-aspa ./data/rpki.co/repo/Mlgt/0/AS204508.asa

| asn    | afi_limit | allowed_upstream |
|--------|-----------|------------------|
| 204508 | none      | 6939             |
|        |           | 7721             |
|        |           | 29632            |
|        |           | 34872            |
|        |           | 34927            |
|        |           | 38230            |
|        |           | 41051            |
|        |           | 50058            |
|        |           | 60326            |
|        |           | 200105           |
|        |           | 208753           |
|        |           | 210633           |
|        |           | 211411           |
|        |           | 212271           |
|        |           | 212895           |

BGPKIT is experimenting with library support for ASPA to support measurement and research use cases. Feedback and feature requests are welcome at data@bgpkit.com.

RFCs for IP Address Usage in Network Documentation and Examples

Mingwei Zhang — Wed, 22 Mar 2023 00:00:00 GMT

Originally published on Hashnode.

Network documentation and topology examples should use address ranges reserved for documentation, rather than globally routed addresses such as 1.1.1.1 or 1.2.3.4.

Two RFCs define standardized IP address ranges reserved specifically for documentation and examples: RFC3849 and RFC5737. RFC3849 outlines the use of the IPv6 address prefix 2001:DB8::/32, while RFC5737 outlines the use of several IPv4 address blocks, including 192.0.2.0/24, 198.51.100.0/24, and 203.0.113.0/24. These addresses are not intended for use in a production network but instead serve as placeholders to ensure accurate and clear documentation.

RFC3849: IPv6 Address Prefix Reserved for Documentation

RFC3849 specifies the IPv6 address prefix 2001:DB8::/32 for use in documentation and example code, not in a production network. For example, a network administrator might use the IPv6 address 2001:DB8::1 in their documentation or example code to represent a specific device or network location. This address would not be used for actual network traffic, but rather as a reference point for documentation purposes.

RFC5737: IPv4 Address Blocks Reserved for Documentation

RFC5737 outlines the use of several IPv4 address blocks for use in documentation and example code. These address blocks are similar to the IPv4 addresses outlined in RFC5735 (which contains more special blocks such as 0.0.0.0/8), but are specifically for use in documentation and examples.

The address blocks outlined in RFC5737 include:

192.0.2.0/24
198.51.100.0/24
203.0.113.0/24

Similarly, these address blocks are not intended for use in a production network, but rather as placeholders for documentation and examples.

Proper Use of Reserved IP Addresses

Using reserved IP addresses for documentation and examples is essential for a number of reasons. First, it ensures that these addresses are not used for general Internet routing, which can help to prevent routing issues and other network problems. Second, it helps to prevent confusion and technical issues that can arise when non-reserved addresses are used in documentation or example code.

In recent years, there have been cases where non-reserved IP addresses have been used in documentation and examples, leading to confusion and technical issues. In a recent YouTube video, Cloudflare's Tom Strickx highlighted this problem and encouraged operators to avoid addresses such as 1.1.1.1 and 1.2.3.4, using the ranges specified in RFC5735, RFC5737, and RFC3849 instead.

https://www.youtube.com/watch?v=SIb_wxO1wt0

Using reserved IP addresses in documentation and examples avoids unintended traffic to real networks and improves clarity for readers. RFC3849 and RFC5737 provide the standard address ranges for these use cases, and network operators and researchers should use them consistently in examples and documentation.

RFC9234 Observed in the Wild

Mingwei Zhang — Thu, 16 Mar 2023 00:00:00 GMT

Originally published on Hashnode.

BGP Route Leaks

Route leaks occur when BGP prefixes are propagated in a way that goes against the expected topology relationships of BGP. For example, this can happen when a route learned from one transit provider is announced to another transit provider or a lateral peer (peer-peer-peer), or when a route learned from one lateral peer is announced to another lateral peer or a transit provider (see RFC7908). These leaks often result from misconfiguration or the absence of BGP route filtering, or from inadequate coordination between autonomous systems (ASes).

Cloudflare Radar includes a public route-leak detection system. The system detects potential route leaks by first inferring inter-AS relationships on a per-prefix basis, then examining each announced AS path for valley-free violations. Detection is useful for visibility, but prevention is the stronger operational goal.

RFC9234

RFC9234 documents an active route-leak prevention approach where it defines new BGP capacities (BGP Roles) exchanged during the eBGP session open time and allows the BGP routers to understand AS relationships between local and remote ASes, and thus prevent the propagation of route leaks. RFC9234 also defines a new BGP attributes type, only-to-customer, which tells the receiving BGP routers whether some routes should never be announced to another provider.

When a pair of eBGP routers both implement RFC9234, they will first confirm the BGP role of each other.

With roles defined, it handles the only-to-customer attributes as follows.

In essence, RFC9234 uses BGP roles and OTC attributes to make sure routes received from a provider or a peer can only be propagated to customers.

Only-to-customer Attribute in the Wild

RFC9234 was published in May 2022. Public discussion of RFC9234 deployment has been limited, and public measurements of deployment on the Internet remain relatively scarce.

To measure RFC9234 deployment, RFC9234 support was added to BGPKIT Parser. The repository also includes example code demonstrating how to parse a RIB file and identify messages that contain Only-to-Customer attributes.

The following output comes from parsing a single route-views2 RIB dump (file link).

https://gist.github.com/digizeph/756bd6b4d6627bf1e2a0a4f87a1c8290

This output shows that at least four ASes (AS6939, AS15562, AS20555, AS212068) were associated with RFC9234 OTC attributes in messages that propagated to the route collector and were preserved in MRT files.

The code that generates this result iterates over parsed BGP messages and checks whether the only_to_customer attribute is present.

use bgpkit_parser::BgpkitParser;

fn main() {
    for elem in BgpkitParser::new(
        "http://archive.routeviews.org/bgpdata/2023.03/RIBS/rib.20230316.0200.bz2",
    )
    .unwrap()
    {
        if let Some(otc) = elem.only_to_customer {
            println!(
                "OTC found: {} for path {}\n{}\n",
                &otc,
                &elem.as_path.as_ref().unwrap(),
                &elem
            );
        }
    }
}

To reproduce this example, install the Rust toolchain and run:

git clone https://github.com/bgpkit/bgpkit-parser
cd bgpkit-parser
cargo run --release --example only-to-customer

Update, Mar. 16, 2023: Job Snijders provided additional context on the source of these OTC attributes: YYCIX.

https://twitter.com/JobSnijders/status/1636291640519282688

2022 Year in Review

Mingwei Zhang — Tue, 31 Jan 2023 00:00:00 GMT

In 2022, BGPKIT as an open-source organization made significant progresses. As the founder, I am grateful for all the opportunities would like to take this time to appreciate all the milestones we achieved. In this post, I will go through some notable changes we made in 2022, and take a look at what we are excited about for the year of 2023.

BGPKIT Parser

There were a number of major features added to BGPKIT Parser in 2022:

v0.7.0 added support for filtering messages by many fields, and allowing reading from uncompressed files.
v0.7.1 added better examples of parallel MRT files processing with rayon.
v0.7.2 added filtering by multiple peer_ips.
v0.8.0 includes many internal refactoring and brought in the new oneio library to improve developer experience.

BGPKIT Broker

In 2022, we have revised the BGPKIT Broker backend to support crawling for estimated file sizes in addition to other fields like timestamps and URLs. This allows us to keep track of MRT file size changes and help users to pick suitable collectors to use, especially at the age where RIB dumps from a single collector could reach over 1 GB in size.

Figure of RIB sizes over time. Blue line is for rrc00, while yellow line is for route-views2.

We also made major revisions to our Rust SDK for more features like .latest() to get the lastest MRT files from each collector.

https://github.com/bgpkit/bgpkit-broker/releases/tag/v0.5.0

Plus, if you would like to deploy a broker instance yourself, we also have made some significant efforts to improve our documentation on self-hosting guide.

https://github.com/bgpkit/bgpkit-broker-backend/blob/main/deployment/README.md

Python Bindings

In addition to core Rust code base for parser and broker, we have also added support for Python bindings for various Rust SDKs. Users can easily parse MRT files directly using pybgpkit Python library. It is also proven to be usable on cloud-based Jupyter notebooks like Google Colab (examples).

https://github.com/bgpkit/pybgpkit

monocle

To ties things together, we have also developed our first investigative tool, monocle , to help users to quick find relevant BGP announcements with a suite of easy-to-use utilities. Users with Rust toolchain installed can run cargo install monocle to install the tool.

https://github.com/bgpkit/monocle

Users can use the following subcommands

parse: parse single MRT files, remotely or locally
search: find and filter BGP messages accross multiple public collectors
time: convert between local time string and Unix timestamps
whois: find out AS names, ASN, registration countries, and organizations.

Web/API and Infrastructure

In 2022, we started to experiment new cloud-based infrastructure, especially cloud-based databases for better API stability and developer experiences. We ended up selecting Supabase as our PostgreSQL production host and a self-hosted instance for backup. We are happy with the performance and cost provided by Supabase and more exicted about the potential capability it brings such as user authentication, cloud storage, local dev schema changes, etc.

Based on the new infrastructure, we have started to test a new integrated API system (still in alpha). This allows us to put all of our data access and processing end-points into one location. Based on the new API, we have also developed a newer version of the BGPKIT Broker statistics page: https://alpha.stats.bgpkit.com/

New Datasets

Apart from provide SDKs and data APIs, we have also started providing free access to historical archives of some data that we find interesting. We blogged about one dataset previous, the peer-stats dataset:

/blog/introducing-peer-stats-dataset

Here is a list of our currently available datasets at https://data.bgpkit.com/

[peer-stats](https://data.bgpkit.com/peer-stats/): route collector peers statistics (IP, ASN, v4/v6 prefixes counts)
[as2rel](https://data.bgpkit.com/as2rel/): AS-level relationship, using all available collectors
[pfx2as](https://data.bgpkit.com/pfx2as/): prefix-to-AS mapping, using all available collectors
[ihr-hegemony](https://data.bgpkit.com/ihr/hegemony/ipv4/global/): mirror of IIJ-IHR's global hegemony score dataset (big shout out to Romain and Internet Health Report for producing this data)

All the above datasets are free to use for research or commercial usages, and here is the acceptable usage agreement.

More Public Repositories

There are more open code repositories set available, some experimental and some for data analysis. You can check out the full list here:

https://github.com/orgs/bgpkit/repositories

Founder's Notes

In later 2022, I have joined Cloudflare to continue working on routing security for public benefits. During the first few months, we have built and shipped our new route-leak detection system under the public Radar platform. BGPKIT suite is now used in production for the BGP data anaysis pipeline at Cloudflare. While working full-time now, I am still committed to maintain the software suite and bringing new features to BGPKIT. For example, this year, I ported back our Kafka support used in Cloudflare to BGPKIT Broker backend. Folks at Cloudflare are doing great things in the open-source realm, and BGPKIT software suite will continue to become more useful to BGP enthuesastics and remain completely open-source.

Quote from Cloudflare's route-leak detection system blog.

Looking at 2023, here are a few things that I am really excited to work on for BGPKIT suite:

continue improve the infrastructure of the system and adding new data processing pipelines
continue improve the parser's performance and reliability
adding new RFC supports to parser (e.g. RFC9234 for route-leak prevention)
productionizing the new API and stats website
write more examples and documentations (don't we all love that)

And yeah, we will continue to be open-source first!

If you like what we do here, please consider subscribe to our blog. For all code repositories, check out our GitHub page.

https://github.com/bgpkit

Introducing Peer-Stats Dataset

Mingwei Zhang — Mon, 16 May 2022 00:00:00 GMT

Public BGP data collector projects like RouteViews and RIPE RIS provide valuable research and operational information for understanding BGP and detecting Internet routing anomalies.

There are many BGP routers involved in BGP collection projects.

A project includes many "collectors," and each serves as a collection of messages from several active BGP peers from different networks. Some bigger collectors collect BGP data from more than one hundred BGP routers. For example, RIPE RIS rrc00 has 112 active BGP peers at writing. (To learn about the complete list of BGP peers from all collectors, try theexperimental toolswe developed.)

Sometimes, too many BGP peers may become problematic.

Not all peers present the same amount of data. Some peers are so-called "full-feed" peers, which are the ones that provide the full routing tables to the collector. In a routing table dump file from the collectors, we can observe the full table of these peers. Some peers, however, only provide a limited number of routing entries to the collectors, not representing the whole routing status from these peers. In a project that tries to rebuild full routing tables, e.g., some BGP hijack detection or anomaly detectors, people prefer to use the full-feed peers as their data source.

At times, we are only interested in data from certain peers. For example, when studying the routing data from a particular network, if the network connects to BGP data collectors, we can directly pull data from the collectors' data. However, it can be troublesome to learn about what collectors have data from certain peers. RIPE RIS provides a nice API for querying such info, but we couldn't find one for RouteViews.

Historical data for such information is also missing. Unfortunately, for the researchers who want to study the evolution of the data collectors, even RIPE RIS's peers API could not help with that.

Introducing BGPKIT Peer-Stats Dataset

Peer-Stats dataset is a publicly available, free-to-use dataset that aims to provide daily collector peer information for all RouteViews and RIPE RIS collectors for ten years.

https://data.bgpkit.com/peer-stats/

The data includes the following fields for each peer of a BGP collector:

asn: Autonomous System Number of the collector peer
ip: the IP address of the collector peer
num_v4_pfxs: the number of IPv4 prefixes propagated from the collector peer
num_v6_pfxs: the number of IPv6 prefixes propagated from the collector peer
num_connected_asns: the number of connected (immediate next hop) ASes from the collector peer

The dataset is organized by the following structure.

- collector
	- year
		- month
			- data files

Screenshots of the dataset file listing site.

Each data file is in JSON format (see the section below) and compressed with bzip2. Users can easily use tools like bzcat and jq to view the data files. For example, you can run the following command to quickly view any of the peer-stats data for the collector rrc00 on 2022-05-01.

curl "https://data.bgpkit.com/peer-stats/rrc00/2022/05/rrc00-2022-05-01-1651363200.bz2" --silent | bzcat | jq

{
  "collector": "rrc00",
  "peers": {
    "102.67.56.1": {
      "asn": 328474,
      "ip": "102.67.56.1",
      "num_connected_asns": 330,
      "num_v4_pfxs": 919443,
      "num_v6_pfxs": 0
    },
    "103.102.5.1": {
      "asn": 131477,
      "ip": "103.102.5.1",
      "num_connected_asns": 184,
      "num_v4_pfxs": 895482,
      "num_v6_pfxs": 0
    },
...

Because all the data files are generated against the midnight UTC RIB dump of the day, you can also easily construct a URL to a data file for any particular date using the following template.

https://data.bgpkit.com/peer-stats/{COLLECTOR}/{YEAR}/{MONTH}/{COLLECTOR}-{YEAR}-{MONTH}-{DAY}-{MIDNIGHT_TIMESTAMP}.bz2

Open-source

We also open-sourced the data collection command-line tool source code on GitHub. Feel free to check it out and run it on your infrastructure if needed.

https://github.com/bgpkit/peer-stats

Credits and Sponsorship

The original idea for this work came from our extensive discussion with Romain Fontugne (follow him on Twitter at @romain_fontugne) from IIJ. This work is made possible by IIJ's generous sponsorship.

Please consider sponsoring us on GitHub if you find our work valuable and would like to see more open-source code and datasets on BGP.

https://github.com/sponsors/bgpkit

KhersonTelecom Outage and Connectivity Change

Mingwei Zhang — Tue, 03 May 2022 00:00:00 GMT

Internet service in Russian-occupied Kherson, Ukraine was disabled at 16:12 UTC (6:12pm local) on Saturday, 30 April. #UkraineRussiaWar

Khersontelecom service was restored ~24hrs later via Russian transit from nearby Crimea. pic.twitter.com/uN31jLrzEc

— Doug Madory (@DougMadory) May 2, 2022

The AS47598 experienced an outage shortly after 2022-04-30T16:10:00 and then resumed connectivity 2022-05-01T16:15:00 (both UTC time). After the outage, the AS47598 is then connected via a different upstream provider, AS201776.

The upstream provide change can also be seen on IIJ's Internet Health Report.

Prefixes

AS47598 announces only one prefix 91.206.110.0/23 (data from Hurricane Electric). Most of the BGP announcements are for this prefix. However, after the provider change happened, there were also a IPv6 prefix announcements for this V6 prefix as well 5bce:6e00::/23. It is possible that this prefix is a V4-translated prefix propagated to a V6 collector peer, but we do not know for sure.

We can also confirm the outage of the prefix with RIPEstat’s Routing History data widget. (uncheck the No low visibility box to reveal the outage).

BGP Messages

We can visualize the overall BGP announcements volume with Cloudflare Radar’s AS-level page.

The following are the UTC timestamps for the corresponding BGP message spikes.

2022-04-30T16:10:00: announcements with old provider 12883 in the paths.
2022-05-01T16:00:00: announcements with the new provider 201776 in the paths
2022-05-03T10:45:00: similar announcements with a new provider in the paths.

During the first gap time (2022-04-30T16:15:00 to 2022-05-01T16:00:00), there were 0 BGP updates for the prefix or from the ASN.

The old provider paths look like this where AS12883 is the next hop for AS47598. See the full list of messages from rrc00 here: https://gist.github.com/digizeph/c58b77f755d7fec8a7969807fb17d5ba.

207564 56655 3257 12883 47598

The new provider paths look like this where AS12389 and AS201776 are the next hops for AS47598. See the full list of messages from rrc00 here: https://gist.github.com/digizeph/896a4a7e4de23082b496b92ab5bdab5b

207564 28824 28824 1299 12389 201776 47598

BGP Data Tooling

The analysis is done using a privately hosted open-source BGPKIT parser web API. You can host it on your infrastructure, and the source code is freely available at https://github.com/bgpkit/pybgpkit-api. Comments and feedback are welcome!

Update on 2022-05-04T09:20:00 Pacific

#Kherson — Internet connectivity is returning to the occupied city in South of #Ukraine, after an outage since Saturday. @Cloudflare data shows growth in requests since 04:15 UTC, and telecom connection was confirmed by the Ukrainian Vice PM @FedorovMykhailo. pic.twitter.com/JxT3kcM234

— Cloudflare Radar (@CloudflareRadar) May 4, 2022

The upstreams for AS47598 has reverted back to the original ASes, and the traffic has started coming back to normal.

Parallel MRT Files Parsing with BGPKIT

Mingwei Zhang — Wed, 16 Mar 2022 00:00:00 GMT

In this post, we will talk about how to implement a Rust workflow that can process a large number of BGP data files as fast as we can. We will use BGPKIT Parser and Broker for data collection and parsing, and Rayon crate for parallelization of the code.

Task Overview

Before we begin to talk about the code design, we first need to introduce the data we are dealing with. We want to process the BGP data collected by various collectors, saved in compressed binary MRT format, and archived to files with a fixed interval. In this post, we are using RouteViews archive data as an example. The average data file size ranges from 2MB to 10MB by different collectors (AMSIX collector for example has pretty large files).

For processing, we can use the simplest task possible to do: sum the number of MRT records in all the files. We want to download and process all the updates files for one hour from all the collectors in RouteViews project. Here is the estimated amount of data we are dealing with:

35 collectors
5-minute interval — 12 files per collector
420 total number of files to download and process
840MB to 4.2GB total download size (it’s somewhere in between)

Ok! Now that we know what we want to do and have a sense of the estimated workload of the overall task, let’s coding!

Photo by Glenn Carstens-Peters on Unsplash

1. Sequential Parsing

Our first attempt to achieve the goal is to design and implement a naive sequential workflow as described below

find all BGP updates files within the hour of interest
iterate through each file, parse the MRT data and count the number of records
sum all record counts and print out the result

For this sequential workflow, we will need to pull in two dependencies into Cargo.toml:

[dependencies]
bgpkit-parser = "0.7.2"
bgpkit-broker = "0.3.2"

The bgpkit-broker handles looking for updates files within the hour, while bgpkit-parser handles parsing each individual file.

Finding files

BGPKIT Broker indexes all available BGP MRT data archive files from both RouteViews and RIPE RIS in close-to-real-time. For each data file, it saves the following information:

project: route-views or riperis
collector: the collector ID, e.g. rrc00 or route-views2
url: the URL to the corresponding MRT file
timestamp: the UNIX time of the start time of the MRT data file.

With all this information indexed, we can then query the backend and retrieve files information as we want. BGPKIT Broker provides both RESTful API, Rust API, as well as a Python API. Here we use the Rust API to pull in the information we need:

let broker = BgpkitBroker::new_with_params(
    "https://api.broker.bgpkit.com/v1",
    QueryParams {
        start_ts: Some(1640995200),
        end_ts: Some(1640998799),
        project: Some("route-views".to_string()),
        data_type: Some("update".to_string()),
        ..Default::default()
    });

for item in &broker {
    println!("processing {:?}...", &item);
}

The above block queries the broker and prints out all information of the retrieved files' metadata. The BgpkitBroker::new_with_params call accepts two parameters, one for the endpoint of the broker instance, and the other specifies the filtering criteria. In this example, we search for all BGP updates files from RouteViews with timestamps between 2022-01-01T00:00:00 and 2022-01-01T00:59:59 UTC. It prints out the output as the following:

processing BrokerItem { collector_id: "route-views.telxatl", timestamp: 1640997000, data_type: "update", url: "https://archive.routeviews.org/route-views.telxatl/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }...
processing BrokerItem { collector_id: "route-views.uaeix", timestamp: 1640997000, data_type: "update", url: "https://archive.routeviews.org/route-views.uaeix/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }...
processing BrokerItem { collector_id: "route-views.wide", timestamp: 1640997000, data_type: "update", url: "https://archive.routeviews.org/route-views.wide/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }...
processing BrokerItem { collector_id: "route-views2", timestamp: 1640997900, data_type: "update", url: "https://archive.routeviews.org/bgpdata/2022.01/UPDATES/updates.20220101.0045.bz2" }...

Parse each MRT file

Previously, in the for loop, we only print out the retrieved meta information of the MRT files. Now let's add the actual parsing of the files into the loop. The code is very simple, as designed:

let mut sum: usize = 0;
for item in &broker {
    println!("processing {}...", &item.url);
    let parser = BgpkitParser::new(&item.url).unwrap();
    let count = parser.into_record_iter().count();
    sum += count;
}

We first define a mutable variable sum outside the loop. Then for each file, we create a new Parser instance by BgpkitParser::_new_(&item.url). Here, as our goal is to count the number of records, we call the parser's .into_record_iter() function to create a iterator over the records of the file, and then .count() to get the count of the records. Lastly, we add the count to the overall sum variable.

Run and timing

For testing, I use a fairly powerful VM on a host with AMD 3950x CPU (32 threads), then build the release build and time the release run. The runtime includes downloading the MRT files to my machine with 1Gpbs down link in Southern California.

cargo build --release
time cargo run --release --bin sequential

It ended up taking about 1 minute and 23 seconds to sequentially parse 144 MRT files from RouteViews for all available ones within the first hour of 2022 (UTC).

total number of records for 144 files is 10554212

real    1m23.081s
user    0m39.535s
sys     0m1.006s

2. Parallel Parsing

Since the parsing of each file is completely independent of each other, we can parse the files in parallel and then sum up the count for each thread at the end. In Rust with Rayon, this conversion is very simple.

Let's first add the dependency of Rayon first:

[dependencies]
bgpkit-parser = "0.7.2"
bgpkit-broker = "0.3.2"
rayon = "1.5.1"

Then we change the broker code one tiny bit to collect all meta information for files into a vector first.

let items = broker.into_iter().collect::<Vec<BrokerItem>>();

This would enable us to fully utilize rayon's great syntax sugar to turn our sequential code into a parallel one.

let sum: usize = items.par_iter().map(|item| {
    println!("processing {}...", &item.url);
    let parser = BgpkitParser::new(&item.url).unwrap();
    let count = parser.into_record_iter().count();
    count
}).sum();

The key difference here is the calling of .par_iter(). It turns a sequential iterator into a parallel iterator, and by default going to utilize all available cores on the host machine for scheduling. Then we call .map() to define the parsing steps for each file, and then call .sum() at the end to add all results up.

The final result is approximately 10x faster than the sequential version, and it took only 8 seconds to parse all MRT files and get the record counts.

total number of records is 10554212

real    0m8.086s
user    0m42.569s
sys     0m1.068s

The full code for this example is as follows:

The source code of the two examples is available on GitHub. Feel free to poke around and tweak it as you wish.

%[https://gist.github.com/digizeph/c23ba39968c6cb4e1ad323520540010f#file-parallel-parsing-rs]

https://github.com/bgpkit/bgpkit-tutorials/tree/main/parallel-parsing

Real-time RIS Live Data with BGPKIT Parser

Mingwei Zhang — Fri, 12 Nov 2021 00:00:00 GMT

In terms of real-time BGP data processing, RIPE NCC provides a great data source: Routing Information Service Live (RIS Live).

To begin with, here is what RIS Live by the creators:

RIS Live is a feed that offers BGP messages in real-time. It collects information from the RIS Route Collectors (RRCs) and uses a WebSocket JSON API to monitor and detect routing events around the world. A non-interactive full stream (“firehose”) is also available.

In essence, RIS Live provides:

a WebSocket interface to stream BGP messages in real-time
ability to subscribe to “sub-streams” with custom filtering messages
JSON-encoded BGP messages as the stream payload
“firehose” HTTPS stream interface as well, without needing to work with websocket.

In this post, we will discuss how to use the RIS Live stream in practice.

RIS Live Message Format

RIS Live has client messages and server messages.

The client messages is used to setup or dismantle “subscriptions”, which essentially tell the server what kind of BGP messages a client would like to receive, and allow the server to send only the interested messages to the client.

A server acknowledges the requests from the client and afterwards start streaming requested data back to the client. At a high-level, a server sends either ris_message or ris_error messages. The ris_message is the main payload that we are interested in, while the ris_error message provides debugging messages for the scenarios where stream or subscription fails.

Subscribe to a WebSocket Stream

RIS Live provides great flexibility for the clients to specify/narrowdown the interested messages, allowing both the server and client process less messages during a streaming session.

host: only messages collected from a particular RRC (e.g. rrc21)
type: only messages of a given type, e.g. UPDATE , OPEN
require : only messages containing a given key, e.g. withdrawals will return only message that contains any withdrawn prefixes
peer: messages from a particular BGP peer
path : ASN or pattern to match the AS Path attribute in BGP update messages
prefix: only messages containing information for a given prefix
moreSpecific and lessSpecific: only messages that are the subprefix or super-prefix of the specified prefix
includeRaw: whether to include the Base64-encoded RAW BGP messages

As an example, let’s take a look at the following message from the official manual:

{
  "host": "rrc01",
  "type": "UPDATE",
  "require": "announcements",
  "path": "64496,64497$"
}

Example subscription message composer on RIS Live official site.

As an example, let’s take a look at the following message from the official manual:

{
  "host": "rrc01",
  "type": "UPDATE",
  "require": "announcements",
  "path": "64496,64497$"
}

collected by rrc01
BGP UPDATE messages
have at least one announced prefix
the last two hops of the AS Path is 66496 and 64497 (the origin)

The ris_message consists of “common header” fields and “data” fields (although they’re on the same level).

The “common header” fields are present for all types of sub-type messages, including timestamp, peer, peer_asn, id, host, type . The rest of the fields are the data fields that are dependent on the type of the messages. For most people, the UPDATE message is what they need. The following JSON block is an example message pulled directly from the demo site.

Example JSON formatted RIS message:

This example shows a BGP announcement of AS132354 originating two prefixes 103.249.208.0/23 and 103.14.184.0/24 , with the next hop to be 37.49.237.228. At this point, the information we see here is pretty similar to what we can see from other BGP MRT reader’s output (e.g. from bgpdump or bgpreader), just in JSON format.

WebSocket or Firehose?

Provided that RIS Live provides both WebSocket and HTTP Firehose, one would naturally wonder which one is the right choice for their application. Here we have a brief comparison between the two in the context of RIS Live.

WebSocket

Good:

easy to customize stream by composing a simple JSON subscribe message
work with various toolings in languages like Python and JavaScript

Bad:

requires extra library dependencies to work with WebSocket
need to write somewhat lengthy to get started (comparing to firehose)

Firehose

Good:

easy to consume by simply calling GET request on the URL
simple single-liner commandline program can start the stream (e.g. a simple curl call), no need complex script

Bad:

customizing stream is doable with XRIS-SUBSCRIBE HTTP request header, but feels clunky and limited
in my personal tests, the stream get disconnected often due to the stream cannot keep up with the data producer. this did not happen with websocket tests.

Summary

If your application could afford additional dependencies or writing extra scripts, WebSocket is the better choice. RIS Live official manual also makes implication that the WebSocket format is the current formally-supported streaming method.

RIS Live Coding Example with BGPKIT Parser

Now that we have a basic idea of what is RIS Live and the basic message format, we can get started working on some code that will actually use RIS Live to do something useful.

In the following example, we will build a short monitoring service that alerts us when Facebook operators announces their DNS IP prefix (see what happened before here). We are going to build the service in Rust with BGPKIT Parser , WebSocket library Tungstenite.

First, lets collect some basic information about what we are going to monitor here:

Facebook’s autonomous system number is 32934. So we will watch for all messages that was originated from AS32934.
Facebook’s DNS server IP prefixes involved in the previous incidence are 129.134.30.0/23 and 185.89.218.0/23 . So we want to carefully watch these two prefixes in our monitoring system.
We want to use one of the RIPE RIS collectors’ data for monitoring, rrc21 is a good choice since it’s being used by RIS Live’s demonstration. You can easily extend this service by tweaking the subscription message later.

OK, we are good to go. Let’s do it!

Setting up the stream

We picked the Tungstenite library as our WebSocket library of choice, partly because it has a very straightforward API design.

Let’s first connect to the websocket server by calling connect function given a websocket URL. One thing to notice is that the URL protocol section here is ws as opposed to the wss mentioned in the RIS Live documentation. For some reason, Tungstenite does not work with wss protocol (with SSL).

use tungstenite::{connect, Message}; 
const RIS_LIVE_URL: &str = "ws://ris-live.ripe.net/v1/ws/?client=rust-bgpkit-parser";
let (mut socket, _response) =
    connect(Url::parse(RIS_LIVE_URL).unwrap())
    .expect("Can't connect to RIS Live websocket server");

Now, with a socket ready, we will first send a subscription message to let server know that we want some messages and we are ready to receive.

let msg = json!({"type": "ris_subscribe", "data": {"host": "rrc21"}}).to_string();
socket.write_message(Message::Text(msg)).unwrap();

Here we composed a simple subscription message that limits the stream to have messages only from rrc21 collector.

Parsing JSON messages

At this point, we have a WebSocket connection to RIS Live server, and have sent out a subscription message to the server. The server should be sending back messages anytime now, and we are ready to consume the stream.

We would like to code the following behavior:

continuously reading the websocket messages;
parse JSON string into internal BGP structs;
check each message if it contains origins (withdraw-only messages does not contain AS paths, and thus no origins either;
if the origin AS is AS32934, and the announced prefix is 129.134.30.0/23 or 185.89.218.0/23 , then we print out the message to output.

loop {
    let msg = socket.read_message().expect("Error reading message").to_string();
    if letOk(elems) = parse_ris_live_message(msg.as_str()) {
        for elem in elems {
            if letSome(origins) = elem.origin_asns.as_ref() {
                if origins.contains(&32934) &&
                    ( elem.prefix.to_string() ==  "129.134.30.0/23".to_string() ||
                        elem.prefix.to_string() ==  "185.89.218.0/23".to_string() )
                {
                    println!("{}", elem);
                }
            }
        }
    }
}

The full example code can be found here:

%[https://gist.github.com/digizeph/fcac3027555c0b744ea0b3a11197b694#file-bgpkit-parser-kafka-bmp-example-rs]

Building More with BGPKIT Tools

As introduced in our previous blog post, we added support of real-time BMP stream to BGPKIT Parser as well. Combining with RIPE RIS Live, and RouteViews BMP stream, we can build a powerful real-time BGP monitoring service directly within BGPKIT Parser. We also offer indexing and processing of historical BGP data as well with BGPKIT Broker.

Our goal at BGPKIT is to design, develop, and deploy the most developer-friendly BGP data processing toolkit. To learn more about our offerings, please check out our website and official Twitter account.

Real-time BMP with BGPKIT Parser

Mingwei Zhang — Wed, 10 Nov 2021 00:00:00 GMT

Real-time BGP data processing is very critical on building monitoring services that can detect BGP issues quickly with minimum delay and react to anomalies quickly and mitigate potential issues.

We are creating a new series of posts describing how we design our software to work with real-time BGP data streams. As an opening, we will describe how we handle data streams with BMP protocol and OpenBMP messages.

BMP

The BGP Monitoring Protocol (BMP) is a protocol that allows monitoring of BGP devices.

The RFC7854 describes the purpose of BMP as:

Many researchers and network operators wish to have access to the contents of routers’ BGP Routing Information Bases (RIBs) as well as a view of protocol updates the router is receiving. This monitoring task cannot be realized by standard protocol mechanisms. Prior to the introduction of BMP, this data could only be obtained through screen scraping.

BMP provides access to the Adj-RIB-In of a peer on an ongoing basis and a periodic dump of certain statistics the monitoring station can use for further analysis. From a high level, BMP can be thought of as the result of multiplexing together the messages received on the various monitored BGP sessions.

There are multiple types of BMP messages, each serving different purposes.

Peer up and down notification: notification about the status of peering sessions to a monitored router;
Initiation message: inform the monitoring station of the routers vendor, software version, and so on;
Termination message: provides information on why a monitored router is terminating a session;
Route monitoring:initial synchronization of the routing table;
Route mirroring: verbatim duplication of messages as received.

For real-time BGP data processing, we are specifically interested in the route monitoring and route-mirroring messages, as we provide the routing information encoded as actual BGP messages.

OpenBMP

OpenBMP is a software implementation of the BMP protocol. It is an open-source project created by Cisco and currently maintained by nice folks from CAIDA/UCSD and RouteViews. It is implemented in C++, can be used with any compliant BMP sender (e.g., router).

Architecture graph of OpenBMP

OpenBMP provides multiple formats for outputting the BMP messages collected from the connected routers, one of which is the raw_bmp format, which is a thin wrapper of the raw BMP messages. The raw_bmp format provides the best performance and allows use to handle the BMP messages directly without having to write a different parser for the plaintext messages.

RouteViews currently provides a OpenBMP Kafka stream that streams BMP messages from their collectors.

BGPKIT Parser with BMP/OpenBMP Support

We develop BGPKIT Parser to provide a one-stop solution for handling all parsing tasks regarding BGP data. Supporting real-time data like BMP is a very important milestone for us.

We have recently developed the full support for BMP messages, and partial support for OpenBMP messages (for raw_bmp type only). This enables us to start working with real-time BMP streams like RouteViews’ Kafka stream.

Below is an example code that takes RouteViews’ Kafka OpenBMP stream and parse the messages into internal data structures:

let mut reader = Cursor::new(Vec::from(kafka_payload));
let header = parse_openbmp_header(&mut reader).unwrap();
if let Ok(msg) = parse_bmp_msg(&mut reader) {
    info!("Parsing OK: {:?}", msg.common_header.msg_type);
    match msg.message_body {
        MessageBody::RouteMonitoring(m) => {
            dbg!(m.bgp_update);
        }
        _ => {}
    }
}

Here is a break down of what it does:

it first creates a bytes reader from the raw Kafka message payload;
then parse OpenBMP message header, which contains some basic information about the BMP session;
then it calls the parse_bmp_msg function to parse the embedded raw BMP messages and print out the BGP update messages if the parsing is successful.

Here is a full code example:

%[https://gist.github.com/digizeph/fcac3027555c0b744ea0b3a11197b694#file-bgpkit-parser-kafka-bmp-example-rs]

We have published the SDK on crates.io and GitHub. Feel free the check out the example code at examples/routeviews-kafka.rs if you are interested.

Introducing BGPKIT Parser

Mingwei Zhang — Mon, 01 Nov 2021 00:00:00 GMT

BGPKIT Parser is an open-source Rust-based MRT/BGP data parser that takes a MRT formatted binary file and turns it into BGP messages. It is one of the most important building block software that enables BGP data processing and analysis tasks.

Design and Features

As mentioned in our previous post introducing BGPKIT Broker, the most used BGP data collection projects, RouteViews and RIPE RIS, both publish their collected BGP data in MRT format on their data platform. The BGPKIT Parser is designed to handle parsing tasks for these data sources.

We design our parser to strictly follow the industry standard, e.g. RFC4271 and RFC6396. BGPKIT Parser is also designed with the following goals:

performant: comparable to C-based implementations like bgpdump or bgpreader.
actively maintained: we consistently introduce feature updates and bug fixes, and support most of the relevant BGP RFCs.
ergonomic API: a three-line for loop can already get you started.
battery-included: ready to handle remote or local, bzip2 or gz data files out of the box.
open-source: we want people to use our parser freely and we can continue develop and improve it based on community feedbacks.

To demonstrate how easy it is to get started using BGPKIT Parser, check out the example below where we print out all BGP messages from a remote MRT file on RouteViews:

Example of reading a remote MRT file from RouteViews and print out BGP messages. Code available at https://gist.github.com/digizeph/9977371653f39a459ff3ae507dc3636c

There are a number of things happens when we call for elem in BgpkitParser::new(url) :

it creates a new BgpkitParser struct instance with the provided URL to the data file;
it tries to retrieve the content of the remote file and download the raw compressed bytes into memory;
it determines the compression type by file suffix and calls corresponding decompression library to create a buffered reader;
it then creates an iterator (used by the for loop) that continuous return new parsed items until it reaches the end of the data stream.

We determined that it is worth the extra binary file size to bring in the network and compression libraries into the project so that the library users will never have to worry about handling data downloading and decompression by themselves again.

The Future

The future of the BGPKIT Parser lies on continuous performance and stability improvements, as well as some exciting features that we are currently planning. Some of the coming features include

adding capability of handling real-time data streams coming from RIPE RIS Live and RotueViews’s Kafka BMP stream;
supporting data serialization back to MRT files (reverse-parsing), which allows users to produce customized MRT files after data processing;
adding WASM support to allow BGP data parsing directly on the web with JavaScript.

Because we are building our software in Rust, we can effortlessly tapping into Rust’s great software ecosystem and continue introducing new features and improvements. The future of BGPKIT Parser is exciting and we can’t wait to bring more features for you to try out!

For more details about the BGPKIT Parser, check out our GitHub repo and our website. Feedback is highly appreciated!

https://github.com/bgpkit/bgpkit-parser

Introducing BGPKIT Broker

Mingwei Zhang — Sun, 31 Oct 2021 00:00:00 GMT

BGPKIT Broker is a data API service that focuses on building a BGP data file index to enable searching for public/private BGP data files with custom filters. It is one of the building block components designed by BGPKIT to facilitate BGP data processing with ease.

The first step to investigate a BGP event.

Imagine this scenario: a malicious player just attempted to hijack a IP prefix using BGP announcements, and you are interested in learning what exactly happened during that half-hour down time of the victim network, how would you start investigating?

Collecting evidences. Luckily for us, the actors on the Internet, good and bad, always leave traces behind if they use BGP as their method. There are a number of reputable public BGP route collectors operating for years collecting all BGP messages received from their connected router peers and dump them into regular dump files. The most used projects are RouteViews and RIPE RIS Data.

Files are all over the places. There are more than 60 different data collectors from the two projects alone, and each publishing their data under separate sites. The two projects also have different data file structure and compression algorithms for their data dump files. It is not hard to go find one data file during the interested event time from one collector, but it will be a real hassle to gather URLs toward all data files that include information within a specified time range.

Collector’s data is published as compressed MRT files at regular frequency.

BGPKIT Broker — A BGP Data File Index API Service

We designed the BGPKIT Broker to resolve one and only one problem: quickly collect links to the BGP data files that matches a filtering criteria.

You can filter BGP data using multiple criteria:

start_ts : UNIX timestamp that all files must be dumped after
end_ts : UNIX timestamp that all files must be dumped before
data_type : the type of the data file, can be update or rib
collector: the collector ID that the files are generated from
project: the data collection project, can be route-views or riperis
page and page_size: the pagination control for collecting a large number of files

Here is an example REST API call: https://api.broker.bgpkit.com/v1/search?data_type=update&start_ts=1633046400&end_ts=1633132800&collector=rrc00&project=riperis&page%20=2&page_size=3

It asks for all data updates files dumped between 1633046400 and 1633132800, from RIPE RIS’s collector rrc00. It also requested for the second page of the results and each page contains 3 items.

DEPRECATED: BGPKIT Broker API service is freely available to use, hosted at https://api.broker.bgpkit.com/v1/. The documentation is available at https://docs.broker.bgpkit.com/

BGPKIT Broker API and SDK has been upgraded to V2 now. The V1 examples are left up for legacy services that depends on it. Please check out the current API documentation at for more:https://api.broker.bgpkit.com/v2/

BGPKIT Broker Rust API

The BGPKIT Broker API service is built entirely using Rust, and of course we also developed native Rust API to access the broker data with ease using Rust.

Example BGPKIT Broker Rust API call

The Rust API is open source under MIT license, and with the free API, you can already build your own workflow today!

https://github.com/bgpkit/bgpkit-broker

Easy On-premise Deployment

For API service like this, especially that it also collects and indexes data of over 10 years span, one might imagine the deployment could be complex and slow.

For BGPKIT Broker, we spent extra efforts to make the API deployment process as quickly as possible, and also efficient on resource consumption. For references, we bootstrapped the entirety of the database in under 5 minutes, and cost less than 1 GB storage to store the information in PostgreSQL database. The whole database and API run fluently with less than 500MB of RAM usage. This allows use to deployment extra instances when under heavy load without costing a fortune.

Users who need dedicated resource allocation for query performance can contact us for private API hosting that are not shared by others. Enterprise option is also available for on-premise deployment with customization consultation available. If you are interested in testing it out, feel free to shoot us an email at contact@bgpkit.com.

For more information, checkout our website!

https://bgpkit.com/broker

BGPKIT Journey Started

Mingwei Zhang — Tue, 26 Oct 2021 00:00:00 GMT

BGPKIT is a small-team start-up that aims to provide comprehensive tool suite to facilitate companies building on-premise BGP data monitoring services. We started our journey of building the best BGP data toolkit for developers in October, 2021. Here is a brief glance on what we are working on and what values we strive to provide.

BGP

Border Gateway Protocol (BGP) is the de facto inter-domain routing protocols being used by every major companies on the Internet. The main purpose of BGP is to allow companies exchange IP prefix reachability information. In other words, companies tell other companies what IP blocks they have, and how to reach these IP blocks. This the key functionality that enables the Internet.

Because the purpose of BGP is to exchange information and allow everyone to know how to reach certain IP blocks, the BGP messages must be propagated globally and publicly. There are a number of data sources available out there that provides BGP data archives (e.g. RouteViews and RIPE RIS), or real-time data like BGP looking glasses or live BGP data stream.

The aforementioned data sources contains a wealth of information if you know what to look for. For example, by looking at BGP information, researchers can infer companies relationships, monitor security incidences. Having handy toolkit in hand enables people to build successful businesses around BGP data.

BGPKIT

At BGPKIT, we build software tools to process BGP data and reveal insights from BGP messages. We aims to provide the best developer experience, and enable customers to build their own BGP data processing pipeline and monitoring services on-premise.

Complete Tool Suite

Our goal is to build complete tool suite for BGP data processing: data collection, parsing, analysis, programmable and visual interface, and data warehousing. Everything you need to handle BGP data.

Rust Implementation

To achieve the best performance and security, we choose to focus on building our tools using the Rust programming language.

We believe that Rust’s ecosystem is now mature enough that building modern features like data streaming, async data workflow, parallel processing, web API, or even porting the entire codebase onto WASM and running it on browsers is a archivable task with reasonable efforts.

Powerful Extensibility

At BGPKIT, we design our libraries to provide powerful API and assist customers to further customize workflow to meet individual needs. We strive to provide the most ergonomic interfaces that allow library consumers to easily integrate our library into theirs.

Embrace Open-Source

We also believe that good tools empowers people, so we open-sourced our building block libraries to the public with very permissive license so that any interested parties can explore and build their own ideas free of charge and limitation.

We are excited to start this journey of building, and we hope to have the chance to work with more people and building more dreams together! Follow us here, on Twitter, or visit our website to learn more!

BGPKIT Blog

Anatomy of the Virtualizor BGP Hijack: Reading the Attack in Raw BGP Bytes

The one-paragraph summary

The hijack path

Reading the first hijacked message, byte by byte

Timeline, verified against raw MRT where possible

How long the hijack "worked" end to end

What the entire public collector fleet saw

Where the flapping came from

Why RPKI didn't stop it

How ASPA would have changed the outcome

Why TLS didn't stop it

Detecting this class of hijack with BGPKIT tooling

Stage 1: notice, a new more-specific inside a valid covering block

Stage 2: corroborate, independent vantage, multiple transit chains

Stage 3: attribute the trust failure, ROA history

Stage 4: pin the evidence, bytes, not summaries

Stage 5: report it

Reproducing this analysis

Sources

Don't Deserialize: Serving 1.5M RPKI ROA Records from a Memory-Mapped Trie

The problem: deserialize-everything-at-startup

rkyv in one paragraph

v1 vs v2 at a glance

The migration

Step 1: prefix-trie with rkyv support

Step 2: rkyv-derivable record types

Step 3: the archive root

Step 4: mmap and query without deserializing

Step 5: safe hot-reload via atomic rename

Step 6: an allocator that gives memory back

Results

Correctness verification

Practical lessons

rkyv archived types are not transparent

prefix-trie's archived joint map exposes internal family fields

Backward compatibility costs one enum, not an architecture

When to use this pattern

Try it against the live API

Wrapping up

monocle v1.4.0: Remote BGP Search, PCH Archive Parsing, and RIB Correctness

Install and run

Run searches where the data and compute are

Parse PCH's Cisco-style BGP-table archives

Keep large filters out of shell arguments

Reconstruct the RIB that existed at the requested time

Pair reconstructed routes with historical RPKI data

bgpkit-parser 0.18.0 Release

Install

RIS Live raw message parsing

Raw BGP attribute retention

Deprecated attribute code handling

New typed BGP path attribute parsers

API changes to note

WebAssembly package

Links

Finding Well-Known BGP Communities with monocle

Why 65535:*?

Setup

Searching MRT updates with monocle

What the scan found

Checking BLACKHOLE behavior against RFC 7999

Checking NOPEER visibility against RFC 3765

A note on policy behavior

Takeaways

Investigating the India–Telegram BGP Hijack with monocle

Prerequisites

Step 1: Establish ground truth - who should originate this prefix?

The single most important check: RPKI validation

List Telegram's signed prefixes

Understand the suspect's connectivity

Step 2: Find the hijack in the data

Why a small collector?

Why -s -S?

The prefix arms race

Verify every hijacked prefix was RPKI-invalid

Step 3: Was the hijacked route actually selected in the routing table?

When did it end?

Step 4: Who propagated the invalid routes?

Step 5: The RPKI timeline - did Telegram have ROAs before the incident?

`prefix-trie`'s archived joint map exposes internal family fields

Why `65535:*`?

Searching MRT updates with `monocle`

Why `-s -S`?

New Feature: `monocle rib`

SDK: `query_total_count()` Method

Using `monocle radar`

`monocle radar stats`

`monocle radar pfx2as`