BGPKIT Blog

monocle v1.3.0: Reconstruct BGP RIB State at Any Timestamp

Wed, 27 May 2026 00:00:00 GMT

We are happy to announce the release of **monocle v1.3.0**, now available on [GitHub](https://github.com/bgpkit/monocle/releases/tag/v1.3.0) and [crates.io](https://crates.io/crates/monocle/1.3.0), also available on Homebrew via `brew install monocle`. This release introduces a major new command, `monocle rib`, which reconstructs BGP Routing Information Base (RIB) state at arbitrary timestamps. ## New Feature: `monocle rib` Most BGP tools show you what happened *during* an update file. `monocle rib` shows you the routing table *as it existed* at any moment in time. Given a target timestamp, the command: 1. Finds the latest RIB dump at or before that time 2. Downloads and replays all overlapping BGP update files up to the exact target timestamp 3. Materializes the final route state per peer and prefix The result is the reconstructed RIB — not raw BGP messages, but the actual route state. ### Basic Usage Reconstruct the RIB for a single timestamp and print to stdout: ```bash monocle rib 2025-09-01T12:00:00Z -c route-views6 ``` Write multiple snapshots to a single SQLite file: ```bash monocle rib \ 2025-09-01T08:00:00Z \ 2025-09-01T12:00:00Z \ --sqlite-path /tmp/rib-snapshots.sqlite3 \ -c route-views6 ``` ### Filtering Apply filters at reconstruction time to narrow the result set: ```bash # Only routes originated by AS13335 (Cloudflare) monocle rib 2025-09-01T12:00:00Z -c route-views6 -o 13335 # Routes for a specific prefix, including more-specifics monocle rib 2025-09-01T12:00:00Z -c route-views6 -p 2606:4700::/32 --include-sub # Routes from full-feed peers only monocle rib 2025-09-01T12:00:00Z -c route-views6 --full-feed-only # Routes where AS path contains a specific ASN monocle rib 2025-09-01T12:00:00Z -c route-views6 -a "13335" ``` ## Performance Notes The `rib` command keeps the working RIB state in memory during reconstruction, avoiding SQLite lookups and writes on the hot path. The updates query window was tightened from +2 hours lookahead to the exact target timestamp, reducing update file downloads by roughly one third for typical requests. ## Installation ```bash cargo install monocle ``` Or download pre-built binaries for Linux (x86_64, aarch64) and macOS (universal) from the [GitHub release page](https://github.com/bgpkit/monocle/releases/tag/v1.3.0).

monocle v1.1.0

Sat, 14 Feb 2026 00:00:00 GMT

monocle `v1.1.0` focuses on interface consistency and day-to-day usability. This release simplifies feature gates, standardizes data refresh APIs, and adds quality-of-life improvements across parsing, search, and configuration workflows. ## TL;DR * Feature flags are now simplified to `lib`, `server`, and `cli`. * Config and update flows are more consistent (`config update`, `config backup`, `config sources`, and `--no-update`). * Data refresh APIs are standardized across ASInfo, AS2Rel, RPKI, and Pfx2as. * Cache TTL defaults are unified at 7 days, with clearer staleness reporting. * Parse/search workflows gain multi-value filters, negation filters, field selection, ordering, timestamp format control, and local cache support. ## What's New ### Simpler feature flags The crate feature model is now reduced to three options: * `lib`: complete library functionality (database + lenses + display) * `server`: WebSocket server support (implies `lib`) * `cli`: full command-line binary (implies `lib` and `server`) This replaces the previous multi-tier setup and makes dependency selection easier for downstream users. ### Standardized data refresh APIs Database refresh behavior is now more uniform across ASInfo, AS2Rel, RPKI, and Pfx2as: * Consistent `needs_*_refresh(ttl)` checks * A shared `RefreshResult` shape with source and load details * Standardized naming (`refresh_*`) with compatibility aliases where needed * URL and local-path loading paths available across repositories This update reduces API drift and makes maintenance code paths more predictable. ### Config command updates Configuration and maintenance commands now use clearer naming: * `monocle config db-refresh` -> `monocle config update` * `monocle config db-backup` -> `monocle config backup` * `monocle config db-sources` -> `monocle config sources` The global no-refresh toggle was also renamed for consistency: * `--no-refresh` -> `--no-update` The following command shows the active configuration, cache TTL settings, database status, and server defaults: ```bash monocle config ``` Example output: ```text monocle Configuration ===================== General: Config file: /home/user/.monocle/monocle.toml Data dir: /home/user/.monocle/ Cache TTL: ASInfo: 7 days AS2Rel: 7 days RPKI: 7 days Pfx2as: 7 days Database: Path: /home/user/.monocle/monocle-data.sqlite3 Status: exists Size: 512.47 MB Schema: initialized (v3) ASInfo: 120953 records (updated: 2026-02-02 19:54:01 UTC) AS2Rel: 877937 records (updated: 2026-02-02 14:25:34 UTC) RPKI: 796899 ROAs, 962 ASPAs (updated: 2026-02-10 19:53:50 UTC) Pfx2as: 1580626 records (updated: 2026-02-02 20:02:10 UTC) ``` ### Better cache control defaults All major data sources now support configurable cache TTL with a 7-day default. This applies to ASInfo, AS2Rel, RPKI, and Pfx2as. `monocle config sources` now reports staleness based on TTL, so it is easier to see what needs updating. The following command shows per-source status, staleness, and last update recency: ```bash monocle config sources ``` Example output: ```text Data Sources: Name Status Stale Last Updated ------------------------------------------------------------ asinfo 120953 records yes a week ago as2rel 877937 records yes a week ago rpki 797861 records no 2 hours ago pfx2as 1580626 records yes a week ago Configuration: ASInfo cache TTL: 7 days AS2Rel cache TTL: 7 days RPKI cache TTL: 7 days Pfx2as cache TTL: 7 days ``` ### RPKI improvements monocle now supports fetching ROAs via RTR (RPKI-to-Router), including endpoint override support and fallback behavior. The following command refreshes only RPKI data and uses the provided RTR endpoint for this run instead of the default configured source. ```bash monocle config update --rpki --rtr-endpoint rtr.rpki.cloudflare.com:8282 ``` ### Parse and search enhancements `parse` and `search` gained several output and filtering improvements: * Multi-value filters with OR semantics * Negation filters using `!` * Validation for ASN/prefix filter inputs * `--fields` for column selection * `--order-by` and `--order` for sorted output * `--time-format` for unix or RFC3339 display * `search --cache-dir` local file + broker query caching The following command searches one hour of updates starting at `2024-01-01`, filters for prefix `1.1.1.0/24`, and caches downloaded MRT files plus broker query results under `/tmp/mrt-cache` for faster repeat runs. ```bash monocle search -t 2024-01-01 -d 1h -p 1.1.1.0/24 --cache-dir /tmp/mrt-cache ``` The following command uses multi-value filters with negation to exclude two origin ASNs while also matching either of two peer ASNs: ```bash monocle search -t 2024-01-01 -d 1h -o '!13335,!15169' -J 174,2914 ``` Negation and positive values cannot be mixed within the same filter field. ## Breaking Changes and Migration Notes ### 1) Feature flag migration If you previously used feature tiers like `database`, `lens-core`, `lens-bgpkit`, `lens-full`, or `display`, switch to: * `lib` for library use * `server` for WebSocket API use * `cli` for full command-line use ### 2) CLI and subcommand renames Update scripts and automation: * `--no-refresh` -> `--no-update` * `config db-refresh` -> `config update` * `config db-backup` -> `config backup` * `config db-sources` -> `config sources` ### 3) Parse/search filter type updates (library API) `ParseFilters` moved from scalar optional fields to vector-based values for multi-value and negation support. Library consumers should update filter construction accordingly. ## Additional Improvements * AS name rendering now prefers PeeringDB naming fields before falling back to AS2Org/core names * Data refresh logging now shows specific reasons (empty vs outdated) * Example layout was reorganized to one example per lens ## Full Change List See the v1.1.0 section in [`CHANGELOG.md`](http://CHANGELOG.md) for the complete list of changes.

BGPKIT Parser v0.14.0 Release and v0.13.0 Highlights

Thu, 25 Dec 2025 00:00:00 GMT

We are pleased to announce the release of **BGPKIT Parser v0.14.0**. This update introduces support for negative filters and the RPKI-to-Router (RTR) protocol. We also want to highlight key features from the recent v0.13.0 release, including enhanced debugging tools. ## v0.14.0 Features ### Negative Filter Support A frequent request has been the ability to filter *out* specific data points. We have added support for negative filters across most filter types, allowing exclusion of specific origins, prefixes, peers, or communities. In the CLI, use the `!=` operator. For example, to process all records *except* those originating from AS 13335: ```bash bgpkit-parser https://spaces.bgpkit.org/parser/update-example.gz --filter "origin_asn!=13335" ``` In Rust code, use the `!` prefix: ```rust let parser = BgpkitParser::new("...") .add_filter("!origin_asn", "13335") .add_filter("!peer_ip", "192.168.1.1"); ``` Supported negative filters include `!origin_asn`, `!prefix`, `!peer_ip`, `!peer_asn`, `!type`, `!as_path`, `!community`, and `!ip_version`. ### RPKI RTR Protocol Support We have added support for the RPKI-to-Router (RTR) protocol, covering both version 0 ([RFC 6810](https://datatracker.ietf.org/doc/html/rfc6810)) and version 1 ([RFC 8210](https://datatracker.ietf.org/doc/html/rfc8210)). The new `models::rpki::rtr` and `parser::rpki::rtr` modules allow developers to build custom RTR clients or servers. Here is how you can use the library to connect to an RTR server and request data: ```rust use bgpkit_parser::models::rpki::rtr::*; use bgpkit_parser::parser::rpki::rtr::{read_rtr_pdu, RtrEncode, RtrError}; use std::net::TcpStream; use std::io::Write; // 1. Connect to the RTR server let mut stream = TcpStream::connect("rtr.rpki.cloudflare.com:8282")?; // 2. Send a Reset Query to request the full database let reset_query = RtrResetQuery::new_v1(); stream.write_all(&reset_query.encode())?; // 3. Read the response PDUs loop { match read_rtr_pdu(&mut stream)? { RtrPdu::IPv4Prefix(p) => { println!("Received IPv4 ROA: {}/{}-{} -> AS{}", p.prefix, p.prefix_length, p.max_length, p.asn); } RtrPdu::EndOfData(_) => break, _ => {} } } ``` We have included a fully functional RTR client example that connects to a server, fetches ROAs, and performs route validation. You can run the example yourself: ```bash cargo run --example rtr_client -- rtr.rpki.cloudflare.com 8282 ``` ## In Case You Missed It: v0.13.0 The v0.13.0 release introduced several improvements for debugging and analyzing MRT data. ### Record-Level Output The CLI supports inspecting individual MRT records rather than just parsed BGP elements. This aids in debugging parser issues or analyzing raw MRT files. Switch to record-level output with `--level records` and choose a format (e.g., JSON): ```bash bgpkit-parser https://spaces.bgpkit.org/parser/update-example.gz --level records --format json ``` ### Raw Bytes Access For developers, `RawMrtRecord` now includes a `header_bytes` field, and the `raw_bytes` field has been renamed to `message_bytes`. This provides access to the exact bytes of the MRT header and the message body as they appeared on the wire, enabling byte-for-byte export and debugging without re-encoding. ### Other Improvements * **Testing & Fuzzing**: Added a `cargo-fuzz` harness and initial fuzz targets. * **Performance**: Continued optimizations for faster processing. Check out the full [CHANGELOG](https://github.com/bgpkit/bgpkit-parser/blob/main/CHANGELOG.md) for more details. --- *Happy Parsing!*

BGPKIT Broker v0.9.0: Better Pagination and New Collectors

Sat, 01 Nov 2025 00:00:00 GMT

We are excited to announce the release of BGPKIT Broker v0.9.0. This release introduces total count support for efficient pagination, making it easier to build data exploration interfaces and manage large result sets. We also expand our collector coverage with two new Internet Exchange points. ## Previously on BGPKIT Broker Since the major v0.7 release that unified the architecture into a single SQLite-backed CLI application, BGPKIT Broker has been serving the community with stable uptime and continuous data coverage. The v0.8 series brought several enhancements including automated backup systems, configuration validation, and convenient query shortcuts. However, one common challenge remained for developers building interfaces on top of Broker: pagination. When querying large time ranges, users needed to fetch all results to know the total count, making it inefficient to build proper pagination controls or display result statistics. ## V0.9: Efficient Pagination Support Version 0.9.0 addresses pagination needs with total count support at both the SDK and API levels. This allows applications to fetch result counts independently from the data itself, enabling responsive user interfaces without over-fetching data. ### SDK: `query_total_count()` Method The SDK now provides a dedicated `query_total_count()` method that returns the total number of matching items without retrieving the actual data. This is useful when you need to know how many results exist before deciding how to paginate through them. ```rust use bgpkit_broker::BgpkitBroker; #[tokio::main] async fn main() { let broker = BgpkitBroker::new() .ts_start("2024-10-01") .ts_end("2024-10-02") .project("route-views"); // Get total count without fetching items let total = broker.query_total_count().await.unwrap(); println!("Total matching items: {}", total); // Now fetch paginated results let result = broker .page_size(100) .page(1) .query() .await .unwrap(); println!("Retrieved {} items out of {} total", result.data.len(), result.total.unwrap_or(0)); } ``` This method executes a `COUNT(*)` query against the database with the same filters as your main query, returning just the number rather than all the data. For large time ranges with thousands of results, this can save significant bandwidth and processing time. ### API: Total Field in Search Results The `/v3/search` endpoint now includes a `total` field in every response, providing the total count of matching items alongside the paginated results. This enhancement makes it straightforward to build pagination controls in web interfaces or CLI tools. ```bash curl "https://api.bgpkit.com/v3/broker/search?ts_start=2024-10-01&ts_end=2024-10-02&page_size=10&page=1" ``` Response: ```json { "total": 1234, "data": [ { "ts_start": 1727740800, "ts_end": 1727740800, "collector": "route-views2", "project": "route-views", "url": "...", ... } ] } ``` With the `total` field, client applications can: * Display "showing X of Y results" to users * Calculate the number of pages needed for pagination * Decide whether to fetch all results or paginate * Provide accurate progress indicators for data processing ### New RouteViews Collectors This release adds support for two new RouteViews collectors: * **hkix.hkg**: Hong Kong Internet Exchange (HKIX) collector in Hong Kong * **ix-br.gru**: [IX.br](http://IX.br) ([PTT.br](http://PTT.br)) in São Paulo These additions enhance BGPKIT Broker's global coverage, providing more diverse vantage points into internet routing behavior. Users who update to v0.9.0 can run `bgpkit-broker update` to automatically bootstrap data for these new collectors. ## Behind the Scenes Improvements Beyond the user-facing features, we also made several code improvements: * Refactored bootstrap download logging to centralize progress reporting and eliminate redundant code paths * Updated the oneio dependency to v0.20.0 with optimized feature flags for better handling rustls providers * Enhanced test coverage overall ## Upgrading to v0.9.0 ### For SDK Users Update your `Cargo.toml` to use the latest v0.9 release: ```toml [dependencies] bgpkit-broker = "0.9" ``` Then run: ```bash cargo update ``` The new `query_total_count()` method is immediately available for use. The `total` field in `BrokerQueryResult` is backward compatible as an optional field. ### For Self-Hosted Instances If you run your own BGPKIT Broker instance, update to the latest version: ```bash cargo install --force bgpkit-broker --version 0.9.0 --features cli ``` Or pull the latest Docker image: ```bash docker pull bgpkit/bgpkit-broker:latest ``` After upgrading, run the update command to add the new collectors: ```bash bgpkit-broker update --db-path your-database.sqlite3 ``` The update process will automatically detect and bootstrap historical data for the two new collectors. ## Looking Forward BGPKIT Broker continues to serve as a fundamental component for BGP data processing pipelines. With v0.9.0, we focused on making the developer experience better for building applications that need pagination and result statistics. We continue to maintain the public Broker instance at [`https://api.bgpkit.com/v3/broker`](https://api.bgpkit.com/v3/broker) with 99.996% uptime, and encourage users to deploy on-premise instances for production pipelines to reduce external dependencies. For the full v0.9.0 release notes, please check out our [GitHub release page](https://github.com/bgpkit/bgpkit-broker/releases/tag/v0.9.0). If you have any comments, please drop us a message on [Twitter](https://twitter.com/bgpkit), [Mastodon](https://infosec.exchange/@bgpkit), [Bluesky](https://bsky.app/profile/bgpkit.com) or [email](mailto:contact@bgpkit.com). ## Get Started Ready to try out the new features? Here are some resources: * **GitHub**: [github.com/bgpkit/bgpkit-broker](http://github.com/bgpkit/bgpkit-broker) * **Examples**: Browse [example code](https://github.com/bgpkit/bgpkit-broker/tree/main/examples) demonstrating common usage patterns * **Public API**: Try queries at [https://api.bgpkit.com](https://api.bgpkit.com) Have questions or feedback? Open an issue on [GitHub](https://github.com/bgpkit/bgpkit-broker/issues) or join discussions in our [Discord Channel](https://discord.gg/XDaAtZsz6b). ## 💖 If you find our libraries and services useful, we would highly appreciate if you consider sponsoring us on [GitHub](https://github.com/sponsors/bgpkit).

Run BGPKIT on Cloudflare Containers

Mon, 30 Jun 2025 00:00:00 GMT

For the longest time, I’ve been using [Cloudflare](https://developers.cloudflare.com/) exclusively for web/API hosting and relatively lightweight tasks. For the most of my work on BGP, there is really not much that I can accomplish with just JavaScript/TypeScript (except maybe working with [RIS Live](https://ris-live.ripe.net/) WebSocket). The computationally intensive nature of the most BGP data processing doesn't naturally fit within the typical Cloudflare developer platform. This changes with the recent [announcement](https://blog.cloudflare.com/containers-are-available-in-public-beta-for-simple-global-and-programmable/) of [Cloudflare Containers](https://developers.cloudflare.com/containers/). In short, it allows developers to build and run custom containers on the Cloudflare’s platform, enabling the heavy workload to mix with other primitives and unify the deployment. In this blog post, I will show you how to build a BGP data search API with BGPKIT and deploy on Cloudflare Containers. The source code is available [on GitHub](https://github.com/bgpkit/bgpkit-cf-containers). ## BGP Data Search API For this example, I will show you how to build a very straightforward HTTP API to allow accepting search parameters and let BGPKIT to fetch and parse BGP archives and return the parsed messages. The API accepts four parameters: `collector`, `prefix`, `ts_start`, and `ts_end`, to filter and parse BGP archives efficiently. * `collector`: the BGP route collector ID to use (e.g. `rrc00` or `route-views2`). We want to limit the search to a single collector. * `prefix`: the prefix of the BGP relevant updates. Open-ended search will burn up resources quick, but you can opt-out this requirement. * `ts_start` and `ts_end`: the starting and ending timestamps. The goal is to limit the search to end up parsing a very small amount of MRT files (e.g. give them the same value will do). We should probably leave large-scale data crunching to a environment with more CPU powers. The parameters are defined in a struct to be able to passed in to a `axum` “GET” route: ```rust #[derive(Deserialize, Serialize)] struct Params { collector: String, prefix: String, ts_start: String, ts_end: String, } ``` With the given parameters, we will first find the relevant MRT files by setting the filters of timestamps and collector to `BgpkitBroker` instance: ```rust let files = match bgpkit_broker::BgpkitBroker::new() .ts_end(ts_end.clone()) .ts_start(ts_start.clone()) .collector_id(collector.clone()) .query(){ Ok(items) => items, Err(e) => { return Json(Result { error: Some(e.to_string()), data: vec![], meta: None, }); } }; ``` For each file, we will parse the whole MRT file and collect BGP updates that is relevant to the target prefix: ```rust for file in files { let mut parser = match bgpkit_parser::BgpkitParser::new(file.url.as_str()){ Ok(parser) => parser, Err(e) => { return Json(Result { error: Some(e.to_string()), data: vec![], meta: None, }); } }; parser = match parser.add_filter("prefix", prefix.as_str()){ Ok(parser) => parser, Err(e) => { return Json(Result { error: Some(e.to_string()), data: vec![], meta: None, }); } }; items.extend(parser.into_elem_iter()); } ``` Because the BGPKIT parser and broker code are implemented in the sync environment, we will need to wrap the code above in a blocking thread in order to use it in async web frameworks like `axum`. ```rust let result = tokio::task::spawn_blocking(move || { // THE BLOCKING CODE PIECES }).await.unwrap(); ``` Please see the full source code here for more details: [https://github.com/bgpkit/bgpkit-cf-containers/blob/main/container-src/src/main.rs](https://github.com/bgpkit/bgpkit-cf-containers/blob/main/container-src/src/main.rs) ## Cloudflare Containers Deployment Now that we have an Rust-based API code working, we will need to 1. containerize the code, 2. put a Cloudflare Container wrapper around it for deployment. The container definition is a typical two-stage build definition, with a builder stage to build the binary application, and a minimum deployment stage to call the executable. The two-stage build is almost necessary as Cloudflare Containers [has limits](https://developers.cloudflare.com/containers/platform-details/#limits) on the size of each image and the total storage per account. The smaller the image the better. ```dockerfile # ---- Build Stage ---- FROM rust:1.86 AS builder WORKDIR /app # Install build dependencies RUN apt-get update && apt-get install -y pkg-config libssl-dev # Build application COPY container-src/Cargo.lock container-src/Cargo.toml ./ COPY container-src/src ./src RUN cargo build --release # ---- Runtime Stage ---- FROM debian:bookworm-slim # Install minimal runtime dependencies RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* WORKDIR /app # Copy the statically linked binary from the builder COPY --from=builder /app/target/release/bgpkit-cf-container /app/bgpkit-cf-container EXPOSE 3000 CMD ["/app/bgpkit-cf-container"] ``` The rest of the task is to build a API app for Workers and configure Containers. The following config is pretty much all it needs to configure the Workers script to build and push the container image, and create a [Durable Object](https://developers.cloudflare.com/durable-objects/) to coordinate and run Containers. ```json "containers": [ { "class_name": "BgpkitContainer", "image": "./Dockerfile", "max_instances": 5 } ], "durable_objects": { "bindings": [ { "class_name": "BgpkitContainer", "name": "BGPKIT_CONTAINER" } ] }, "migrations": [ { "new_sqlite_classes": [ "BgpkitContainer" ], "tag": "v1" } ] ``` The main Workers script is only 22 lines long: ```typescript export class BgpkitContainer extends Container { defaultPort = 3000; sleepAfter = '5m'; } // Create Hono app with proper typing for Cloudflare Workers const app = new Hono<{ Bindings: { BGPKIT_CONTAINER: DurableObjectNamespace }; }>(); app.get("/search", async (c) => { if (!c.req.query('collector') || !c.req.query('prefix') || !c.req.query('ts_start') || !c.req.query('ts_end')) { return c.json({ error: "Missing required query parameters: collector, prefix, ts_start, ts_end" }, 400); } const container = getContainer(c.env.BGPKIT_CONTAINER); return await container.fetch(c.req.raw); }); export default app; ``` The important pieces are: * `class BgpkitContainer extends Container` block defines the port to use and configures how long the container should run after the last query. In this example, containers will be killed after 5 minutes of inactivity. It is crucial to realize that Cloudflare Containers are not a drop-in replacement of some other container deployment platforms like fly.io or Railway, as the workload for Containers are intended to be short-lived (ping me if this changes) and scales horizontally depending on the requests amount. * the `getContainer` function here tries to reach a container. If the intended container is overloaded, it may create a new container on demand. You may choose to use the `getRandom` function to round-robin containers. See [the docs](https://developers.cloudflare.com/containers/scaling-and-routing/) for more. * the `container.fetch(c.req.raw)` forwards the query to the container, including the query parameters, which will then be handled by the running container. ## Example Queries The following example will reach the Workers script (handled by Hono), and then reach the container to run the actual BGP data crunching task. (This URL won’t actually work as we don’t have budget to provide such service openly. Feel free to deploy it on your own account to try it out.) [https://EXAMPLE.bgpkit.workers.dev/search?collector=rrc00&prefix=1.1.1.0/24&ts\_start=1751231488&ts\_end=1751231488](https://bgpkit-cf-containers.bgpkit.workers.dev/search?collector=rrc00&prefix=103.228.200.0/24&ts_start=1751231488&ts_end=1751231488)

BGPKIT Broker 0.7 Release

Sat, 22 Jun 2024 00:00:00 GMT

[BGPKIT Broker](https://github.com/bgpkit/bgpkit-broker) is a fundamental component to our design of a all-purpose BGP data processing pipeline. In short, it is a BGP data file meta information "broker" that tells the data consumers what MRT files from RouteViews and RIPE RIS are available for any given time range in question. It commonly serves as a data input entry point for data pipelines. For instance, here is a simple diagram for a system that creates a semi-real-time BGP data stream with BGPKIT Broker and Parser (a very common use case for these two libraries). BGPKIT Broker periodically crawls the websites of RIPE RIS and RouteViews MRT data pages of their collectors and index meta information into a database. Downstream consumers can ask and retrieve new files and process the files into BGP messages. # Previously on BGPKIT Broker In the BGPKIT Broker version 0.1 to 0.6, a working broker instance consists of three individual components: a crawler, a Postgres database, and an API. Each of the three component runs independently, and requires independent configuration, cronjobs, deployment, and all these goodies. For example, to run BGPKIT Broker v0.6, a user will need to configure and run * a PostgreSQL database with proper credentials and schema set up; * a cronjob instance that periodically crawls the data sources, with optional locks to prevent overlapping executions in case some crawl became slow; * a API application likely sitting behind a configured reverse proxy like Caddy to serve the data. It's fun and exciting to set up all these for the first time, but quickly became tiring and too complex for repeated set up or bootstrapping for new users. # V0.7: one CLI app that does everything We completely revamped the architecture for BGPKIT Broker in V0.7 to merge every functionality needed for a running Broker instance into one single command-line application: `bgpkit-broker`. V0.7 provides a single application to configure, run, debug, query everything on BGPKIT Broker. To achieve this redesign, we made some major changes to our architecture. ## SQLite instead of PostgreSQL There are two major topics to concern when choosing a backend database for BGPKIT Broker: performance and portability. ### SQLite is more than fast enough BGPKIT Broker indexes metadata for **all collectors** from RouteViews and RIPE RIS, which includes time, URL, type, size of every RIB dump and updates MRT files from these two public archives. Dating all the way back to 1999, we have indexed roughly 48 million MRT files' metadata. With a single index on the timestamp of files, we are able to search data files in less than 0.5s for any queries, which is more than fast enough for our use cases. We admit that we have spent our time for "early optimization" and in the end, the simple schema out-weighs the small performance gains. ### Backup and bootstrap with just one file Now in terms of portability, we can appreciate enough the beauty of single-file database like SQLite. In our current production setup, we periodically backup the database, and it literally involves just copying a single file to another directory (well, we also upload it to Cloudflare R2 for safekeeping). Portability also means users can move their instance anywhere they want with ease. This is definitely the case for V0.7 where new users can bootstrap by simply download a SQLite file (our CLI provides all that functionality), and move to new locations by scp it to anywhere they desire. Here is a video demonstrating bootstrapping a local BGPKIT Broker SQLite database with the new `bgpkit-broker bootstrap` command. %[https://youtu.be/SsCyfQ5q0G0] ## New file notification via NATS Before V0.7, pipelines that needs to continuously processing new MRT files will need to "pull" data from BGPKIT Broker instance periodically and keep track of the latest files processed. We consider this a hassle that developers should not be dealing with and thus introduced a new [`NATS`](https://nats.io/)\-based message channel allowing data consumers to subscribe to the public/private NATS channel where a Broker instance may publish new file notification to. We dedicated `nats.broker.bgpkit.com` as the public endpoint for any NATS consumers to connect to. Whenever a new file becomes available in Broker, it will publish a new file notification with all metadata as in the database entry to the public channel. Consumers (e.g. data pipelines) can use the `NatsNotifier::new(None).start_subscription()` to start waiting for new files. The following snippet below shows how a simple pipeline can use this feature in a loop. ```rust let mut notifier = match NatsNotifier::new(url).await { Ok(n) => n, Err(e) => { error!("{}", e); return; } }; if let Err(e) = notifier.start_subscription(subject).await { error!("{}", e); return; } while let Some(item) = notifier.next().await { if pretty { println!("{}", serde_json::to_string_pretty(&item).unwrap()); } else { println!("{}", item); } } ``` We also implemented a simple new file watcher in the app as `bgpkit-broker live` subcommand. It will start a subscription to the public BGPKIT NATS endpoint and print out new file data as they come to the channel. ## One command to serve and update As mentioned previously, the new `bgpkit-broker` application includes everything one needs to start a instance. Once one bootstrapped the database to a local sqlite file (via `bgpkit-broker bootstrap ` command), all they need to start a auto-updating API is to run `bgpkit-broker serve ` . ```bash bgpkit-broker serve --help Serve the Broker content via RESTful API Usage: bgpkit-broker serve [OPTIONS] Arguments: broker db file location Options: -i, --update-interval update interval in seconds [default: 300] --no-log disable logging -b, --bootstrap bootstrap the database if it does not exist --env -s, --silent disable bootstrap progress bar -h, --host host address [default: 0.0.0.0] -p, --port port number [default: 40064] -r, --root root path, useful for configuring docs UI [default: /] --no-update disable updater service --no-api disable API service -h, --help Print help -V, --version Print version ``` The `serve` subcommand will also start a thread that periodically crawl and update the SQLite database to make sure the API always serve the up-to-date data. Noticed that error message? It's by design as it tries to connect to notification channel for new files as a default behavior for a service, but not NATS URL is configured. We use the `BGPKIT_BROKER_NATS_URL` environment variable to configure the NATS channel to use. We also allow users to optionally configure a heartbeat URL to monitor the data updating status. After every success data crawling run, Broker will try to execute a HTTP GET to a URL if `BGPKIT_BROKER_HEARTBEAT_URL` is set in the environment. This is useful to monitor the running status of the Broker instance without the need of setting up a cronjob. We use [Better Stack's Uptime](https://betterstack.com/) monitoring service for page and heartbeat monitoring, and the public Broker instance is running V0.7 with the heartbeat URL set to this service. All status information can be found at [https://status.bgpkit.com/](https://status.bgpkit.com/) # Production-ready, on-prem deployment Although BGPKIT Broker has not yet reached V1.0, we consider it to be feature-complete and production-ready. Ever since V0.2, we have made our better efforts on not introducing any breaking changes and the service has been serving the community with a stable uptime ever since. We believe all libraries running in production should at least be 1.0, and thus **we will release V1.0 soon this summer**. We also made significant efforts in V0.7 release to make sure BGPKIT Broker is as portable as possible. New users can spin up a fully functioning Broker instance with just two commands: `bgpkit-broker bootstrap` and `bgpkit-broker serve`, all within 5 minutes. With V0.7 released, we **encourage all data pipeline designers to deploy a Broker instance on-premise**, ensuring data pipelines are self-containing and reduce external dependencies as much as possible. We also will continue maintain our public instance to our best efforts (we are currently at **99.996% uptime**). Thanks to [our sponsors](https://github.com/sponsors/bgpkit), we are able to keep the services up as we do, and we plan to continue serving the community the same way in the foreseeable future. For the full V0.7 release notes, please check out our [GitHub release page](https://github.com/bgpkit/bgpkit-broker/releases/tag/v0.7.0). If you have any comments, please drop us a message at [Twitter](https://twitter.com/bgpkit), [Mastodon](https://infosec.exchange/@bgpkit), or [email](mailto:contact@bgpkit.com).

💖

If you find our libraries and services useful, we would highly appreciate if you consider sponsor us on GitHub.

Command-line Routing Stats with monocle and Cloudflare Radar API

Sun, 21 Apr 2024 00:00:00 GMT

BGPKIT `monocle` is a command-line utility program that helps users quickly pull Internet routing-related information from publicly available sources. [https://github.com/bgpkit/monocle](https://github.com/bgpkit/monocle) In BGPKIT `monocle` version V0.5, we add support for querying [Cloudflare Radar](https://radar.cloudflare.com/)'s new BGP [routing statistics](https://developers.cloudflare.com/api/operations/radar-get-bgp-routes-stats) and [prefix-to-origin mapping](https://developers.cloudflare.com/api/operations/radar-get-bgp-pfx2as) APIs, the same APIs that power the [Cloudflare Radar routing section](https://radar.cloudflare.com/routing). `monocle` users can now quickly glance overview of routing stats for any given ASN, country, or the whole Internet. Users can also quickly look up prefix origins and examine their RPKI validation status as well as prefix visibility on the global routing tables. ## Using `monocle radar` We added a new `monocle radar` command group in V0.5, which contains the following to subcommands: * `monocle radar stats [QUERY]`: get routing stats (like prefix count, rpki invalid count) for a given country or ASN. * `monocle radar pfx2as [QUERY] [--rpki-status valid|invalid|unknown]`: get prefix to origin mapping for a given prefix or ASN ```plaintext mingwei@terrier ~ % monocle radar Cloudflare Radar API lookup (set CF_API_TOKEN to enable) Usage: monocle radar Commands: stats get routing stats pfx2as look up prefix to origin mapping on the most recent global routing table snapshot help Print this message or the help of the given subcommand(s) Options: -h, --help Print help -V, --version Print version ``` ### Cloudflare API token needed Since the `monocle radar` command relies on querying data using Cloudflare Radar public API, we also need to specify a user API token as the environment variable `CF_API_TOKEN`. Obtaining an API token is free and only needs a Cloudflare account. Interested users can follow their [official tutorial](https://developers.cloudflare.com/radar/get-started/first-request/) to obtain a token. The environment variable can be set in a `.env` file in the current directory, or set in ~`/.bashrc` or `~/.profile` etc. ### `monocle radar stats` Users can query the routing statistics for a given country or ASN. For example, `monocle radar stats us` returns the routing stats for the United States, while `monocle radar stats 174` returns the stats for Cogent (`AS174`). The displayed table is further divided into three rows, one for overall counting, and one for IPv4 and IPv6-specific counting. For each row, we show the following fields: * `origins`: the number of origins ASes registered in the given country * `prefixes`: the number of prefixes originated by the given ASN or ASes registered in the given country * `rpki_valid/invalid/unknown`: the number of RPKI valid/invalid/unknown prefix routes (prefix-origin mapping) on the global routing table and their percentage of the overall routes. ### `monocle radar pfx2as` Users can query the prefix-to-origin API to get the mapping of origin ASes and their originated prefixes on the global routing table. In the following example, `monocle radar pfx2as 174 --rpki-status invalid`, we ask for all the prefixes originated by `AS174` with the RPKI validation status to be invalid. This command returns us the list of RPKI invalid prefixes originated by `AS174` at the time of generating the dataset. ### Questions it can answer now (more in the future) Here is a selected list of questions that `monocle radar` command can answer you: * How many ASes are there on the Internet that announce at least one prefix? (81,770) * How many of these ASes announce only IPv6 prefixes? (6,853) * How many prefixes are there on the global routing table? (1,205,218) * How many prefixes do `AS400644` announce? (1) * Which AS(es) originates `1.1.1.0/24`? (AS13335) * How many prefixes originated by `AS174` are NOT covered by some RPKI ROA? (a lot, 94%+) * How about the RPKI valid ratio for the Philippines? (77%, nice!) ## Powered by Cloudflare Radar free API > Cloudflare Radar is a hub that showcases global Internet traffic, attack, and technology trends and insights. What [Cloudflare Radar](https://radar.cloudflare.com/) shines is its data openness. Everything you see on the Cloudflare Radar website is powered by their free [publicly available APIs](https://developers.cloudflare.com/api/operations/radar-get-bgp-pfx2as-moas). It's a treasure trove there, and all users need is a [free API token](https://developers.cloudflare.com/radar/get-started/first-request/) to access everything. At BGPKIT, we think we can further improve the usability of the API by exposing them as a proper Rust SDK: [`radar-rs`](https://github.com/bgpkit/radar-rs). This is our (unofficial) effort on bringing the Cloudflare Radar's rich data to Rust developers. For example, `monocle radar` is powered by this SDK.

2022 Year in Review

Tue, 31 Jan 2023 00:00:00 GMT

In 2022, BGPKIT as an open-source organization made significant progresses. As the founder, I am grateful for all the opportunities would like to take this time to appreciate all the milestones we achieved. In this post, I will go through some notable changes we made in 2022, and take a look at what we are excited about for the year of 2023. --- ## BGPKIT Parser There were a number of major features added to BGPKIT Parser in 2022: * [v0.7.0](https://github.com/bgpkit/bgpkit-parser/releases/tag/v0.7.0) added support for filtering messages by many fields, and allowing reading from uncompressed files. * [v0.7.1](https://github.com/bgpkit/bgpkit-parser/releases/tag/v0.7.1) added better examples of parallel MRT files processing with `rayon`. * [v0.7.2](https://github.com/bgpkit/bgpkit-parser/releases/tag/v0.7.2) added filtering by multiple `peer_ip`s. * [v0.8.0](https://github.com/bgpkit/bgpkit-parser/releases/tag/v0.7.1) includes many internal refactoring and brought in the new [oneio](https://github.com/bgpkit/oneio) library to improve developer experience. ## BGPKIT Broker In 2022, we have revised the BGPKIT Broker backend to support crawling for estimated file sizes in addition to other fields like timestamps and URLs. This allows us to keep track of MRT file size changes and help users to pick suitable collectors to use, especially at the age where RIB dumps from a single collector could reach over 1 GB in size. Figure of RIB sizes over time. Blue line is for rrc00, while yellow line is for route-views2. We also made major revisions to our Rust SDK for more features like `.latest()` to get the lastest MRT files from each collector. [https://github.com/bgpkit/bgpkit-broker/releases/tag/v0.5.0](https://github.com/bgpkit/bgpkit-broker/releases/tag/v0.5.0) Plus, if you would like to deploy a broker instance yourself, we also have made some significant efforts to improve our documentation on self-hosting guide. [https://github.com/bgpkit/bgpkit-broker-backend/blob/main/deployment/README.md](https://github.com/bgpkit/bgpkit-broker-backend/blob/main/deployment/README.md) ## Python Bindings In addition to core Rust code base for parser and broker, we have also added support for Python bindings for various Rust SDKs. Users can easily parse MRT files directly using `pybgpkit` Python library. It is also proven to be usable on cloud-based Jupyter notebooks like Google Colab ([examples](https://colab.research.google.com/drive/1AuNnzT43LYAZNnp1muhJTy0rvv3qbCX6)). [https://github.com/bgpkit/pybgpkit](https://github.com/bgpkit/pybgpkit) ## monocle To ties things together, we have also developed our first investigative tool, `monocle` , to help users to quick find relevant BGP announcements with a suite of easy-to-use utilities. Users with Rust toolchain installed can run `cargo install monocle` to install the tool. [https://github.com/bgpkit/monocle](https://github.com/bgpkit/monocle) Users can use the following subcommands * `parse`: parse single MRT files, remotely or locally * `search`: find and filter BGP messages accross multiple public collectors * `time`: convert between local time string and Unix timestamps * `whois`: find out AS names, ASN, registration countries, and organizations. ## Web/API and Infrastructure In 2022, we started to experiment new cloud-based infrastructure, especially cloud-based databases for better API stability and developer experiences. We ended up selecting [Supabase](https://supabase.com/) as our PostgreSQL production host and a self-hosted instance for backup. We are happy with the performance and cost provided by Supabase and more exicted about the potential capability it brings such as user authentication, cloud storage, local dev schema changes, etc. Based on the new infrastructure, we have started to test a new integrated API system ([still in alpha](https://alpha.api.bgpkit.com/docs/)). This allows us to put all of our data access and processing end-points into one location. Based on the new API, we have also developed a newer version of the BGPKIT Broker statistics page: [https://alpha.stats.bgpkit.com/](https://alpha.stats.bgpkit.com/) ## New Datasets Apart from provide SDKs and data APIs, we have also started providing free access to historical archives of some data that we find interesting. We blogged about one dataset previous, the peer-stats dataset: [/blog/introducing-peer-stats-dataset](/blog/introducing-peer-stats-dataset) Here is a list of our currently available datasets at [https://data.bgpkit.com/](https://data.bgpkit.com/) * `[peer-stats](https://data.bgpkit.com/peer-stats/)`: route collector peers statistics (IP, ASN, v4/v6 prefixes counts) * `[as2rel](https://data.bgpkit.com/as2rel/)`: AS-level relationship, using all available collectors * `[pfx2as](https://data.bgpkit.com/pfx2as/)`: prefix-to-AS mapping, using all available collectors * `[ihr-hegemony](https://data.bgpkit.com/ihr/hegemony/ipv4/global/)`: mirror of [IIJ-IHR](https://ihr.iijlab.net/ihr/en-us)'s global hegemony score dataset (big shout out to [Romain](https://twitter.com/romain_fontugne) and [Internet Health Report](https://twitter.com/ihr_alerts) for producing this data) All the above datasets are free to use for research or commercial usages, and here is the [acceptable usage agreement](https://bgpkit.com/aua). ## More Public Repositories There are more open code repositories set available, some experimental and some for data analysis. You can check out the full list here: [https://github.com/orgs/bgpkit/repositories](https://github.com/orgs/bgpkit/repositories) --- ## Founder's Notes In later 2022, I have joined Cloudflare to continue working on routing security for public benefits. During the first few months, we have built and shipped our [new route-leak detection system](https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/) under the public [Radar](https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar/) platform. BGPKIT suite is now used in production for the BGP data anaysis pipeline at Cloudflare. While working full-time now, I am still committed to maintain the software suite and bringing new features to BGPKIT. For example, this year, I ported back our Kafka support used in Cloudflare to BGPKIT Broker backend. Folks at Cloudflare are doing great things in the open-source realm, and BGPKIT software suite will continue to become more useful to BGP enthuesastics and remain completely open-source. Quote from Cloudflare's [route-leak detection system blog](https://blog.cloudflare.com/route-leak-detection-with-cloudflare-radar). Looking at 2023, here are a few things that I am really excited to work on for BGPKIT suite: 1. continue improve the infrastructure of the system and adding new data processing pipelines 2. continue improve the parser's performance and reliability 3. adding new RFC supports to parser (e.g. [RFC9234](https://datatracker.ietf.org/doc/rfc9234/) for route-leak prevention) 4. productionizing the new API and stats website 5. write more examples and documentations (don't we all love that) And yeah, we will continue to be open-source first! [](https://bgpkit.com) --- If you like what we do here, please consider subscribe to our blog. For all code repositories, check out our GitHub page. [https://github.com/bgpkit](https://github.com/bgpkit)

Introducing Peer-Stats Dataset

Mon, 16 May 2022 00:00:00 GMT

Public BGP data collector projects like RouteViews and RIPE RIS provide valuable research and operational information for understanding BGP and detecting Internet routing anomalies. **There are many BGP routers involved in BGP collection projects.** A project includes many "collectors," and each serves as a collection of messages from several active BGP peers from different networks. Some bigger collectors collect BGP data from more than one hundred BGP routers. For example, RIPE RIS `rrc00` has 112 active BGP peers at writing. *(To learn about the complete list of BGP peers from all collectors, try the*[*experimental tools*](https://github.com/bgpkit/bgpkit-labs/tree/main/collector-peers)*we developed.)* **Sometimes, too many BGP peers may become problematic.** Not all peers present the same amount of data. Some peers are so-called "full-feed" peers, which are the ones that provide the full routing tables to the collector. In a routing table dump file from the collectors, we can observe the full table of these peers. Some peers, however, only provide a limited number of routing entries to the collectors, not representing the whole routing status from these peers. In a project that tries to rebuild full routing tables, e.g., some BGP hijack detection or anomaly detectors, people prefer to use the full-feed peers as their data source. At times, we are only interested in data from certain peers. For example, when studying the routing data from a particular network, if the network connects to BGP data collectors, we can directly pull data from the collectors' data. However, it can be troublesome to learn about what collectors have data from certain peers. RIPE RIS provides a [nice API](https://stat.ripe.net/data/ris-peers/data.json) for querying such info, but we couldn't find one for RouteViews. Historical data for such information is also missing. Unfortunately, for the researchers who want to study the evolution of the data collectors, even RIPE RIS's peers API could not help with that. ## Introducing BGPKIT Peer-Stats Dataset `Peer-Stats` dataset is a publicly available, free-to-use dataset that aims to provide daily collector peer information for all RouteViews and RIPE RIS collectors for ten years. [https://data.bgpkit.com/peer-stats/](https://data.bgpkit.com/peer-stats/) The data includes the following fields for each peer of a BGP collector: 1. `asn`: Autonomous System Number of the collector peer 2. `ip`: the IP address of the collector peer 3. `num_v4_pfxs`: the number of IPv4 prefixes propagated from the collector peer 4. `num_v6_pfx`s: the number of IPv6 prefixes propagated from the collector peer 5. `num_connected_asns`: the number of connected (immediate next hop) ASes from the collector peer The dataset is organized by the following structure. ```plaintext - collector - year - month - data files ``` Screenshots of the dataset file listing site. Each data file is in JSON format (see the section below) and compressed with bzip2. Users can easily use tools like `bzcat` and `jq` to view the data files. For example, you can run the following command to quickly view any of the peer-stats data for the collector `rrc00` on 2022-05-01. ```bash curl "https://data.bgpkit.com/peer-stats/rrc00/2022/05/rrc00-2022-05-01-1651363200.bz2" --silent | bzcat | jq ``` ```json { "collector": "rrc00", "peers": { "102.67.56.1": { "asn": 328474, "ip": "102.67.56.1", "num_connected_asns": 330, "num_v4_pfxs": 919443, "num_v6_pfxs": 0 }, "103.102.5.1": { "asn": 131477, "ip": "103.102.5.1", "num_connected_asns": 184, "num_v4_pfxs": 895482, "num_v6_pfxs": 0 }, ... ``` Because all the data files are generated against the midnight UTC RIB dump of the day, you can also easily construct a URL to a data file for any particular date using the following template. `https://data.bgpkit.com/peer-stats/{COLLECTOR}/{YEAR}/{MONTH}/{COLLECTOR}-{YEAR}-{MONTH}-{DAY}-{MIDNIGHT_TIMESTAMP}.bz2` ## Open-source We also open-sourced the data collection command-line tool source code on GitHub. Feel free to check it out and run it on your infrastructure if needed. [https://github.com/bgpkit/peer-stats](https://github.com/bgpkit/peer-stats) --- ## Credits and Sponsorship The original idea for this work came from our extensive discussion with Romain Fontugne (follow him on Twitter at [@romain\_fontugne](https://twitter.com/romain_fontugne)) from IIJ. This work is made possible by IIJ's generous sponsorship. Please consider sponsoring us on GitHub if you find our work valuable and would like to see more open-source code and datasets on BGP. [https://github.com/sponsors/bgpkit](https://github.com/sponsors/bgpkit)

KhersonTelecom Outage and Connectivity Change

Tue, 03 May 2022 00:00:00 GMT

> > > Internet service in Russian-occupied Kherson, Ukraine was disabled at 16:12 UTC (6:12pm local) on Saturday, 30 April. [#UkraineRussiaWar](https://twitter.com/hashtag/UkraineRussiaWar?src=hash&ref_src=twsrc%5Etfw) > > Khersontelecom service was restored ~24hrs later via Russian transit from nearby Crimea. [pic.twitter.com/uN31jLrzEc](https://t.co/uN31jLrzEc) > > — Doug Madory (@DougMadory) [May 2, 2022](https://twitter.com/DougMadory/status/1521102562509873152?ref_src=twsrc%5Etfw) The `AS47598` experienced an outage shortly after `2022-04-30T16:10:00` and then resumed connectivity `2022-05-01T16:15:00` (both UTC time). After the outage, the `AS47598` is then connected via a different upstream provider, `AS201776`. The upstream provide change can also be seen on IIJ's [Internet Health Report](https://ihr.iijlab.net/ihr/en-us/networks/AS47598?af=4&last=3&date=2022-05-01&rov_tb=routes). Prefixes -------- `AS47598` announces only one prefix `91.206.110.0/23` (data from [Hurricane Electric](https://bgp.he.net/AS47598#_prefixes)). Most of the BGP announcements are for this prefix. However, after the provider change happened, there were also a IPv6 prefix announcements for this V6 prefix as well `5bce:6e00::/23`. It is possible that this prefix is a V4-translated prefix propagated to a V6 collector peer, but we do not know for sure. We can also confirm the outage of the prefix with RIPEstat’s [Routing History data widget](https://stat.ripe.net/widget/routing-history#w.resource=91.206.110.0/23&w.starttime=2022-04-25T00:00:00&w.endtime=2022-05-02T00:00:00). (uncheck the `No low visibility` box to reveal the outage). BGP Messages ------------ We can visualize the overall BGP announcements volume with [Cloudflare Radar’s AS-level page](https://radar.cloudflare.com/asn/47598?date_filter=last_7_days). The following are the UTC timestamps for the corresponding BGP message spikes. * `2022-04-30T16:10:00`: announcements with old provider `12883` in the paths. * `2022-05-01T16:00:00`: announcements with the new provider `201776` in the paths * `2022-05-03T10:45:00`: similar announcements with a new provider in the paths. During the first gap time (2022-04-30T16:15:00 to 2022-05-01T16:00:00), there were 0 BGP updates for the prefix or from the ASN. The old provider paths look like this where `AS12883` is the next hop for `AS47598`. See the full list of messages from `rrc00` here: [https://gist.github.com/digizeph/c58b77f755d7fec8a7969807fb17d5ba](https://gist.github.com/digizeph/c58b77f755d7fec8a7969807fb17d5ba). 207564 56655 3257 12883 47598 The new provider paths look like this where `AS12389` and `AS201776` are the next hops for `AS47598`. See the full list of messages from `rrc00` here: [https://gist.github.com/digizeph/896a4a7e4de23082b496b92ab5bdab5b](https://gist.github.com/digizeph/896a4a7e4de23082b496b92ab5bdab5b) 207564 28824 28824 1299 12389 201776 47598 BGP Data Tooling ---------------- The analysis is done using a privately hosted open-source BGPKIT parser web API. You can host it on your infrastructure, and the source code is freely available at [https://github.com/bgpkit/pybgpkit-api](https://github.com/bgpkit/pybgpkit-api). Comments and feedback are welcome! * * * ### Update on 2022-05-04T09:20:00 Pacific > [#Kherson](https://twitter.com/hashtag/Kherson?src=hash&ref_src=twsrc%5Etfw) — Internet connectivity is returning to the occupied city in South of [#Ukraine](https://twitter.com/hashtag/Ukraine?src=hash&ref_src=twsrc%5Etfw), after an outage since Saturday. [@Cloudflare](https://twitter.com/Cloudflare?ref_src=twsrc%5Etfw) data shows growth in requests since 04:15 UTC, and telecom connection was confirmed by the Ukrainian Vice PM [@FedorovMykhailo](https://twitter.com/FedorovMykhailo?ref_src=twsrc%5Etfw). [pic.twitter.com/JxT3kcM234](https://t.co/JxT3kcM234) > > — Cloudflare Radar (@CloudflareRadar) [May 4, 2022](https://twitter.com/CloudflareRadar/status/1521812037055176705?ref_src=twsrc%5Etfw) The upstreams for `AS47598` has reverted back to the original ASes, and the traffic has started coming back to normal.

Parallel MRT Files Parsing with BGPKIT

Wed, 16 Mar 2022 00:00:00 GMT

In this post, we will talk about how to implement a Rust workflow that can process a **large number** of BGP data files **as fast as we can**. We will use BGPKIT [Parser](https://bgpkit.com/parser) and [Broker](https://bgpkit.com/broker) for data collection and parsing, and [Rayon](https://github.com/rayon-rs/rayon) crate for parallelization of the code. ## Task Overview Before we begin to talk about the code design, we first need to introduce the data we are dealing with. We want to process the BGP data collected by various collectors, saved in compressed binary MRT format, and archived to files with a fixed interval. In this post, we are using [RouteViews](http://archive.routeviews.org/) archive data as an example. The average data file size ranges from 2MB to 10MB by different collectors ([AMSIX collector](http://archive.routeviews.org/route-views.amsix/bgpdata/2021.10/UPDATES/) for example has pretty large files). For processing, we can use the simplest task possible to do: *sum the number of MRT records in all the files*. We want to download and process all the updates files for one hour from all the collectors in RouteViews project. Here is the estimated amount of data we are dealing with: * 35 collectors * 5-minute interval — 12 files per collector * **420 total number of files** to download and process * **840MB to 4.2GB total download size** (it’s somewhere in between) Ok! Now that we know what we want to do and have a sense of the estimated workload of the overall task, let’s coding! Photo by [Glenn Carstens-Peters](https://unsplash.com/@glenncarstenspeters?utm_source=medium&utm_medium=referral) on [Unsplash](https://unsplash.com/?utm_source=medium&utm_medium=referral) --- ## 1\. Sequential Parsing Our first attempt to achieve the goal is to design and implement a naive sequential workflow as described below 1. find all BGP updates files within the hour of interest 2. iterate through each file, parse the MRT data and count the number of records 3. sum all record counts and print out the result For this sequential workflow, we will need to pull in two dependencies into `Cargo.toml`: ```ini [dependencies] bgpkit-parser = "0.7.2" bgpkit-broker = "0.3.2" ``` The `bgpkit-broker` handles looking for updates files within the hour, while `bgpkit-parser` handles parsing each individual file. ### Finding files BGPKIT Broker indexes all available BGP MRT data archive files from both RouteViews and RIPE RIS in close-to-real-time. For each data file, it saves the following information: * `project`: `route-views` or `riperis` * `collector`: the collector ID, e.g. `rrc00` or `route-views2` * `url`: the URL to the corresponding MRT file * `timestamp`: the UNIX time of the *start time* of the MRT data file. With all this information indexed, we can then query the backend and retrieve files information as we want. BGPKIT Broker provides both [RESTful API](https://docs.broker.bgpkit.com), [Rust API](https://github.com/bgpkit/bgpkit-broker), as well as a [Python API](https://pypi.org/project/pybgpkit/). Here we use the Rust API to pull in the information we need: ```rust let broker = BgpkitBroker::new_with_params( "https://api.broker.bgpkit.com/v1", QueryParams { start_ts: Some(1640995200), end_ts: Some(1640998799), project: Some("route-views".to_string()), data_type: Some("update".to_string()), ..Default::default() }); for item in &broker { println!("processing {:?}...", &item); } ``` The above block queries the broker and prints out all information of the retrieved files' metadata. The `BgpkitBroker::new_with_params` call accepts two parameters, one for the endpoint of the broker instance, and the other specifies the filtering criteria. In this example, we search for all BGP updates files from RouteViews with timestamps between `2022-01-01T00:00:00` and `2022-01-01T00:59:59` UTC. It prints out the output as the following: ```plaintext processing BrokerItem { collector_id: "route-views.telxatl", timestamp: 1640997000, data_type: "update", url: "http://archive.routeviews.org/route-views.telxatl/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }... processing BrokerItem { collector_id: "route-views.uaeix", timestamp: 1640997000, data_type: "update", url: "http://archive.routeviews.org/route-views.uaeix/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }... processing BrokerItem { collector_id: "route-views.wide", timestamp: 1640997000, data_type: "update", url: "http://archive.routeviews.org/route-views.wide/bgpdata/2022.01/UPDATES/updates.20220101.0030.bz2" }... processing BrokerItem { collector_id: "route-views2", timestamp: 1640997900, data_type: "update", url: "http://archive.routeviews.org/bgpdata/2022.01/UPDATES/updates.20220101.0045.bz2" }... ``` ### Parse each MRT file Previously, in the for loop, we only print out the retrieved meta information of the MRT files. Now let's add the actual parsing of the files into the loop. The code is very simple, as designed: ```rust let mut sum: usize = 0; for item in &broker { println!("processing {}...", &item.url); let parser = BgpkitParser::new(&item.url).unwrap(); let count = parser.into_record_iter().count(); sum += count; } ``` We first define a mutable variable `sum` outside the loop. Then for each file, we create a new Parser instance by `BgpkitParser::_new_(&item.url)`. Here, as our goal is to count the number of records, we call the parser's `.into_record_iter()` function to create a iterator over the records of the file, and then `.count()` to get the count of the records. Lastly, we add the count to the overall sum variable. ### Run and timing For testing, I use a fairly powerful VM on a host with AMD 3950x CPU (32 threads), then build the release build and time the release run. The runtime includes downloading the MRT files to my machine with 1Gpbs down link in Southern California. ```bash cargo build --release time cargo run --release --bin sequential ``` It ended up taking about **1 minute and 23 seconds** to sequentially parse 144 MRT files from RouteViews for all available ones within the first hour of 2022 (UTC). ```plaintext total number of records for 144 files is 10554212 real 1m23.081s user 0m39.535s sys 0m1.006s ``` ## 2\. Parallel Parsing Since the parsing of each file is completely independent of each other, we can parse the files in parallel and then sum up the count for each thread at the end. In Rust with Rayon, this conversion is very simple. Let's first add the dependency of Rayon first: ```ini [dependencies] bgpkit-parser = "0.7.2" bgpkit-broker = "0.3.2" rayon = "1.5.1" ``` Then we change the broker code one tiny bit to collect all meta information for files into a vector first. ```rust let items = broker.into_iter().collect::>(); ``` This would enable us to fully utilize `rayon`'s great syntax sugar to turn our sequential code into a parallel one. ```rust let sum: usize = items.par_iter().map(|item| { println!("processing {}...", &item.url); let parser = BgpkitParser::new(&item.url).unwrap(); let count = parser.into_record_iter().count(); count }).sum(); ``` The key difference here is the calling of `.par_iter()`. It turns a sequential iterator into a parallel iterator, and by default going to utilize all available cores on the host machine for scheduling. Then we call `.map()` to define the parsing steps for each file, and then call `.sum()` at the end to add all results up. The final result is approximately **10x faster** than the sequential version, and it took only **8 seconds** to parse all MRT files and get the record counts. ```plaintext total number of records is 10554212 real 0m8.086s user 0m42.569s sys 0m1.068s ``` The full code for this example is as follows: --- The source code of the two examples is available on GitHub. Feel free to poke around and tweak it as you wish. %[https://gist.github.com/digizeph/c23ba39968c6cb4e1ad323520540010f#file-parallel-parsing-rs] [https://github.com/bgpkit/bgpkit-tutorials/tree/main/parallel-parsing](https://github.com/bgpkit/bgpkit-tutorials/tree/main/parallel-parsing)

Real-time RIS Live Data with BGPKIT Parser

Fri, 12 Nov 2021 00:00:00 GMT

In terms of real-time BGP data processing, [RIPE NCC](https://www.ripe.net/) provides a great data source: [Routing Information Service Live (RIS Live)](https://ris-live.ripe.net/). To begin with, here is what [RIS Live](https://ris-live.ripe.net/) by the creators: > RIS Live is a feed that offers BGP messages in real-time. It collects information from the RIS Route Collectors (RRCs) and uses a WebSocket JSON API to monitor and detect routing events around the world. A non-interactive full stream (“firehose”) is also available. In essence, RIS Live provides: * a WebSocket interface to stream BGP messages in real-time * ability to subscribe to “sub-streams” with custom filtering messages * JSON-encoded BGP messages as the stream payload * “firehose” HTTPS stream interface as well, without needing to work with websocket. In this post, we will discuss how to use the RIS Live stream in practice. ## RIS Live Message Format RIS Live has *client messages* and *server messages*. The client messages is used to setup or dismantle “subscriptions”, which essentially tell the server what kind of BGP messages a client would like to receive, and allow the server to send only the interested messages to the client. A server acknowledges the requests from the client and afterwards start streaming requested data back to the client. At a high-level, a server sends either `ris_message` or `ris_error` messages. The `ris_message` is the main payload that we are interested in, while the `ris_error` message provides debugging messages for the scenarios where stream or subscription fails. ## Subscribe to a WebSocket Stream RIS Live provides great flexibility for the clients to specify/narrowdown the interested messages, allowing both the server and client process less messages during a streaming session. * `host`: only messages collected from a particular RRC (e.g. `rrc21`) * `type`: only messages of a given type, e.g. `UPDATE` , `OPEN` * `require` : only messages containing a given key, e.g. `withdrawals` will return only message that contains any withdrawn prefixes * `peer`: messages from a particular BGP peer * `path` : ASN or pattern to match the AS Path attribute in BGP update messages * `prefix`: only messages containing information for a given prefix * `moreSpecific` and `lessSpecific`: only messages that are the subprefix or super-prefix of the specified prefix * `includeRaw`: whether to include the Base64-encoded RAW BGP messages As an example, let’s take a look at the following message from the official manual: ```json { "host": "rrc01", "type": "UPDATE", "require": "announcements", "path": "64496,64497$" } ``` Example subscription message composer on [RIS Live official site](https://ris-live.ripe.net/). As an example, let’s take a look at the following message from the official manual: ```json { "host": "rrc01", "type": "UPDATE", "require": "announcements", "path": "64496,64497$" } ``` * collected by `rrc01` * BGP UPDATE messages * have at least one announced prefix * the last two hops of the AS Path is 66496 and 64497 (the origin) The `ris_message` consists of “common header” fields and “data” fields (although they’re on the same level). The “common header” fields are present for all types of sub-type messages, including `timestamp`, `peer`, `peer_asn`, `id`, `host`, `type` . The rest of the fields are the data fields that are dependent on the type of the messages. For most people, the `UPDATE` message is what they need. The following JSON block is an example message pulled directly from the demo site. Example JSON formatted RIS message: This example shows a BGP announcement of AS132354 originating two prefixes `103.249.208.0/23` and `103.14.184.0/24` , with the next hop to be `37.49.237.228`. At this point, the information we see here is pretty similar to what we can see from other BGP MRT reader’s output (e.g. from `bgpdump` or `bgpreader`), just in JSON format. ## WebSocket or Firehose? Provided that RIS Live provides both WebSocket and HTTP Firehose, one would naturally wonder which one is the right choice for their application. Here we have a brief comparison between the two in the context of RIS Live. **WebSocket** Good: * easy to customize stream by composing a simple JSON subscribe message * work with various toolings in languages like Python and JavaScript Bad: * requires extra library dependencies to work with WebSocket * need to write somewhat lengthy to get started (comparing to firehose) **Firehose** Good: * easy to consume by simply calling GET request on the URL * simple single-liner commandline program can start the stream (e.g. a simple `curl` call), no need complex script Bad: * customizing stream is doable with `XRIS-SUBSCRIBE` HTTP request header, but feels clunky and limited * in my personal tests, the stream get disconnected often due to the stream cannot keep up with the data producer. this did not happen with websocket tests. ### Summary If your application could afford additional dependencies or writing extra scripts, WebSocket is the better choice. RIS Live official manual also makes implication that the WebSocket format is the current formally-supported streaming method. --- ## RIS Live Coding Example with BGPKIT Parser Now that we have a basic idea of what is RIS Live and the basic message format, we can get started working on some code that will actually use RIS Live to do something useful. In the following example, we will build a short monitoring service that alerts us when Facebook operators announces their DNS IP prefix (see what happened before [here](https://blog.cloudflare.com/october-2021-facebook-outage/)). We are going to build the service in Rust with [BGPKIT Parser](https://bgpkit.com/parser) , WebSocket library [Tungstenite](https://github.com/snapview/tungstenite-rs). First, lets collect some basic information about what we are going to monitor here: 1. Facebook’s autonomous system number is 32934. So we will watch for all messages that was originated from AS32934. 2. Facebook’s DNS server IP prefixes involved in the previous incidence are `129.134.30.0/23` and `185.89.218.0/23` . So we want to carefully watch these two prefixes in our monitoring system. 3. We want to use one of the RIPE RIS collectors’ data for monitoring, `rrc21` is a good choice since it’s being used by RIS Live’s demonstration. You can easily extend this service by tweaking the subscription message later. OK, we are good to go. Let’s do it! ## Setting up the stream We picked the Tungstenite library as our WebSocket library of choice, partly because it has a very straightforward API design. Let’s first connect to the websocket server by calling `connect` function given a websocket URL. One thing to notice is that the URL protocol section here is `ws` as opposed to the `wss` mentioned in the RIS Live documentation. For some reason, Tungstenite does not work with `wss` protocol (with SSL). ```rust use tungstenite::{connect, Message}; const RIS_LIVE_URL: &str = "ws://ris-live.ripe.net/v1/ws/?client=rust-bgpkit-parser"; let (mut socket, _response) = connect(Url::parse(RIS_LIVE_URL).unwrap()) .expect("Can't connect to RIS Live websocket server"); ``` Now, with a socket ready, we will first send a subscription message to let server know that we want some messages and we are ready to receive. ```rust let msg = json!({"type": "ris_subscribe", "data": {"host": "rrc21"}}).to_string(); socket.write_message(Message::Text(msg)).unwrap(); ``` Here we composed a simple subscription message that limits the stream to have messages only from `rrc21` collector. ## Parsing JSON messages At this point, we have a WebSocket connection to RIS Live server, and have sent out a subscription message to the server. The server should be sending back messages anytime now, and we are ready to consume the stream. We would like to code the following behavior: 1. continuously reading the websocket messages; 2. parse JSON string into internal BGP structs; 3. check each message if it contains origins (withdraw-only messages does not contain AS paths, and thus no origins either; 4. if the origin AS is AS32934, and the announced prefix is `129.134.30.0/23` or `185.89.218.0/23` , then we print out the message to output. ```rust loop { let msg = socket.read_message().expect("Error reading message").to_string(); if letOk(elems) = parse_ris_live_message(msg.as_str()) { for elem in elems { if letSome(origins) = elem.origin_asns.as_ref() { if origins.contains(&32934) && ( elem.prefix.to_string() == "129.134.30.0/23".to_string() || elem.prefix.to_string() == "185.89.218.0/23".to_string() ) { println!("{}", elem); } } } } } ``` The full example code can be found here: %[https://gist.github.com/digizeph/fcac3027555c0b744ea0b3a11197b694#file-bgpkit-parser-kafka-bmp-example-rs] --- ## Building More with BGPKIT Tools As introduced in our [previous blog post](/blog/real-time-bgp-data-processing-1-bmp-and-openbmp), we added support of real-time BMP stream to BGPKIT Parser as well. Combining with RIPE RIS Live, and RouteViews BMP stream, we can build a powerful real-time BGP monitoring service directly within BGPKIT Parser. We also offer indexing and processing of historical BGP data as well with [BGPKIT Broker](https://bgpkit.com/broker). Our goal at BGPKIT is to design, develop, and deploy the most developer-friendly BGP data processing toolkit. To learn more about our offerings, please check out our website and official [Twitter account](https://twitter.com/bgpkit).

Real-time BMP with BGPKIT Parser

Wed, 10 Nov 2021 00:00:00 GMT

Real-time BGP data processing is very critical on building monitoring services that can detect BGP issues quickly with minimum delay and react to anomalies quickly and mitigate potential issues. We are creating a new series of posts describing how we design our software to work with real-time BGP data streams. As an opening, we will describe how we handle data streams with BMP protocol and OpenBMP messages. --- # BMP The BGP Monitoring Protocol (BMP) is a protocol that allows monitoring of BGP devices. The [RFC7854](https://datatracker.ietf.org/doc/html/rfc7854) describes the purpose of BMP as: > Many researchers and network operators wish to have access to the contents of routers’ BGP Routing Information Bases (RIBs) as well as a view of protocol updates the router is receiving. This monitoring task cannot be realized by standard protocol mechanisms. Prior to the introduction of BMP, this data could only be obtained through screen scraping. > > BMP provides access to the Adj-RIB-In of a peer on an ongoing basis and a periodic dump of certain statistics the monitoring station can use for further analysis. From a high level, BMP can be thought of as the result of multiplexing together the messages received on the various monitored BGP sessions. There are multiple types of BMP messages, each serving different purposes. * Peer up and down notification: notification about the status of peering sessions to a monitored router; * Initiation message: inform the monitoring station of the routers vendor, software version, and so on; * Termination message: provides information on why a monitored router is terminating a session; * **Route monitoring**:initial synchronization of the routing table; * **Route mirroring**: verbatim duplication of messages as received. For real-time BGP data processing, we are specifically interested in the route monitoring and route-mirroring messages, as we provide the routing information encoded as actual BGP messages. --- # OpenBMP [OpenBMP](https://www.openbmp.org/) is a software implementation of the BMP protocol. It is an open-source project created by Cisco and currently maintained by nice folks from [CAIDA/UCSD](https://www.caida.org/) and [RouteViews](http://routeviews.org/). It is implemented in C++, can be used with any compliant BMP sender (e.g., router). Architecture graph of OpenBMP OpenBMP provides multiple formats for outputting the BMP messages collected from the connected routers, one of which is the `raw_bmp` format, which is a thin wrapper of the raw BMP messages. The `raw_bmp` format provides the best performance and allows use to handle the BMP messages directly without having to write a different parser for the plaintext messages. RouteViews currently provides a OpenBMP Kafka stream that streams BMP messages from their collectors. --- # BGPKIT Parser with BMP/OpenBMP Support We develop BGPKIT Parser to provide a one-stop solution for handling all parsing tasks regarding BGP data. Supporting real-time data like BMP is a very important milestone for us. We have recently developed the full support for BMP messages, and partial support for OpenBMP messages (for `raw_bmp` type only). This enables us to start working with real-time BMP streams like RouteViews’ Kafka stream. Below is an example code that takes RouteViews’ Kafka OpenBMP stream and parse the messages into internal data structures: ```rust let mut reader = Cursor::new(Vec::from(kafka_payload)); let header = parse_openbmp_header(&mut reader).unwrap(); if let Ok(msg) = parse_bmp_msg(&mut reader) { info!("Parsing OK: {:?}", msg.common_header.msg_type); match msg.message_body { MessageBody::RouteMonitoring(m) => { dbg!(m.bgp_update); } _ => {} } } ``` Here is a break down of what it does: * it first creates a bytes reader from the raw Kafka message payload; * then parse OpenBMP message header, which contains some basic information about the BMP session; * then it calls the `parse_bmp_msg` function to parse the embedded raw BMP messages and print out the BGP update messages if the parsing is successful. Here is a full code example: %[https://gist.github.com/digizeph/fcac3027555c0b744ea0b3a11197b694#file-bgpkit-parser-kafka-bmp-example-rs] We have published the SDK on [crates.io](https://crates.io/crates/bgpkit-parser) and [GitHub](https://github.com/bgpkit/bgpkit-parser). Feel free the check out the example code at [examples/routeviews-kafka.rs](https://github.com/bgpkit/bgpkit-parser/blob/main/examples/real-time-routeviews-kafka-openbmp.rs) if you are interested.

Introducing BGPKIT Parser

Mon, 01 Nov 2021 00:00:00 GMT

BGPKIT Parser is an open-source Rust-based MRT/BGP data parser that takes a [MRT](https://datatracker.ietf.org/doc/html/rfc6396) formatted binary file and turns it into BGP messages. It is one of the most important building block software that enables BGP data processing and analysis tasks. ## Design and Features As mentioned in our previous post [introducing BGPKIT Broker](https://medium.com/bgpkit/introducing-bgpkit-broker-b734dac4661e), the most used BGP data collection projects, RouteViews and RIPE RIS, both publish their collected BGP data in MRT format on their data platform. The BGPKIT Parser is designed to handle parsing tasks for these data sources. We design our parser to strictly follow the industry standard, e.g. [RFC4271](https://datatracker.ietf.org/doc/html/rfc4271) and [RFC6396](https://datatracker.ietf.org/doc/html/rfc6396). BGPKIT Parser is also designed with the following goals: * **performant**: comparable to C-based implementations like `bgpdump` or `bgpreader`. * **actively maintained**: we consistently introduce feature updates and bug fixes, and support most of the relevant BGP RFCs. * **ergonomic API**: a three-line for loop can already get you started. * **battery-included**: ready to handle remote or local, `bzip2` or `gz` data files out of the box. * **open-source**: we want people to use our parser freely and we can continue develop and improve it based on community feedbacks. To demonstrate how easy it is to get started using BGPKIT Parser, check out the example below where we print out all BGP messages from a remote MRT file on RouteViews: Example of reading a remote MRT file from RouteViews and print out BGP messages. Code available at [https://gist.github.com/digizeph/9977371653f39a459ff3ae507dc3636c](https://gist.github.com/digizeph/9977371653f39a459ff3ae507dc3636c) There are a number of things happens when we call `for elem in BgpkitParser::new(url)` : 1. it creates a new BgpkitParser struct instance with the provided URL to the data file; 2. it tries to retrieve the content of the remote file and download the raw compressed bytes into memory; 3. it determines the compression type by file suffix and calls corresponding decompression library to create a buffered reader; 4. it then creates an iterator (used by the for loop) that continuous return new parsed items until it reaches the end of the data stream. We determined that it is worth the extra binary file size to bring in the network and compression libraries into the project so that the library users will never have to worry about handling data downloading and decompression by themselves again. ## The Future The future of the BGPKIT Parser lies on continuous performance and stability improvements, as well as some exciting features that we are currently planning. Some of the coming features include 1. adding capability of handling **real-time data streams** coming from RIPE RIS Live and RotueViews’s Kafka BMP stream; 2. supporting **data serialization** back to MRT files (reverse-parsing), which allows users to produce customized MRT files after data processing; 3. adding **WASM support** to allow BGP data parsing directly on the web with JavaScript. Because we are building our software in Rust, we can effortlessly tapping into Rust’s great software ecosystem and continue introducing new features and improvements. The future of BGPKIT Parser is exciting and we can’t wait to bring more features for you to try out! For more details about the BGPKIT Parser, check out our GitHub repo and our website. Feedback is highly appreciated! [https://github.com/bgpkit/bgpkit-parser](https://github.com/bgpkit/bgpkit-parser)

Introducing BGPKIT Broker

Sun, 31 Oct 2021 00:00:00 GMT

BGPKIT Broker is a data API service that focuses on building a BGP data file index to enable searching for public/private BGP data files with custom filters. It is one of the building block components designed by BGPKIT to facilitate BGP data processing with ease. ## The first step to investigate a BGP event. Imagine this scenario: a malicious player just attempted to hijack a IP prefix using BGP announcements, and you are interested in learning what exactly happened during that half-hour down time of the victim network, how would you start investigating? **Collecting evidences**. Luckily for us, the actors on the Internet, good and bad, always leave traces behind if they use BGP as their method. There are a number of **reputable public BGP route collectors** operating for years collecting all BGP messages received from their connected router peers and dump them into regular dump files. The most used projects are [RouteViews](http://routeviews.org/) and [RIPE RIS Data](https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data). **Files are all over the places.** There are more than 60 different data collectors from the two projects alone, and each publishing their data under separate sites. The two projects also have different data file structure and compression algorithms for their data dump files. It is not hard to go find one data file during the interested event time from one collector, but it will be a real hassle to gather URLs toward **all data files** that include information within a specified time range. Collector’s data is published as compressed MRT files at regular frequency. ## BGPKIT Broker — A BGP Data File Index API Service We designed the BGPKIT Broker to resolve one and only one problem: quickly collect links to the BGP data files that matches a filtering criteria. You can filter BGP data using multiple criteria: * `start_ts` : UNIX timestamp that all files must be dumped after * `end_ts` : UNIX timestamp that all files must be dumped before * `data_type` : the type of the data file, can be `update` or `rib` * `collector`: the collector ID that the files are generated from * `project`: the data collection project, can be `route-views` or `riperis` * `page` and `page_size`: the pagination control for collecting a large number of files Here is an example REST API call: [https://api.broker.bgpkit.com/v1/search?data\_type=update&start\_ts=1633046400&end\_ts=1633132800&collector=rrc00&project=riperis&page%20=2&page\_size=3](https://api.broker.bgpkit.com/v1/search?data_type=update&start_ts=1633046400&end_ts=1633132800&collector=rrc00&project=riperis&page%20=2&page_size=3) It asks for all data **updates** files dumped between **1633046400** and **1633132800**, from **RIPE RIS**’s collector **rrc00**. It also requested for the **second page** of the results and **each page contains 3 items**. **DEPRECATED**: BGPKIT Broker API service is freely available to use, hosted at [https://api.broker.bgpkit.com/v1/](https://api.broker.bgpkit.com/v1/). The documentation is available at [https://docs.broker.bgpkit.com/](https://docs.broker.bgpkit.com/) **BGPKIT Broker API and SDK has been upgraded to V2 now. The V1 examples are left up for legacy services that depends on it. Please check out the current API documentation at for more:**[**https://api.broker.bgpkit.com/v2/**](https://api.broker.bgpkit.com/v2/) ## BGPKIT Broker Rust API The BGPKIT Broker API service is built entirely using Rust, and of course we also developed native Rust API to access the broker data with ease using Rust. Example BGPKIT Broker Rust API call The Rust API is open source under MIT license, and with the free API, you can already build your own workflow today! [https://github.com/bgpkit/bgpkit-broker](https://github.com/bgpkit/bgpkit-broker) ## Easy On-premise Deployment For API service like this, especially that it also collects and indexes data of over 10 years span, one might imagine the deployment could be complex and slow. For BGPKIT Broker, we spent extra efforts to make the API deployment process as quickly as possible, and also efficient on resource consumption. For references, we bootstrapped the entirety of the database in **under 5 minutes**, and cost **less than 1 GB storage** to store the information in PostgreSQL database. The whole database and API run fluently with **less than 500MB of RAM** usage. This allows use to deployment extra instances when under heavy load without costing a fortune. Users who need dedicated resource allocation for query performance can contact us for private API hosting that are not shared by others. Enterprise option is also available for on-premise deployment with customization consultation available. If you are interested in testing it out, feel free to shoot us an email at contact@bgpkit.com. For more information, checkout our website! [https://bgpkit.com/broker](https://bgpkit.com/broker)

BGPKIT Journey Started

Tue, 26 Oct 2021 00:00:00 GMT

[BGPKIT](https://bgpkit.com/) is a small-team start-up that aims to provide comprehensive tool suite to facilitate companies building on-premise BGP data monitoring services. We started our journey of building the best BGP data toolkit for developers in October, 2021. Here is a brief glance on what we are working on and what values we strive to provide. [](https://bgpkit.com) # BGP Border Gateway Protocol (BGP) is the de facto inter-domain routing protocols being used by every major companies on the Internet. **The main purpose of BGP is to allow companies exchange IP prefix reachability information.** In other words, companies tell other companies what IP blocks they have, and how to reach these IP blocks. This the key functionality that enables the Internet. Because the purpose of BGP is to exchange information and allow everyone to know how to reach certain IP blocks, the BGP messages must be propagated globally and publicly. There are a number of data sources available out there that provides BGP data archives (e.g. [RouteViews](http://www.routeviews.org/routeviews/) and [RIPE RIS](https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data)), or real-time data like BGP [looking glass](https://lg.he.net/)es or [live BGP data stream](https://ris-live.ripe.net/). The aforementioned data sources contains a wealth of information if you know what to look for. For example, by looking at BGP information, researchers can [infer companies relationships](https://www.caida.org/catalog/datasets/as-relationships/), monitor [security](https://www.bgpmon.net/) [incidences](https://radar.qrator.net/). Having handy toolkit in hand enables people to build successful businesses around BGP data. # BGPKIT At BGPKIT, we build software tools to process BGP data and reveal insights from BGP messages. We aims to provide the best developer experience, and enable customers to build their own BGP data processing pipeline and monitoring services on-premise. ## Complete Tool Suite Our goal is to build complete tool suite for BGP data processing: data collection, parsing, analysis, programmable and visual interface, and data warehousing. Everything you need to handle BGP data. ## Rust Implementation To achieve the best performance and security, we choose to focus on building our tools using the [Rust programming language](https://www.rust-lang.org/). We believe that Rust’s ecosystem is now mature enough that building modern features like data streaming, async data workflow, parallel processing, web API, or even porting the entire codebase onto WASM and running it on browsers is a archivable task with reasonable efforts. ## Powerful Extensibility At BGPKIT, we design our libraries to provide powerful API and assist customers to further customize workflow to meet individual needs. We strive to provide the most ergonomic interfaces that allow library consumers to easily integrate our library into theirs. ## Embrace Open-Source We also believe that good tools empowers people, so we open-sourced our building block libraries to the public with very permissive license so that any interested parties can explore and build their own ideas free of charge and limitation. We are excited to start this journey of building, and we hope to have the chance to work with more people and building more dreams together! Follow us here, on [Twitter](https://twitter.com/bgpkit), or visit our website to learn more!