Rust library

Oneio

All-in-one IO for Rust: read/write local and remote files, with transparent compression support and optional sync/async APIs.

GitHub Crates.io Docs.rs

Quick start

Read a remote, compressed file (HTTP + gzip) into a string. OneIO auto-detects compression from the file extension.

Read a remote .gz file

use oneio;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Works with compression and remote files automatically
    let content = oneio::read_to_string(
        "https://spaces.bgpkit.org/oneio/test_data.txt.gz"
    )?;
    println!("{}", content);
    Ok(())
}

What OneIO is for

• Download + read files the same way (local paths or URLs)
• Stream with readers/writers to avoid loading everything into memory
• Keep dependency surface small with feature flags

Installation

Add `oneio` to your project, then enable only the features you need. The examples below follow the upstream README.

Default

Cargo.toml

# Cargo.toml
[dependencies]
oneio = "0.20"  # Default: gz, bz, https

Minimal / custom feature sets

Feature selection

# Local files only
oneio = { version = "0.20", default-features = false, features = ["gz", "bz"] }

# HTTPS with default rustls
oneio = { version = "0.20", default-features = false, features = ["https", "gz"] }

# S3-compatible storage
oneio = { version = "0.20", default-features = false, features = ["s3", "https", "gz"] }

# Async operations
oneio = { version = "0.20", features = ["async"] }

Examples

Common usage patterns: read lines, get a streaming reader, and write with automatic compression.

Read line by line

read_lines()

use oneio;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let lines = oneio::read_lines("https://spaces.bgpkit.org/oneio/test_data.txt.gz")?
        .map(|line| line.unwrap())
        .collect::<Vec<String>>();

    println!("{} lines", lines.len());
    Ok(())
}

Streaming reader

get_reader()

use oneio;
use std::io::Read;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut reader = oneio::get_reader("tests/test_data.txt.gz")?;
    let mut buffer = Vec::new();
    reader.read_to_end(&mut buffer)?;
    println!("read {} bytes", buffer.len());
    Ok(())
}

Write with compression

get_writer()

use oneio;
use std::io::Write;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut writer = oneio::get_writer("output.txt.gz")?;
    writer.write_all(b"Hello, compressed world!")?;
    drop(writer); // Important: close the writer

    let content = oneio::read_to_string("output.txt.gz")?;
    assert_eq!(content, "Hello, compressed world!");
    Ok(())
}

Async APIs (feature: async)

Async IO

use oneio;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let content = oneio::read_to_string_async("https://example.com/data.json.gz").await?;
    println!("{}", content);

    oneio::download_async(
        "https://example.com/data.csv.gz",
        "local_data.csv.gz"
    ).await?;

    Ok(())
}

Note: Async compression support is limited (see upstream README for details).

Features

OneIO is designed to be compiled “just enough” for your use case (protocols, compression formats, TLS backend, async, etc.).

Unified IO for local + remote

Read and write local files, HTTP/HTTPS, and optional backends like S3/FTP behind a single API.

Automatic compression handling

Detects compression by extension (.gz, .bz2, .lz4, .xz, .zst, …) and reads/writes accordingly.

Sync + async support

Use synchronous helpers for CLI/tools, or async APIs for services and pipelines (feature: async).

Feature-flag friendly

Compile only what you need (compression/protocol/TLS options) to keep builds minimal.

CLI

OneIO also ships a small command-line tool (feature: `cli`) for inspecting, downloading, and piping data.

Install

Install CLI

# Install the CLI tool
cargo install oneio --features cli

Usage

Examples

# Read and print a remote compressed file
oneio https://example.com/data.txt.gz

# Download a file
oneio -d https://example.com/largefile.bz2

# Pipe to other tools (example)
oneio https://api.example.com/data.json.gz | jq '.results | length'