delta.sharing

delta.sharing reads Delta Sharing tables from R. Discover shares, schemas, and tables, then read snapshots or change data feeds as tibble, data.frame, or Arrow objects.

Reads are powered by Delta Kernel, including support for deletion vectors and column mapping.

See the package website for the guides and the complete reference.

Installation

Install the released version from CRAN:

install.packages("delta.sharing")

Or install the development version from GitHub:

# install.packages("pak")
pak::pak("zacdav-db/delta-sharing-r")

Installation from source requires Cargo and rustc >= 1.88.

Quick start

The public example server needs no registration or private credential:

library(delta.sharing)

client <- sharing_client(demo_profile())
housing <- client$table("delta_sharing.default.boston-housing")

housing$snapshot(
  columns = c("chas", "medv"),
  limit = 5
)$to_tibble()

demo_profile() retrieves the public profile maintained by the Delta Sharing project.

For your own share, pass the path to its profile, discover the available tables, and create a reusable table handle:

client <- sharing_client("~/config.share")
client$list_tables("sales", "default")

orders <- client$table("sales.default.orders")

The Getting started guide walks through profiles, discovery, table metadata, and reads.

Read snapshots and changes

Read the latest snapshot, optionally selecting columns and limiting rows:

orders_tbl <- orders$snapshot(
  columns = c("order_id", "status", "amount"),
  limit = 1000
)$to_tibble()

Snapshots can also target a specific version or timestamp:

orders$snapshot(version = 42)$to_tibble()
orders$snapshot(timestamp = "2026-01-01T00:00:00Z")$to_tibble()

Read an inclusive change data feed range:

changes_tbl <- orders$changes(
  starting_version = 120,
  ending_version = 125
)$to_tibble()

to_tibble() is the usual choice for R analysis. Use to_data_frame() when a base R data.frame is required. Both automatically return BIGINT columns as bit64::integer64, including small values, empty results, and nested columns. The value -9223372036854775808 raises a conversion error because bit64 reserves it for missing values; use an Arrow materializer to retain it.

For Arrow workflows, to_arrow() returns an in-memory arrow::Table and to_arrow_reader() returns a lazy arrow::RecordBatchReader. The arrow package is a required dependency. to_arrow_stream() exposes the lower-level Arrow C stream interface.

Selected files are downloaded concurrently and cached for the R session. See the Performance and caching guide for cold and repeated reads, cache lifetime, concurrency, batching, and tuning.

Query with DuckDB

DuckDB can query a lazy Arrow reader without first creating an R data.frame. This requires the optional DBI, duckdb, and withr packages.

snapshot <- housing$snapshot(
  columns = c("chas", "medv")
)
reader <- snapshot$to_arrow_reader()

con <- DBI::dbConnect(duckdb::duckdb())
duckdb::duckdb_register_arrow(con, "housing", reader)

summary <- withr::with_options(
  list(arrow.use_threads = FALSE),
  DBI::dbGetQuery(con, "
    SELECT chas, count(*) AS homes, avg(medv) AS mean_value
    FROM housing
    GROUP BY chas
    ORDER BY chas
  ")
)
summary
#>   chas homes mean_value
#> 1    0   471   22.29553
#> 2    1    35   30.17500

duckdb::duckdb_unregister_arrow(con, "housing")
DBI::dbDisconnect(con)

Once registered, let Arrow manage the reader's lifetime: do not call reader$Close() while a scanner may still be reading ahead.

Use snapshot$to_arrow() instead when the same result will be queried more than once. This materializes the result in Arrow memory.

Performance

These results are medians of 3 end-to-end to_tibble() snapshot reads using 4 concurrent downloads. The cached read repeats the same query after its selected files have been staged locally.

Apple M2 Pro (12 cores), 32 GB RAM, R 4.5.1; VPN connection: 92 Mbps down, 111 ms base round-trip latency.

Rows	R result size	Empty cache	Cached
10,000	0.38 MiB	5.15 s	0.82 s
1,000,000	38.1 MiB	8.16 s	1.69 s
10,000,000	381 MiB	28.3 s	6.45 s

Each end-to-end measurement includes the Delta Sharing request, local log construction, Delta Kernel scan, and tibble materialization. Reproduce the benchmark with bench/snapshot.R.