A general model for single-cell omics data — built from axes and fields — and the lightweight glue that moves data losslessly between AnnData, Seurat, SingleCellExperiment, and pagoda/conos, including their disk-backed forms (backed AnnData, Seurat v5/BPCells, SCE/HDF5Array) — so even datasets too large for memory convert in bounded memory.
L★ represents a dataset as axes (the entities you index by — cells, genes, samples, clusters) and
fields (typed data over them — counts, embeddings, graphs, labels, designs). Because everything is
just axes and fields, one small model spans the diversity of real single-cell work that a fixed
cells × genes container strains on — for example a multi-sample (even cross-species) integration kept
as a collection of heterogeneous samples rather than one concatenated matrix; a CITE-seq object with
a second, protein feature axis; or a case-control cohort carrying a statistical design over its
samples. The routine count-matrix-plus-a-clustering case stays just as simple, while the harder cases
use the same vocabulary instead of an opaque uns/misc blob (see Why lstar?).
In the short term, the most immediately useful thing this buys you is moving data between the formats people already use. Each existing container — AnnData (Python), Seurat and SingleCellExperiment (R), pagoda/conos — fixes a few named slots; routing a dataset through L★ converts one to another while preserving the meaning of each piece and reporting anything a target can't hold instead of dropping it silently.
lstar is available in Python, R, C++, and the browser (JS/WebAssembly) (all sharing one fast C++ core), reads and writes a portable Zarr-based format (v3 by default, v2 on request), and is built to scale. Everything heavy can be streamed in bounded memory — convert a multi-gigabyte dataset, write a store, or compute per-gene statistics without ever loading the whole matrix, so work that needs a big machine today runs on a laptop (see Large data: lazy reads and streaming). You can also open a million-cell dataset over the network and read just the parts you need.
Status: early-development alpha; the Python package ships on PyPI as
lstar-sc. Working today: read/write the same store from Python, C++, and R; profiles for AnnData, Seurat (legacy v2 → v5), SingleCellExperiment, and Conos; the collection model; lazy/streaming reads; a browser/WebAssembly data layer.
Three things are hard with today's fixed-schema containers, and L★ is designed around them:
cells × genes slot, so they end
up as opaque blobs in uns/misc. In L★ they are ordinary, typed, queryable fields.If you only ever need to move data between AnnData, Seurat, and SCE, point 1 is reason enough to use lstar. Points 2 and 3 are why the model is shaped the way it is.
One command — lstar convert detects each format from its path, routes through the L★ store (in-process
for Python formats, an Rscript bridge for Seurat/SCE), and reports what crossed:
lstar convert pbmc.h5ad pbmc.rds # AnnData (Python) -> Seurat (R), bridged automatically
lstar convert atlas.h5ad atlas.lstar.zarr # -> a portable L* store (--to sce for SingleCellExperiment)
lstar convert pbmc.rds pbmc.h5ad --report # + a fidelity report (every field, and what was `dropped`)
Two things make it more than a one-liner:
a fidelity report (--report / --report-json) lists every axis and field with its role, state,
and provenance, and — crucially — dropped: what the target couldn't represent, made visible
rather than silently lost.
a native-acceptance check (--check, on by default; --strict to gate the exit code) opens the
result in its own library and runs a canonical-ops smoke (scanpy / Seurat / scran), so you know the
native analysis tools will accept it — not just that the bytes round-tripped.
a package-free fallback (--backend auto|native|direct): each conversion uses the format's native
package when it's installed, else lstar's own codec — so you don't need the domain packages for the
common cases. What works without the native packages:
| convert (no native package) | needs only |
|---|---|
.h5ad ↔ store — read and write |
lstar + h5py
|
Seurat .rds ↔ store — read and write |
lstar + base R (no SeuratObject) |
SCE .rds → store — read
|
lstar + base R (no SingleCellExperiment) |
store → SCE .rds (write) · .h5mu ↔ store |
native-only — needs SingleCellExperiment / mudata
|
At a wall (an unknown on-disk version, a BPCells-backed matrix) it stops and names exactly what to
install. The heavy analysis packages (scanpy / full Seurat / scran) are never needed to convert —
only for the optional --check. Details: docs/conversions.md.
Under the hood it is just write_Y(read_X(...)) with the on-disk L★ store as the bridge between the two
languages, which you can also drive directly:
python3 -c 'import anndata as ad, lstar; from lstar.profiles.anndata import read_anndata
lstar.write(read_anndata(ad.read_h5ad("pbmc.h5ad")), "pbmc.lstar.zarr")' # AnnData -> L* store
Rscript -e 'library(lstar); saveRDS(write_seurat(lstar_read("pbmc.lstar.zarr")), "pbmc.rds")' # -> Seurat
The shared-vocabulary core — raw counts, normalized/scaled expression, PCA (scores and gene
loadings), UMAP/t-SNE, clusterings, cell/gene metadata — survives. Whatever the target can't hold (e.g.
neighbor graphs through Seurat) is listed in the dataset's dropped manifest, so nothing vanishes
unannounced. A runnable, commented version is
examples/convert_h5ad_to_seurat.sh.
See docs/conversions.md for the full glue guide (every reader/writer, the conversion matrix, what is preserved vs. recorded as dropped, version detection) and docs/mapping.md for the deterministic role→slot contract — what lands where in each target, and the native-acceptance check that verifies the native tools won't choke.
If you want to author or inspect L★ data, the model is just axes (the things you index by) and fields (typed data over them):
import scipy.sparse as sp, lstar
ds = lstar.Dataset(kind="sample")
ds.add_axis("cells", [f"cell{i}" for i in range(100)])
ds.add_axis("genes", [f"g{i}" for i in range(50)])
# A field declares what it IS (a `measure` over cells × genes) — no fixed "X" slot.
ds.add_field("counts", sp.random(100, 50, density=0.1, format="csc"),
role="measure", span=["cells", "genes"], state="raw")
lstar.write(ds, "sample.lstar.zarr")
ds2 = lstar.read("sample.lstar.zarr") # also readable from R and C++
A field's role (measure, embedding, loading, relation, label, …) says what kind of object
it is. A new kind of result is a new field with a role — never a change to the format. See
docs/model.md.
Collections, not one big matrix. A multi-sample study is stored as a samples axis plus
per-sample cells.{s}/genes.{s} axes and measures (samples may differ in cells and genes), with a
union cells axis for the joint analysis (embedding, clusters, and the integration graph as a
relation). The R package ingests a Conos object (write_conos) and a split Seurat v5 assay
this way — see examples/conos_collection_demo.R.
Versions are recognized, not assumed. Formats change shape across releases, so the readers detect
the variant and adapt — even a legacy v2 seurat object (the pre-Assay S4 class, read via its raw
slots) through v3/v4 Assay vs. v5 Assay5 (with a fallback for SeuratObject < 5),
pagoda2's getRawCounts() accessor vs. the legacy $counts slot, AnnData's .raw slot. The detected
<format>@<version> is recorded, so a downstream reader knows what produced the data.
Single-cell stores get big — hundreds of thousands of cells, tens of thousands of genes. lstar is built so you never hold a whole dataset in memory to work with it: the heavy operations stream the matrix in blocks, so peak memory stays bounded and roughly flat as the data grows.
h5ad → L* conversion of the Tabula Muris Senis droplet atlas (subsampled from 25k to 245k cells, up to 502M nonzeros): the in-memory path's peak RAM grows with the matrix (to ~4 GB) while streaming stays ~flat (~0.3 GB, ~13× less at full size), for a small, roughly constant time premium. Reproduce with examples/streaming_scaling.py.
convert_anndata (h5ad → L*) and convert_to_h5ad
(L* → h5ad) move data between formats with a backed read + block-by-block write, never materializing
the matrix; lstar.write(..., stream=True) does the same for any lazy/backed source. A multi-gigabyte
atlas converts in a few hundred MB.lstar.read(path, lazy=True) reads only the small manifest; the heavy
arrays stay on disk (or on the server) until you touch them. Opening a 78-million-nonzero matrix this
way costs a few megabytes of memory instead of hundreds.ds = lstar.read("big.lstar.zarr", lazy=True) # opens in MBs, not GBs
# per-gene mean/variance over log-normalized counts, streamed in bounded memory:
mean, var, nnz = lstar.stream_col_stats(ds.field("counts").values,
lognorm=True, # normalize on the fly; the dense matrix is never built
n_threads=8) # use as many cores as you like
top_variable_genes = var.argsort()[::-1][:2000]
When you write a store, chunking and compression make these reads cheap (a lazy read fetches only the chunks it needs):
import numcodecs
lstar.write(ds, "big.lstar.zarr", chunk_elems=1_000_000, compressor=numcodecs.GZip(5))
In practice this is fast and frugal: opening that 40,220 × 20,138 matrix lazily uses ~9 MB instead of
~780 MB, per-gene statistics stream in bounded memory, and the heavy reductions run on a shared C++
core (used automatically when available, ~8× faster on 16 threads, identical results in Python, R, and
the browser). Measurements and the full picture are in misc/plan1.md §12.
| what it is | |
|---|---|
Python (python/) |
the lstar package on zarr-python, with an optional compiled C++ accelerator |
R (R/) |
the lstar package; the format profiles (Seurat, SCE, Conos) live here |
C++ (core/) |
libstar, the header-only core: the model, Zarr IO (v2/v3, chunked, gzip/zstd, sharding), and the fast kernels |
Browser/Node (js/) |
a TypeScript data layer over libstar compiled to WebAssembly (the same reader R/Python/C++ use) — reads v2/v3, and writes stores (chunked, sharded, gzip/zstd) for viewers |
docs/ principles, the model & format specs, conversions, worked examples
core/ libstar — the C++ core
python/ R/ the Python and R packages
js/ the browser/WASM data layer
conformance/ the shared round-trip / cross-format / cross-language test suite
examples/ runnable, commented end-to-end demos
misc/ the design proposal (Lstar_proposal.md) + plans
lstar convert CLI).The full normative specification (the model, the Zarr schema, and the bidirectional profile rule
catalog for every format) is the proposal, misc/Lstar_proposal.md.
MIT.