h2oplyr provides familiar transformation verbs while
keeping the observations inside an H2O cluster. The only
external runtime dependency is h2o. The
tidytable API inspires parts of the interface, but neither
tidytable nor dplyr participates in a
calculation. knitr and rmarkdown are optional
documentation-building dependencies.
The workflow is:
CSV file → H2OFrame → h2o_tbl → remote operations → H2OFrame
An h2o_tbl is a small R object containing a remote frame
handle and grouping column names. R assembles expressions and inspects
metadata; H2O processes the rows. Scalar reductions and category domains
can reach R, but no transformation uses a local data frame, matrix or
vector as its computational backend. For display only, the previews
below request a few scalar values and categorical labels. Displaying a
value necessarily brings that value to the reader; it does not change
where selection, aggregation, joins and reshaping run.
This tutorial is for R users familiar with selecting columns, filtering rows and grouping data. You will learn to inspect remote results, handle missing values, check joins and verify a reshape round trip. Java and a matching H2O client/server are the only external runtime requirements.
This tutorial uses measurements of 344 penguins from three species in
the Palmer Archipelago, collected during 2007–2009. The CSV is included
in the package, so examples do not depend on a future download or a
palmerpenguins installation.
| Practical question | Tools used below | Visible evidence |
|---|---|---|
| Which variables and rows do I need? | h_select(), h_filter(),
h_drop_na() |
Values, column names and row counts |
| How do I derive and summarize measurements? | h_mutate(), h_summarize(),
h_across() |
Derived units and species means |
| Can I combine and reshape without losing information? | h_left_join(), pivot verbs |
Preserved row counts and matching values |
| How do ordering and sampling affect the result? | Slice verbs, h_arrange(),
h_lag() |
Sample rows, tied maxima and lag values |
Reading the examples. Each transformation is followed by values returned from H2O and a green check based on an executed assertion. A preview is a small window into the result; the caption reports the full result dimensions.
Install the package archive using
install.packages(path, repos = NULL, type = "source") after
installing h2o. Start a matching H2O server explicitly:
# Run this yourself, once per session. Loading h2oplyr never starts a server.
h2o::h2o.init(nthreads = 2, max_mem_size = "2G")
library(h2oplyr)These helpers belong to this vignette, not to the package API.
preview_h2o() accepts a remote table, a title, a few column
names and a row limit (at most six). It prints HTML and returns no local
analytical table. Numbers use scalar H2O reductions; text uses the
categorical domain of a single selected value. H2O may internally cache
small metadata previews. This is a deliberately small, slow display
routine, not a data export facility. verify_output() stops
the render if its condition fails. All helpers are defined here before
their first use.
# Escape labels before placing them inside HTML.
html_text <- function(x) {
x <- gsub("&", "&", x, fixed = TRUE)
x <- gsub("<", "<", x, fixed = TRUE)
gsub(">", ">", x, fixed = TRUE)
}
# Show bounded remote values without creating a local data.frame.
preview_h2o <- function(x, title, columns = names(as_h2o(x)), rows = 4L) {
frame <- as_h2o(x)
stopifnot(inherits(frame, "H2OFrame"), rows >= 1L, rows <= 6L,
all(columns %in% names(frame)))
types <- unlist(h2o::h2o.getTypes(frame[, columns]), use.names = FALSE)
shown <- min(rows, nrow(frame))
cat('\n<div class="result"><h3>', html_text(title), '</h3>', sep = "")
cat('<p class="result-meta">H2OFrame · ', nrow(frame), ' rows × ',
ncol(frame), ' columns · showing ', shown, ' rows and ',
length(columns), ' columns</p><div class="table-scroll"><table><thead><tr>',
sep = "")
for (j in seq_along(columns)) {
alignment <- if (types[j] %in% c("int", "real")) ' class="num"' else ""
cat('<th scope="col"', alignment, '>', html_text(columns[j]), '</th>', sep = "")
}
cat('</tr></thead><tbody>')
for (i in seq_len(shown)) {
cat('<tr>')
for (j in seq_along(columns)) {
# Two-step indexing keeps the cell as a remote frame in H2O.
cell <- frame[i, ][, columns[j]]
stopifnot(inherits(cell, "H2OFrame"))
numeric_column <- types[j] %in% c("int", "real")
alignment <- if (numeric_column) ' class="num"' else ""
if (h2o::h2o.sum(is.na(cell)) == 1) {
value <- '<span class="missing">NA</span>'
} else if (numeric_column) {
number <- h2o::h2o.sum(cell)
value <- if (number == floor(number)) sprintf("%.0f", number) else sprintf("%.2f", number)
} else {
# Re-encode in H2O so the domain contains only this cell's label.
label <- h2o::h2o.levels(h2o::h2o.asfactor(h2o::h2o.ascharacter(cell)))
stopifnot(length(label) == 1L)
value <- html_text(as.character(label[[1L]]))
}
cat('<td', alignment, '>', value, '</td>', sep = "")
}
cat('</tr>')
}
cat('</tbody></table></div></div>\n\n')
invisible(NULL)
}
# Only a successful, evaluated assertion receives a green check.
verify_output <- function(condition, label) {
stopifnot(isTRUE(condition))
cat('\n<p class="check"><strong>Verified.</strong> ', html_text(label), '</p>\n\n', sep = "")
}# Import the bundled CSV and confirm its dimensions.
penguins <- h_penguins()
stopifnot(nrow(as_h2o(penguins)) == 344)h_penguins() uploads the local CSV through
h2o.uploadFile() and declares "NA" as missing
in every column. Species, island and sex are categorical; measurements
are numeric. For your own files, upload directly with
h2o::h2o.uploadFile() and wrap the returned frame with
h2o_tbl().
The default table printer displays metadata only. This vignette adds
bounded value previews so you can inspect the results directly.
h_collect(penguins) and as_h2o(penguins) both
return the remote H2OFrame, preserving the backend
contract. This is an intentional change from the original standalone
prototype.
The eight variables are species, island, bill length and depth (mm),
flipper length (mm), body mass (g), sex and year. A missing measurement
appears as NA, not as zero. The same penguin can have one
measurement missing and others present.
Recognize the object. The result is an
H2OFrame wrapped by h2o_tbl. The small HTML
table above is a display of selected values, not a new data backend.
# Keep relevant columns, then rename and order the bill measurements.
bills <- h_select(penguins, species, starts_with("bill_"))
measurements <- h_select(penguins, where(is.numeric))
renamed <- h_rename(bills, bill_length = bill_length_mm)
relocated <- h_relocate(renamed, species, .before = bill_length)Selection operates on column names and H2O schema types.
where() supports the base predicates
is.numeric, is.character and
is.factor; arbitrary predicates are deliberately rejected.
all_of() requires every requested column, while
any_of() allows absent names. Ranges such as
species:island and exclusions such as -year
are supported. A grouped selection retains its grouping columns.
Takeaway. Selection changes the schema, not the observations. All three previews retain the original 344 rows.
# Compare complete-case filtering with two measurement thresholds.
complete_penguins <- h_drop_na(penguins)
stopifnot(nrow(as_h2o(complete_penguins)) == 333)
cat("Complete observations:", nrow(as_h2o(complete_penguins)), "\n")
large <- h_filter(penguins, body_mass_g >= 5000)
cat("Penguins weighing at least 5000 g:", nrow(as_h2o(large)), "\n")
bill_range <- h_filter(penguins, h_between(bill_length_mm, 40, 50))
cat("Penguins with bill length from 40 to 50 mm:", nrow(as_h2o(bill_range)), "\n")A missing filter condition is dropped. The expression interpreter
propagates missing comparisons explicitly, because H2O’s raw comparison
behavior can differ from R’s. h_filter_out() negates the
combined predicate and also drops missing conditions. Use
is.na() when you intend to select missing measurements.
# Select the missing body masses explicitly.
missing_mass <- h_filter(penguins, is.na(body_mass_g))
stopifnot(nrow(as_h2o(missing_mass)) == 2)Dropping all incomplete rows yields 333 penguins, whereas a mass-only analysis can retain 342. Those are different analysis populations. Replacing missing body mass with zero would invent a biologically impossible measurement, so this tutorial removes missing mass only where it is needed.
Takeaway. Choose the missing-data rule for the question: requiring every variable to be complete removes more rows than requiring body mass alone.
# Drop only missing mass, then derive units and a mass category.
prepared <- penguins |>
h_drop_na(body_mass_g) |>
h_mutate(
mass_kg = body_mass_g / 1000,
mass_band = h_case_when(
body_mass_g >= 5000 ~ "large",
body_mass_g >= 3500 ~ "medium",
TRUE ~ "small"
)
)
stopifnot(nrow(as_h2o(prepared)) == 342)
cat("Total measured mass (kg):",
h2o::h2o.sum(as_h2o(prepared)[, "mass_kg"]), "\n")Named mutations run sequentially, so a later expression can use a
newly created column. Arithmetic, comparisons, selected H2O mathematical
and text functions, and the package’s vector helpers are supported.
Calls to arbitrary R functions or other package namespaces raise an
error. h_transmute() retains only the new named columns.
Grouped mutation is not implemented: ungroup explicitly rather than
accidentally calculate a global value inside a group.
The native |> pipe and the package’s %.%
pipe can both chain these verbs.
Takeaway. Read the derived columns side by side. The numerical assertion checks every measured mass, not just the displayed rows.
# Compute counts and means inside H2O, grouped by species.
by_species <- h_summarize(
penguins,
observations = n(),
mean_mass_g = mean(body_mass_g, na.rm = TRUE),
mean_bill_mm = mean(bill_length_mm, na.rm = TRUE),
.by = species
)The output has one row per species. n() counts all
observations; each mean removes missing values independently. Gentoo
penguins have the highest mean body mass in this sample. This is a
descriptive comparison, not a causal effect of species or island.
.by applies grouping to this call only. The equivalent
persistent grouping form is
h_group_by(penguins, species) |> h_summarize(...);
summaries themselves return ungrouped results. .add = TRUE
adds grouping columns, and h_group_vars() reports the
current grouping metadata.
Supported reductions are n(),
n_distinct(column), sum, mean,
min, max, sd, var
and median. Each reduction takes a column, optionally with
na.rm. Create transformed columns with
h_mutate() before aggregating. The adapter rejects
empty-input summaries explicitly.
Takeaway. Group sizes count penguins, while each mean uses its own non-missing measurements. A group count is not necessarily the denominator of every mean.
# Apply unit conversion and grouped means to multiple columns.
scaled <- h_mutate(
bills,
h_across(starts_with("bill_"), list(cm = function(z) z / 10))
)
average_bills <- bills |>
h_drop_na() |>
h_group_by(species) |>
h_summarize_across(starts_with("bill_"), list(average = mean))h_across() builds column expressions with names such as
bill_length_mm_cm. The one-argument function body is
interpreted through the same remote expression rules as
h_mutate(). Summary function identity is used, so an alias
such as average = mean still computes the mean. Across
summaries default to na.rm = FALSE; this example
deliberately removes missing bill measurements first. The standalone
h_mutate_across() wrapper is retained for
compatibility.
Takeaway. Across applies the same expression to several columns; the suffix records the transformation or summary that produced each output.
# Attach a unique species summary to each observed mass.
enriched <- h_left_join(
prepared, by_species,
by = "species", relationship = "many-to-one"
)
stopifnot(nrow(as_h2o(enriched)) == nrow(as_h2o(prepared)))
enriched <- h_mutate(enriched, mass_minus_species_mean = body_mass_g - mean_mass_g)The relationship check ensures the right table has at most one record
per species, preventing accidental multiplication of penguin rows.
Inner, left, right and full equijoins are available.
h_semi_join() selects left rows with a match;
h_anti_join() selects those without one. Named keys map
different column names, for example
by = c(species = "species_code").
A full join is implemented using a left join plus unmatched right
rows, all inside H2O. Missing-key matching follows H2O, and join row
order is unspecified. Always use h_arrange() when
subsequent calculations depend on ordering.
Takeaway. A valid many-to-one join must preserve the left row count. Sorting the preview makes it readable; join order itself is unspecified.
# Use species-level summaries so the ID is naturally unique.
wide <- h_summarize(
penguins,
bill_length_mm = mean(bill_length_mm, na.rm = TRUE),
bill_depth_mm = mean(bill_depth_mm, na.rm = TRUE),
.by = species
)
long <- h_pivot_longer(
wide, starts_with("bill_"),
names_to = "measurement", values_to = "mean_mm"
)
stopifnot(nrow(as_h2o(long)) == 6)
restored <- h_pivot_wider(long, measurement, mean_mm)
stopifnot(nrow(as_h2o(restored)) == 3)The long result contains six species/measurement combinations.
Widening gets the output names from the remote categorical domain and
combines values with remote joins. Repeated ID/name combinations are
rejected: decide on an aggregation before reshaping. This avoids
silently choosing one measurement or creating a local list-column.
values_fill replaces missing result cells, including
explicitly missing input values.
For text transformations, h_separate() uses H2O
regular-expression splitting. h_unite() uses H2O
categorical interaction with encoded original underscores; its current
separator cannot contain dollar signs or backslashes. High-cardinality
interaction domains can consume substantial cluster memory.
Takeaway. The round-trip check compares both measurements across all species. Matching row counts alone would not prove that the values survived.
# Sample reproducibly, retain tied maxima and compute a sorted lag.
sample_penguins <- h_slice_sample(prepared, n = 12, seed = 2026)
stopifnot(nrow(as_h2o(sample_penguins)) == 12)
heaviest <- h_slice_max(prepared, body_mass_g, n = 3, with_ties = TRUE)
cat("Rows among the three heaviest positions, including ties:",
nrow(as_h2o(heaviest)), "\n")
ordered <- h_arrange(prepared, body_mass_g)
ordered <- h_mutate(ordered, previous_mass = h_lag(body_mass_g))Sampling uses H2O random columns. Sampling with replacement creates
draw indices in H2O and joins them to rows. With
seed = NULL (the default), no seed is passed to H2O; supply
a seed explicitly when reproducibility is needed. Identical seeds are
reproducible on the same frame and cluster layout; they are not a
promise of identical samples across versions or distributed layouts.
with_ties = TRUE can return more than n
rows.
Lag, lead, cumulative operations and h_fill() use
current row order. Penguins are not a longitudinal time series: the lag
example illustrates an ordered comparison, not a temporal
interpretation. Grouped windows and grouped slicing are rejected in this
release.
Takeaway. A seeded sample is reproducible within the documented setting. A lag is meaningful only after choosing an order; penguins here are not a longitudinal panel.
Every exported function has an executable example with an assertion. The package also includes regression tests for missing values, aggregation, ties, empty slices, join multiplicity, invalid expressions and input preservation. Integration tests compare H2O function bindings before and after execution without modifying the H2O namespace. A static audit rejects local-table calls in package functions. Additional maintainer-only download instrumentation is kept outside the package.
Run all examples and regressions against a cluster you started explicitly:
h2o::h2o.init(nthreads = 2, max_mem_size = "2G")
library(h2oplyr)
source(system.file("examples", "examples.R", package = "h2oplyr"))
for (nm in names(examples)) eval(examples[[nm]])
source(system.file("examples", "regressions.R", package = "h2oplyr"))This package is an explicit subset of tidy-style manipulation, not a
complete drop-in replacement for dplyr or
tidytable. It has no local rowwise/map backend,
non-equality joins, automatic list-column nesting, arbitrary R
callbacks, grouped mutation or implicit duplicate-cell pivot
aggregation. h_bind_rows() requires matching column-name
sets. h_nest_by() returns a list of remote group handles
and is intended for a modest number of groups.
When no cluster is connected, rebuilding this vignette retains the code but skips H2O execution. The distributed HTML was built against an explicitly connected server. The package never starts or shuts down a user’s cluster while loading, printing, or running its ordinary examples.
The green checks accompany the tutorial’s calculations. The installed package also contains 73 asserted API examples and semantic regression tests; those are a separate suite, not 73 examples all reproduced on this page. The commands in “Connect once” and “Validation” are instructions to run yourself, rather than claims that this HTML launched a cluster or reran the complete package test suite.
Code-only rebuild. No H2O server was connected. The transformation chunks and their previews were skipped; no successful execution is claimed.
Try changing the mass threshold or selecting another species, then
rerun the section and inspect both the values and the assertion. Do not
infer a successful transformation merely from seeing an object name or
from a silent stopifnot().
Data: Horst AM, Hill AP, Gorman KB (2020), palmerpenguins: Palmer
Archipelago (Antarctica) penguin data, doi:10.5281/zenodo.3960218.
The data are available under CC0. Measurements were collected by Kristen
Gorman and Palmer Station LTER. The research description is Gorman KB,
Williams TD, Fraser WR (2014), doi:10.1371/journal.pone.0090081.
The bundled CSV’s source, retrieval date and SHA-256 are recorded in
system.file("extdata", "PROVENANCE.md", package = "h2oplyr").