cmrlifecycle

Small package for auditing the update/revision lifecycle of NASA CMR granules -- i.e. figuring out when a collection's files actually change, as distinct from CMR metadata churn, so that a downstream mirror (e.g. a COG conversion published to source.coop) knows when it needs to re-derive a file. Originally built against MUR-JPL-L4-GLOB-v4.1; the core is now collection-agnostic, with a companion layer for discovering and sanity-checking collections before you audit them.

Start here

• Just want one collection's lifecycle? vignette("cmr-quickstart", package = "cmrlifecycle") -- a live, worked walkthrough: find a collection, sanity-check it, audit it, plot it, and handle the common edge cases (opaque GranuleUR, swath granule volume, cloud/S3 access).
• Curious what exploring beyond one collection turned up? vignette("cmr-landscape", package = "cmrlifecycle") -- the full narrative: real bugs found and fixed, provider-specific metadata quirks, and the tools built in response, all with baked-in evidence.
• Want current numbers, not a fixed write-up? vignette("cmr-current-snapshot", package = "cmrlifecycle") -- re-runs a subset of the landscape checks live, every time it's built.

The rest of this README is a compact reference; the vignettes are where the depth and worked examples live.

Installation

From this repo:

# remotes::install_github("hypertidy/cmrlifecycle")

Usage

library(cmrlifecycle)

# don't have a short_name yet?
cmr_find_collections(keyword = "sea surface temperature", provider = "POCLOUD")

# sanity check before auditing: is it alive, and what does it cover?
cmr_collection_liveness("MUR-JPL-L4-GLOB-v4.1")
cmr_find_collections(short_name = "MUR-JPL-L4-GLOB-v4.1")

# audit the granule-level revision/production lifecycle
rev <- cmr_audit_revisions("MUR-JPL-L4-GLOB-v4.1", from = as.Date("2024-01-01"))
plot_cmr_lifecycle(rev)

See vignette("cmr-quickstart") for the full walkthrough, including opaque-GranuleUR collections (e.g. LAADS swath products), spatial subsetting for high-volume swaths (bounding_box/polygon), and cloud/S3 access.

What cmr_audit_revisions() gives you

One row per granule:

column	meaning
granule_ur, concept_id	CMR identifiers
data_url	first GET DATA (https) download URL from UMM RelatedUrls
s3_url	first GET DATA VIA DIRECT ACCESS (s3://) URL -- NA if the collection doesn't publish direct S3 access
n_rev	CMR meta revision-id -- how many times the metadata record was touched (often bulk churn, not a data-change signal)
insert_date / update_date	UMM ProviderDates Insert/Update
rev_date	CMR meta revision-date (latest revision)
production	DataGranule.ProductionDateTime -- when the science file itself was generated. This is the churn-immune signal.
temporal_start	UMM TemporalExtent Beginning/SingleDateTime -- the granule's acquisition start
*_lag	each date above minus gran_date (days)

gran_date is parsed from a leading YYYYMMDD in granule_ur by default (GHRSST-style naming); pass gran_date_fun for other conventions -- see vignette("cmr-quickstart")'s "when your collection doesn't play by MUR's rules" section for the opaque- GranuleUR and swath-volume cases this covers.

Key finding this was built to support

n_rev is metadata churn, not a change-of-data-file signal; production is the reliable one. The staleness rule for a mirror is:

stale <- mirror_mtime < as.Date(production)

joined against e.g. rclone lsjson mtimes of the mirror; nc_bytes is a free secondary check (a regenerated file rarely keeps its exact size).

Notes / limits

• all_revisions=true is not supported for granule searches (CMR 400s) -- only the first Insert, latest Update, and a revision count are recoverable, not intermediate revision timestamps.
• data_url/s3_url pick the first matching UMM RelatedUrls entry; inspect RelatedUrls directly if a collection publishes multiple mirrors and you need a different one.
• s3_url is only cheap/direct from compute in the same AWS region as the bucket, and always requires an Earthdata Login credential exchange -- check cmr_find_collections()'s region/ s3_credentials_endpoint first. See vignette("cmr-landscape") for why this matters.
• ends_at_present/temporal_end (declared collection-level metadata) vary a lot in reliability by provider -- use cmr_collection_liveness() to check real recent granule activity instead when it matters. See vignette("cmr-landscape") for the full investigation (PO.DAAC vs. LAADS at catalog scale).
• Not every CMR "collection" has granules -- cmr_find_collections()'s granule_count flags this directly; see vignette("cmr-landscape") for a real example (an AADC record with a single static download and zero UMM-G granules).

Files

• R/cmr-lifecycle.R -- cmr_find_granules(), cmr_audit_revisions(), plot_cmr_lifecycle()
• R/cmr-collections.R -- cmr_find_collections(), cmr_all_collections(), cmr_collection_liveness()
• R/globals.R, R/cmrlifecycle-package.R -- package plumbing (NSE variable declarations, package-level doc)
• vignettes/cmr-quickstart.Rmd -- live, in-depth single-collection walkthrough
• vignettes/cmr-landscape.Rmd -- narrative write-up of the broader exploration, with baked-in evidence
• vignettes/cmr-current-snapshot.Rmd -- lightweight companion that re-runs a subset of the landscape checks live on every build
• snapshots/ -- dated, rendered copies of cmr-current-snapshot.Rmd, kept as a historical record (not part of the built package)
• data-raw/cmr-granules.R -- the original MUR-specific driver script (not installed as part of the package), kept as a worked example
• data-raw/cmr_revisions_*.parquet -- output of a run of that script