The drift report now surfaces per-column numeric spread changes: a new
"Numeric spread changes (min / max / sd)" table shows each stat's previous
and current value with its shift, and compare_snapshots() returns them as
spread_changes. (The values were already computed and stored in
column_snapshots every run; they were never read back.) Three new
comparison rules — max_numeric_sd_shift_pct, max_numeric_min_shift_pct,
max_numeric_max_shift_pct — breach-flag a shift, settable at rules level
or per column like their mean-shift sibling. They have no default:
unset, the shifts are reported but never judged, so existing configs and
reports gain information without gaining new failure modes. Generated
global configs list them commented out with an example value. Internally,
the drift side's per-column threshold resolution was generalised from a
mean-shift-only helper to one rule-keyed resolver
(.column_comparison_overrides()) covering all four per-column comparison
rules.
validate_config()'s unknown-key warning now distinguishes a likely typo
from a possibly-newer vocabulary key: a near-miss still gets "Did you
mean...", while an unknown key resembling nothing known states the
installed dqcheckr version and that a newer release may define it — so a
config written against a newer dqcheckr explains itself on an older one
instead of reading as a mistake.
A snapshot_db pointing at a directory (or any path SQLite cannot open) now
raises the typed dqcheckr_db_error instead of leaking a raw RSQLite error.
file.exists() is TRUE for a directory, so such a path slipped past the
missing-file guard and failed opaquely at connect; connect-time failures are
now typed at the single connection helper, covering every database path
(read and write), with a directory-specific message.
compare_snapshots() now checks that the database has a column_snapshots
table alongside the existing snapshots-table check, raising the same typed
dqcheckr_schema_error. A database carrying only snapshots (a partial
restore or export of run history) passed every guard and then failed deep in
the drift computation with a raw untyped "no such table" error.
The database initialiser now refuses a foreign column_snapshots table with
the same typed dqcheckr_schema_error it already raised for a foreign
snapshots table. A snapshot_db pointing at a database that held another
application's column_snapshots used to be adopted silently: dqcheckr's
snapshots table was planted inside the foreign file and every subsequent
run then failed to record its snapshot ("Snapshot not recorded"), so the
dataset could never accumulate history and a database dqcheckr did not
create had been modified. The refusal rolls back inside the initialiser's
transaction, leaving the foreign file exactly as it was.
col_threshold() no longer aborts with an untyped subscript out of bounds
error when a column_rules entry is a bare value where a rule map belongs
(e.g. column_rules: {mycol: 5}). The same guard now also covers malformed
threshold values inside a valid rule map — a quoted number ("0.5"), a
vector, or a word crashed the exported check functions with untyped base-R
errors (threshold * 100 on a character; if on a length-2 condition)
while the drift side silently ignored the identical override. A value that
is not a single finite numeric now simply does not resolve at its level and
the descent continues to the rules-level threshold, then the default — so
the run and drift surfaces agree. The run path already rejected such
configs via validate_config(); this guards the exported check functions
when they are called directly. The table-level twin table_threshold()
(min_row_count, max_row_count, max_file_size_mb) now applies the same
single-finite-numeric discipline: a malformed value falls through to the
default instead of crashing check_min_row_count() untyped — where the
pre-crash status comparison was lexicographic (50 < "100" is FALSE), so
the crash was the only thing preventing a silently wrong verdict.
sniff_dataset() no longer reports a single-quote quote_char as
"detected" on the strength of the file merely containing no apostrophes. On
an unquoted file with stray double-quotes (inch marks: 5" nail), the
default-quote pass fails by line merge and the ' candidate used to win
vacuously — writing a config that asserts quoting the file does not use,
which a later delivery with an unclosed leading apostrophe would repay by
row-merging the rest of the file into one field. A non-default quote is now
accepted only when some sampled field actually starts and ends with it;
otherwise detection re-runs under the default quote with readr's tokenizer
(field-start-only quote semantics, the same as the runtime reader).
sniff_dataset() now detects the delimiter of an all-fields-quoted CSV whose
quoted free-text fields carry embedded newlines (so a logical record spans
several physical lines) -- e.g. a ServiceNow export. The per-physical-line
field counter cannot see across a multi-line quoted record and used to reject
every delimiter, mis-classifying the file as single-column; when the strict
pass finds nothing, a readr-based fallback re-detects the delimiter, guarded
so a lone stray delimiter in one row cannot fake a multi-column file.
generate_dataset_config() can no longer emit a config its own
validate_config() cannot parse: a mapping key derived from a column name is
emitted in explicit-key block form (? key / : value) once it would exceed
libyaml's 1024-character inline-key limit, and .y_quote() now escapes
embedded newlines/tabs plus every other C0 control character, DEL, and the
C1 range as \xNN (YAML's printable set excludes them and libyaml refuses a
document carrying one raw — a stray vertical tab from an Excel cell break was
enough), so even a pathological (or mis-sniffed) column name yields parseable
YAML that round-trips to the delivery's exact spelling. As a structural
backstop, both generators now parse the emitted YAML back before renaming
it into place: any future emitter gap aborts typed
(dqcheckr_generate_error) with no file written, instead of leaving a
created-but-unreadable config that the create-only guard would then defend
from regeneration. A failure of the write itself is likewise typed
(dqcheckr_write_error, the class already used when the rename into place
fails) rather than escaping as a raw writeLines error.
infer_col_type() now recognises ISO-style date-time strings
("%Y-%m-%d %H:%M:%S" and "%Y-%m-%dT%H:%M:%S") as the date type, so a
timestamp column (e.g. a ServiceNow opened_at export) classifies as date
rather than falling through to character. The new shapes are fully anchored,
so they match only a genuine <date><sep>HH:MM:SS value and never overlap the
date-only shapes. The GUI's wizard-preview type inference is kept in sync.
New vignette "The dqcheckr workflow: generate, edit, validate, run" — a
fully executed walk-through of the whole loop on a bundled example:
bootstrap a deployment with the two generators, read the generated
self-documenting YAML (including the duplicate-header rename and the
positional-list warning), edit a rule, validate (including the did-you-mean
typo path), run, list the history, compare two deliveries, and the
packed-FWF TODO and never-overwrite special cases. Because it executes,
R CMD check runs the entire chain as a living test.
New validate_config(dataset_name, config_dir): validation of the global
and dataset YAML against the config vocabulary, reporting every finding in
one pass (severities error/warning/note) instead of aborting on the first.
The severity rule: config mistakes are errors — wrong types/ranges,
misplaced rule keys (rules-level vs per-column), fwf_col_names/
fwf_widths length mismatches, duplicate output column names, unresolved
TODO width placeholders, a missing file source — while delivery-facing
findings are warnings, because they can equally mean the supplier changed
the delivery, and drift must be recorded, not crash the run. Unknown keys
get a did-you-mean suggestion and warn, so hand-kept extra keys round-trip.
The validator never crashes on malformed values (any internal error becomes
an error-severity finding). An unreadable dataset config aborts with
distinct classes — dqcheckr_missing_file, dqcheckr_empty_config,
dqcheckr_config_parse_error, dqcheckr_invalid_config — while an empty
or comments-only global config is tolerated as all-defaults with a
warning (matching how the package has always run).
When the delivery file the config points at is resolvable, validation
additionally cross-checks the config against the file's header only
(never the body — cheap even for multi-GB files on a network share):
col_names length vs the physical column count, key_columns/
expected_columns/column_types/column_rules naming columns that
exist, and fwf_widths summing to the record length — all warnings, per
the severity rule. The CSV header is parsed with readr (honouring the
configured delimiter, quote, and encoding), so a legal quoted field carrying
an embedded newline is counted as one column rather than mis-split by a
line-based read into a spurious column-count warning. When no delivery is resolvable the header tier is
skipped with the reason stated in the result and by print() — a verdict
always says which tier it reached.
run_dq_check() validates first, before any other work: error findings
abort with a dqcheckr_validation_error whose message lists every finding,
and no snapshot row is written — an unrunnable config can no longer
leave a pending orphan in the database. Warning findings are printed
and persisted into the run's results as VC-01 ("Config validation")
WARN checks — they land in the report, the warn counts, and the overall
status, so delivery drift the runtime checks cannot see (e.g. a surplus
column readr silently auto-names) still leaves a recorded trace.
Note-severity findings (e.g. both current_file and folder set, where
current_file wins) are now surfaced in the run log too, rather than being
silently dropped on the run path. A
corrupt dataset config aborts the run with the typed classes above instead
of the YAML parser's raw error. The description key the GUI wizard
writes is part of the config vocabulary (never read by the checks). The
two change-thresholds (max_numeric_mean_shift_pct,
max_row_count_change_pct) accept any value >= 0 — they bound an
unbounded relative change, and deployed configs legally carry values
above 1 — with a warning when a value looks like a raw percentage.
CP-04 (numeric mean shift) now honours a per-column
max_numeric_mean_shift_pct set in column_rules, which the GUI has always
written but the check silently ignored — the read goes through the same
column > rules > default resolution as every other per-column threshold.
The drift report applies the same per-column values, so the run report and
compare_snapshots() cannot contradict each other on one snapshot pair. A
malformed per-column value is now rejected up front by validation — both
run_dq_check() and compare_snapshots() validate the config before doing
any work (see below) — rather than silently tolerated; the internal drift
resolver still warns and falls back to the rules level as a last-resort
safety net if ever reached with a raw value.
compare_snapshots() now validates the dataset configuration before reading
any thresholds — the same gate run_dq_check() applies — so a malformed
threshold (or any error-severity config finding) is rejected with a
dqcheckr_validation_error on the drift path too, not tolerated there while
the run path aborts. Run and drift can no longer disagree on what counts as a
runnable config. A snapshot-only comparison may still omit current_file/
folder: a drift diffs snapshots already in the database and reads no
delivery file, so that run-only requirement is exempted and does not block
it. Relatedly, a corrupt config now surfaces the same typed
dqcheckr_config_parse_error on the compare_snapshots() and list_runs()
paths (and load_config() generally) as run_dq_check() already did, rather
than leaking the YAML parser's raw error.
Two more config mistakes now fail with a clear, typed message instead of a
confusing downstream error. An explicit expected_columns: [] is an
error-severity validation finding that names the fix — an empty list is never
a way to disable the schema-contract check; omit the key instead — rather
than being read as "expect zero columns" and failing every column. And a
directory given where a current_file/previous_file is expected aborts with
dqcheckr_invalid_config pointing at folder: mode, instead of failing deep
in the CSV reader.
New list_runs(dataset_name, config_dir, n): name-based run history. Resolves
the snapshot database from the dataset's merged configuration exactly as
run_dq_check() does and returns the recent-runs data frame from
read_recent_snapshots(), so the workflow can be driven end to end by dataset
names — the returned id column is what compare_snapshots() takes to
compare a specific pair of runs.
Internal: a config vocabulary (R/vocabulary.R) now records every YAML key
the package consumes — scope, type, default, positional flag, description —
as the single source of truth the upcoming validator checks against and the
config generators emit from. In support, the per-site default literals were
consolidated into shared constants (.default_read, .default_qc_rules,
joining the existing .default_paths/.default_comparison_rules):
max_missing_rate's 0.05 previously lived independently in two files.
No behaviour change.
New sniff_dataset(path): pure structure inference over a delivery file —
format (CSV vs fixed-width), encoding (the same full-file streamed UTF-8
scan and single-byte fallback as read_dataset()), delimiter/quote, header
presence, column names (duplicate header names renamed positionally with the
originals recorded), per-column types via infer_col_type() (one
implementation — the sniff cannot disagree with run-time classification),
and key-column candidates. Fixed-width boundaries come from
readr::fwf_empty(), with the widths anchored at position 0 so a leading
blank margin (or a right-aligned first column) folds into column 1 instead
of shifting the whole grid; a packed file (no blank gutters) is reported
with an explicit fwf_packed marker rather than a wrong confident guess. The
same explicit marker is used for any fixed-width file that is not
single-byte-per-character — non-UTF-8 multibyte, UTF-16/32, and also valid
multibyte UTF-8 (accented text): readr::fwf_empty() returns byte offsets
while the sample is sliced by characters, and readr::read_fwf() itself
slices by bytes at run time, so a confident guess for such a file could
silently truncate its last column (and no single width vector fits a file
whose accented and plain rows differ in byte length). Widths are left as a
TODO for review instead of being sniffed wrongly. The fixed-width gate
itself tests byte lengths as well as character lengths, so a genuine
byte-fixed multibyte file (uniform bytes, varying characters — the on-disk
norm, since readr::read_fwf() slices by bytes) reaches that explicit
TODO path instead of falling through to a confidently wrong single-column
CSV whose "header" is the first data record; the minimum packed-record
length is likewise measured in bytes. No side effects; each
headline field's origin is recorded as detected/default/generated. This is
the detection half of the config generator.
New generate_dataset_config(path, config_dir, dataset_name): sniffs a
delivery and writes a fully-optioned, self-documenting YAML config. Every
config key appears exactly once — detected values live, optional settings
commented out with their defaults — and each key carries the same
description the validator's vocabulary uses. Duplicate and empty header
names (a trailing delimiter) are renamed positionally with # was "..."
annotations and csv_skip: 1, with rename suffixes bumped past any name
already in the header; positional lists are always emitted complete under a
do-not-comment warning; fixed-width files get a character-position ruler
comment, and a packed file gets explicit TODO widths that
validate_config() flags as an error, so run_dq_check() refuses to run
until they are filled in. Emitted values are
YAML-safe (backslashes escaped, map keys quoted, paths in forward-slash
form with relative paths kept relative); columns whose sampled type is
unknown stay unpinned. Create-only: an existing config aborts with
dqcheckr_config_exists and is never touched (the never-overwrite rule).
Both generators write crash-safely — the YAML is emitted to a temp file and
renamed into place only once complete, so an interrupted generation can
never leave a truncated config that the create-only guard would then treat
as "already generated" (the accepted residual concurrent-generation window
is documented in the specification vignette, Accepted design decisions).
New generate_global_config(config_dir): writes a fully-optioned
dqcheckr.yml — the shared infrastructure paths set live to their built-in
defaults, and a commented default_rules block listing every rule key with
its default value and description (rules with no default show an example
and a note that unset keeps the check off). Change a threshold by
uncommenting one line. Create-only, like the dataset generator. A brand-new
deployment now bootstraps with exactly two generator calls and one
run_dq_check().
The per-column check loops (QC-07/08/09/10/11/13/15, SC-01/02, and the
comparison checks CP-04/05/06/07) no longer grow their result list with
c(results, list(...)) inside a for loop, which reallocated the whole list
on every column and made a run O(columns^2). They now build results with
lapply() and compact once, so wide deliveries (100+ columns) build their
results in linear time. Check output is unchanged.
The missing/empty predicate (is.na(x) | x == "") and the four
comparison-check default thresholds are each now defined once and shared by
the QC checks, the comparison checks, the drift report, and the snapshot
writer, instead of being reimplemented at each call site. This removes the
risk of, for example, the QC report and the drift report applying different
default thresholds to the same rule after only one copy was edited.
The shared Quarto render pipeline (render into a temp dir, verify an output
file was produced, abort with dqcheckr_render_error if not, then move the
result into place) is now a single internal helper (.quarto_render_to_file())
called by both the main report writer and the drift report writer, instead of
being duplicated near-verbatim in each. A future change to the render pipeline
now lands in one place rather than needing to be mirrored by hand.
fwf_widths (must abort with dqcheckr_invalid_config), and
render_report() when Quarto returns without writing an output file (must
abort with dqcheckr_render_error rather than name a report that does not
exist).QC-07 (check_numeric_stats()) no longer reports Inf/NaN summary
statistics. as.numeric("Inf") is Inf, not NA, so a literal infinity in
a delivery classified the column as numeric and survived the check's
is.na() filter — min/max reported ±Inf, and with both signs present the
mean came back NaN. The aggregates are now taken over the finite subset,
and the number of excluded values is appended to observed, so an infinity
in a delivery is still visible rather than silently dropped. (Snapshot
statistics were never affected: compute_col_stats() already filtered
non-finite inputs and serialised its aggregates through .finite_or_na().)
Duplicate per-column statistics no longer multiply a column's rows in the
drift report. column_snapshots carries no uniqueness constraint (a
constraint would turn a duplicate into a lost snapshot row through the
non-fatal write path), and custom checks may emit several results for one
column under one check_id — so two rows for the same (column, statistic)
are reachable on a clean run. Both consumers join on the column name, and a
join row-multiplies: the column appeared twice in every per-column drift
table (four times if duplicated in both snapshots), breach flags included.
compare_snapshots() now collapses duplicates to the first value and warns,
naming what was collapsed. See Accepted design decisions (AD-03) in the
specification vignette.
A custom-check result carrying a non-character field (e.g. a plain list
built without dq_result(), with observed = 42) no longer costs the run
its entire snapshot row. Such a result passed run_custom_checks()'s
field-presence validation but failed the snapshot writer's typed extraction
much later, dropping the whole history row and all column statistics under
a misleading "SQLite write failed" warning — while the report still
rendered, so the loss was easy to miss. run_custom_checks() now
normalises every string field at the boundary (length-1 atomic values are
coerced with as.character(), matching what dq_result() produces;
anything else aborts with dqcheckr_invalid_custom_checks, naming the
element and field), and dq_result() itself now coerces check_id,
check_name, and column the same way it already coerced observed and
message.
The CP-04 numeric-mean-shift comparison no longer aborts the whole run when a
numeric column contains a literal Inf/-Inf value. as.numeric("Inf") is
Inf (not NA), so such a column still classifies as numeric and reached the
mean comparison; a non-finite mean slipped past the guard and made the shift
NaN, which crashed the pipeline at if (shift_pct > threshold). The check now
filters to finite values first (as check_outliers() and compute_col_stats()
already do), so an Inf-bearing column is handled as a warning rather than a
crash. R's own write.csv() emits the literal Inf, so this is a realistic
corrupted-delivery input.
compare_snapshots() now rejects a NULL or empty dataset_name with a clear
dqcheckr_invalid_argument error instead of aborting with an empty message. A
NULL bound as SQL NULL (matching no rows) and the "need 2 snapshots" abort
then collapsed to a zero-length, blank-text condition.
read_recent_snapshots() now clamps a negative n to 0 rather than returning
the entire history. SQLite reads LIMIT -1 as "no limit", so a negative n
silently returned every row instead of capping the result as documented.
compare_snapshots() now returns the rendered drift report's path as a
report_path element on its result (NULL when no report was written), so a
caller can link to the report directly instead of reconstructing the filename
from a slug pattern -- the reconstruction approach is fragile and broke when
the drift filename gained its snapshot ids.
A snapshot's render_status now carries a "pending" state while its report
is still rendering. The row is written as "pending" and only flipped to
"success" (with report_file) once the report is confirmed on disk, or to
"failed" if the render is skipped or errors. Previously the row was written
as "success" with a NULL report_file before the render finished, which a
concurrent reader could not tell apart from a completed pre-0.2.3 row and would
turn into a broken report link. Consumers linking to a report should treat a
"pending" row as not-yet-available rather than reconstructing a filename.
The drift report no longer silently reports a report that was never written.
When Quarto returned without error but produced no output file,
compare_snapshots() still announced (and, with open_report = TRUE, tried to
open) the intended path. The drift writer now raises the same typed error the
main report already did, and compare_snapshots() downgrades it to a warning
with no report link -- the computed drift is still returned.
Drift report filenames now include both compared snapshot ids
(drift_dataset_20260718_010203_1_2.html), so two comparisons of one dataset
started in the same wall-clock second (a loop over id pairs, two GUI users) no
longer collide on a single filename and silently overwrite each other -- the
same protection report_filename() already gives the main report.
Column statistics can no longer store a non-finite value even when the
aggregate itself overflows. An earlier fix excluded non-finite inputs
(Inf/-Inf from a corrupted delivery), but sd() of finite-but-very-large
values still overflows the double range internally and returned Inf, which
was stored as the literal string "Inf" and poisoned drift arithmetic. Every
numeric aggregate is now serialised through a finiteness guard (non-finite ->
NA), and the drift reader maps any non-finite value parsed from an older
database to NA as well, so a snapshot written before this fix cannot
re-poison a comparison.
Report filenames now include the snapshot id
(dataset_20260718_010203_47.html), so two runs of one dataset that start in
the same wall-clock second no longer collide on a single filename. Previously
the second run silently overwrote the first run's report while the first run's
snapshot kept pointing at it. The filename is now written to the snapshot's
report_file column by an update after the report is confirmed rendered,
rather than optimistically at insert time, so the column can never name a
report that was not written.
Configuration lookups no longer partial-match. R's $ operator falls back to
a prefix match when the exact element is absent, so a config with no
column_rules: key but a parked or mistyped one (column_rules_disabled,
column_rules_old, ...) had that section silently drive per-column
thresholds -- producing wrong PASS/FAIL verdicts from a section the config
did not contain. Every config-key access now uses exact [[ ]] indexing, so
a parked or renamed section is correctly treated as absent.
QC-16 no longer reports a spurious clean pass for a delivery whose encoding it
did not actually verify. A declared multi-byte or unknown encoding
(UTF-16LE, UTF-32, Shift-JIS, GB18030, ...) used to be reported as
"a single-byte encoding; every byte is valid by construction" -- which is
false, and which also silently disabled the crash guard that scanning gives
UTF-8 deliveries. Such an encoding is now read as declared but reported as a
WARN stating it was not validity-checked. Genuine single-byte encodings
(ISO-8859-x, Windows-125x) still PASS.
The UTF-8 validity scan now streams the file in bounded chunks instead of reading it into one in-memory vector, so an arbitrarily large delivery is verified in flat memory rather than exhausting it on the network-share hosts dqcheckr deploys to.
Non-finite values in a numeric column (Inf/-Inf, e.g. from a corrupted
upstream delivery whose CSV contains the literal text "Inf") are no longer
mishandled. compute_col_stats() excludes them from the mean, standard
deviation, minimum, and maximum, so the snapshot database can no longer store
the literal string "NaN"/"Inf" and poison later drift comparisons; the finite
mean shift a contaminated delivery causes is still reported as drift.
check_outliers() (QC-15), which previously aborted with an uninformative
"missing value where TRUE/FALSE needed" on such a column, now runs cleanly.
A zero-column delivery (for example an empty file) is now recorded. Previously
compute_col_stats() returned NULL, and the snapshot write then failed and
lost the entire run; the run is now snapshotted with a column count of zero.
Failures that were previously swallowed into silent, benign-looking results are now surfaced.
read_recent_snapshots() and list_snapshots() no longer report a
corrupt, locked, or unreadable database as an empty history. They emit a
warning naming the cause before returning the empty frame, so "the read
failed" is distinguishable from "no runs yet".run_dq_check() now appends
"[snapshot NOT recorded]" to its result line instead of printing an
unqualified success while the history row was lost.Concurrent runs sharing one snapshot database no longer lose a snapshot or die
during migration. Connections now set PRAGMA busy_timeout, so a run that
meets a database another run is writing waits for the lock instead of failing
instantly (its snapshot was previously swallowed into a warning). Schema
creation and the auto-migration of older databases now run inside a single
BEGIN IMMEDIATE transaction, so two first-runs after an upgrade can no
longer both add the same column (the loser used to die with "duplicate column
name"). WAL journalling is intentionally not used, as it is unsafe on the
network file systems dqcheckr is deployed on.
init_snapshot_db() no longer modifies a snapshots table it did not create.
If the target database already contains an unrelated table of that name, it
now aborts with a typed dqcheckr_schema_error instead of altering the user's
table. Column-existence checks during migration are now case-insensitive,
matching SQLite, so a column stored as e.g. Report_File is recognised rather
than re-added.
The drift report now flags a missing-rate or non-numeric-rate change in the
same direction as the corresponding comparison check. Both .compute_drift()
columns used abs(), so a column that improved between deliveries (fewer
missing values, or less non-numeric junk) was flagged as breaching, while the
CP-03 and CP-07 checks -- reading the same max_missing_rate_change_pp /
max_non_numeric_rate_change_pp thresholds -- passed it. Drift now breaches
only on an increase, matching the checks. (The numeric-mean-shift drift keeps
its two-directional test, matching the two-directional CP-04 check.)
A run whose HTML report is not produced no longer records a successful-looking
snapshot. When the Quarto CLI is absent the report is skipped with a warning,
but the snapshot row previously kept render_status = "success" and a
report_file naming a report that was never written, so history readers (and
the GUI's "Open report" link) pointed at a file that 404s. The snapshot is now
reconciled against what actually happened: if no report file exists the row is
marked render_status = "failed" and its report_file is cleared. Rendering
that returns without producing a file now raises instead of reporting success,
and report_file is guaranteed to name a report that exists.
Column type inference no longer misclassifies a value whose prefix happens
to be a date. as.Date() matches a prefix and silently ignores trailing
characters, so "2024-01-15x" and the 9-digit id "202401159" (whose first
eight digits parse under %Y%m%d) were both classified as "date" — which
made corrupt dates pass QC-06 and stripped the numeric checks (QC-07/08/11)
from id columns of eight or more digits. Each date format is now gated on an
anchored shape before calendar validation. The documented caveat is
unchanged: a genuine eight-digit value that is also a valid %Y%m%d date
still classifies as a date; a nine-digit id now correctly classifies as
numeric.
read_recent_snapshots() now returns the full set of columns even for a
snapshot database created before 0.2.3. Previously it ran SELECT * without
migrating, so an older database returned rows with the newer columns (such as
report_file) absent rather than NA, and callers relying on those
columns errored instead of degrading. The missing columns are now filled in
after the read with the same defaults a migration would apply; the database
file itself is not modified, so reads remain safe on read-only or
network-shared databases.
starwars_csv demonstration data (inst/demonstrations/data/)
is now shipped. An unanchored .gitignore rule had excluded it, so a fresh
clone was missing the file that 27 example blocks and two tests depend on,
and R CMD check failed on a clean checkout.starwars_fwf) now has its data file. A new
inst/demonstrations/makedata.R derives starwars.fwf from the CSV, so the
FWF half of demo.R runs end-to-end instead of aborting on a missing file.encoding of ASCII (or a formal alias such as US-ASCII) is now
read as UTF-8. ASCII is a strict subset of UTF-8, so this is lossless — and
it removes a hard R session crash ("Invalid multibyte sequence" inside
vroom/iconv, not a catchable R error) when a delivery declared ASCII
contains a byte above 127 beyond whatever sample an encoding sniffer
originally looked at.read_dataset() now validity-scans the entire file before parsing. A
delivery that is not valid UTF-8 no longer risks crashing or silently
producing mojibake: it is read with a single-byte fallback encoding, the
run completes, and QC-16 reports a FAIL naming the detector's best guess at
the actual encoding (suppliers can change export encodings between
deliveries, so this is checked per delivery). Valid files and declared
single-byte encodings (which have no invalid byte sequences) report PASS.
New dependency: stringi.run_dq_check() with an untyped "missing value where TRUE/FALSE needed"
error. Missing rates are defined as 0 for empty inputs, and QC-14 gains an
unconditional "Empty file" FAIL sub-check so an empty delivery always fails
the run — with a snapshot and report — instead of crashing it.detect_files() no longer considers subdirectories of folder when
picking the current/previous file by modification time.dq_result(threshold = NULL) now yields an NA threshold instead of
failing with "argument is of length zero"; invalid vector status values
produce a clear typed error.column_rules pattern now produces a QC-13 FAIL
result naming the pattern instead of aborting the run; an invalid
column_order_severity is rejected at load_config() time with a typed
dqcheckr_invalid_config error instead of aborting mid-check.run_custom_checks() now validates each returned element (required fields
and a valid status) and aborts with dqcheckr_invalid_custom_checks naming
the offending element, instead of failing later with misleading errors.snapshots row without its column_snapshots stats;
column-level custom results are batch-inserted.run_timestamp and the report filename are now derived from
one timestamp taken at the start of the run. Previously they came from two
separate clock reads, so links that reconstruct the report filename from
the snapshot timestamp (as the GUI does) could intermittently point at a
file with a name one second off.compare_snapshots() no longer announces (and offers to open) a drift
report path when Quarto is unavailable and no report was written.file.rename() fails across filesystems (e.g. a
network-share report directory); previously the move failed silently."2.10" matches an allowed value of 2.1.read_recent_snapshots()'s empty-database fallback now has the same
columns as the live query (it was missing comparison_mode,
render_status, and type_changed_cols_vs_previous).report_file column in the snapshots table stores the rendered
report's filename outright (auto-migrated into existing databases), so
consumers no longer need to reconstruct it from the run timestamp.
read_recent_snapshots() returns it; NA for pre-0.2.3 rows.run_dq_check() and compare_snapshots() now document that relative
snapshot_db / report_output_dir paths resolve against the working
directory, not config_dir.infer_col_type() documents its date-format precedence, the
all-must-parse rule, and the all-8-digit-identifier caveat, with a pointer
to column_types overrides.run_qc_checks() and the individual check functions
gain an optional types argument; previously each of QC-06/07/08/11/15,
the column statistics, and five comparison checks re-ran full-column type
inference (five date parses plus a numeric parse) independently — the
dominant cost on large files.infer_col_type() rejects non-matching date formats from a 100-value head
sample before scanning the full column (results are identical; only the
rejection path gets cheaper).compute_col_stats() builds one data frame per column instead of one per
statistic row, cutting allocations on wide files.csv_skip (parallel to fwf_skip):
read_dataset() now forwards skip = config$csv_skip %||% 0L to
readr::read_delim(). This lets a config supply an explicit col_names
list and drop the file's original header row — required for delivery
files whose header repeats column names (e.g. a header that repeats
Name/Amount per bundled record), where the real header is unusable and
must be replaced positionally. Defaults to 0L, so existing configs are
byte-for-byte unaffected.?dqcheckr package help now resolves (the "_PACKAGE" doc block no
longer carries @noRd).csv_skip value (skip exceeding the row
count yields a zero-row frame).expect_error().@importFrom declarations for template-only packages.read_pass_rate_trend() function from drift.R.CP-07 type-change comparison now uses a single guard so it only runs the
non-numeric rate comparison when both snapshots classify the column as
numeric — eliminates spurious WARNs on type-changed columns.read_dataset() now forwards the col_names config key to
readr::read_delim(), so headerless CSVs are read correctly.read_dataset() now forwards the quote_char config key to
readr::read_delim() as quote =.rlang::abort() calls (27 sites) now carry a typed, two-level class
hierarchy (c("<specific>", "dqcheckr_error")), letting callers catch
errors broadly or precisely.helper.R.compare_snapshots() and list_snapshots() functions for drift analysis.
Compares any two historical snapshots and optionally renders an HTML drift
report with per-column statistical drift, schema changes, and trend charts.resolve_col_type() function: returns the effective type for a column,
respecting per-column type overrides set in the column_types config key.QC-15 outlier detection check (check_outliers()). Configured via
max_z_score and/or iqr_fence_multiplier; skipped silently when neither is
set.check_key_uniqueness() (QC-12) now supports composite keys: set
key_columns to a character vector in the dataset YAML.check_min_row_count() (QC-14) gains max_row_count and max_file_size_mb
thresholds.col_threshold() and table_threshold() added as internal helpers;
column_rules per-column threshold overrides are now correctly stored in
column_snapshots.threshold (G-01).render_report() uses
quarto::quarto_render() and returns NULL with a warning when Quarto CLI
is not installed.compare_schema() (CP-02) split into three separate result objects:
CP-02a (new columns), CP-02b (dropped columns), CP-02c (type changes).compare_non_numeric_rate() (CP-07) now always emits a result for every
eligible column, including PASS for columns where the rate did not increase
(G-02).run_timestamp is now stored in ISO-8601 UTC format
(2026-01-01T12:00:00Z) rather than local time (RC-07).snapshots table gains three columns on first use: comparison_mode,
render_status, and type_changed_cols_vs_previous. Existing 0.1.x
databases are auto-migrated on the first 0.2.0 run.PRAGMA foreign_keys = ON).detect_files() uses filename as a secondary sort when two files share the
same modification time, making folder-mode ordering deterministic (RC-01).check_allowed_values() (QC-09), compare_new_values() (CP-05), and
compare_dropped_values() (CP-06) cap observed at 20 values with an
"... and N more" suffix (RC-04, RC-05).check_non_numeric() (QC-11) gains a warn_non_numeric_rate config key for
a separate WARN threshold (C-01).compare_missing_rate() (CP-03) gains a missing_rate_change_severity
config key (warn / fail) (B-07).compare_column_order() (CP-08) gains a column_order_severity config key
that overrides the format-based default (C-02).compute_col_stats() stores the numeric mean under the key
numeric_parseable_mean (renamed from numeric_mean) to clarify that
non-parseable values are excluded from the calculation (C-04).compare_snapshots() uses the full merged per-dataset config for drift
threshold comparisons, so *** markers match the original check run for
datasets with rule_overrides (G-05/G-06).compute_col_stats() unused qc_results parameter removed (D-01).flag_new_columns, flag_dropped_columns, flag_type_changes (in CP-02) and
flag_column_order_change (CP-08) are now honoured. Setting any flag to false
suppresses the corresponding check from the report. Schema changes are still
tracked in the SQLite snapshot regardless of flags.type_inference_threshold is now configurable per dataset via rule_overrides
in the dataset YAML (or default_rules in the global config). Previously fixed
at 90%, it now defaults to 90% if not set. Affects QC-06, QC-07, QC-08, QC-11,
CP-02, CP-04, CP-05, CP-06, and CP-07.Initial release.
dqcheckr.yml and per-dataset YAML files.