Rule-driven, auditable cleaning of survey and assessment response data, implementing the WeianData Detect → Execute → Report workflow.
Status: Active · Owner: WEIAN DATA Engineering · License: GPL (>= 2)
DCC treats data cleaning as an auditable pipeline rather than a pile of ad-hoc edits. Raw data is immutable; every correction is driven by a declarative, versioned rule and recorded at the cell level, so any value in the cleaned dataset can be traced back to the finding and rule that changed it, and a whole run can be reproduced from a manifest.
It is built for survey and educational-assessment data: multi-format and multi-encoding input, response-quality detectors, answer-key scoring, multi-form item-bank alignment, and one normalized report model rendered for survey staff, statistical programmers, and AI agents.
# install.packages("remotes")
remotes::install_github("weiandata/DCC")
All formally supported input backends are installed with DCC: delimited and
fixed-width text, JSON/JSONL, XLS/XLSX/XLSB/ODS, SPSS/Stata/SAS/XPT,
Parquet/Feather, and RDS. dcc_capabilities()$formats reports each backend's
installed status and limitations; opening a file alone does not make a format
Stable.
For survey staff the shortest safe workflow uses a protected Excel plan and four simple functions:
library(DCC)
dcc_template("DCC-cleaning-plan.xlsx")
dcc_check("responses.csv", "DCC-cleaning-plan.xlsx", "dcc-check")
dcc_run("responses.csv", plan = "DCC-cleaning-plan.xlsx",
output_dir = "dcc-preview") # safe preview default
dcc_help("PLAN_COLUMN_TYPE") # plain-language fix
Only yellow cells are editable. The bilingual template is strict: unknown or
renamed sheets and columns fail with an exact Excel cell address. After review,
run again with a new output directory and mode = "execute". See the
Chinese and English
guides and the synthetic example.
Statistical programmers may use the same dcc_plan through
dcc_read_plan()/dcc_validate_plan(). AI agents can use JSON with
dcc_schema("plan"); Excel and JSON validate to the same contract.
library(DCC)
# 1. A small response file and a declarative rule set
csv <- tempfile(fileext = ".csv")
writeLines(c("sid,score", "S1,90", "S2,150", "S3,70"), csv)
rules_file <- tempfile(fileext = ".yaml")
writeLines(c(
"checks:",
" - id: R001",
" type: range",
" variable: score",
" min: 0",
" max: 100"
), rules_file)
# 2. Read -> Detect -> Execute
x <- dcc_read(csv)
rules <- dcc_rules(rules_file)
found <- dcc_detect(x, rules, id_var = "sid") # S2 is out of range
res <- dcc_execute(x, found,
actions = list(R001 = "set_na"), # blank the bad cell
id_var = "sid")
dcc_cleaned(res) # the corrected data
dcc_audit_log(res) # one row per change, with old/new value and rule
# 3. Report, trace, reproduce
model <- dcc_report_model(res)
dcc_report_staff(model, file.path(tempdir(), "staff"))
dcc_report_statistical(model, file.path(tempdir(), "statistical"),
table_format = "csv")
dcc_report_machine(model, file.path(tempdir(), "machine"))
dcc_report(res, tempfile(fileext = ".html")) # compatible legacy report
dcc_trace(res, "S2", "score") # full history of one cell
dcc_rerun(dcc_manifest(res))$reproduced # TRUE: byte-identical rerun
See vignette("dcc-pipeline", package = "DCC") for the full walkthrough.
Detect. dcc_read() remains the compatibility reader with encoding and
type inference. Strict workflows use dcc_import() with declared source
columns, canonical names, types, roles, missing codes, worksheet/range, and
encoding. The canonical result exposes dcc_dictionary() and
dcc_missing_states(). The registry covers CSV/TSV/TXT/FWF, JSON/JSONL,
XLS/XLSX/XLSB/ODS, SPSS/Stata/SAS/XPT, Parquet/Feather, and RDS, including
bounded gzip and explicit-member ZIP input. dcc_detect() evaluates a
versioned dcc_rules() set: range/set/expression checks plus five
response-quality detectors — missing items, straight-lining, response time,
trap items, and score anomalies — producing a structured dcc_findings object.
Execute. dcc_execute() applies declarative actions (exclude, set_na,
recode, flag) mapped to rule IDs. The whole plan is validated before any
data changes — unknown action IDs, unmapped recodes, and missing or duplicated
record IDs are errors, and findings without an explicit action are returned
unhandled rather than silently dispositioned. Every finding also has one
terminal disposition available through dcc_dispositions() (changed,
excluded, flagged, skipped, failed, or unhandled). Input data is never mutated;
every change is written to a cell-level audit log that carries the exact
finding_id it came from, with old/new value, triggering rule, method,
timestamp, and rule/key hashes. dcc_score() and dcc_map_forms() handle
answer-key scoring and multi-form item-bank alignment.
Report. dcc_report_model() creates the single validated source of truth
for a run. dcc_report_staff() writes a bilingual, redacted-by-default Excel,
HTML, and text result; dcc_report_statistical() writes complete analytical
tables, methods, provenance, and artifact hashes; dcc_report_machine() writes
versioned JSON/JSONL plus its schemas. dcc_run() can publish all three
atomically under staff/, statistical/, and machine/. They share one run
ID, counts, reconciliation, and hashes. PDF is optional and is not part of the
base output contract. The original dcc_report() remains available for
compatibility. dcc_trace() gives cell-level lineage; dcc_manifest() /
dcc_rerun() provide one-command, hash-verified reproduction.
Scale. Core work runs on data.table for million-row in-memory workloads.
dcc_detect_chunked() streams larger-than-memory files with an adaptive backend
(data.table for delimited text, arrow for Parquet/Feather), with findings
identical to the in-memory path.
Contracts. dcc_capabilities() returns a versioned, machine-readable list
of every Stable, Experimental, and Planned feature, rule type, action type, and
input format, including extensions, backend, semantics, and limitations.
dcc_doctor(formats = "all") checks installed backend versions and platform
constraints. dcc_schema() returns published JSON Schemas for strict plans,
findings, audit logs, dispositions, provenance, rule files, action maps, and
manifests, so programmatic and AI callers work against a stable contract.
vignette("dcc-pipeline") — end-to-end walkthroughAll example and test data are synthetic. Do not commit credentials, secrets, personal information, or unapproved datasets; follow SECURITY.md for private vulnerability reporting.
Released under the GNU General Public License version 2 or later. Copyright © 2026 WEIAN DATA TECH (Beijing) Co., Ltd. (惟安数据科技（北京）有限公司). Dependency copyright boundaries are recorded in inst/COPYRIGHTS.