CAGED microdata straight into your 'DuckDB', with a complete pipeline ready for analysis.
datacaged is an R package that automates the entire workflow with CAGED (Cadastro Geral de Empregados e Desempregados — Brazil's General Register of Employed and Unemployed Workers) microdata. Data is downloaded from a HuggingFace repository via HTTPS, parsed with streaming decompression, and stored in a local 'DuckDB' database for fast, scalable queries with dplyr or SQL.
Automated download from HuggingFace (HTTPS — no FTP required)
Parallel downloads (workers = 3 by default)
Stream decompression of .7z files — no temporary disk extraction
Efficient storage in a local 'DuckDB' database
Full pipeline in a single function call
Integration with dplyr for data analysis
Compatible with Windows, macOS and Linux
Support for multiple datasets:
Stable version from CRAN:
install.packages("datacaged")
Development version from GitHub:
# install.packages("pak")
pak::pak("gecomt/datacaged")
| Feature | Windows | macOS | Linux |
|---|---|---|---|
| Download (HTTPS) | Yes | Yes | Yes |
| Parallel downloads | Yes | Yes | Yes |
| Novo CAGED (2020+, LZMA) | Yes | Yes | Yes |
| Legacy CAGED (pre-2020, PPMd) | Requires 7-Zip | Requires 7-Zip | Requires 7-Zip |
| DuckDB | Yes | Yes | Yes |
Legacy CAGED (pre-2020) uses PPMd compression which requires 7-Zip. Novo CAGED (2020+) works on all platforms without any extra software.
Windows — install the 7-Zip installer (standard installation to C:\Program Files\7-Zip is automatically detected).
macOS
brew install 7-zip
Linux (Debian/Ubuntu)
sudo apt install 7zip
Linux (Fedora/RHEL)
sudo dnf install 7zip
library(datacaged)
# Full pipeline — downloads, parses and writes to DuckDB
caged_load(
years = 2022:2023,
months = seq_len(12L),
db_path = file.path(tempdir(), "caged.duckdb")
)
# Connect and query
con <- caged_connect(file.path(tempdir(), "caged.duckdb"))
library(dplyr)
tbl(con, "caged_mov") |>
filter(uf == 35, competenciamov >= 202201L) |>
group_by(competenciamov) |>
summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |>
collect()
DBI::dbDisconnect(con, shutdown = TRUE)
# Parallel downloads (default: 3 workers)
caged_download(years = 2023, months = 1:12, workers = 3)
# Sequential (for slow connections or debugging)
caged_download(years = 2023, months = 1:12, workers = 1)
# Set globally
options(datacaged.workers = 4)
Parallel downloads process MOV, FOR and EXC files simultaneously per month — approximately 3× faster than sequential for Novo CAGED.
HuggingFace (HTTPS)
↓
caged_download() ← parallel, with local cache
↓
caged_parse() ← stream decompression via archive_read()
↓
caged_to_duckdb() ← bulk insert, deduplication by period
↓
caged_connect() ← dplyr / SQL / DBI
| Function | Description |
|---|---|
caged_load() |
Full pipeline (download → parse → DuckDB) |
caged_adjustments_load() |
Adjustments pipeline |
caged_download() |
Download .7z files with cache |
caged_download_layouts() |
Download official MTE layouts (from HuggingFace) |
caged_parse() |
Parse a single .7z file |
caged_parse_batch() |
Batch parse |
caged_to_duckdb() |
Write to DuckDB |
caged_connect() |
Open DBI connection |
caged_info() |
Database statistics |
caged_status() |
Check HuggingFace connectivity |
caged_hf_files() |
List available competencies |
caged_update() |
Download only new competencies |
caged_to_parquet() |
Export tables to Parquet files |
Console messages (errors, warnings and progress) are written in English and translated to Portuguese. With R set to Portuguese they appear in Portuguese automatically; to switch within a session:
Sys.setLanguage("pt_BR") # Portuguese
Sys.setLanguage("en") # English
| Table | Content | Period |
|---|---|---|
caged_mov |
Movements | 2020+ |
caged_for |
Late declarations | 2020+ |
caged_exc |
Exclusions | 2020+ |
caged_antigo |
Historical data | 2007–2019 |
caged_ajustes |
Retroactive adjustments | 2002–2019 |
MIT © Alexsandro Prado
Author: Alexsandro Prado
Email: alexsandro.prado@ufersa.edu.br
GitHub: https://github.com/gecomt/datacaged