                        Changes in version 0.5.0                        

This release makes the build of the datasets reproducible and prepares
monthly data updates. The data of the six datasets of 0.4.0 changes only
through the fixes listed below, and aqua joins as the seventh dataset.
No new journal issues are added yet; the first monthly update follows as
0.6.0.

Coverage

Five datasets are updated monthly from now on. uncnewsletter and
datapapers are frozen: still built with every release, no longer
updated.

| Dataset | Updates | Articles | Years | Latest issue |
|---|---|---:|---|---| | washdev | monthly | 1,173 | 2011 to 2026 |
Vol. 16 Issue 6 | | ws | monthly | 4,884 | 2001 to 2026 | Vol. 26
Issue 6 | | jwh | monthly | 2,013 | 2003 to 2026 | Vol. 24 Issue 6 | |
aqua | monthly | 1,819 | 1998 to 2026 | Vol. 75 Issue 6 | | ploswater |
monthly | 434 | 2022 to 2026 | Vol. 5 Issue 7, published up
to 2026-07-06 | | uncnewsletter | frozen | 173 | 2020 to 2023 |
newsletter ceased in May 2024 | | datapapers | frozen | 8 | 2018 to 2025
| harvested on 23 July 2026 |

Reproducible build

  - The datasets are built by a targets pipeline from the raw snapshots
    and decision sheets committed in data-raw/, with the package
    versions pinned in renv.lock. One command rebuilds everything:
    Rscript data-raw/build.R. Manual corrections moved from the code
    into decision sheets keyed on the article identifier.

  - A validation gate checks every dataset before anything is written:
    documented columns, a unique key, no email addresses or credentials,
    statement text wherever a statement is flagged, UN country names,
    and no article lost since the last release.

  - A workflow on GitHub rebuilds the datasets on every push and pull
    request and fails when they differ from the committed data
    (data-raw/check_reproducible.R).

  - Zenodo files the release as a dataset in the openwashdata community,
    from the new .zenodo.json. The releases 0.1.0, 0.3.0 and 0.4.0 were
    never archived there.

New features

  - New dataset aqua: all 1,819 articles of the journal AQUA - Water
    Infrastructure, Ecosystems and Society in the journal's online
    archive, 1998 to 2026, with the same columns as ws and jwh. 539
    articles carry a data availability statement; 529 of these map to a
    shared das_type and the remaining 10 keep their full text and are
    listed in data-raw/aqua-das-review.csv.
    
    The 0.4.0 notes said AQUA needed a re-scrape because its statement
    text had never been captured. That was wrong. The statements were in
    the raw snapshot all along. The build read the snapshot with guessed
    column types, the journal's first statement sits beyond the rows a
    guess looks at, so the statement column was read as logical and came
    back empty. The build now states every column type (#52).

Bug fixes

  - washdev: 33 correspondence author countries were three letter ISO
    codes ("IND", "GBR") in a column that otherwise holds United Nations
    country names. 32 are now the UN name. The 33rd, an affiliation in
    Taiwan, is missing, as it already was in the first author column:
    the UN names have no entry for it. The article is listed in
    data-raw/washdev-country-review.csv.

  - ploswater: two articles appeared twice (10.1371/journal.pwat.0000024
    and 10.1371/journal.pwat.0000520), each as two rows identical in
    every column. The dataset now has 434 rows, one per article. The
    cause was a search query paged without a sort order.

  - Credentials are removed from the text of the datasets. Where a
    statement linked a restricted Zenodo record through an access token,
    the token parameter is now dropped and the plain record URL kept; up
    to v0.4.0 the URL kept the parameter with a placeholder value and
    did not resolve (ploswater, one article). Where authors wrote a
    password into their statement, such as the login of an FTP site, the
    password is replaced by [removed] and the rest of the sentence kept
    (ploswater and ws, one statement each).

  - ws: one data availability statement kept an author email address in
    das_type. The masking of addresses inside text, introduced in
    v0.4.0, skipped that column because it is a factor. The address is
    now masked there as it already was in das.

Changes

  - washdev, ws, jwh: the levels of das_type now list the shared
    statement types first ("available in online repository", "in paper",
    "on request"), followed by the statements no rule mapped. Before,
    the three types sat scattered among the statements in alphabetical
    order. Values are unchanged; code that relies on the integer codes
    of the factor needs checking.

  - Credentials are removed from the raw snapshots in data-raw/, as they
    already were from the datasets. The download links of supplementary
    files in washdev.csv, ws.csv, jwh.csv and aqua.csv lose their signed
    query and keep the file path (1,752 links of 1,532 articles, all
    expired since August 2026). In ploswater.csv the access tokens in
    the URLs of one statement are dropped. A password that authors wrote
    into their statement is replaced by [removed] (ploswater.csv and
    ws.csv, one statement each). The scrapers apply the same rules to
    new articles. The datasets are unchanged. The git history was not
    rewritten, so commits up to and including v0.4.0 still hold these
    values.

  - The raw snapshots (washdev.csv, ws.csv, jwh.csv, aqua.csv and
    ploswater.csv) no longer have the columns first_author_email and
    correspondence_author_email, and the scrapers no longer collect
    author email addresses. Contact addresses that some article pages
    print after the affiliation are masked in the same way as in the
    datasets (washdev.csv 10 articles, aqua.csv 4), and the scraper
    masks them in new articles. No other value in the snapshots changed,
    and the datasets are unchanged.
    
    The git history was not rewritten. Commits up to and including
    v0.4.0 still hold the removed values, and the archive of v0.0.1 on
    Zenodo holds the email columns of washdev.csv. Addresses that
    authors wrote inside a data availability statement stay in the raw
    statement text, which is the published statement, and are masked in
    the datasets as before.

Data acquisition

  - The scraper of the IWA journals is incremental. A run lists the
    journal issues again from the newest known publication year, scrapes
    the issues that are missing from the raw snapshot, and reads the two
    most recent issues again to fetch articles that were added late. An
    issue that the site lists before its articles are online is asked
    for again on the next run. The functions that decide what a run
    fetches are tested without network.

  - The scraper stores an issue complete or not at all, and writes no
    row for an article whose page was not read. A page counts as read
    only when the site served the kind of page that was asked for. Its
    "Not Found" page loads completely too and was taken for an article
    without authors, statement or supplement, or for an issue without
    articles.

  - washdev is scraped by the same scraper as ws, jwh and aqua
    (data-raw/iwa_scraping.R). The separate data-raw/washdev_scraping.R
    and the runner data-raw/run_iwa_scrapes.R are removed. The old
    washdev scraper read a throttled page as "no more issues" and stored
    an article that failed to load as an empty row. The index column
    that the first Python scraper left in data-raw/washdev.csv is
    removed. The dataset washdev is unchanged.

  - The signed download links of supplementary files, which work for
    about three weeks, are written to data-raw/private/, which is not
    committed. data-raw/suppfiles_download.R reads them there.

  - One command runs the monthly acquisition for every live source:
    Rscript data-raw/update_sources.R. It runs the four IWA journals in
    sequence with a cooldown, then PLOS Water, each as its own R
    process. A source that fails does not stop the others, and
    data-raw/update-log.csv records per journal and run how many journal
    issues and rows were added.

  - The PLOS Water downloader can run incrementally. It failed on the
    second run, because it read the existing snapshot with guessed
    column types and could not combine the publication date with the new
    rows. It now pages the search in a fixed sort order and keeps one
    row per DOI. The two repeated rows are removed from
    data-raw/ploswater.csv; the dataset has been without them since the
    fix listed above. The unsorted paging had also skipped two articles
    (10.1371/journal.pwat.0000223 and 10.1371/journal.pwat.0000302).
    They enter the dataset with the next data update.

  - A workflow opens an issue on the 5th of each month with the
    checklist of the monthly data update
    (data-raw/monthly-data-update.md).

Documentation

  - The help pages and the README say which datasets are updated monthly
    and which are frozen, and the README has a coverage table that is
    computed from the datasets.

  - uncnewsletter: the citations column is now described in the help
    page and the data dictionary. It has been in the dataset since its
    first release without documentation. The source of the counts is not
    recorded.

                        Changes in version 0.4.0                        

Breaking changes

  - The scraped author email addresses are removed from every dataset.
    The columns first_author_email and correspondence_author_email no
    longer exist in washdev, ploswater, uncnewsletter, ws or jwh. They
    were published up to and including v0.3.0. The addresses are
    personal data and earn nothing analytically, since the research
    questions use author country, das_type, keyword frequency and
    supplementary counts. A CC BY table of corresponding author
    addresses is a ready-made mailing list, which is the concrete harm.
    The addresses remain in data-raw/ for provenance; code that read
    either column needs updating. Addresses that authors wrote into the
    statements themselves are masked in place, keeping the domain so the
    statement still reads correctly (52 statements across the five
    datasets).

  - The flat-file exports in inst/extdata/ are no longer built into the
    installed package. They are still generated and still live in the
    repository and the Zenodo deposit, but shipping a CSV and an XLSX
    per dataset for six datasets would push the built tarball past the 5
    MB CRAN guidance. Calls to system.file("extdata", ..., package =
    "washopenresearch") no longer resolve; read the datasets directly
    instead, for example data(ws). inst/CITATION is unaffected and still
    ships, so citation("washopenresearch") is unchanged.

  - Access tokens are stripped from repository URLs. A few statements
    carried a Zenodo pre-signed link (?token=<JWT>) granting access to
    an otherwise restricted record; the token is replaced and the record
    URL kept, following the same reasoning as the expired Silverchair
    signatures in #10.

New features

  - Two new datasets covering the remaining IWA journals scraped in the
    same run as washdev (#32-#34). ws holds all 4,884 articles of Water
    Supply from 2001 to 2026, and jwh holds all 2,013 articles of the
    Journal of Water and Health from 2003 to 2026, both collected with
    data-raw/iwa_scraping.R and sharing the washdev schema. Together
    with washdev and ploswater this takes the corpus to 8,403 screened
    articles. Data availability statements were mapped to the shared
    das_type levels for 1,596 of 1,623 statements in ws and 717 of 733
    in jwh; the unmapped tail keeps the full statement text and is
    listed in data-raw/ws-das-review.csv and data-raw/jwh-das-review.csv
    (#12).

  - A fourth IWA journal, AQUA, was scraped in the same run but is not
    exported. 539 of its 1,819 rows carry has_das TRUE while das and
    das_type are empty, so the statement text was never captured and the
    dataset's central variable would ship empty. It needs a re-scrape
    first.

  - Yash Dubey is added as an author (#31). The contribution predates
    the R port: the Selenium-based scraper and a washdev data update,
    committed in December 2024 under the GitHub Action identity, which
    is why it was missed when the author list was last reviewed.

                        Changes in version 0.3.0                        

New features

  - New function das_in_paper_support() classifies how a "data in paper"
    data availability statement is backed by the recorded supplement
    fields (#47). The modal claim "all relevant data are included in the
    paper or its supplementary information" splits into "no supplement"
    (the claim rests on the printed tables alone), "unstructured
    supplement" (pdf or images only), "structured supplement" (docx,
    xlsx and similar), and "open supplement" (csv, txt, json, xml).
    Across washdev and the three IWA journal snapshots (2,599
    claims), 71.5% have no supplement, 26.5% a structured one, 2.1% an
    unstructured one, and none an open format.
    data-raw/das_in_paper_support.R reproduces the summary in
    data-raw/das-in-paper-support-summary.csv. A follow-up cross-check
    against OpenAlex article types
    (data-raw/das_in_paper_article_types.R, committed DOI-to-type
    lookup) shows the no-supplement claims are almost entirely
    substantive research articles: only 0.5% are front matter or
    reviews, so the article-type filter proposed in #47 does not shrink
    the population. Verifying the remaining 1,847 bare claims requires
    reading the articles' tables.
  - Supplement content audit (#47 tier 3): the 741 supplementary files
    of the IWA snapshots whose pre-signed CDN links were still valid
    were downloaded (checksummed manifest in
    data-raw/suppfiles/manifest.csv; the signatures lapse 2026-08-18
    to 2026-08-30, so 572 further links were already dead) and
    classified with transparent structural heuristics
    (data-raw/suppfiles_parse.R: pandoc-converted docx tables,
    readxl/xlsx sheet metrics). Of the 290 in-paper claims whose
    structured supplement was in hand, 54% share prose only, 21% summary
    tables, and 20% tables shaped like observations
    (data-raw/das_in_paper_suppfile_audit.R). xlsx files are the
    exception: 25 of 29 hold observation-shaped sheets. Heuristics are
    recorded per file and await validation against a manual sample.
  - New scripted acquisition pipeline for a fourth dataset, datapapers,
    covering WASH-related data papers in seven dedicated data journals
    (Scientific Data, Data in Brief, Gates Open Research, F1000Research,
    GigaScience, GigaByte, and Data (MDPI)) (#28). The pipeline lives in
    data-raw/01_datapapers_acquire.R (Crossref/Europe PMC harvest with a
    committed raw snapshot), data-raw/02_datapapers_screen.R (relevance
    screening captured in a committed decision sheet keyed on DOI), and
    data-raw/03_datapapers_process.R (harmonisation to the shared schema
    and export). The first harvest and screening round yielded 8 papers,
    shipped as the new datapapers dataset with CSV and XLSX exports in
    inst/extdata/.

Minor improvements and fixes

  - datapapers now carries the repository links its papers deposit to:
    data_repo_url and data_repo were NA for all 8 papers because the
    Crossref relation metadata was empty and the fallback planned for
    #27 never ran. The links were verified against Crossref relations,
    DataCite resource types, and the articles' availability sections,
    and are recorded in data-raw/datapapers_repo_fixes.csv, applied
    during processing. Seven papers use general repositories (GBIF, IEEE
    DataPort, Figshare, Dryad, NCBI BioProject, Zenodo); none uses a
    WASH sector platform.
  - Expired pre-signed CDN links in washdev$supp_url are rewritten to
    stable DOI URLs, and Google Scholar alert redirects in
    uncnewsletter$paper_url are decoded to their target URLs (#10).
    The 343 Silverchair links carried a January 2024 expiry, and the
    high-entropy signature tokens tripped secret scanners; the article
    DOI is recovered from the link path, so no re-collection is needed.
    Two helpers in data-raw/helpers.R, canonicalize_silverchair_url()
    and decode_scholar_redirect(), do the rewrites reproducibly.
  - The list-column collapsing helper and shared country-cleaning steps
    moved to data-raw/helpers.R, sourced by all processing scripts.
  - data-raw/README.md documents the run order and provenance of every
    committed snapshot and decision sheet.

                        Changes in version 0.2.0                        

New features

  - New dataset ploswater with all 436 articles of the journal PLOS
    Water from its first volume (2022) to July 2026, collected through
    the public PLOS API (#14). Beyond the shared schema it records
    das_repo_url (links and dataset DOIs in the data availability
    statement), das_repo_name (the recognized repository behind them),
    article_type, and publication_date (#15). Data availability
    statements are mandatory at PLOS: all 333 research articles carry
    one, and 177 articles link out to a data location.
  - washdev now covers volume 1 (2011) through volume 16 issue 6
    (June 2026) with 1173 observations, up from 932 (#12). The 241 new
    rows cover volumes 14 to 16.
  - washdev gains a doi column, filled for the newly scraped articles;
    earlier rows will be backfilled via Crossref (#20).
  - Data acquisition is now fully R. The washdev scraper was ported from
    Python/Selenium to data-raw/washdev_scraping.R using chromote and
    rvest, with incremental updates (#11). The Python tooling in
    inst/python/, including a 17 MB chromedriver binary, was removed
    (#17).

Bug fixes

  - The manual supplement-type corrections for uncnewsletter were
    indexed against washdev paperids and landed on the wrong rows, and
    correspondence_author_affiliation_country was never cleaned because
    the cleaned values were written into
    first_author_affiliation_country, overwriting it. Both are fixed and
    uncnewsletter was regenerated; country values changed on 56 rows and
    supplement types on 16 rows (#13).
  - Two author names in washdev (for example "Inês Freire Machete")
    carried Mac Roman bytes that made the xlsx export fail; the raw data
    is repaired at read time (#13).
  - .Rbuildignore excluded the whole inst/ directory from the built
    package, so citation("washopenresearch") and the inst/extdata files
    were missing from installed packages. The rule is now scoped
    correctly (#17).

Minor improvements

  - The data dictionary and roxygen documentation use the actual
    variable names first_author_affiliation_country and
    correspondence_author_affiliation_country for washdev (previously
    documented as *_affiliation_region) and document the url_source and
    doi variables. The uncnewsletter dictionary entry for issue_url is
    corrected from "Volume number of the journal" (integer) to the
    newsletter issue URL (character).
  - uncnewsletter is documented as a frozen source: the newsletter
    ceased publication in May 2024 (#17).
  - For newly scraped washdev rows, mixed supplementary file types are
    recorded as " & "-joined lists (one type per file) instead of the
    literal "misc", which previously required manual repair.
  - Data values that no cleaning rule could resolve are written to
    review files under data-raw/ (*-das-review.csv,
    *-country-review.csv) instead of being fixed by hand, so every
    correction stays in reproducible R code (#12, #15).

                        Changes in version 0.1.0                        

Breaking changes

  - washdev and uncnewsletter no longer contain list-columns (#8). The
    multi-value variables supp_file_type, supp_url, das_repo_url, and
    keywords are now character columns in which multiple values are
    separated by "; ". Flat-file exports such as write.csv() now work
    directly on both datasets. Code that used tidyr::unnest() or
    unlist() on these columns should split the strings instead, for
    example with tidyr::separate_rows(supp_file_type, sep = "; ") or
    stringr::str_split(keywords, "; ").
  - Depends was raised from R (>= 2.10) to R (>= 3.5), required by the
    serialization format of the regenerated data files.

Minor improvements and fixes

  - The CSV and XLSX exports in inst/extdata/ now show multi-value cells
    as "; "-separated strings instead of R code literals such as
    c("pdf", "docx").
  - The variable descriptions in the package documentation and the data
    dictionary describe the new delimited format and the correct
    variable types.
  - The article "Missed Opportunity: where is WASH research data gone?"
    uses tidyr::separate_rows() in place of tidyr::unnest() to expand
    supp_file_type.
  - The word cloud figure in the README has alt text.

                        Changes in version 0.0.1                        

  - Initial release with the washdev and uncnewsletter datasets on data
    availability statements in WASH research publications.
