Web data

textpress is organized around four actions – fetch, read, process, search. This vignette covers the first two. fetch_rss() retrieves recent entries from publisher-supported RSS and Atom feeds; read_urls() scrapes their content into a node-level data frame while retaining that provenance. Together they turn selected news sources into an analysis-ready corpus in a few lines.

Select, fetch, and read

Filter the bundled rss_politics catalog directly, use fetch_rss() to retrieve recent entries from those publishers, then filter the returned metadata locally before read_urls() scrapes and parses the selected articles. The filtering below is ordinary R filtering, not a query sent to a search service.

library(textpress)
library(dplyr)
library(DT)

feeds <- textpress::rss_politics |>
  filter(category %in% c("polling", "public_opinion"))

web_urls <- textpress::fetch_rss(feeds$url) |>
  filter(grepl(
    "immigration|poll|survey",
    paste(title, description),
    ignore.case = TRUE
  ))

Scrape and parse the URLs returned above into a node-level data frame with $text and $meta components.

web_text_list <- web_urls |>
  textpress::read_urls(cores = 4)

Build a text snippet from the first 15 words of each article, join to metadata, and display as an interactive table.

snippets <- web_text_list$text |>
  group_by(doc_id) |>
  summarise(text = {
    words <- unlist(strsplit(paste(text, collapse = " "), "\\s+"))
    paste(paste(words[seq_len(min(15, length(words)))], collapse = " "), "...")
  }, .groups = "drop")

metas_dt <- web_text_list$meta |>
  filter(!is.na(h1_title) & nzchar(trimws(h1_title))) |>
  left_join(snippets, by = "doc_id") |>
  arrange(desc(date)) |>
  mutate(
    title_link = paste0(
      '<a href="', url, '" target="_blank">', h1_title, '</a>'
    )
  )

DT::datatable(
  metas_dt |> select(date, source, title_link, text),
  options = list(columnDefs = list(
    list(targets = 2, orderable = FALSE)
  )),
  escape   = FALSE,
  rownames = FALSE
)

Summary

fetch_rss() and read_urls() are source-to-corpus entry points for a textpress pipeline. fetch_rss() returns publisher-provided article URLs with feed metadata; read_urls() scrapes and parses them into $text (one row per node) and $meta (one row per URL, including discovery provenance).