textpress

textpress is an R toolkit for building text corpora and searching them -- no custom object classes, just plain data frames from start to finish. It collects URLs from durable sources -- RSS and Atom feeds, Wikipedia and its citations, or URLs you already have -- then reads, processes, and searches the resulting text through a consistent four-step API: Fetch, Read, Process, Search.

Installation

From CRAN:

install.packages("textpress")

Development version:

remotes::install_github("jaytimm/textpress")

A source-to-corpus example

Select a small set of known publishers, retrieve their recent feed entries, then read the linked pages. read_urls() retains the RSS discovery metadata.

feeds <- textpress::rss_local_rags |>
  subset(state_abbr == "NM") |>
  head(2)

articles <- textpress::fetch_rss(feeds$url)
corpus <- textpress::read_urls(articles)

corpus$text
corpus$meta

There are three ways into the same pipeline:

Starting point	Entry point	Best for
Known publishers	`rss_politics` or `rss_local_rags` -> `fetch_rss()`	Recent, source-led collections
A topic	`fetch_wiki_urls()` -> `fetch_wiki_refs()`	Topic- and citation-led discovery
Existing URLs	`read_urls()` directly	Curated or externally collected corpora

Each route converges on read_urls() -> nlp_*() -> search_*().

The `textpress` API

Conventions: corpus is a data frame with a text column plus identifier column(s) passed to by (default doc_id). All outputs are plain data frames or data.tables; pipe-friendly.

1. Fetch (`fetch_*`)

Collect URLs and provenance from stable sources -- not full article text. textpress does not provide general-purpose live web search. Pass fetch results to read_urls() to retrieve their content.

fetch_rss(feed_url) -- Retrieve recent entries from RSS and Atom feeds; bundled catalogs are available as rss_politics and rss_local_rags.
fetch_wiki_urls(query, limit) -- Wikipedia article URLs matching a search phrase.
fetch_wiki_refs(url, n) -- External citation URLs from a Wikipedia article's References section.

2. Read (`read_*`)

Scrape and parse URLs into a structured corpus.

read_urls(x, ...) -- URL vector or fetch-result table → list(text, meta). Passing a table preserves its discovery metadata and doc_id. text is one row per node; meta is one row per URL.

3. Process (`nlp_*`)

Prepare text for search or indexing.

nlp_split_paragraphs() -- Break documents into structural blocks.
nlp_split_sentences() -- Segment blocks into individual sentences.
nlp_tokenize_text() -- Normalize text into a clean token stream.
nlp_index_tokens() -- Build a weighted BM25 index for ranked retrieval.
nlp_roll_chunks() -- Roll sentences into fixed-size chunks with surrounding context (RAG-style).

4. Search (`search_*`)

Four retrieval modes over the same corpus. Data-first, pipe-friendly.

Function	Query type	Use case
`search_regex(corpus, query)`	Regex pattern	Specific strings, KWIC with inline highlighting.
`search_dict(corpus, terms)`	Term vector	Exact phrases and MWEs; built-in `dict_generations`, `dict_political`.
`search_index(index, query)`	Keywords	BM25 ranked retrieval over a token index.
`search_vector(embeddings, query)`	Numeric vector	Semantic nearest-neighbor search; use `util_fetch_embeddings()` to embed.

RAG & LLM pipelines

textpress is designed to compose cleanly into retrieval-augmented generation pipelines.

Hybrid retrieval -- run search_index() and search_vector() over the same chunks, then merge with reciprocal rank fusion (RRF).

Context assembly -- nlp_roll_chunks() with context_size > 0 gives each chunk a focal sentence plus surrounding context, so retrieved passages are self-contained when passed to an LLM.

Because the stages exchange plain data frames, their outputs can also be inspected, filtered, saved, or passed to an LLM without conversion to a custom corpus class.

Vignettes

Web data -- fetch_rss() + read_urls()
Basic NLP -- sentence splitting, tokenization, span-aware casting
Wikipedia data -- fetch_wiki_urls() + fetch_wiki_refs()
Regex search -- search_regex(), KWIC
Dictionary search -- search_dict(), PMI co-occurrence
Semantic search -- RAG pipeline: embeddings, BM25, hybrid RRF retrieval, LLM extraction

License

Citation

citation("textpress")

Issues

Report bugs or request features at https://github.com/jaytimm/textpress/issues