                        Changes in version 4.5.1                        

Bug fixes

  - Fixed tests that failed under tm 0.7-20, which no longer attaches
    the NLP package, causing CRAN check errors on r-devel.

  - Fixed a bug in tokens() when the object's concatenator is an empty
    character.

Changes and additions

  - Verbose messages from corpus, tokens, dfm and fcm methods now
    summarise the object returned, in the form "Returning tokens
    of 5,394 documents (55,866 tokens, 1,000 types) from tokens_trim()",
    instead of comparing counts before and after the operation.
    fcm_select(), fcm_keep() and fcm_remove() now report on the fcm they
    return, not on an intermediate dfm. (#2016, #2518)

  - The summary line printed by print() for corpus, tokens, dfm and fcm
    objects now has a common format, e.g. "tokens of 9 documents (6,855
    tokens, 1,812 types)." It replaces "Corpus consisting of", "Tokens
    consisting of", "Document-feature matrix of:" and "Feature
    co-occurrence matrix of:", and adds the number of characters
    (corpus), tokens and types (tokens), occurrences (dfm), and
    co-occurrences and sparsity (fcm). Code or tests that match the old
    text will need updating. (#2518)

  - featfreq() and docfreq() now work on tokens and tokens_xptr objects,
    computing feature and document frequencies directly without first
    creating a dfm. docfreq() supports all weighting schemes for these
    objects, but threshold is only supported for a dfm. (#2523)

                         Changes in version 4.5                         

Bug fixes

  - Fix dictionary() to handle separator correctly when a YAML file is
    provided.

Changes and additions

  - Added tokens_recompile() to reassign token IDs and remove duplicate
    or unused types from tokens and tokens_xptr objects. (#2506)

  - Query functions such as types(), ntype(), and ntoken() no longer
    recompile tokens_xptr objects. Subsetting documents now preserves
    the complete type table, consistently with dfm objects. (#2506)

  - Improve print methods for corpus, tokens, dfm, fcm, dictionary and
    kwic objects to wrap long lines at getOption("width").

  - Added tokens_match() to match token IDs in separately tokenized
    objects.

  - Added an as.matrix() method for tokens objects, returning a dense
    [document × position] integer matrix of token IDs, with length,
    extract, and drop arguments. (#2497)

  - as.tensor() for tokens now returns a dense tensor rather than a
    sparse COO tensor, gains an extract argument for selecting
    documents, and passes ... through to torch::torch_tensor(). (#2497)

  - Added docvars() and docvars<-() methods for tokens_xptr objects,
    allowing document variables to be read and assigned without
    recompiling the tokens. (#2497)

  - Added trim = TRUE to dfm() to allow users to disable trimming of
    features.

  - Add the drop0() method for dfm and fcm (#2460).

  - Added the max_n argument to dfm_trim() and tokens_trim() to keep
    only the most frequent features, with ties broken by feature order.
    (#2496)

  - Improved the C++ code for tokens and tokens_xptr objects to reduce
    memory usage.

                  Changes in version 4.4 (2026-04-06)                   

Changes and additions

  - Added levels argument to dictionary() to select levels (#2001).

  - Added normalize argument to tokens() to replace Unicode quotation
    marks and hyphens with their ASCII equivalent (#2480).

  - Added as.tensor() method for tokens objects to convert tokens to
    sparse tensors compatible with the \pkg{torch} package (#2474).

  - Added tokens_annotate() to insert tags to a tokens object using a
    dictionary.

  - Added tokenize argument to dictionary() to match dictionary values
    and tokens more easily in Japanese and Chinese (#2476).

  - Update corpus_reshape() to segment paragraphs more accurately
    (#2468).

  - Update index() and kwic() to skip nested matches in a similar way as
    tokens_lookup() (#2063).

  - Update corpus() to always append segment numbers to docnames when
    unique_docnames = FALSE. This argument is also added to as.tokens()
    and docnames<- for the same behaviour (#2486).

                 Changes in version 4.3.1 (2025-07-10)                  

Changes and additions

  - Improved the speed of tokens_trim() when tokens are very large.

Bug fixes

  - Fixed the Trump 2025 speech in data_corpus_inaugural, which was
    mistakenly a duplicate of Biden 2021.

                 Changes in version 4.3.0 (2025-05-20)                  

Changes and additions

  - Added corpus_chunk() for chunking texts into smaller documents.

  - Significantly reduce the memory usage for the c operation on large
    tokens and tokens_xptr objects.

  - Further improvements to the verbose messages for corpus, tokens, dfm
    and fcm objects.

  - tokens_ngrams() now includes a new argument apply_if, functioning
    similar to this argument in tokens_compound() and tokens_lookup()
    (#2390).

  - Replaced remove_unigram with match_pattern in object2id() to control
    the matching of single-word patterns or multi-word patterns.

  - data_corpus_inaugural now updated for Trump 2025.

                 Changes in version 4.2.0 (2025-01-08)                  

Changes and additions

  - Made the c operation on tokens and tokens_xptr objects significantly
    faster.

  - New, and more consistent verbose messages for tokens and dfm
    objects.

  - Preserve the default concatenator of tokens objects in
    tokens_compound() (#2432).

  - Make the c operation on tokens and tokens_xptr objects significantly
    faster.

Bug fixes and stability enhancements

  - Fix a bug in dfm_lookup() that leads to wrong feature names when
    exclusive = TRUE (#2424).

                         Changes in version 4.1                         

Bug fixes and stability enhancements

  - Improved the handling of invisible control characters causing some
    tokens operations to crash (#2407).

  - Addressed #2358 and #2359 more thoroughly, at the C++ level, to
    convert list(integer(), integer(),...) to std::vector<std::vector.
    This function removes negative values and NA_INTEGER.

  - Removed RcppArmadillo as a dependency in an effort to avoid UBSAN
    warnings in #2417.

Changes and additions

  - Added tokens_trim() function similar to dfm_trim() (#2419).

  - Added keep_unigrams argument to tokens_compound(), to keep in the
    returned object the unigrams that are to be compounded (#2399).

  - print.tokens() now allows passing arguments to base print() via ...,
    providing for instance the ability to print tokens without surround
    quotes using quote = FALSE (#2381).

                 Changes in version 4.0.2 (2024-04-24)                  

Bug fixes and stability enhancements

  - Rewrite the recompilation function to serialize tokens more
    systematically (#2375).

                 Changes in version 4.0.1 (2024-04-08)                  

Bug fixes and stability enhancements

  - Make detection of Intel oneAPI Threads Building Blocks (TBB) library
    more reliable.

                 Changes in version 4.0.0 (2024-04-04)                  

Changes and additions

  - Introduces the tokens_xptr objects that extend the tokens objects
    with external pointers for a greater efficiency. Once tokens objects
    are converted to tokens_xptr objects using as.tokens_xptr(),
    tokens_*.tokens_xptr() methods are called automatically.

  - Improved C++ functions to allow the users to change the number of
    threads for parallel computing in more flexible manner using
    quanteda_options(). The value of threads can be changed in the
    middle of analysis pipeline.

  - Makes "word4" the default (word) tokeniser, with improved
    efficiency, language handling, and customisation options.

  - Replaced all occurrences of the magrittr %>% pipe with the R pipe |>
    introduced in R 4.1, although the %>% pipe is still re-exported and
    therefore available to all users of quanteda without loading any
    additional packages.

  - Added min_ntoken and max_ntoken to tokens_subset() and dfm_subset()
    to extract documents based on number of tokens easily. It is
    equivalent to selecting documents using ntoken().

  - Added a new argument apply_if that allows a tokens-based operation
    to apply only to documents that meet a logical condition. This
    argument has been added to tokens_select(), tokens_compound(),
    tokens_replace(), tokens_split(), and tokens_lookup(). This is
    similar to applying purrr::map_if() to a tokens object, but is
    implemented within the function so that it can be performed
    efficiently in C++.

  - Added new arguments append_key, separator and concatenator to
    tokens_lookup(). These allow tokens matched by dictionary values to
    be retained with their keys appended to them, separated by
    separator. The addition of the concatenator argument allows
    additional control at the lookup stage for tokens that will be
    concatenated from having matched multi-word dictionary values.
    (#2324)

  - Added a new argument remove_padding to ntoken() and ntype() that
    allows for not counting padding that might have been left over from
    tokens_remove(x, padding = TRUE). This changes the previous number
    of types from ntype() when pads exist, by counting pads by default.
    (#2336)

  - Removed dependency on RcppParallel to improve the stability of the
    C++ code. This change requires the users of Linux-like OS to install
    the Intel TBB library manually to enable parallel computing.

Removals

  - bootstrap_dfm() was removed for character and corpus objects. The
    correct way to bootstrap sentences is not to tokenize them as
    sentences and then bootstrap them from the dfm. This is consistent
    with requiring the user to tokenise objects prior to forming dfms or
    other "downstream" objects.

  - dfm() no longer works on character or corpus objects, only on tokens
    or other dfm objects. This was deprecated in v3 and removed in v4.

  - Very old arguments to dfm() options that were not visible but worked
    with warnings (such as stem = TRUE) are removed.

  - Deprecated or renamed arguments formerly passed in tokens() that
    formerly mapped to the v3 arguments with a warning are removed.

  - Methods for readtext objects are removed, since these are data.frame
    objects that are straightforward to convert into a corpus object.

  - topfeatures() no longer works on an fcm object. (#2141)

Deprecations

  - Some on-the-fly calculations applied to character or corpus objects
    that require a temporary tokenisation are now deprecated. This
    includes:
    
      - nsentence() -- use lengths(tokens(x, what = "sentence"))
        instead;
      - ntype() -- use ntype(tokens(x)) instead; and.
      - ntoken() -- use ntoken(tokens(x)) instead.
      - char_ngrams() -- use tokens_ngrams(tokens(x)) instead.

  - corpus.kwic() is deprecated, with the suggestion to form a corpus
    from using tokens_select(x, window = ...) instead.

Bug fixes and stability enhancements

  - tokens_group() works efficiently even when the number of documents
    and groups are very large.

                 Changes in version 3.3.1 (2023-05-18)                  

Bug fixes and stability enhancements

  - Fixed a potential crash when calling tokens_compound() with patterns
    containing paddings (#2254).

  - Updated for compatibility with (forthcoming) Matrix 1.5.5 handling
    of dimnames() for empty dimensions.

  - restores readtext object class method extensions, to work better
    with the readtext package.

  - Removes some unused internal methods, such as docvars.kwic() that
    were not exported despite matching exported generics.

                 Changes in version 3.3.0 (2023-04-07)                  

Changes and additions

  - Implements a "word4" tokeniser that is based on new RBBI
    (RuleBasedBreakIterator) rules, implemented in a new .yml file that
    can be edited and changed by users, but whose defaults represent a
    significant improvement in pattern handling for words, sentences,
    and other forms of patterns. These rules are customised from the ICU
    rules for breaks, with the standard and customised rules found now
    in the breakrules/ system folder, so that they could, in principle,
    be modified by the user.

  - Other minor changes:
    
      - changes how elapsed time is recorded, by creating a global
        environment to record these in (aaa.R)
      - improves several of the R-coded patterns that apply to "word2":
          - the hashtag pattern (`pattern_hashtag)
          - the separator pattern (by adding \\p{M}).
          - the URL pattern
      - creates a new tokens_restore(), implemented in C++, to replace
        the older preserve_special() that rejoined splits created by the
        default stringi tokeniser machinery.
      - makes some technical improvements to internal tokenisation
        functions, such as moving the ellipsis to the end of the
        function, to allow more modularity in developing future
        tokenisers.

Bug fixes and stability enhancements

  - dfm_group() now works correctly with an empty dfm (#2225).
  - convert(x, to = "stm") no longer vulnerable to large numbers of
    removed features as in #2189.

                        Changes in version 3.2.5                        

Changes and additions

  - segid() is added to extract document serial numbers from corpus,
    tokens or dfm objects.

                 Changes in version 3.2.4 (2022-12-08)                  

Bug fixes and stability enhancements

Fixes test failures caused by recent changes to Matrix package
behaviours on some operating systems.

                 Changes in version 3.2.3 (2022-08-29)                  

Bug fixes and stability enhancements

  - Matrix package calls updated for compatibility with Matrix 1.4.2.
    (#2182)
  - Changes to C++ code for fcm() to prevent some (chance) errors
    downstream in LSX. (#2181)

                 Changes in version 3.2.2 (2022-08-09)                  

Bug fixes and stability enhancements

  - fcm() computes the marginal frequency of upper-case tokens correctly
    (#2176).
  - tokens_chunk() keeps all the docid, including those of empty
    documents, in the original object.
  - tokens_select() recycles values when the length of startpos or
    endpos is less than ndoc(x).
  - tokens_lookup() and dfm_lookup() can apply very large dictionaries
    (more than 100,000 keys).

                 Changes in version 3.2.1 (2022-03-01)                  

Bug fixes and stability enhancements

  - dfm_lookup() ignores matches of multiple dictionary values in the
    same key in a similar way as tokens_lookup() (#2159).

Changes and additions

  - A new split_tags argument has been added to tokens(), to provide the
    user with an option not to preserve social media tags (addresses
    #2156).

                         Changes in version 3.2                         

Bug fixes and stability enhancements

  - dfm() returns a dfm with the identical column order even if
    tokens_compound() or tokens_ngrams() is used in the upstream
    (#2100).
  - dfm_group() with NA values in a grouping variable now drops those,
    similar to the behaviour of tokens_group() and corpus_group()
    (#2134).

Changes and additions

  - char_wordstem() now has a a new argument check_whitespace, which
    will not throw an error when lower-casing text containing a
    whitespace character.
  - dfm_remove() now has a new argument padding = FALSE that when TRUE,
    collects counts of the removed features in the first column. This
    produces results consistent with what is compiled as a dfm built
    from tokens where some have been removed with padding = TRUE
    (#2152).

                         Changes in version 3.1                         

Bug fixes and stability enhancements

  - Improved and more consistent handling of empty corpus, tokens and
    dfm objects, to address #2110.
  - rbind.dfm() now preserves docvars (#2109).
  - Document name for Biden's 2021 Inaugural Address in
    data_corpus_inaugural is now consistent with all other documents.
  - Fix #2127 that caused subsetting to change document names.

Changes and additions

  - phrase() now has a separator argument (#2124).

Deprecations

  - phrase() methods for tokens, collocations, and lists are deprecated
    in favour of as.phrase() (#2129).

                         Changes in version 3.0                         

quanteda 3.0 is a major release that improves functionality, completes
the modularisation of the package begun in v2.0, further improves
function consistency by removing previously deprecated functions, and
enhances workflow stability and consistency by deprecating some shortcut
steps built into some functions.

Changes and additions

  - Modularisation: We have now separated the textplot_*() functions
    from the main package into a separate package quanteda.textplots,
    and the textstat_*() functions from the main package into a separate
    package quanteda.textstats. This completes the modularisation begun
    in v2 with the move of the textmodel_*() functions to the separate
    package quanteda.textmodels. quanteda now consists of core functions
    for textual data processing and management.

  - The package dependency structure is now greatly reduced, by
    eliminating some unnecessary package dependencies, through
    modularisation, and by addressing complex downstream dependencies in
    packages such as stopwords. v3 should serve as a more lightweight
    and more consistent platform for other text analysis packages to
    build on.

  - We have added non-standard evaluation for by and groups arguments to
    access object docvars:
    
      - The *_sample() functions' argument by, and groups in the
        *_group() functions, now take unquoted document variable
        (docvar) names directly, similar to the way the subset argument
        works in the *_subset() functions.
      - Quoted docvar names no longer work, as these will be evaluated
        literally.
      - The by = "document" formerly sampled from docid(x), but this
        functionality is now removed. Instead, use by = docid(x) to
        replicate this functionality.
      - For groups, the default is now docid(x), which is now documented
        more completely. See ?groups and ?docid.

  - dfm() has a new argument, remove_padding, for removing the "pads"
    left behind after removing tokens with padding = TRUE. (For other
    extensive changes to dfm(), see "Deprecated" below.)

  - tokens_group(), formerly internal-only, is now exported.

  - corpus_sample(), dfm_sample(), and tokens_sample() now work
    consistently (#2023).

  - The kwic() return object structure has been redefined, and built
    with an option to use a new function index() that returns token
    spans following a pattern search. (#2045 and #2065)

  - The punctuation regular expression and that for matching social
    media usernames has now been redefined so that the valid Twitter
    username @_ is now counted as a "tag" rather than as "punctuation".
    (#2049)

  - The data object data_corpus_inaugural has been updated to include
    the Biden 2021 inaugural address.

  - A new system of validators for input types now provides better
    argument type and value checking, with more consistent error
    messages for invalid types or values.

  - Upon startup, we now message the console with the Unicode and ICU
    version information. Because we removed our redefinition of View()
    (see below), the former conflict warning is now gone.

  - as.character.corpus() now has a use.names = TRUE argument, similar
    to as.character.tokens() (but with a different default value).

Deprecations

The main potentially breaking changes in version 3 relate to the
deprecation or elimination of shortcut steps that allowed functions that
required tokens inputs to skip the tokens creation step. We did this to
require users to take more direct control of tokenization options, or to
substitute the alternative tokeniser of their choice (and then coercing
it to tokens via [as.tokens()]). This also allows our function behaviour
to be more consistent, with each function performing a single task,
rather than combining functions (such as tokenisation and constructing a
matrix).

The most common example involves constructing a dfm directly from a
character or corpus object. Formerly, this would construct a tokens
object internally before creating the dfm, and allowed passing arguments
to tokens() via .... This is now deprecated, although still functional
with a warning.

We strongly encourage either creating a tokens object first, or piping
the tokens return to dfm() using %>%. (See examples below.)

We have also deprecated direct character or corpus inputs to [kwic()],
since this also requires a tokenised input.

The full listing of deprecations is:

  - dfm.character() and dfm.corpus() are deprecated. Users should create
    a tokens object first, and input that to dfm().

  - dfm(): As of version 3, only tokens objects are supported as inputs
    to dfm(). Calling dfm() for character or corpus objects is still
    functional, but issues a warning. Convenience passing of arguments
    to tokens() via ... for dfm() is also deprecated, but undocumented,
    and functions only with a warning. Users should now create a tokens
    object (using tokens() from character or corpus inputs before
    calling dfm().

  - kwic(): As of version 3, only tokens objects are supported as inputs
    to kwic(). Calling kwic() for character or corpus objects is still
    functional, but issues a warning. Passing arguments to tokens() via
    ... in kwic() is now disabled. Users should now create a tokens
    object (using tokens() from character or corpus inputs before
    calling kwic().

  - Shortcut arguments to dfm() are now deprecated. These are still
    active, with a warning, although they are no longer documented.
    These are:
    
      - stem -- use tokens_wordstem() or dfm_wordstem() instead.
      - select, remove -- use tokens_select() / dfm_select() or
        tokens_remove() / dfm_remove() instead.
      - dictionary, thesaurus -- use tokens_lookup() or dfm_lookup()
        instead.
      - valuetype, case_insensitive -- these are disabled; for the
        deprecated arguments that take these qualifiers, they are fixed
        to the defaults "glob" and TRUE.
      - groups -- use tokens_group() or dfm_group() instead.

  - texts() and texts<- are deprecated.
    
      - Use as.character.corpus() to turn a corpus into a simple named
        character vector.
      - Use corpus_group() instead of texts(x, groups = ...) to
        aggregate texts by a grouping variable.
      - Use [<- instead of texts()<- for replacing texts in a corpus
        object.

Removals

  - See note above under "Changes" about the textplot_*() and
    textstat_*() functions.

  - The following functions have been removed:
    
      - all methods for defunct corpuszip objects.
      - View() functions
      - as.wfm() and as.DocumentTermMatrix() (the same functionality is
        available via convert())
      - metadoc() and metacorpus()
      - corpus_trimsentences() (replaced by corpus_trim())
      - all of the tortl functions
      - all legacy functions related to the ancient "corpuszip" corpus
        variant.

  - dfm objects can no longer be used as a pattern in dfm_select()
    (formerly deprecated).

  - dfm_sample():
    
      - no longer has a margin argument. Instead, dfm_sample() now
        samples only on documents, the same as corpus_sample() and
        tokens_sample(); and
      - no longer works with by = "document" -- use by = docid(x)
        instead.

  - dictionary_edit(), char_edit(), and list_edit() are removed.

  - dfm_weight() - formerly deprecated "scheme" options are now removed.

  - tokens() - formerly deprecated options remove_hyphens and
    remove_twitter are now removed. (Use split_hyphens instead, and the
    default tokenizer always now preserves Twitter and other social
    media tags.)

  - Special versions of head() and tail() for corpus, dfm, and fcm
    objects are now removed, since the base methods work fine for these
    objects. The main consequence was the removal of the nf option from
    the methods for dfm and fcm objects, which limited the number of
    features. This can be accomplished using the index operator [
    instead, or for printing, by specifying print(x, max_nfeat = 6L)
    (for instance).

Bug fixes and stability enhancements

  - Fixed a bug causing topfeatures(x, group = something) to fail with
    weighted dfms (#2032).

  - kwic() is more stable and does not crash when a vector is supplied
    as the window argument (#2008).

  - Allow use of multi-threading with more than two threads by fixing
    quanteda_options().

  - Mentions of the now-removed ngrams option in dfm(x, ...) has now
    been removed from the dfm documentation. (#1990)

  - Handling for some early-cycle v2 dfm object is improved, to ensure
    that they are updated to the latest object format. (#2097)

                 Changes in version 2.1.2 (2020-09-23)                  

Changes

  - textstat_keyness() performance is now improved through
    implementation in (multi-threaded) C++.

Bug fixes and stability enhancements

  - Fixes breaking tests and examples on Solaris platform as well as
    other changes introduced by changes to the stringi package.

                 Changes in version 2.1.1 (2020-07-27)                  

Changes

Bug fixes and stability enhancements

  - corpus_reshape() now allows reshaping back to documents even when
    segmented texts were of zero length. (#1978)
  - Special handling applied for Solaris to some issues breaking on that
    build, relating to the cacheing in
    summary.corpus()/textstat_summary().

                 Changes in version 2.1.0 (2020-07-05)                  

Changes

  - Added block_size to quanteda_options() to control the number of
    documents in blocked tokenization.
  - Fixed print.dictionary2() to control the printing of nested levels
    with max_nkey (#1967)
  - Added textstat_summary() to provide detailed information about dfm,
    tokens and corpus objects. It will replace summary() in future
    versions.
  - Fixed a performance issue causing slowdowns in tokenizing (using the
    default what = "word") corpora with large numbers of documents that
    contain social media tags and URLs that needed to be preserved (such
    a large corpus of Tweets).
  - Updated the (default) "word" tokenizer to preserve hashtags and
    usernames better with non-ASCII text, and made these patterns
    user-configurable in quanteda_options(). The following are now
    preserved: "#政治" as well as Weibo-style hashtags such as "#英国首相#".
  - convert(x, to = "data.frame") now outputs the first column as
    "doc_id" rather than "document" since "document" is a commonly
    occurring term in many texts. (#1918)
  - Added new methods char_select(), char_keep(), and char_remove() for
    easy manipulation of character vectors.
  - Added dictionary_edit() for easy, interactive editing of
    dictionaries, plus the functions char_edit() and list_edit() for
    editing character and list of character objects.
  - Added a method to textplot_wordcloud() that plots objects from
    textstat_keyness(), to visualize keywords either by comparison or
    for the target category only.
  - Improved the performance of kwic() (#1840).
  - Added new logsmooth scheme to dfm_weight().
  - Added new textstat_summary() method, which returns summary
    information about the tokens/types/features etc in an object. It
    also caches summary information so that this can be retrieved on
    subsequent calls, rather than re-computed.

Bug fixes and stability enhancements

  - Stopped returning NA for non-existent features when n > nfeat(x) in
    textstat_frequency(x, n). (#1929)
  - Fixed a problem in dfm_lookup() and tokens_lookup() in which an
    error was caused when no dictionary key returned a single match
    (#1946).
  - Fixed a bug that caused a textstat_simil/dist object converted to a
    data.frame to drop its document2 labels (#1939).
  - Fixed a bug causing dfm_match() to fail on a dfm that included
    "pads" (""). (#1960)
  - Updated the data_dfm_lbgexample object using more modern dfm
    internals.
  - Updates textstat_readability(), textstat_lexdiv(), and nscrabble()
    so that empty texts are not dropped in the result. (#1976)

                 Changes in version 2.0.1 (2020-03-18)                  

Changes

  - Moved data_corpus_irishbudget2010 and data_corpus_dailnoconf1991 to
    the quanteda.textmodels package.
  - Em dashes and double dashes between words, whether surrounded by a
    space or not, are now converted to " - " to distinguish them from
    infix hyphens. (#1889)
  - Verbose output for dfm and tokens creation is now corrected and more
    consistent. (#1894)

Bug fixes and stability enhancements

  - Number removal is now both improved and fixed (#1909).
  - Fixed an issue causing CRAN errors in pre-v4, related to the new
    default of stringsAsFactors = FALSE for data.frame objects.
  - An error in the print method for dfm objects is now fixed (#1897)
  - Fixed a bug in tokens_replace() when the pattern was not matched
    (#1895)
  - Fixed the names of dimensions not exchanging when a dfm was
    transposed (#1903)

                         Changes in version 2.0                         

Changes

quanteda 2.0 introduces some major changes, detailed here.

1.  New corpus object structure.
    
    The internals of the corpus object have been redesigned, and now are
    based around a character vector with meta- and system-data in
    attributes. These are all updated to work with the existing
    extractor and replacement functions. If you were using these before,
    then you should not even notice the change. Docvars are now handled
    separately from the texts, in the same way that docvars are handled
    for tokens objects.

2.  New metadata handling.
    
    Corpus-level metadata is now inserted in a user metadata list via
    meta() and meta<-(). metacorpus() is kept as a synonym for meta(),
    for backwards compatibility. Additional system-level corpus
    information is also recorded, but automatically when an object is
    created.
    
    Document-level metadata is deprecated, and now all document-level
    information is simply a "docvar". For backward compatibility,
    metadoc() is kept and will insert document variables (docvars) with
    the name prefixed by an underscore.

3.  Corpus objects now store default summary statistics for efficiency.
    When these are present, summary.corpus() retrieves them rather than
    computing them on the fly.

4.  New index operators for core objects. The main change here is to
    redefine the $ operator for corpus, tokens, and dfm objects (all
    objects that retain docvars) to allow this operator to access single
    docvars by name. Some other index operators have been redefined as
    well, such as [.corpus returning a slice of a corpus, and [[.corpus
    returning the texts from a corpus.
    
    See the full details at
    https://github.com/quanteda/quanteda/wiki/indexing_core_objects.

5.  *_subset() functions.
    
    The subset argument now must be logical, and the select argument has
    been removed. (This is part of base::subset() but has never made
    sense, either in quanteda or base.)

6.  Return format from textstat_simil() and textstat_dist().
    
    Now defaults to a sparse matrix from the Matrix package, but
    coercion methods are provided for as.data.frame(), to make these
    functions return a data.frame just like the other textstat
    functions. Additional coercion methods are provided for as.dist(),
    as.simil(), and as.matrix().

7.  settings functions (and related slots and object attributes) are
    gone. These are now replaced by a new meta(x, type = "object") that
    records object-specific meta-data, including settings such as the n
    for tokens (to record the ngrams).

8.  All included data objects are upgraded to the new formats. This
    includes the three corpus objects, the single dfm data object, and
    the LSD 2015 dictionary object.

9.  New print methods for core objects (corpus, tokens, dfm, dictionary)
    now exist, each with new global options to control the number of
    documents shown, as well as the length of a text snippet (corpus),
    the tokens (tokens), dfm cells (dfm), or keys and values
    (dictionary). Similar to the extended printing options for dfm
    objects, printing of corpus objects now allows for brief summaries
    of the texts to be printed, and for the number of documents and the
    length of the previews to be controlled by new global options.

10. All textmodels and related functions have been moved to a new
    package quanteda.textmodels. This makes them easier to maintain and
    update, and keeps the size of the core package down.

11. quanteda v2 implements major changes to the tokens() constructor.
    These are designed to simplify the code and its maintenance in
    quanteda, to allow users to work with other (external) tokenizers,
    and to improve consistency across the tokens processing options.
    Changes include:
    
      - A new method tokens.list(x, ...) constructs a tokens object from
        named list of characters, allowing users to tokenize texts using
        some other function (or package) such as tokenize_words(),
        tokenize_sentences(), or tokenize_tweets() from the tokenizers
        package, or the list returned by spacyr::spacy_tokenize(). This
        allows users to use their choice of tokenizer, as long as it
        returns a named list of characters. With tokens.list(), all
        tokens processing (remove_*) options can be applied, or the list
        can be converted directly to a tokens object without processing
        using as.tokens.list().
    
      - All tokens options are now intervention options, to split or
        remove things that by default are not split or removed. All
        remove_* options to tokens() now remove them from tokens objects
        by calling tokens.tokens(), after constructing the object.
        "Pre-processing" is now actually post-processing using
        tokens_*() methods internally, after a conservative tokenization
        on token boundaries. This both improves performance and improves
        consistency in handling special characters (e.g. Twitter
        characters) across different tokenizer engines. (#1503, #1446,
        #1801)
    
    Note that tokens.tokens() will remove what is found, but cannot
    "undo" a removal -- for instance it cannot replace missing
    punctuation characters if these have already been removed.
    
      - The option remove_hyphens is removed and deprecated, but
        replaced by split_hyphens. This preserves infix (internal)
        hyphens rather than splitting them. This behaviour is
        implemented in both the what = "word" and what = "word2"
        tokenizer options. This option is FALSE by default.
    
      - The option remove_twitter has been removed. The new what =
        "word" is a smarter tokenizer that preserves social media tags,
        URLs, and email-addresses. "Tags" are defined as valid social
        media hashtags and usernames (using Twitter rules for validity)
        rather than removing the # and @ punctuation characters, even if
        remove_punct = TRUE.

New features

  - Changed the default value of the size argument in dfm_sample() to
    the number of features, not the number of documents. (#1643)
  - Fixes a few CRAN-related issues (compiler warnings on Solaris and
    encoding warnings on r-devel-linux-x86_64-debian-clang.)
  - Added startpos and endpos arguments to tokens_select(), for
    selecting on token positions relative to the start or end of the
    tokens in each document. (#1475)
  - Added a convert() method for corpus objects, to convert them into
    data.frame or json formats.
  - Added a spacy_tokenize() method for corpus objects, to provide
    direct access via the spacyr package.

Behaviour changes

  - Added a force = TRUE option and error checking for the situations of
    applying dfm_weight() or dfm_group() to a dfm that has already been
    weighted. (#1545) The function textstat_frequency() now allows
    passing this argument to dfm_group() via .... (#1646)
  - textstat_frequency() now has a new argument for resolving ties when
    ranking term frequencies, defaulting to the "min" method. (#1634)
  - New docvars accessor and replacement functions are available for
    corpus, tokens, and dfm objects via $. (See Index Operators for Core
    Objects above.)
  - textstat_entropy() now produces a data.frame that is more consistent
    with other textstat methods. (#1690)

Bug fixes and stability enhancements

  - docnames now enforced to be character (formerly, could be numeric
    for some objects).
  - docnames are now enforced to be strictly unique for all object
    classes.
  - Grouping operations in tokens_group() and dfm_group() are more
    robust to using multiple grouping variables, and preserve these
    correctly as docvars in the new dfm. (#1809)
  - Some fixes to documented ... objects in two functions that were
    previously causing CRAN check failures on the release of 1.5.2.

Other improvements

  - All of the (three) included corpus objects have been cleaned up and
    augmented with improved meta-data and docvars. The inaugural speech
    corpus, for instance, now includes the President's political party
    affiliation.

                 Changes in version 1.5.2 (2019-11-26)                  

New features

  - Added Yule's I to textstat_lexdiv().
  - Added forward compatibility for newer (v2) corpus class objects.
  - Added a new function featfreq() to compute the overall feature
    frequencies from a dfm.

Bug fixes

  - Fixed a bug in tokens_lookup() when exclusive = FALSE and the tokens
    object has paddings. (#1743)
  - Fixed a bug in tokens_replace() (#1765).

                 Changes in version 1.5.1 (2019-07-30)                  

New features

  - Added omit_empty as an argument to convert(), to allow the user to
    control whether empty documents are excluded from converted dfm
    objects for certain formats. (#1660)

Bug fixes and stability enhancements

  - Fixed a bug that affects the new textstat_dist() and
    textstat_simil(). (#1730)
  - Fixed a bug in how textstat_dist() and textstat_simil() class
    symmetric matrices.

                 Changes in version 1.5.0 (2019-07-04)                  

New features

  - Add flatten and levels arguments to as.list.dictionary2() to enable
    more flexible conversion of dictionary objects. (#1661)
  - In corpus_sample(), the size now works with the by argument, to
    control the size of units sampled from each group.
  - Improvements to textstat_dist() and textstat_simil(), see below.
  - Long tokens are not discarded automatically in the call to tokens().
    (#1713)

Behaviour changes

  - textstat_dist() and textstat_simil() now return sparse symmetric
    matrix objects using classes from the Matrix package. This replaces
    the former structure based on the dist class. Computation of these
    classes is now also based on the fast implementation in the proxyC
    package. When computing similarities, the new min_simil argument
    allows a user to ignore certain values below a specified similarity
    threshold. A new coercion method as.data.frame.textstat_simildist()
    now exists for converting these returns into a data.frame of
    pairwise comparisons. Existing methods such as as.matrix(),
    as.dist(), and as.list() work as they did before.
  - We have removed the "faith", "chi-squared", and "kullback" methods
    from textstat_dist() and textstat_simil() because these were either
    not symmetric or not invariant to document or feature ordering.
    Finally, the selection argument has been deprecated in favour of a
    new y argument.
  - textstat_readability() now defaults to measure = "Flesch" if no
    measure is supplied. This makes it consistent with textstat_lexdiv()
    that also takes a default measure ("TTR") if none is supplied.
    (#1715)
  - The default values for max_nchar and min_nchar in tokens_select()
    are now NULL, meaning they are not applied if the user does not
    supply values. Fixes #1713.

Bug fixes and stability enhancements

  - kwic.corpus() and kwic.tokens() behaviour now aligned, meaning that
    dictionaries are correctly faceted by key instead of by value.
    (#1684)
  - Improved formatting of tokens() verbose output. (#1683)
  - Subsetting and printing of subsetted kwic objects is more robust.
    (#1665)
  - The "Bormuth" and "DRP" measures are now fixed for
    textstat_readability(). (#1701)

                 Changes in version 1.4.1 (2019-02-26)                  

Bug fixes and stability enhancements

  - Fixed an issue with special handling of whitespace variants that
    caused a test to fail when running Ubuntu 18.10 system with
    libicu-dev version 63.1 (#1604).
  - Fixed the operation of docvars<-.corpus() in a way that solves #1603
    (reassignment of docvar names).

                 Changes in version 1.4.0 (2019-01-30)                  

Bug fixes and stability enhancements

  - Fixed bug in dfm_compress() and dfm_group() that changed or deleted
    docvars attributes of dfm objects (#1506).
  - Fixed a bug in textplot_xray() that caused incorrect facet labels
    when a pattern contained multiple list elements or values (#1514).
  - kwic() now correctly returns the pattern associated with each match
    as the "keywords" attribute, for all pattern types (#1515)
  - Implemented some improvements in efficiency and computation of
    unusual edge cases for textstat_simil() and textstat_dist().

New features

  - textstat_lexdiv() now works on tokens objects, not just dfm objects.
    New methods of lexical diversity now include MATTR (the
    Moving-Average Type-Token Ratio, Covington & McFall 2010) and MSTTR
    (Mean Segmental Type-Token Ratio).
  - New function tokens_split() allows splitting single into multiple
    tokens based on a pattern match. (#1500)
  - New function tokens_chunk() allows splitting tokens into new
    documents of equally-sized "chunks". (#1520)
  - New function textstat_entropy() now computes entropy for a dfm
    across feature or document margins.
  - The documentation for textstat_readability() is vastly improved, now
    providing detailing all formulas and providing full references.
  - New function dfm_match() allows a user to specify the features in a
    dfm according to a fixed vector of feature names, including those of
    another dfm. Replaces dfm_select(x, pattern) where pattern was a
    dfm.
  - A new argument vertex_labelsize added to textplot_network() to allow
    more precise control of label sizes, either globally or
    individually.

Behaviour changes

  - tokens.tokens(x, remove_hyphens = TRUE) where x was generated with
    remove_hyphens = FALSE now behaves similarly to how the same tokens
    would be handled had this option been called on character input as
    tokens.character(x, remove_hyphens = TRUE). (#1498)

                 Changes in version 1.3.14 (2018-11-19)                 

Bug fixes and stability enhancements

  - Improved the robustness of textstat_keyness() (#1482).
  - Improved the accuracy of sparsity reporting for the print method of
    a dfm (#1473).
  - Diagonals on a textstat_simil() return object coerced to matrix now
    default to 1.0, rather than 0.0 (#1494).

New Features

  - Added the following measures to textstat_lexdiv(): Yule's K,
    Simpson's D, and Herdan's Vm.

                 Changes in version 1.3.13 (2018-11-01)                 

Bug fixes and stability enhancements

  - Fixed a bug causing incorrect counting in fcm(x, ordered = TRUE).
    (#1413) Also set the condition that window can be of size 1
    (formerly the limit was 2 or greater).
  - Fixed deprecation warnings from adding a dfm as docvars, and this
    now imports the feature names as docvar names automatically.
    (related to #1417)
  - Fixed behaviour from tokens(x, what = "fasterword",
    remove_separators = TRUE) so that it correctly splits words
    separated by \n and \t characters. (#1420)
  - Add error checking for functions taking dfm inputs in case a dfm has
    empty features (#1419).
  - For textstat_readability(), fixed a bug in Dale-Chall-based measures
    and in the Spache word list measure. These were caused by an
    incorrect lookup mechanism but also by limited implementation of the
    wordlists. The new wordlists include all of the variations called
    for in the original measures, but using fast fixed matching. (#1410)
  - Fixed problems with basic dfm operations (rowMeans(), rowSums(),
    colMeans(), colSums()) caused by not having access to the Matrix
    package methods. (#1428)
  - Fixed problem in textplot_scale1d() when input a predicted
    wordscores object with se.fit = TRUE (#1440).
  - Improved the stability of textplot_network(). (#1460)

New Features

  - Added new argument intermediate to textstat_readability(x, measure,
    intermediate = FALSE), which if TRUE returns intermediate quantities
    used in the computation of readability statistics. Useful for
    verification or direct use of the intermediate quantities.
  - Added a new separator argument to kwic() to allow a user to define
    which characters will be added between tokens returned from a
    keywords in context search. (#1449)
  - Reimplemented textstat_dist() and textstat_simil() in C++ for
    enhanced performance. (#1210)
  - Added a tokens_sample() function (#1478).

Behaviour changes

  - Removed the Hamming distance method from textstat_dist() (#1443),
    based on the reasoning in #1442.
  - Removed the "chisquared" and "chisquared2" distance measures from
    textstat_simil(). (#1442)

                 Changes in version 1.3.4 (2018-07-15)                  

Bug fixes and stability enhancements

  - Keep encodings of types when a tokens object is recompiled. (#1387)
  - More robust handling in predict.textmodel_worscores() when training
    and test feature sets are difference (#1380).
  - char_segment() and corpus_segment() are more robust to whitespace
    characters preceding a pattern (#1394).
  - tokens_ngrams() is more robust to handling large numbers of
    documents (#1395).
  - corpus.data.frame() is now robust to handling data.frame inputs with
    improper or missing variable names (#1388).

New Features

  - Added as.igraph.fcm() method for converting an fcm object into an
    igraph graph object.
  - Added a case_insensitive argument to char_segment() and
    corpus_segment().

                 Changes in version 1.3.0 (2018-06-05)                  

New Features

  - Added to = "tripletlist" output type for convert(), to convert a dfm
    into a simple triplet list. (#1321)
  - Added tokens_tortl() and char_tortl() to add markers for
    right-to-left language tokens and character objects. (#1322)

Behaviour changes

  - Improved corpus.kwic() by adding new arguments split_context and
    extract_keyword.
  - dfm_remove(x, selection = anydfm) is now equivalent to dfm_remove(x,
    selection = featnames(anydfm)). (#1320)
  - Improved consistency of predict.textmodel_nb() returns, and added
    type =  argument. (#1329)

Bug fixes

  - Fixed a bug in textmodel_affinity() that caused failure when the
    input dfm had been compiled with tolower = FALSE. (#1338)
  - Fixed a bug affecting tokens_lookup() and dfm_lookup() when nomatch
    is used. (#1347)
  - Fixed a problem whereby NA texts created a "document" (or tokens)
    containing "NA" (#1372)

                 Changes in version 1.2.0 (2018-04-15)                  

New Features

  - Added an nsentence() method for spacyr parsed objects. (#1289)

Bug fixes and stability enhancements

  - Fix bug in nsyllable() that incorrectly handled cased words, and
    returned wrong names with use.names = TRUE. (#1282)
  - Fix the overwriting of summary.character() caused by previous import
    of the network package namespace. (#1285)
  - dfm_smooth() now correctly sets the smooth value in the dfm (#1274).
    Arithmetic operations on dfm objects are now much more consistent
    and do not drop attributes of the dfm, as sometimes happened with
    earlier versions.

Behaviour changes

  - tokens_toupper() and tokens_tolower() no longer remove unused token
    types. Solves #1278.
  - dfm_trim() now takes more options, and these are implemented more
    consistently. min_termfreq and max_termfreq have replaced min_count
    and max_count, and these can be modified using a termfreq_type
    argument. (Similar options are implemented for docfreq_type.) Solves
    #1253, #1254.
  - textstat_simil() and textstat_dist() now take valid dfm indexes for
    the relevant margin for the selection argument. Previously, this
    could also be a direct vector or matrix for comparison, but this is
    no longer allowed. Solves #1266.
  - Improved performance for dfm_group() (#1295).

                 Changes in version 1.1.1 (2018-03-07)                  

New Features

  - Added as.dfm() methods for tm DocumentTermMatrix and
    TermDocumentMatrix objects. (#1222)
  - predict.textmodel_wordscores() now includes an include_reftexts
    argument to exclude training texts from the predicted model object
    (#1229). The default behaviour is include_reftexts = TRUE, producing
    the same behaviour as existed before the introduction of this
    argument. This allows rescaling based on the reference documents
    (since rescaling requires prediction on the reference documents) but
    provides an easy way to exclude the reference documents from the
    predicted quantities.
  - textplot_wordcloud() now uses code entirely internal to quanteda,
    instead of using the wordcloud package.

Bug fixes and stability enhancements

  - Fixed a problem in the examples for textplot_scale1d() by adjusting
    the refscores for data_corpus_irishbudget2010.
  - Eliminated unnecessary dependency on the digest package.
  - Updated the vignette title to be less generic.
  - Improved the robustness of dfm_trim() and dfm_weight() for
    previously weighted dfm objects and when supplied thresholds are
    proportions instead of counts. (#1237)
  - Fixed a problem in summary.corpus(x, n = 101) when ndoc(x) > 100
    (#1242).
  - Fixed a problem in predict.textmodel_wordscores(x, rescaling = "mv")
    that always reset the reference values for rescaling to the first
    and second documents (#1251).
  - Issues in the color generation and labels for textplot_keyness() are
    now resolved (#1233, #1233).

Performance improvements

  - textmodel methods are now exported, to facilitate extension packages
    for other textmodel methods (e.g. wordshoal).

Behaviour changes

  - Changed the default in textmodel_wordfish() to sparse = FALSE, in
    response to #1216.
  - dfm_group() now preserves docvars that are constant for the group
    aggregation (#1228).
  - The default threads is now 2, to comply with CRAN policies. (The
    user can increase this via quanteda_options(threads = ...).

                 Changes in version 1.0.0 (2018-01-28)                  

New Features

  - Added vertex_labelfont to textplot_network().
  - Added textmodel_lsa() for Latent Semantic Analysis models.
  - Added textmodel_affinity() for the Perry and Benoit (2017) class
    affinity scaling model.
  - Added Chinese stopwords.
  - Added a pkgdown vignette for applications in the Chinese language.
  - Added textplot_network() function.
  - The stopwords() function and the associated internal data object
    data_char_stopwords have been removed from quanteda, and replaced by
    equivalent functionality in the stopwords package.
  - Added tokens_subset(), now consistent with other *_subset()
    functions (#1149).

Bug fixes and stability enhancements

  - Performance has been improved for fcm() and for
    textmodel_wordfish().
  - dfm() now correctly passes through all ... arguments to tokens().
    (#1121)
  - All dfm_*() functions now work correctly with empty dfm objects.
    (#1133)
  - Fixed a bug in dfm_weight() for named weight vectors (#1150)
  - Fixed a bug preventing textplot_influence() from working (#1116).

Behaviour Changes

  - The convenience wrappers to convert() are simplified and no longer
    exported. To convert a dfm, convert() is now the only official
    function.
  - nfeat() replaces nfeature(), which is now deprecated. (#1134)
  - textmodel_wordshoal() has been removed, and relocated to a new
    package (wordshoal).
  - The generic wrapper function textmodel(), which used to be a gateway
    to specific textmodel_*() functions, has been removed.
  - (Most of) the textmodel_*() have been reimplemented to make their
    behaviour consistent with the lm/glm() families of models, including
    especially how the predict, summary, and coef methods work (#1007,
    #108).
  - The GitHub home for the repository has been moved to
    https://github.com/quanteda/quanteda.

                Changes in version 0.99.12 (2017-10-06)                 

New Features

  - tokens_segment() has a new window argument, permitting selection
    within an asymmetric window around the pattern of selection. (#521)
  - tokens_replace() now allows token types to be substituted directly
    and quickly.
  - textmodel_affinity() now adds functionality to fit the Perry and
    Benoit (2017) class affinity model.
  - Added a spacy_parse method for corpus objects. Also restored
    quanteda methods for spacyr spacy_parsed objects.

Bug fixes and stability enhancements

  - Improved documentation for textmodel_nb() (#1010), and made output
    quantities from the fitted NB model regular matrix objects instead
    of Matrix classes.

Behaviour Changes

  - All of the deprecated functions are now removed. (#991)
  - tokens_group() is now significantly faster.
  - The deprecated "list of characters" tokenize() function and all
    methods associated with the tokenizedTexts object types have been
    removed.
  - Added convenience functions for keeping tokens or features:
    tokens_keep(), dfm_keep(), and fcm_keep(). (#1037)
  - textmodel_NB() has been replaced by textmodel_nb().

                 Changes in version 0.99.9 (2017-09-22)                 

New Features

  - Added methods for changing the docnames of tokens and dfm objects
    (#987).
  - Added new function textmodel_lsa() for Latent Semantic Analysis.

Bug fixes and stability enhancements

  - The computation of tfidf has been more thoroughly described in the
    documentation for this function (#997).
  - Fixed a bug discovered in #1011 for unused keys in
    tokens_lookup(..., exclusive = FALSE).

                  Changes in version 0.99 (2017-08-15)                  

New Features

  - Added tokens_segment(), which works on tokens objects in the same
    way as corpus_segment() does on corpus objects (#902).
  - Added magrittr pipe support (#927). %>% can now be used with
    quanteda without needing to attach magrittr (or, as many users
    apparently believe, the entire tidyverse.)
  - corpus_segment() now behaves more logically and flexibly, and is
    clearly differentiated from corpus_reshape() in terms of its
    functionality. Its documentation is also vastly improved. (#908)
  - Added data_dictionary_LSD2015, the Lexicoder Sentiment 2015
    dictionary (#963).
  - Significant improvements to the performance of tokens_lookup() and
    dfm_lookup() (#960).
  - New functions head.corpus(), tail.corpus() provide fast subsetting
    of the first or last documents in a corpus. (#952)

Bug fixes and stability enhancements

  - Fixed a problem when applying purrr::map() to dfm() (#928).
  - Added documentation for regex2fixed() and associated functions.
  - Fixed a bug in textstat_collocations.tokens() caused by "documents"
    containing only "" as tokens. (#940)
  - Fixed a bug caused by cbind.dfm() when features shared a name
    starting with quanteda_options("base_featname") (#946)
  - Improved dictionary handling and creation now correctly handles
    nested LIWC 2015 categories. (#941)
  - Number of threads now set correctly by quanteda_options(). (#966)

Behaviour changes

  - summary.corpus() now generates a special data.frame, which has its
    own print method, rather than requiring verbose = FALSE to suppress
    output (#926).
  - textstat_collocations() is now multi-threaded.
  - head.dfm(), tail.dfm() now behave consistently with base R methods
    for matrix, with the added argument nfeature. Previously, these
    methods printed the subset and invisibly returned it. Now, they
    simply return the subset. (#952)
  - Dictionary keys are now unique, and if multiple, identical keys are
    defined for a dictionary when constructed, the values will be merged
    into the consolidated key. (#959)

                Changes in version 0.9.9-65 (2017-05-26)                

New features

  - Improvements and consolidation of methods for detecting multi-word
    expressions, now active only through textstat_collocations(), which
    computes only the lambda method for now, but does so accurately and
    efficiently. (#753, #803). This function is still under development
    and likely to change further.
  - Added new quanteda_options that affect the maximum documents and
    features displayed by the dfm print method (#756).
  - ngram formation is now significantly faster, including with skips
    (skipgrams).
  - Improvements to topfeatures():
      - now accepts a groups argument that can be used to generate lists
        of top (or bottom) features in a group of texts, including by
        document (#336).
      - new argument scheme that takes the default of (frequency)
        "count" but also a new "docfreq" value (#408).
  - New wrapper phrase() converts whitespace-separated multi-word
    patterns into a list of patterns. This affects the feature/pattern
    matching in tokens/dfm_select/remove, tokens_compound,
    tokens/dfm_lookup, and kwic. phrase() and the associated changes
    also make the behaviour of using character vectors, lists of
    characters, dictionaries, and collocation objects for pattern
    matches far more consistent. (See #820, #787, #740, #837, #836,
    #838)
  - corpus.Corpus() for creating a corpus from a tm Corpus now works
    with more complex objects that include document-level variables,
    such as data from the manifestoR package (#849).
  - New plot function textplot_keyness() plots term "keyness", the
    association of words with contrasting classes as measured by
    textstat_keyness().
  - Added corpus constructor for corpus objects (#690).
  - Added dictionary constructor for dictionary objects (#690).
  - Added a tokens constructor for tokens objects (#690), including
    updates to tokens() that improve the consistency and efficiency of
    the tokenization.
  - Added new quanteda_options(): language_stemmer and
    language_stopwords, now used for default in *_wordstem functions and
    stopwords() for defaults, respectively. Also uses this option in
    dfm() when stem = TRUE, rather than hard-wiring in the "english"
    stemmer (#386).
  - Added a new function textstat_frequency() to compile feature
    frequencies, possibly by groups. (#825)
  - Added nomatch option to tokens_lookup() and dfm_lookup(), to provide
    tokens or feature counts for categories not matched to any
    dictionary key. (#496)

Behaviour changes

  - The functions sequences() and collocations() have been removed and
    replaced by textstat_collocations().
  - (Finally) we added "will" to the list of English stopwords (#818).
  - dfm objects with one or both dimensions having zero length, and
    empty kwic objects now display more appropriately in their print
    methods (per #811).
  - Pattern matches are now implemented more consistently across
    functions. In functions such as *_select, *_remove, tokens_compound,
    features has been replaced by pattern, and in kwic, keywords has
    been replaced by pattern. These all behave consistently with respect
    to pattern, which now has a unified single help page and parameter
    description.(#839) See also above new features related to phrase().
  - We have improved the performance of the C++ routines that handle
    many of the tokens_* functions using hashed tokens, making some of
    them 10x faster (#853).
  - Upgrades to the dfm_group() function now allow "empty" documents to
    be created using the fill = TRUE option, for making documents
    conform to a selection (similar to how dfm_select() works for
    features, when supplied a dfm as the pattern argument). The groups
    argument now behaves consistently across the functions where it is
    used. (#854)
  - dictionary() now requires its main argument to be a list, not a
    series of elements that can be used to build a list.
  - Some changes to the behaviour of tokens() have improved the
    behaviour of remove_hyphens = FALSE, which now behaves more
    correctly regardless of the setting of remove_punct (#887).
  - Improved cbind.dfm() function allows cbinding vectors, matrixes, and
    (recyclable) scalars to dfm objects.

Bug fixes and stability enhancements

  - For the underlying methods behind textstat_collocations(), we
    corrected the word matching, and lambda and z calculation methods,
    which were slightly incorrect before. We also removed the chi2, G2,
    and pmi statistics, because these were incorrectly calculated for
    size > 2.
  - LIWC-formatted dictionary import now robust to assignment to term
    assignment to missing categories.
  - textmodel_NB(x, y, distribution = "Bernoulli") was previously
    inactive even when this option was set. It has now been fully
    implemented and tested (#776, #780).
  - Separators including rare spacing characters are now handled more
    robustly by the remove_separators argument in tokens(). See #796.
  - Improved memory usage when computing ntoken() and ntype(). (#795)
  - Improvements to quanteda_options() now does not throw an error when
    quanteda functions are called directly without attaching the
    package. In addition, quanteda options can be set now in .Rprofile
    and will not be overwritten when the options initialization takes
    place when attaching the package.
  - Fixed a bug in textstat_readability() that wrongly computed the
    number of words with fewer than 3 syllables in a text; this affected
    the FOG.NRI and the Linsear.Write measures only.
  - Fixed mistakes in the computation of two docfreq schemes: "logave"
    and "inverseprob".
  - Fixed a bug in the handling of multi-thread options where the
    settings using quanteda_options() did not actually set the number of
    threads. In addition, we fixed a bug causing threading to be turned
    off on macOS (due to a check for a gcc version that is not used for
    compiling the macOS binaries) prevented multi-threading from being
    used at all on that platform.
  - Fixed a bug causing failure when functions that use
    quanteda_options() are called without the namespace or package being
    attached or loaded (#864).
  - Fixed a bug in overloading the View method that caused all named
    objects in the RStudio/Source pane to be named "x". (#893)

                Changes in version 0.9.9-50 (2017-04-20)                

New features

  - Corpus construction using corpus() now works for a tm::SimpleCorpus
    object. (#680)
  - Added corpus_trim() and char_trim() functions for selecting
    documents or subsets of documents based on sentence, paragraph, or
    document lengths.
  - Conversion of a dfm to an stm object now passes docvars through in
    the $meta of the return object.
  - New dfm_group(x, groups = ) command, a convenience wrapper around
    dfm.dfm(x, groups = ) (#725).
  - Methods for extending quanteda functions to readtext objects updated
    to match CRAN release of readtext package.
  - Corpus constructor methods for data.frame objects now conform to the
    "text interchange format" for corpus data.frames, automatically
    recognizing doc_id and text fields, which also provides
    interoperability with the readtext package. corpus construction
    methods are now more explicitly tailored to input object classes.

Bug fixes and stability enhancements

  - dfm_lookup() behaves more robustly on different platforms,
    especially for keys whose values match no features (#704).
  - textstat_simil() and textstat_dist() no longer take the n argument,
    as this was not sorting features in correct order.
  - Fixed failure of tokens(x, what = "character") when x included
    Twitter characters @ and # (#637).
  - Fixed bug #707 where ntype.dfm() produced an incorrect result.
  - Fixed bug #706 where textstat_readability() and textstat_lexdiv()
    for single-document returns when drop = TRUE.
  - Improved the robustness of corpus_reshape().
  - print, and head, and tail methods for dfm are more robust (#684).
  - Fixed bug in convert(x, to = "stm") caused by zero-count documents
    and zero-count features in a dfm (#699, #700, #701). This also
    removes docvar rows from $meta when this is passed through the dfm,
    for zero-count documents.
  - Corrected broken handling of nested Yoshikoder dictionaries in
    dictionary(). (#722)
  - dfm_compress now preserves a dfm's docvars if collapsing only on the
    features margin, which means that dfm_tolower() and dfm_toupper() no
    longer remove the docvars.
  - fcm_compress() now retains the fcm class, and generates and error
    when an asymmetric compression is attempted (#728).
  - textstat_collocations() now returns the collocations as character,
    not as a factor (#736)
  - Fixed a bug in dfm_lookup(x, exclusive = FALSE) wherein an empty dfm
    ws returned with there was no no match (#116).
  - Argument passing through dfm() to tokens() is now robust, and
    preserves variables defined in the calling environment (#721).
  - Fixed issues related to dictionaries failing when applying str(),
    names(), or other indexing operations, which started happening on
    Linux and Windows platforms following the CRAN move to 3.4.0. (#744)
  - Dictionary import using the LIWC format is more robust to improperly
    formatted input files (#685).
  - Weights applied using dfm_weight() now print friendlier error
    messages when the weight vector contains features not found in the
    dfm. See this Stack Overflow question for the use case that sparked
    this improvement.

                Changes in version 0.9.9-24 (2017-02-13)                

New features

  - corpus_reshape() can now go from sentences and paragraph units back
    to documents.
  - Added a by =  argument to corpus_sample(), for use in bootstrap
    resampling of sub-document units.
  - Added an experimental method bootstrap_dfm() to generate a list of
    dimensionally-equivalent dfm objects based on sentence-level
    resampling of the original documents.
  - Added option to tokens() and dfm() for passing docvars through to to
    tokens and dfm objects, and added docvars() and metadoc() methods
    for tokens and dfm class objects. Overall, the code for docvars and
    metadoc is now more robust and consistent.
  - docvars() on eligible objects that contain no docvars now returns an
    empty 0 x 0 data.frame (in the spirit of #242).
  - Redesigned textmodel_scale1d now produces sorted and grouped
    document positions for fitted wordfish models, and produces a
    ggplot2 plot object.
  - textmodel_wordfish() now preserves sparsity while processing the
    dfm, and uses a fast approximation to an SVD to get starting values.
    This also dramatically improves performance in computing this model.
    (#482, #124)
  - The speed of kwic() is now dramatically improved, and also returns
    an indexed set of tokens that makes subsequent commands on a kwic
    class object much faster. (#603)
  - Package options (for verbose, threads) can now be set or queried
    using quanteda_options().
  - Improved performance and better documentation for corpus_segment().
    (#634)
  - Added functions corpus_trimsentences() and char_trimsentences() to
    remove sentences from a corpus or character object, based on token
    length or pattern matching.
  - Added options to textstat_readability(): min_sentence_length and
    max_sentence_length. (#632)
  - Indexing now works for dictionaries, for slicing out keys and values
    ([), or accessing values directly ([[). (#651)
  - Began the consolidation of collocation detection and scoring into a
    new function textstat_collocations(), which combines the existing
    collocations() and sequences() functions. (#434) Collocations now
    behave as sequences for other functions (such as tokens_compound())
    and have a greatly improved performance for such uses.

Behaviour changes

  - docvars() now permits direct access to "metadoc" fields (starting
    with _, e.g. _document)
  - metadoc() now returns a vector instead of a data.frame for a single
    variable, similar to docvars()
  - Most verbose options now take the default from getOption("verbose")
    rather than fixing the value in the function signatures. (#577)
  - textstat_dist() and textstat_simil() now return a matrix if a
    selection argument is supplied, and coercion to a list produces a
    list of distances or similarities only for that selection.
  - All remaining camelCase arguments are gone. For commonly used ones,
    such as those in tokens(), the old arguments (e.g. removePunct)
    still produce the same behaviour but with a deprecation warning.
  - Added n_target and n_reference columns to textstat_keyness() to
    return counts for each category being compared for keyness.

Bug fixes

  - Fixed an problem in tokens generation for some irregular characters
    (#554).
  - Fixed a problem in setting the parallel thread size on single-core
    machines (#556).
  - Fixed problems for str() on a corpus with no docvars (#571).
  - removeURL in tokens() now removes URLs where the first part of the
    URL is a single letter (#587).
  - dfm_select now works correctly for ngram features (#589).
  - Fixed a bug crashing corpus constructors for character vectors with
    duplicated names (the cause of #580).
  - Fixed a bug in the behaviour for dfm_select(x, features) when
    features was a dfm, that failed to produce the intended featnames
    matches for the output dfm.
  - Fixed a bug in corpus_segment(x, what = "tags") when a document
    contained a whitespace just before a tag, at the beginning of the
    file, or ended with a tag followed by no text (#618, #634).
  - Fixed some problems with dictionary construction and reading some
    dictionary formats (#454, #455, #459).

                Changes in version 0.9.9-17 (2017-01-27)                

New features

  - textstat_keyness() now returns a data.frame with p-values as well as
    the test statistic, and rownames containing the feature. This is
    more consistent with the other textstat functions.
  - tokens_lookup() implements new rules for nested and linked sequences
    in dictionary values. See #502.
  - tokens_compound() has a new join argument for better handling of
    nested and linked sequences. See #517.
  - Internal operations on tokens are now significantly faster due to a
    reimplementation of the hash table functions in C++. (#510)
  - dfm() now works with multi-word dictionaries and thesauruses, which
    previously worked only with tokens_lookup().
  - fcm() is now parallelized for improved performance on multi-core
    systems.

Bug fixes

  - Fixed C++ incompatibilities on older platforms due to compiler
    incompatibilities with the required TBB libraries (for
    multi-threading) (#531, #532, #535), in addition to safeguarding
    against other compiler warnings across a variety of new tested
    undefined behaviours.
  - Fixed a bug in convert(x, to = "lsa") that transposed row and column
    names (#526)
  - Added missing fcm() method for corpus objects (#538)
  - Fixed some minor issues with reading in Lexicoder format
    dictionaries (Improvements to Lexicoder dictionary handling

quanteda 0.9.9-3

Bug fixes

  - Fixed a bug causing dfm and tokens to break on > 10,000 documents.
    (#438)
  - Fixed a bug in tokens(x, what = "character", removeSeparators =
    TRUE) that returned an empty string.
  - Fixed a bug in corpus.VCorpus if the VCorpus contains a single
    document. (#445)
  - Fixed a bug in dfm_compress in which the function failed on
    documents that contained zero feature counts. (#467)
  - Fixed a bug in textmodel_NB that caused the class priors Pc to be
    refactored alphabetically instead of in the order of assignment
    (#471), also affecting predicted classes (#476).

New features

  - New textstat function textstat_keyness() discovers words that occur
    at differential rates between partitions of a dfm (using
    chi-squared, Fisher's exact test, and the G^2 likelihood ratio test
    to measure the strength of associations).
  - Added 2017-Trump to the inaugural corpus datasets
    (data_corpus_inaugual and data_char_inaugural).
  - Improved the groups argument in texts() (and in dfm() that uses this
    function), which will now coerce to a factor rather than requiring
    one.
  - Added a dfm constructor from dfm objects, with the option of
    collapsing by groups.
  - Added new arguments to sequences(): ordered and max_length, the
    latter to prevent memory leaks from extremely long sequences.
  - dictionary() now accepts YAML as an input file format.
  - dfm_lookup and tokens_lookup now accept a levels argument to
    determine which level of a hierarchical dictionary should be
    applied.
  - Added min_nchar and max_nchar arguments to dfm_select.
  - dictionary() can now be called on the argument of a list() without
    explicitly wrapping it in list().
  - fcm now works directly on a dfm object when context = "documents".

This release has some major changes to the API, described below.

Data objects

Renamed data objects

new name | original name | notes :--------|:------------- |:-----
data_char_sampletext | exampleString | data_char_mobydick |
mobydickText| data_dfm_LBGexample | LBGexample | data_char_sampletext |
exampleString |

Renamed internal data objects

The following objects have been renamed, but will not affect user-level
functionality because they are primarily internal. Their man pages have
been moved to a common ?data-internal man page, hidden from the index,
but linked from some of the functions that use them.

new name | original name | notes :--------|:------------- |:-----
data_int_syllables | englishSyllables | (used by textcount_syllables())
data_char_wordlists | wordlists | (used by readability())
data_char_stopwords | .stopwords | (used by stopwords()

Deprecated data objects

In v.0.9.9 the old names remain available, but are deprecated.

new name | original name | notes :--------|:------------- |:-----
data_char_ukimmig2010 | ukimmigTexts | data_corpus_irishbudget2010 |
ie2010Corpus | data_char_inaugural | inaugTexts | data_corpus_inaugural
| inaugCorpus |

Deprecated functions

The following functions will still work, but issue a deprecation
warning:

new function | deprecated function | constructs:
:--------|:------------- |:------- tokens | tokenize() | tokens class
object corpus_subset | subset.corpus | corpus class object
corpus_reshape | changeunits | corpus class object corpus_sample |
sample| corpus class object corpus_segment | segment| corpus class
object dfm_compress | compress | dfm class object dfm_lookup |
applyDictionary | dfm class object dfm_remove | removeFeatures.dfm | dfm
class object dfm_sample | sample.dfm | dfm class object dfm_select |
selectFeatures.dfm | dfm class object dfm_smooth | smoother | dfm class
object dfm_sort | sort.dfm | dfm class object dfm_trim | trim.dfm | dfm
class object dfm_weight | weight | dfm class object textplot_wordcloud |
plot.dfm | (plot) textplot_xray | plot.kwic | (plot)
textstat_readability | readability | data.frame textstat_lexdiv | lexdiv
| data.frame textstat_simil | similarity | dist textstat_dist |
similarity | dist featnames | features | character nsyllable | syllables
| (named) integer nscrabble | scrabble | (named) integer tokens_ngrams |
ngrams | tokens class object tokens_skipgrams | skipgrams | tokens class
object tokens_toupper | toUpper.tokens, toUpper.tokenizedTexts | tokens,
tokenizedTexts tokens_tolower | toLower.tokens, toLower.tokenizedTexts |
tokens, tokenizedTexts char_toupper | toUpper.character,
toUpper.character | character char_tolower | toLower.character,
toLower.character | character tokens_compound | joinTokens,
phrasetotoken | tokens class object

New functions

The following are new to v0.9.9 (and not associated with deprecated
functions):

new function | description | output class :--------|:-------------
|:------- fcm() | constructor for a feature co-occurrence matrix | fcm
fcm_select | selects features from an fcm | fcm fcm_remove | removes
features from an fcm | fcm fcm_sort | sorts an fcm in alphabetical order
of its features| fcm fcm_compress | compacts an fcm | fcm fcm_tolower |
lowercases the features of an fcm and compacts | fcm fcm_toupper |
uppercases the features of an fcm and compacts | fcm dfm_tolower |
lowercases the features of a dfm and compacts | dfm dfm_toupper |
uppercases the features of a dfm and compacts | dfm sequences |
experimental collocation detection | sequences

Deleted functions and data objects

new name | reason :--------|:------------- encodedTextFiles.zip | moved
to the readtext package describeTexts | deprecated several versions ago
for summary.character textfile | moved to package readtext encodedTexts
| moved to package readtext, as data_char_encodedtexts findSequences |
replaced by sequences

Other new features

  - to = "lsa" functionality added to convert() (#414)
  - Much faster pattern matching in general, through an overhaul of how
    valuetype matches work for many functions.
  - Added experimental View methods for kwic objects, based on
    Javascript Datatables.
  - kwic is completely rewritten, now uses fast hashed index matching in
    C++ and fully implements vectorized matches (#306) and all
    valuetypes (#307).
  - tokens_lookup, tokens_select, and tokens_remove are faster and use
    parallelization (based on the TBB library).
  - textstat_dist and textstat_simil add fast, sparse, and parallel
    computation of many new distance and similarity matrices.
  - Added textmodel_wordshoal fitting function.
  - Add max_docfreq and min_docfreq arguments, and better verbose
    output, to dfm_trim (#383).
  - Added support for batch hashing of tokens through tokens(), for more
    memory-efficient token hashing when dealing with very large numbers
    of documents.
  - Added support for in-memory compressed corpus objects.
  - Consolidated corpus-level metadata arguments in corpus() through the
    metacorpus list argument.
  - Added Greek stopwords. (See #282).
  - Added index handling [, [[, and $ for (hashed) tokens objects.
  - Now using ggplot2.
  - Added tokens methods for collocations() and kwic().
  - Much improved performance for tokens_select() (formerly
    selectFeatures.tokens()).
  - Improved ngrams() and joinTokens() performance for hashed tokens
    class objects.
  - Improved dfm.character() by using new tokens() constructor to create
    hashed tokenized texts by default when creating a dfm, resulting in
    performance gains when constructing a dfm. Creating a dfm from a
    hashed tokens object is now 4-5 times faster than the older
    tokenizedTexts object.
  - Added new (hashed) tokens class object.
  - Added plot method for fitted textmodel_wordscores objects.
  - Added fast tokens_lookup() method (formerly applyDictionary()), that
    also works with dictionaries that have multi-word keys. Addresses
    but does not entirely yet solve #188.
  - Added sparsity() function to compute the sparsity of a dfm.
  - Added feature co-occurrence matrix functions (fcm).

                 Changes in version 0.9.8 (2016-07-28)                  

New Features

  - Improved the performance of selectFeatures.tokenizedTexts().
  - Improved the performance of rbind.dfm().
  - Added support for different docvars when importing multiple files
    using textfile(). (#147)
  - Added support for comparison dispersion plots in plot.kwic(). (#146)
  - Added a corpus constructor method for kwic objects.
  - Substantially improved the performance of convert(x, to = "stm") for
    dfm export, including adding an argument for meta-data (docvars, in
    quanteda parlance). (#209)
  - Internal rewrite of textfile(), now supports more file types, more
    wildcard patterns, and is far more robust generally.
  - Add support for loading external dictionary formats:

  - Yoshikoder,
  - Lexicoder v2 and v3 (#228)

  - Autodetect dictionary file format from file extension, so no longer
    require format keyword for loading dictionaries (#227)
  - Improved compatibility with rOpenSci guidelines (#218):
      - Use httr to get remote files
      - Use messages() to display messages rather than print or cat
      - Reorganise sections in README file
  - Added new punctuation argument to collocations() to provide new
    options for handling collocations separated by punctuation
    characters (#220).

Bug fixes

  - (0.9.8.7) Solved #267 in which fcm(x, tri = TRUE) temporarily
    created a dense logical matrix.
  - (0.9.8.7) Added feature co-occurrence matrix functions (fcm).
  - (0.9.8.5) Fixed an incompatibility in sequences.cpp with Solaris x86
    (#257)
  - (0.9.8.4) Fix bug in verbose output of dfm that causes misreporting
    of number of features (#250)
  - (0.9.8.4) Fix a bug in selectFeatures.dfm() that ignored
    case_insensitive = TRUE settings (#251) correct the documentation
    for this function.
  - (0.9.8.3) Fix a bug in tf(x, scheme = "propmax") that returned a
    wrong computation; correct the documentation for this function.
  - (0.9.8.2) Fixed a bug in textfile() causing all texts to have the
    same name, for types using the "textField" argument (a single file
    containing multiple documents).
  - Fixed bug in phrasetotoken() where if pattern included a + for
    valuetype = c("glob", "fixed") it threw a regex error. #239
  - Fixed bug in textfile() where source is a remote .zip set. (#172)
  - Fixed bug in wordstem.dfm() that caused an error if supplied a dfm
    with a feature whose total frequency count was zero, or with a
    feature whose total docfreq was zero. Fixes #181.
  - Fix #214 "mysterious stemmed token" bug in wordstem.dfm(),
    introduced in fixing #181.
  - Fixed previously non-functional toLower =  argument in
    dfm.tokenizedTexts().
  - Fixed some errors in the computation of a few readability formulas
    (#215).
  - Added filenames names to text vectors returned by textfile (#221).
  - dictionary() now works correctly when reading LIWC dictionaries
    where all terms belong to one key (#229).
  - `convert(x, to = "stm") now indexes the dfm components from 1, not 0
    (#222).
  - Remove temporary stemmed token (#214).
  - Fixed bug in textmodel_NB() for non-"uniform" priors (#241)

Changes

  - Added warn = FALSE to the readLines() calls in textfile(), so that
    no warnings are issued when files are read that are missing a final
    EOL or that contain embedded nuls.
  - trim() now prints an output message even when no features are
    removed (#223)
  - We now skip some platform-dependent tests on CRAN, travis-ci and
    Windows.

                        Changes in version 0.9.6                        

  - Improved Naive Bayes model and prediction, textmodel(x, y, method =
    "NB"), now works correctly on k > 2.

  - Improved tag handling for segment(x, what = "tags")

  - Added valuetype argument to segment() methods, which allows faster
    and more robust segmentation on large texts.

  - corpus() now converts all hyphen-like characters to simple hyphen

  - segment.corpus() now preserves all existing docvars.

  - corpus documentation now removes the description of the corpus
    object's structure since too many users were accessing these
    internal elements directly, which is strongly discouraged, as we are
    likely to change the corpus internals (soon and often). Repeat after
    me: "encapsulation".

  - Improve robustness of corpus.VCorpus() for constructing a corpus
    from a tm Corpus object.

  - Add UTF-8 preservation to ngrams.cpp.

  - Fix encoding issues for textfile(), improve functionality.

  - Added two data objects: Moby Dick is now available as mobydickText,
    without needing to access a zipped text file; encodedTextFiles.zip
    is now a zipped archive of different encodings of (mainly) the UN
    Declaration of Human Rights, for testing conversions from 8-bit
    encodings in different (non-Roman) languages.

  - phrasetotoken() now has a method correctly defined for corpus class
    objects.

  - lexdiv() now works just like readability(), and is faster (based on
    data.table) and the code is simpler.

  - removed quanteda::df() as a synonym for docfreq(), as this
    conflicted with stats::df().

  - added version information when package is attached.

  - improved rbind() and cbind() methods for dfm. Both now take any
    length sequence of dfms and perform better type checking.
    rbind.dfm() also knits together dfms with different features, which
    can be useful for information and retrieval purposes or machine
    learning.

  - selectFeatures(x, anyDfm) (where the second argument is a dfm) now
    works with a selection = "remove" option.

  - tokenize.character adds a removeURL option.

  - added a corpus method for data.frame objects, so that a corpus can
    be constructed directly from a data.frame. Requires the addition of
    a textField argument (similar to textfile).

  - added compress.dfm() to combine identically named columns or rows.
    #123

  - Much better phrasetotoken(), with additional methods for all
    combinations of corpus/character v.
    dictionary/character/collocations.

  - Added a weight(x, type, ...) signature where the second argument can
    be a named numeric vector of weights, not just a label for a type of
    weight. Thanks
    https://stackoverflow.com/questions/36815926/assigning-weights-to-different-features-in-r/36823475#36823475.

  - as.data.frame for dfms now passes ... to as.data.frame.matrix.

  - Fixed bug in predict.fitted_textmodel_NB() that caused a failure
    with k > 2 classes (#129)

  - Improved dfm.tokenizedTexts() performance by taking care of
    zero-token documents more efficiently.

  - dictionary(file = "liwc_formatted_dict.dic", format = "LIWC") now
    handles poorly formatted dictionary files better, such as the Moral
    Foundations Dictionary in the examples for ?dictionary.

  - added as.tokenizedTexts to coerce any list of characters to a
    tokenizedTexts object.

Bug fixes >= 0.9.6-3

  - Fix bug in phrasetotoken, signature 'corpus,ANY' that was causing an
    infinite loop.

  - Fixed bug introduced in commit b88287f (0.9.5-26) that caused a
    failure in dfm() with empty (zero-token) documents. Also fixes Issue
    #168.

  - Fixed bug that caused dfm() to break if no features or only one
    feature was found.

  - Fixed bug in predict.fitted_textmodel_NB() that caused a failure
    with k > 2 classes (#129)

Bug fixes

  - Fixed a false-alarm warning message in textmodel_wordfish()

  - Argument defaults for readability.corpus() now same as
    readability.character(). Fixes #107.

  - Fixed a bug causing LIWC format dictionary imports to fail if extra
    characters followed the closing % in the file header.

  - Fixed a bug in applyDictionary(x, dictionary, exclusive = FALSE)
    when the dictionary produced no matches at all, caused by an attempt
    to negative index a NULL. #115

  - Fixed #117, a bug where wordstem.tokenizedTexts() removed attributes
    from the object, causing a failure of dfm.tokenizedTexts().

  - Fixed #119, a bug in selectFeatures.tokenizedTexts(x, features,
    selection = "remove") that returned a NULL for a document's tokens
    when no matching pattern for removal was found.

  - Improved the behaviour of the removeHyphens option to tokenize()
    when what = "fasterword" or what "fastestword".

  - readability() now returns measures in order called, not function
    definition order.

  - textmodel(x, model = "wordfish") now removes zero-frequency
    documents and words prior to calling Rcpp.

  - Fixed a bug in sample.corpus() that caused an error when no docvars
    existed. #128

                 Changes in version 0.9.4 (2016-02-21)                  

  - Added presidents' first names to inaugCorpus

  - Added textmodel implementation of multinomial and Bernoulli Naive
    Bayes.

  - Improved documentation.

  - Added c.corpus() method for concatenating arbitrarily large sets of
    corpus objects.

  - Default for similarity() is now margin = "documents" -- prevents
    overly massive results if selection = NULL.

  - Defined rowMeans() and colMeans() methods for dfm objects.

  - Enhancements to summary.character() and summary.corpus(): Added n =
    to summary.character(); added pass-through options to tokenize() in
    summary.corpus() and summary.character() methods; added toLower as
    an argument to both.

  - Enhancements to corpus object indexing, including [[ and [[<-.

Bug fixes

  - Fixed a bug preventing smoother() from working.

  - Fixed a bug in segment.corpus(x, what = "tag") that was failing to
    recover the tag values after the first text.

  - Fix bug in plot.dfm(x, comparison = TRUE) method causing warning
    about rowMeans() failing.

  - Fixed an issue for mfdict <- dictionary(file = "http://ow.ly/VMRkL",
    format = "LIWC") causing it to fail because of the irregular
    combination of tabs and spaces in the dictionary file.

  - Fixed an exception thrown by wordstem.character(x) if one element of
    x was NA.

  - dfm() on a text or tokenized text containing an NA element now
    returns a row with 0 feature counts. Previously it returned a count
    of 1 for an NA feature.

  - Fix issue #91 removeHyphens = FALSE not working in tokenise for some
    multiple intra-word hyphens, such as "one-of-a-kind"

  - Fixed a bug in as.matrix.similMatrix() that caused scrambled
    conversion when feature sets compared were unequal, which normally
    occurs when setting similarity(x, n = <something>) when n <
    nfeature(x)

  - Fixed a bug in which a corpusSource object (from textfile()) with
    empty docvars prevented this argument from being supplied to
    corpus(corpusSourceObject, docvars = something).

  - Fixed inaccurate documentation for weight(), which previously listed
    unavailable options.

  - More accurate and complete documentation for tokenize().

  - traps an exception when calling wordstem.tokenizedTexts(x) where x
    was not word tokenized.

  - Fixed a bug in textfile() that prevented passthrough arguments in
    ..., such as fileEncoding = or encoding =

  - Fixed a bug in textfile() that caused exceptions with input
    documents containing docvars when there was only a single column of
    docvars (such as .csv files)

                        Changes in version 0.9.2                        

  - added new methods for similarity(), including sparse matrix
    computation for method = "correlation" and "cosine". (More planned
    soon.) Also allows easy conversion to a matrix using as.matrix() on
    similarity lists.

  - more robust implementation of LIWC-formatted dictionary file imports

  - better implementation of tf-idf, and relative frequency weighting,
    especially for very large sparse matrix objects. tf(), idf(), and
    tfidf() now provide relative term frequency, inverse document
    frequency, and tf-idf directly.

  - textmodel_wordfish() now accepts an integer dispersionFloor argument
    to constrain the phi parameter to a minimum value (of
    underdispersion).

  - textfile() now takes a vector of filenames, if you wish to construct
    these yourself. See ?textfile examples.

  - removeFeatures() and selectFeatures.collocations() now all use a
    consistent interface and same underlying code, with removeFeatures()
    acting as a wrapper to selectFeatures().

  - convert(x, to = "stm") now about 3-4x faster because it uses index
    positions from the dgCMatrix to convert to the sparse matrix format
    expected by stm.

Bug fixes

  - Fixed a bug in textfile() preventing encodingFrom and encodingTo
    from working properly.

  - Fixed a nasty bug problem in convert(x, to = "stm") that mixed up
    the word indexes. Thanks Felix Haass for spotting this!

  - Fixed a problem where wordstem was not working on ngram=1 tokenized
    objects

  - Fixed toLower(x, keepAcronyms = TRUE) that caused an error when x
    contained no acronyms.

  - Creating a corpus from a tm VCorpus now works if a "document" is a
    vector of texts rather than a single text

  - Fixed a bug in texts(x, groups = MORE THAN ONE DOCVAR) that now
    groups correctly on combinations of multiple groups

                        Changes in version 0.9.0                        

  - trim() now accepts proportions in addition to integer thresholds.
    Also accepts a new sparsity argument, which works like tm's
    removeSparseTerms(x, sparse = ) (for those who really want to think
    of sparsity this way).

  - [i] and [i, j] indexing of corpus objects is now possible, for
    extracting texts or docvars using convenient notation. See ?corpus
    Details.

  - ngrams() and skipgrams() now use the same underlying function, with
    skip replacing the previous window argument (where a skip = window
    - 1). For efficiency, both are now implemented in C++.

  - tokenize() has a new argument, removeHyphens, that controls the
    treatment of intra-word hyphens.

  - Added new measures from readability for mean syllables per word and
    mean words per sentence directly.

  - wordstem now works on ngrams (tokenizedTexts and dfm objects).

  - Enhanced operation of kwic(), including the definition of a kwic
    class object, and a plot method for this object (produces a
    dispersion plot).

  - Lots more error checking of arguments passed to ... (and potentially
    misspecified or misspelled). Addresses Issue #62.

  - Almost all methods are now methods defined for objects, from a
    generic.

  - texts(x, groups = ) now allows groups to be factors, not just
    document variable labels. There is a new method for
    texts.character(x, groups = ) which is useful for supplying a factor
    to concatenate character objects by group.

Bug Fixes

  - corrected inaccurate printing of valuetype in verbose note of
    selectFeatures.dfm(). (Did not affect functionality.)

  - fixed broken quanteda.R demo, expanded demonstration code.

                        Changes in version 0.8.6                        

  - removeFeatures.dfm(x, stopwords), selectFeatures.dfm(x, features),
    and dfm(x, ignoredFeatures) now work on objects created with ngrams.
    (Any ngram containing a stopword is removed.) Performance on these
    functions is already good but will be improved further soon.

  - selectFeatures(x, features = ) is now possible, to produce a
    selection of features from x identical to those in . Not only are
    only features kept in x that are in , but also features in  not in x
    are added to x as padded zero counts. This functionality can also be
    accessed via dfm(x, keptFeatures = ). This is useful when new data
    used in a test set needs to have identical features as a training
    set dfm constructed at an earlier stage.

  - head.dfm() and tail.dfm() methods added.

  - kwic() has new formals and new functionality, including a completely
    flexible set of matching for phrases, as well as control over how
    the texts and matching keyword(s) are tokenized.

  - segment(x, what = "sentence"), and changeunits(x, to = "sentences")
    now uses tokenize(x, what = "sentence"). Annoying warning messages
    now gone.

  - smoother() and weight() formal "smooth" now changed to "smoothing"
    to avoid clashes with stats::smooth().

  - Updated corpus.VCorpus() to work with recent updates to the tm
    package.

  - added print method for tokenizedTexts

Bug fixes

  - fixed signature error message caused by weight(x, "relFreq") and
    weight(x, "tfidf"). Both now correctly produce objects of class
    dfmSparse.

  - fixed bug in dfm(, keptFeatures = "whatever") that passed it through
    as a glob rather than a regex to selectFeatures(). Now takes a
    regex, as per the manual description.

  - fixed textfeatures() for type json, where now it can call
    jsonlite::fromJSON() on a file directly.

  - dictionary(x, format = "LIWC") now expanded to 25 categories by
    default, and handles entries that are listed on multiple lines in
    .dic files, such as those distributed with the LIWC.

                        Changes in version 0.8.4                        

  - ngrams() rewritten to accept fully vectorized arguments for n and
    for window, thus implementing "skip-grams". Separate function
    skipgrams() behaves in the standard "skipgram" fashion. bigrams(),
    deprecated since 0.7, has been removed from the namespace.

  - corpus() no longer checks all documents for text encoding; rather,
    this is now based on a random sample of max()

  - wordstem.dfm() both faster and more robust when working with large
    objects.

  - toLower.NULL() now allows toLower() to work on texts with no words
    (returns NULL for NULL input)

  - textfile() now works on zip archives of *.txt files, although this
    may not be entirely portable.

Bug fixes

  - fixed bug in selectFeatures() / removeFeatures() that returned zero
    features if no features were found matching removal pattern

  - corpus() previously removed document names, now fixed

  - non-portable \donttest{} examples now removed completely from all
    documentation

                        Changes in version 0.8.2                        

  - 0.8.2-1: Changed R version dependency to 3.2.0 so that Mac binary
    would build on CRAN.

  - 0.8.2-1: sample.corpus() now samples documents from a corpus, and
    sample.dfm() samples documents or features from a dfm. trim() method
    for with nsample argument now calls sample.dfm().

  - sample.corpus() now samples documents from a corpus, and
    sample.dfm() samples documents or features from a dfm. trim() method
    for with nsample argument now calls sample.dfm().

  - tokenize improvements for what = "sentence": more robust to
    specifying options, and does not split sentences after common
    abbreviations such as "Dr.", "Prof.", etc.

  - corpus() no longer automatically converts encodings detected as
    non-UTF-8, as this detection is too imprecise.

  - new function scrabble() computes English Scrabble word values for
    any text, applying any summary numerical function.

  - dfm() now 2x faster, replacing previous data.table matching with
    direct construction of sparse matrix from match().
    Code is also much simpler, based on using three new functions that
    are also available directly:
    
      - new "dfm" method for removeFeatures()
      - new "dfm" method: selectFeatures() that is now how features can
        be added or removed from a dfm, based on vectors of regular
        expressions, globs, or fixed matching
      - new "dfm" method: applyDictionary() that can replace features
        through matching with values in key-value lists from a
        dictionary class objects, based on vectors of regular
        expressions, globs, or fixed matching for dictionary values. All
        functionality for applying dictionaries now takes place through
        applyDictionary().

Bug Fixes

  - fixed the problem that document names were getting erased in
    corpus() because stringi functions were removing them
  - fixed problem in tokenize(x, "character", removePunct = TRUE) that
    deleted texts that had no punctuation to begin with
  - fixed problem in dictionary(, format = "LIWC") causing import to
    fail for some LIWC dictionaries.
  - fixed problem in tokenize(x, ngrams = N) where N > length(x). Now
    returns NULL instead of an erroneously tokenized set of ngrams.
  - Fixed a bug in subset.corpus() related to environments that
    sometimes caused the method to break if nested in function
    environments.

Deletions

  - clean() is no more.

API changes

  - addto option removed from dfm()

Imminent Changes

  - change behaviour of ignoredFeatures and removeFeatures() applied to
    ngrams; change behaviour of stem = TRUE applied to ngrams (in dfm())
  - create ngrams.tokenizedTexts() method, replacing current ngrams(),
    bigrams()

                        Changes in version 0.8.0                        

Syntax changes and workflow streamlining

The workflow is now more logical and more streamlined, with a new
workflow vignette as well as a design vignette explaining the principles
behind the workflow and the commands that encourage this workflow. The
document also details the development plans and things remaining to be
done on the project.

Encoding detection and conversion

Newly rewritten command encoding() detects encoding for character,
corpus, and corpusSource objects (created by textfile). When creating a
corpus using corpus(), detection is automatic to UTF-8 if an encoding
other than UTF-8, ASCII, or ISO-8859-1 is detected.

Major infrastructural changes

The tokenization, cleaning, lower-casing, and dfm construction functions
now use the stringi package, based on the ICU library. This results not
only in substantial speed improvements, but also more correctly handles
Unicode characters and strings.

  - tokenize() and clean() now using stringi, resulting in much faster
    performance and more consistent behaviour across platforms.

  - tokenize() now works on sentences

  - summary.corpus() and summary.character() now use the new
    tokenization functions for counting tokens

  - dfm(x, dictionary = mydict) now uses stringi and is both more
    reliable and many many times faster.

  - phrasetotoken() now using stringi.

  - removeFeatures() now using stringi and fixed binary matches on
    tokenized texts

Other changes

  - textfile has a new option, cache = FALSE, for not writing the data
    to a temporary file, but rather storing the object in memory if that
    is preferred.

  - language() is removed. (See Encoding... section above for changes to
    encoding().)

  - new object encodedTexts contains some encoded character objects for
    testing.

  - ie2010Corpus now has UTF-8 encoded texts (previously was Unicode
    escaped for non-ASCII characters)

  - texts() and docvars() methods added for corpusSource objects.

  - new methods for tokenizedTexts objects: dfm(), removeFeatures(), and
    syllables()

  - syllables() is now much faster, using matching through stringi and
    merging using data.table.

  - added readability() to compute (fast!) readability indexes on a text
    or corpus

  - tokenize() now creates ngrams of any length, with two new arguments:
    ngrams =  and concatenator = "_". The new arguments to tokenize()
    can be passed through from dfm().

Bug fixes

  - fixed a problem in textfile() causing it to fail on Windows machines
    when loading *.txt

  - nsentence() was not counting sentences correctly if the text was
    lower-cased - now issues an error if no upper-case characters are
    detected. This was also causing readability() to fail.

                        Changes in version 0.7.3                        

  - added an ntoken() method for dfm objects.

  - fixed a bug wherein convert(anydfm, to = "tm") created a
    DocumentTermMatrix, not a TermDocumentMatrix. Now correctly creates
    a TermDocumentMatrix. (Both worked previously in topicmodels::LDA()
    so many users may not notice the change.)

                 Changes in version 0.7.2 (2015-04-07)                  

  - phrasetotokens works with dictionaries and collocations, to
    transform multi-word expressions into single tokens in texts or
    corpora

  - dictionaries now redefined as S4 classes

  - improvements to collocations(), now does not include tokens that are
    separated by punctuation

  - created tokenizeOnly*() functions, for testing tokenizing separately
    from cleaning, and a cleanC(), where both new separate functions are
    implemented in C

  - tokenize() now has a new option, cpp=TRUE, to use a C++ tokenizer
    and cleaner, resulting in much faster text tokenization and
    cleaning, including that used in dfm()

  - textmodel_wordfish now implemented entirely in C for speed. No std
    errors yet but coming soon. No predict method currently working
    either.

  - ie2010Corpus, and exampleString now moved into quanteda (formerly
    were only in quantedaData because of non-ASCII characters in each -
    solved with native2ascii and \uXXXX encodings).

  - All dependencies, even conditional, to the quantedaData and austin
    packages have been removed.

                        Changes in version 0.7.1                        

Many major changes to the syntax in this version.

  - trimdfm, flatten.dictionary, the textfile functions, dictionary
    converters are all gone from the NAMESPACE

  - formals changed a bit in clean(), kwic().

  - compoundWords() -> phrasetotoken()

  - Cleaned up minor issues in documentation.

  - countSyllables data object renamed to englishSyllables.Rdata, and
    function renamed to syllables().

  - stopwordsGet() changed to stopwords(). stopwordsRemove() changed to
    removeFeatures().

  - new dictionary() constructor function that also does import and
    conversion, replacing old readWStatdict and readLIWCdict functions.

  - one function to read in text files, called textsource, that does the
    work for different file types based on the filename extension, and
    works also for wildcard expressions (that can link to directories
    for example)

                        Changes in version 0.7.0                        

  - dfm now sparse by default, implemented as subclasses of the Matrix
    package. Option dfm(..., matrixType="sparse") is now the default,
    although matrixType="dense" will still produce the old S3-class dfm
    based on a regular matrix, and all dfm methods will still work with
    this object.

  - Improvements to: weight(), print() for dfms.

  - New methods for dfms: docfreq(), weight(), summary(), as.matrix(),
    as.data.frame.

                        Changes in version 0.6.6                        

  - No more depends, all done through imports. Passes clean check. The
    start of our reliance more on the master branch rather than having
    merges from dev to master happen only once in a blue moon.

  - bigrams in dfm() when bigrams=TRUE and ignoredFeatures= now removed
    if any bigram contains an ignoredFeature

  - stopwordsRemove() now defined for sparse dfms and for collocations.

  - stopwordsRemove() now requires an explicit stopwords= argument, to
    emphasize the user's responsibility for applying stopwords.

                        Changes in version 0.6.5                        

  - New engine for dfm now implemented as standard, using data.table and
    Matrix for fast, efficient (sparse) matrixes.

  - Added trigram collocations (n=3) to collocations().

  - Improvements to clean(): Minor fixes to clean() so that
    removeDigits=TRUE removes "€10bn" entirely and not just the "€10".
    clean() now removes http and https URLs by default, although does
    not preserve them (yet). clean also handles numbers better, to
    remove 1,000,000 and 3.14159 if removeDigits=TRUE but not crazy8
    or 4sure.

  - dfm works for documents that contain no features, including for
    dictionary counts. Thanks to Kevin Munger for catching this.

                        Changes in version 0.6.4                        

  - first cut at REST APIs for Twitter and Facebook

  - some minor improvements to sentence segmentation

  - improvements to package dependencies and imports - but this is
    ongoing!

  - Added more functions to dfms, getting there...

  - Added the ability to segment a corpus on tags (e.g. ##TAG1 text
    text, ##TAG2) and have the document split using the tags as a
    delimiter and the tag then added to the corpus as a docvar.

                        Changes in version 0.6.3                        

  - added textmodel_lda support, including LDA, CTM, and STM. Added a
    converter dfm2stmformat() between dfm and stm's input format.

  - as.dfm works now for data.frame objects

  - added Arabic to list of stopwords. (Still working on a stemmer for
    Arabic.)

                        Changes in version 0.6.2                        

  - The first appearance of dfms(), to create a sparse Matrix using the
    Matrix package. Eventually this will become the default format for
    all but small dfms. Not only is this far more efficient, it is also
    much faster.

  - Minor speed gains for clean() -- but still much more work to be done
    with clean().

                        Changes in version 0.6.1                        

  - started textmodel_wordfish, textmodel_ca. textmodel_wordfish takes
    an mcmc argument that calls JAGS wordfish.

  - now depends on ca, austin rather than importing them

  - dfm subsetting with [,] now works

  - docnames()[], []<-, docvars()[] and []<- now work correctly

                        Changes in version 0.6.0                        

  - Added textmodel for scaling and prediction methods, including for
    starters, wordscores and naivebayes class models. LIKELY TO BE BUGGY
    AND QUIRKY FOR A WHILE.

  - Added smoothdfm() and weight() methods for dfms.

  - Fixed a bug in segmentSentence().

                        Changes in version 0.5.8                        

Classification and scaling methods

  - New dfm methods for fitmodel(), predict(), and specific model
    fitting and prediction methods called by these, for classification
    and scaling of different "textmodel" types, such as wordscores and
    Naive Bayes (for starters).

                        Changes in version 0.5.7                        

  - added compoundWords() to turn space-delimited phrases into single
    "tokens". Works with dfm(, dictionary=) if the text has been
    pre-processed with compoundWords() and the dictionary joins phrases
    with the connector ("_"). May add this functionality to be more
    automatic in future versions.

  - new keep argument for trimdfm() now takes a regular expression for
    which feature labels to retain. New defaults for minDoc and minCount
    (1 each).

  - added nfeature() method for dfm objects.

New arguments for dfm()

  - thesaurus: works to record equivalency classes as lists of words or
    regular expressions for a given key/label.

  - keep: regular expression pattern match for features to keep

                        Changes in version 0.5.6                        

  - added readLIWCdict() to read LIWC-formatted dictionaries

  - fixed a "bug"/feature in readWStatDict() that eliminated wildcards
    (and all other punctuation marks) - now only converts to lower.

  - improved clean() functions to better handle Twitter, punctuation,
    and removing extra whitespace

                        Changes in version 0.5.5                        

  - fixed broken dictionary option in dfm()

  - fixed a bug in dfm() that was preventing clean() options from being
    passed through

  - added Dice and point-wise mutual information as association measures
    for collocations()

  - added: similarity() to implement similarity measures for documents
    or features as vector representations

  - begun: implementing dfm resample methods, but this will need more
    time to work.
    (Solution: a three way table where the third dim is the resampled
    text.)

  - added is.resample() for dfm and corpus objects

  - added Twitter functions: getTweets() performs a REST search through
    twitteR, corpus.twitter creates a corpus object with test and
    docvars form each tweet (operational but needs work)

  - added various resample functions, including making dfm a
    multi-dimensional object when created from a resampled corpus and
    dfm(, bootstrap=TRUE).

  - modified the print.dfm() method.

                        Changes in version 0.5.4                        

  - updated corpus.directory to allow specification of the file
    extension mask

  - updated docvars<- and metadoc<- to take the docvar names from the
    assigned data.frame if field is omitted.

  - added field to docvars()

  - enc argument in corpus() methods now actually converts from enc to
    "UTF-8"

  - started working on clean to give it exceptions for @ # _ for twitter
    text and to allow preservation of underscores used in
    bigrams/collocations.

  - Added: a + method for corpus objects, to combine a corpus using this
    operator.

  - Changed and fixed: collocations(), which was not only fatally slow
    and inefficient, but also wrong. Now is much faster and O(n) because
    it uses data.table and vector operations only.

  - Added: resample() for corpus texts.

                        Changes in version 0.5.3                        

  - added statLexdiv() to compute the lexical diversity of texts from a
    dfm.

  - minor bug fixes; update to print.corpus() output messages.

  - added a wrapper function for SnowballC::wordStem, called wordstem(),
    so that this can be imported without loading the whole package.

                        Changes in version 0.5.2                        

  - Added a corpus constructor method for the VCorpus class object from
    the tm package.

  - added zipfiles() to unzip a directory of text files from disk or a
    URL, for easy import into a corpus using
    corpus.directory(zipfiles())

                        Changes in version 0.5.1                        

  - Fixed all the remaining issues causing warnings in R CMD CHECK, now
    all are fixed.
    Mostly these related to documentation.

  - Fixed corpus.directory to better implementing naming of docvars, if
    found.

  - Moved twitter.R to the R_NEEDFIXING until it can be made to pass
    tests. Apparently setup_twitter_oauth() is deprecated in the latest
    version of the twitteR package.

                        Changes in version 0.5.0                        

Lots of new functions

  - plot.dfm method for producing word clouds from dfm objects

  - print.dfm, print.corpus, and summary.corpus methods now defined

  - new accessor functions defined, such as docnames(), settings(),
    docvars(), metadoc(), metacorpus(), encoding(), and language()

  - replacement functions defined that correspond to most of the above
    accessor functions, e.g. encoding(mycorpus) <- "UTF-8"

  - segment(x, to=c("tokens", "sentences", "paragraphs", "other", ...)
    now provides an easy and powerful method for segmenting a corpus by
    units other than just tokens

  - a settings() function has been added to manage settings that would
    commonly govern how texts are converted for processing, so that
    these can be preserved in a corpus and applied to operations that
    are relevant. These settings also propagate to a dfm for both
    replication purposes and to govern operations for which they would
    be relevant, when applied to a dfm.

Old functions vastly improved

  - better ways now exist to manage corpus internals, such as through
    the accessor functions, rather than trying to access the internal
    structure of the corpus directly.

  - basic functions such as tokenize(), clean(), etc are now faster,
    neater, and operate generally on vectors and return consistent
    object types

Better object and class design

  - the corpus object has been redesigned with more flexible components,
    including a settings list, better corpus-level metadata, and smarter
    implementation of document-level attributes including user-defined
    variables (docvars) and document- level meta-data (metadoc)

  - the dfm now has a proper class definition, including additional
    attributes that hold the settings used to produce the dfm.

  - all important functions are now defined as methods for classes of
    built-in (e.g. character) objects, or quanteda objects such as a
    corpus or dfm. Lots of functions operate on both, for instance
    dfm.corpus(x) and dfm.character(x).

more complete documentation

  - all functions are now documented and have working examples

  - quanteda.pdf provides a pdf version of the function documentation in
    one easy-to-access document
