Introduction to LiblineaR

LiblineaR wraps the LIBLINEAR C/C++ library for large-scale regularized linear classification and regression. This vignette is a practical guide to the parts of the API that are easy to get wrong: which type to pick, what bias/epsilon/svr_eps actually default to, how sparse input is handled, how class weighting (wi) works, and the two ways to search for a good cost.

library(LiblineaR)
data(iris)

Choosing a type

type selects both the loss function and the regularization. Two families:

Classification (type 0-7):

type Regularization Loss Solver
0 L2 logistic primal (Newton)
1 L2 L2-loss SVM (hinge²) dual (coordinate descent)
2 L2 L2-loss SVM primal (Newton)
3 L2 L1-loss SVM (hinge) dual
4 L2 Crammer & Singer multi-class SVM dual
5 L1 L2-loss SVM dual
6 L1 logistic dual
7 L2 logistic dual

Regression (type 11-13), all L2-regularized support vector regression:

type Loss Solver
11 L2-loss (epsilon-insensitive²) primal
12 L2-loss dual
13 L1-loss dual

Rules of thumb:

  • For an ordinary classification problem, type=0 (logistic regression) or type=2 (L2-loss SVM) are the usual defaults; both give one weight vector per class for multi-class problems via one-vs-rest, except type=4, which always fits one weight vector per class simultaneously (relevant if you read $W’s shape — see below).
  • type=1/2 and type=0/7 are the dual/primal formulations of the same problem respectively, and converge to (numerically close to) the same model — a useful sanity check if you’re unsure which to trust.
  • L1-regularized types (5, 6) push weights toward exact zero — useful for feature selection on high-dimensional data.
  • For regression, type=11 (primal) is usually fastest; type=12/13 differ in whether large errors are penalized quadratically or linearly.
x <- iris[, 1:4]
y <- iris[, 5]

m_lr  <- LiblineaR(x, y, type = 0)   # logistic regression
m_svm <- LiblineaR(x, y, type = 2)   # L2-loss SVM

dim(m_lr$W)   # one row per class (3 classes, multi-class problem)
#> [1] 3 5

bias, epsilon, svr_eps: what the defaults actually mean

bias (default 1): if bias > 0, every row gets an extra constant feature appended with that value ([data; bias]) — this is what lets the model fit an intercept. If bias <= 0, no bias term is added at all (the decision boundary is forced through the origin). For backward compatibility, bias=TRUE/FALSE are also accepted (TRUE behaves like 1, FALSE like 0, i.e. no bias).

epsilon (default NULL): the solver’s stopping tolerance. Leave it at the default — NULL lets LIBLINEAR apply its own per-solver default (0.01 for primal solvers, 0.1 for dual solvers; these differ because primal and dual solvers measure convergence on different quantities). Passing a numeric value overrides that for every solver uniformly, which is rarely what you want unless you’re deliberately trading convergence tightness for speed.

svr_eps (regression only, default 0.1 if left NULL): the width of the epsilon-insensitive tube — errors smaller than this aren’t penalized at all. There’s no universally good default; it depends on the scale of your target variable, so it’s worth setting explicitly for regression:

xr <- as.matrix(iris[, 1:3])
yr <- iris[, 4]
m_svr <- LiblineaR(xr, yr, type = 11, svr_eps = 0.05)

Sparse input

data (and predict()’s newx) accept dense matrices/data frames, or sparse matrices of class matrix.csr/matrix.csc/matrix.coo (package SparseM) or dgCMatrix/dgRMatrix/dgTMatrix (package Matrix). The type is detected automatically — no separate argument needed. All six sparse classes and dense input give identical coefficients and predictions on the same data; pick whichever integrates better with the rest of your pipeline.

if (requireNamespace("Matrix", quietly = TRUE)) {
  x_sparse <- Matrix::Matrix(as.matrix(x), sparse = TRUE)
  m_sparse <- LiblineaR(x_sparse, y, type = 0)
  identical(dim(m_sparse$W), dim(m_lr$W))
}
#> 'as(<matrix>, "dgRMatrix")' is deprecated.
#> Use 'as(as(as(., "dMatrix"), "generalMatrix"), "RsparseMatrix")' instead.
#> See help("Deprecated") and help("Matrix-deprecated").
#> [1] TRUE

Class weighting with wi

wi reweights each class’s effective regularization constant (C_class = cost * wi[class], default weight 1 for every class not named). This is the tool for imbalanced data: naming only the minority class with a higher weight pushes the solver to trade some overall accuracy for better recall on that class — a deliberate trade-off, not a bug, and one you should expect to see reflected in a lower raw accuracy alongside a better balanced accuracy.

# Not all classes need to be named -- only the one(s) you want to reweight.
m_weighted <- LiblineaR(x, y, type = 0, wi = c(setosa = 5))

Finding a good cost: heuristicC(), cross, and findC

Three complementary tools:

  • heuristicC(data): a fast, closed-form heuristic (Joachims’ SVM-light heuristic) giving a reasonable starting point for cost, computed directly from the data with no training involved.
  • cross=k: runs k-fold cross-validation at the given cost and returns the CV accuracy (classification) or MSE (regression) as a single number — no model object. Useful for evaluating one specific cost.
  • findC=TRUE: automatic search for a good cost, using repeated cross-validation internally. Only supported for type=0 and type=2 (the primal L2-regularized solvers); any other type raises an error. Returns the best cost found, not a model — retrain with that value to get the actual model.
co <- heuristicC(x)
co
#> [1] 0.1274724

acc <- LiblineaR(x, y, type = 0, cost = co, cross = 5)
acc
#> [1] 0.8266667

best_cost <- LiblineaR(x, y, type = 0, findC = TRUE, cross = 5)
best_cost
#> [1] 64

m_final <- LiblineaR(x, y, type = 0, cost = best_cost)

Predicting

predict() accepts a vector (single feature) or a matrix/data frame with the same columns as training — reordered and with any extra columns dropped automatically, matched by column name if newx has names.

p <- predict(m_final, x)
mean(as.character(p$predictions) == as.character(y))
#> [1] 0.98

# Probabilities are only available for logistic regression (type 0, 6, 7).
p_proba <- predict(m_final, x, proba = TRUE)
head(p_proba$probabilities)
#>         setosa versicolor    virginica
#> [1,] 0.9156984 0.08430157 3.081650e-14
#> [2,] 0.7794057 0.22059429 3.818377e-13
#> [3,] 0.8521161 0.14788391 1.977643e-13
#> [4,] 0.7929417 0.20705826 1.138778e-12
#> [5,] 0.9320938 0.06790622 2.842446e-14
#> [6,] 0.9728372 0.02716275 7.286927e-14