aiDIF: Differential Item Functioning for AI-Scored Assessments

When AI systems score essays, short answers, or structured responses, a fairness question follows: does the scoring engine shift item difficulties differently for different groups?

aiDIF separates three quantities:

• DIF under human scoring — the group contrast in one condition
• DIF under AI scoring — the same contrast in the other
• Differential AI Scoring Bias (DASB) — the change in that contrast

An item can carry large DIF under both conditions and have a DASB of zero: the AI reproduced what the humans were doing. That distinction is the point of the package.

Installation

install.packages("aiDIF")

# development version
# remotes::install_github("causalfragility-lab/aiDIF")

Quick start

library(aiDIF)

eg  <- make_aidif_eg()
mod <- fit_aidif(eg$human, eg$ai, metric = "linked", adjust = "BH")

print(mod)
summary(mod)

plot(mod, type = "dif_forest")
plot(mod, type = "dasb")
plot(mod, type = "weights")
plot(mod, type = "rho")

After calibration with mirt

human_in <- as_aidif(human_mod, groups = c("reference", "focal"))
ai_in    <- as_aidif(ai_mod,    groups = c("reference", "focal"))

fit <- fit_aidif(human_in, ai_in)

Identification

DASB is a difference of differences across four calibrations. If each carries its own arbitrary metric origin, the constants contribute a common offset to every item's DASB, confounded with a uniform effect.

• metric = "common" assumes linking was established upstream.
• metric = "linked" estimates the offset with the same bi-square M-estimator used for within-condition scaling, assuming most items are free of scoring-condition-by-group effects.

Do not subtract intercepts from separately normalised scales without one of these.

Scoring designs

• design = "independent" (default) sums the four marginal variances.
• design = "paired" accepts cross-condition covariance via cross_cov when the same responses were scored twice.

Because the cross-condition covariance is normally positive, the independence variance is an upper bound for paired data: the default is conservative, not anti-conservative.

Scope

aiDIF operates after IRT calibration. It works from item parameter estimates and their asymptotic covariance matrices and never sees respondent-level responses. It does not perform calibration, and it does not replace mirt, difR, or lordif for general DIF work.

Citation

citation("aiDIF")

The robust scaling method is due to Halpin, P. F. (2024). Differential item functioning via robust scaling. Psychometrika, 89(3), 796–821. doi:10.1007/s11336-024-09957-6

License

GPL (>= 3)