Manuscript-Ready Reporting Examples
Purpose
This vignette provides reporting scaffolds for the three flagship workflows. The examples are intentionally conservative: statistical screening is described as evidence for retention or review, not as proof that an item or scale is content valid. Final decisions should also consider construct-domain coverage, item wording, qualitative judge feedback, and the intended use of the measure.
The bundled CSV files are synthetic, deterministic, and regenerated
from data-raw/build-example-data.R. They are useful for
reproducing the examples and for seeing the expected input shape before
analyzing a new study.
Item-sort study
sort_dat <- read_example("sort_example.csv")
sort_fit <- sort_validity(sort_dat)
sort_sum <- summary(sort_fit)
sort_fit$results
#> item target n_total n n_missing n_target competitor n_other_max psa
#> 1 A1 A 20 20 0 18 B; C 1 0.90
#> 2 A2 A 20 20 0 15 B 3 0.75
#> 3 B1 B 20 20 0 17 A 2 0.85
#> 4 B2 B 20 20 0 13 A 5 0.65
#> 5 C1 C 20 20 0 18 A; B 1 0.90
#> 6 C2 C 20 20 0 14 B 4 0.70
#> psa_low psa_high csv p_value critical_n_target passes_chance
#> 1 0.6989664 0.9721335 0.85 0.0002012253 15 TRUE
#> 2 0.5312991 0.8881383 0.60 0.0206947327 15 TRUE
#> 3 0.6395811 0.9476313 0.75 0.0012884140 15 TRUE
#> 4 0.4328543 0.8188082 0.40 0.1315879822 15 FALSE
#> 5 0.6989664 0.9721335 0.85 0.0002012253 15 TRUE
#> 6 0.4810272 0.8545228 0.50 0.0576591492 15 FALSE
#> recommendation issue
#> 1 Retain Supported
#> 2 Retain Supported
#> 3 Retain Supported
#> 4 Review Target favored, exact criterion not met
#> 5 Retain Supported
#> 6 Review Target favored, exact criterion not met
#> interpretation
#> 1 Target assignment meets the exact retention criterion (strongest competitor: B; C).
#> 2 Target assignment meets the exact retention criterion (strongest competitor: B).
#> 3 Target assignment meets the exact retention criterion (strongest competitor: A).
#> 4 The target was at least as common as any competitor but did not meet the exact retention criterion (strongest competitor: A); review before deciding whether to revise or remove the item.
#> 5 Target assignment meets the exact retention criterion (strongest competitor: A; B).
#> 6 The target was at least as common as any competitor but did not meet the exact retention criterion (strongest competitor: B); review before deciding whether to revise or remove the item.
#> status
#> 1 Supported
#> 2 Supported
#> 3 Supported
#> 4 Review
#> 5 Supported
#> 6 Review
sort_fit$scale_summary
#> target n_items n_items_usable n_retain n_review mean_psa psa_strength
#> 1 A 2 2 2 0 0.825 Strong
#> 2 B 2 2 1 1 0.750 Moderate
#> 3 C 2 2 1 1 0.800 Moderate
#> mean_csv csv_strength orbiting_r benchmark_set
#> 1 0.725 Strong NA Overall (not correlation-normed)
#> 2 0.575 Moderate NA Overall (not correlation-normed)
#> 3 0.675 Strong NA Overall (not correlation-normed)
#> benchmark_applicable overall_strength
#> 1 TRUE Strong
#> 2 TRUE Moderate
#> 3 TRUE Moderate
#> evidence
#> 1 The weaker of Psa and Csv falls in the Strong band of published scales (Colquitt et al., 2019).
#> 2 The weaker of Psa and Csv falls in the Moderate band of published scales (Colquitt et al., 2019); review the weaker items before finalizing.
#> 3 The weaker of Psa and Csv falls in the Moderate band of published scales (Colquitt et al., 2019); review the weaker items before finalizing.Methods scaffold
Report who completed the sort, how the construct definitions were presented, the available assignment alternatives, and the a priori screening rule. A concise methods statement can follow this structure:
Candidate items were evaluated in an item-sort pretest in which judges assigned each item to the construct definition that best represented its content. We quantified definitional correspondence using the proportion of substantive agreement (Psa) and definitional distinctiveness using the substantive-validity coefficient (Csv). Item-level screening used the exact target-assignment test described by Howard and Melloy (2016), with the null target-assignment probability and alpha specified a priori. Target-scale mean Psa and Csv were interpreted against Colquitt et al. (2019) norms only when the judge population matched the intended use of those norms.
Results scaffold
For this bundled example, 20 judges evaluated 6 items. 4 items met the exact screening criterion, 2 were flagged for review, and 0 had insufficient usable assignments.
A manuscript table can usually be built directly from:
sort_fit$results[c(
"item", "target", "n", "n_target", "competitor",
"psa", "csv", "p_value", "status", "recommendation"
)]
#> item target n n_target competitor psa csv p_value status
#> 1 A1 A 20 18 B; C 0.90 0.85 0.0002012253 Supported
#> 2 A2 A 20 15 B 0.75 0.60 0.0206947327 Supported
#> 3 B1 B 20 17 A 0.85 0.75 0.0012884140 Supported
#> 4 B2 B 20 13 A 0.65 0.40 0.1315879822 Review
#> 5 C1 C 20 18 A; B 0.90 0.85 0.0002012253 Supported
#> 6 C2 C 20 14 B 0.70 0.50 0.0576591492 Review
#> recommendation
#> 1 Retain
#> 2 Retain
#> 3 Retain
#> 4 Review
#> 5 Retain
#> 6 ReviewDo not report Review as synonymous with deletion. A
review flag identifies an item for substantive inspection; retaining an
item for domain coverage can be a reasonable decision when that
rationale is documented.
Construct-rating study
rating_dat <- read_example("rating_example.csv")
rating_fit <- rating_validity(rating_dat, scale_min = 1, scale_max = 5)
rating_sum <- summary(rating_fit)
rating_fit$results
#> item target n_raters n_complete n_incomplete n_constructs target_mean
#> 1 A1 A 24 24 0 3 4.666667
#> 2 A2 A 24 24 0 3 3.666667
#> 3 B1 B 24 24 0 3 4.666667
#> 4 B2 B 24 24 0 3 3.666667
#> 5 C1 C 24 24 0 3 4.666667
#> 6 C2 C 24 24 0 3 3.666667
#> target_mean_complete strongest_competitor competitor_mean htc htd
#> 1 4.666667 C 2.333333 0.9333333 0.6666667
#> 2 3.666667 B 3.333333 0.7333333 0.2083333
#> 3 4.666667 C 2.333333 0.9333333 0.6666667
#> 4 3.666667 A 3.333333 0.7333333 0.2083333
#> 5 4.666667 B 2.333333 0.9333333 0.6666667
#> 6 3.666667 B 3.333333 0.7333333 0.2083333
#> F df1 df2 p_omnibus epsilon_gg p_value partial_eta2
#> 1 237.0769 2 46 5.921046e-25 0.7647059 1.274299e-19 0.9115646
#> 2 37.3750 2 46 2.291004e-10 0.6956522 7.375102e-08 0.6190476
#> 3 237.0769 2 46 5.921046e-25 0.7647059 1.274299e-19 0.9115646
#> 4 37.3750 2 46 2.291004e-10 0.6956522 7.375102e-08 0.6190476
#> 5 237.0769 2 46 5.921046e-25 0.7647059 1.274299e-19 0.9115646
#> 6 37.3750 2 46 2.291004e-10 0.6956522 7.375102e-08 0.6190476
#> min_mean_diff max_contrast_p contrast_pass recommendation
#> 1 2.3333333 1.122784e-13 TRUE Retain
#> 2 0.3333333 5.173102e-02 FALSE Review
#> 3 2.3333333 1.122784e-13 TRUE Retain
#> 4 0.3333333 5.173102e-02 FALSE Review
#> 5 2.3333333 1.122784e-13 TRUE Retain
#> 6 0.3333333 5.173102e-02 FALSE Review
#> issue
#> 1 Supported
#> 2 Target highest, planned contrasts incomplete
#> 3 Supported
#> 4 Target highest, planned contrasts incomplete
#> 5 Supported
#> 6 Target highest, planned contrasts incomplete
#> interpretation
#> 1 The intended construct is rated higher than all orbiting constructs and all planned contrasts meet the screening criterion (strongest competitor: C).
#> 2 The intended construct has the highest mean but the full inferential screening criterion was not met (strongest competitor: B); review the weakest target-orbiting comparison before revising or removing the item.
#> 3 The intended construct is rated higher than all orbiting constructs and all planned contrasts meet the screening criterion (strongest competitor: C).
#> 4 The intended construct has the highest mean but the full inferential screening criterion was not met (strongest competitor: A); review the weakest target-orbiting comparison before revising or removing the item.
#> 5 The intended construct is rated higher than all orbiting constructs and all planned contrasts meet the screening criterion (strongest competitor: B).
#> 6 The intended construct has the highest mean but the full inferential screening criterion was not met (strongest competitor: B); review the weakest target-orbiting comparison before revising or removing the item.
#> status
#> 1 Supported
#> 2 Review
#> 3 Supported
#> 4 Review
#> 5 Supported
#> 6 Review
rating_fit$scale_summary
#> target n_items n_htc n_htd n_retain n_review n_insufficient mean_htc
#> 1 A 2 2 2 1 1 0 0.8333333
#> 2 B 2 2 2 1 1 0 0.8333333
#> 3 C 2 2 2 1 1 0 0.8333333
#> htc_strength mean_htd htd_strength overall_strength orbiting_r benchmark_set
#> 1 Weak 0.4375 Very Strong Weak NA overall
#> 2 Weak 0.4375 Very Strong Weak NA overall
#> 3 Weak 0.4375 Very Strong Weak NA overall
#> evidence
#> 1 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.
#> 2 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.
#> 3 The weaker of HTC and HTD falls in the Weak band of published scales (Colquitt et al., 2019); review item wording, construct boundaries, and the choice of orbiting constructs, and consider pretesting the revised items again.Methods scaffold
Judges rated every candidate item against each focal and orbiting construct definition using the same response scale. We summarized correspondence with HTC and distinctiveness with HTD. Because the same judges rated the competing definitions, item-level inference used a repeated-measures design. Planned paired contrasts compared each item’s intended definition with every orbiting definition; omnibus Greenhouse-Geisser-corrected inference was used when applicable. Scale-level HTC/HTD norms from Colquitt et al. (2019) were treated as empirical benchmarks rather than universal item cutoffs.
Results scaffold
The example contains 6 items rated by 24 judges against 3 construct definitions. 3 items were supported by the complete screening rule and 3 were flagged for review.
rating_fit$results[c(
"item", "target", "n_complete", "strongest_competitor",
"htc", "htd", "p_value", "max_contrast_p", "status", "recommendation"
)]
#> item target n_complete strongest_competitor htc htd p_value
#> 1 A1 A 24 C 0.9333333 0.6666667 1.274299e-19
#> 2 A2 A 24 B 0.7333333 0.2083333 7.375102e-08
#> 3 B1 B 24 C 0.9333333 0.6666667 1.274299e-19
#> 4 B2 B 24 A 0.7333333 0.2083333 7.375102e-08
#> 5 C1 C 24 B 0.9333333 0.6666667 1.274299e-19
#> 6 C2 C 24 B 0.7333333 0.2083333 7.375102e-08
#> max_contrast_p status recommendation
#> 1 1.122784e-13 Supported Retain
#> 2 5.173102e-02 Review Review
#> 3 1.122784e-13 Supported Retain
#> 4 5.173102e-02 Review Review
#> 5 1.122784e-13 Supported Retain
#> 6 5.173102e-02 Review ReviewFor review items, report the strongest orbiting competitor. That information turns a generic statement about weak distinctiveness into a specific diagnostic about where construct overlap may be occurring.
Expert-panel study
Relevance
expert_rel <- read_example("expert_relevance_example.csv")
expert_rel_matrix <- as.matrix(expert_rel[setdiff(names(expert_rel), "expert")])
expert_fit <- expert_validity(
expert_rel_matrix,
mode = "relevance",
lo = 1,
hi = 4
)
expert_sum <- summary(expert_fit)
expert_fit$results
#> item N n_missing V ci_low ci_high ci_method A
#> 1 Item1 8 0 1.0000000 0.8620238 1.0000000 Penfield-Giacobbi score 8
#> 2 Item2 8 0 0.9166667 0.7415120 0.9768412 Penfield-Giacobbi score 8
#> 3 Item3 8 0 0.8333333 0.6414693 0.9332132 Penfield-Giacobbi score 8
#> 4 Item4 8 0 0.5833333 0.3883467 0.7553240 Penfield-Giacobbi score 6
#> 5 Item5 8 0 0.4166667 0.2446760 0.6116533 Penfield-Giacobbi score 2
#> I_CVI I_CVI_low I_CVI_high Pc kappa_mod cvi_criterion kappa_quality
#> 1 1.00 0.67559244 1.0000000 0.00390625 1.0000000 0.875 Excellent
#> 2 1.00 0.67559244 1.0000000 0.00390625 1.0000000 0.875 Excellent
#> 3 1.00 0.67559244 1.0000000 0.00390625 1.0000000 0.875 Excellent
#> 4 0.75 0.40927543 0.9285208 0.10937500 0.7192982 0.875 Good
#> 5 0.25 0.07147921 0.5907246 0.10937500 0.1578947 0.875 Low
#> ci_width recommendation
#> 1 0.1379762 Strong support
#> 2 0.2353292 Strong support
#> 3 0.2917439 Strong support
#> 4 0.3669773 Review
#> 5 0.3669773 Review
#> interpretation
#> 1 The item meets the common panel-size CVI guideline and shows excellent chance-corrected agreement; Aiken's V and its score interval quantify relevance level and precision.
#> 2 The item meets the common panel-size CVI guideline and shows excellent chance-corrected agreement; Aiken's V and its score interval quantify relevance level and precision.
#> 3 The item meets the common panel-size CVI guideline and shows excellent chance-corrected agreement; Aiken's V and its score interval quantify relevance level and precision.
#> 4 The item does not meet the common panel-size CVI guideline; review wording, relevance, construct coverage, and expert comments before revising or removing it.
#> 5 The item does not meet the common panel-size CVI guideline; review wording, relevance, construct coverage, and expert comments before revising or removing it.
#> status
#> 1 Supported
#> 2 Supported
#> 3 Supported
#> 4 Review
#> 5 Review
expert_fit$scale_summary
#> n_items n_experts_min n_experts_max mean_Aiken_V S_CVI_Ave S_CVI_UA agreement
#> 1 5 8 8 0.75 0.8 0.6 0.6934253
#> agreement_low agreement_high n_strong_support n_support n_review
#> 1 0.07142857 0.8603241 3 0 2
#> n_insufficient
#> 1 0Experts rated the relevance of each candidate item on a bounded ordinal scale. We summarized relevance using Aiken’s V with Penfield-Giacobbi score confidence intervals and calculated I-CVI with Polit-Beck-Owen modified kappa. S-CVI/Ave and S-CVI/UA were reported at the scale level. Panel-size CVI guidelines were used as review aids and were considered alongside written expert feedback and construct coverage.
In this example, 8 experts evaluated 5 items. The workflow identifies 3 supported items and 2 review items under its quantitative rules.
Essentiality
expert_ess <- read_example("expert_essentiality_example.csv")
expert_ess_matrix <- as.matrix(expert_ess[setdiff(names(expert_ess), "expert")])
ess_fit <- expert_validity(expert_ess_matrix, mode = "essentiality")
ess_fit$results
#> item ne N cvr p_value critical_ne critical_cvr pass
#> 1 Item1 11 12 0.8333333 0.003173828 10 0.6666667 TRUE
#> 2 Item2 10 12 0.6666667 0.019287109 10 0.6666667 TRUE
#> 3 Item3 8 12 0.3333333 0.193847656 10 0.6666667 FALSE
#> 4 Item4 6 12 0.0000000 0.612792969 10 0.6666667 FALSE
#> recommendation
#> 1 Supported
#> 2 Supported
#> 3 Review
#> 4 Review
#> interpretation
#> 1 Essential ratings meet the exact one-sided binomial criterion for this panel size.
#> 2 Essential ratings meet the exact one-sided binomial criterion for this panel size.
#> 3 Essential ratings do not meet the exact panel-size criterion; review the item and expert rationale before deciding whether to revise or remove it.
#> 4 Essential ratings do not meet the exact panel-size criterion; review the item and expert rationale before deciding whether to revise or remove it.
#> status
#> 1 Supported
#> 2 Supported
#> 3 Review
#> 4 ReviewFor an essentiality task, state that CVR is tied to a different expert judgment than relevance. Report the effective panel size and exact critical essential count for each item rather than borrowing a relevance/CVI threshold.
Congruence
expert_ioc <- read_example("expert_congruence_example.csv")
ioc_fit <- expert_validity(expert_ioc, mode = "congruence")
ioc_fit$results
#> item target target_ioc strongest_competitor competitor_ioc margin
#> 1 I1 A 1.0000000 B -0.6666667 1.6666667
#> 2 I2 B 1.0000000 A -0.6666667 1.6666667
#> 3 I3 C 0.8333333 B 0.5000000 0.3333333
#> recommendation
#> 1 Target favored
#> 2 Target favored
#> 3 Target favored
#> interpretation
#> 1 The intended objective has the highest IOC; use the margin and expert comments to judge practical distinctiveness.
#> 2 The intended objective has the highest IOC; use the margin and expert comments to judge practical distinctiveness.
#> 3 The intended objective has the highest IOC; use the margin and expert comments to judge practical distinctiveness.
#> status
#> 1 Supported
#> 2 Supported
#> 3 SupportedFor IOC, report the intended objective, target IOC, strongest competing objective, and target-minus-competitor margin. The margin is diagnostic evidence about alignment; it is not a newly invented significance test.
Figures across review stages
Most studies gather more than one kind of content evidence.
content_evidence() brings the stages together, in the order
they ran, and draws two figures for a paper or poster. Here the twelve
items of vignette("one-item-set-both-stages") are first
rated for relevance by eight experts, and the items the panel carries go
on to the item sort. The relevance ratings are constructed for this
example; data-raw/build-walkthrough-panel.R says what each
item’s ratings are built to show.
relevance <- read_example("walkthrough_relevance.csv")
panel <- content_handoff(expert_validity(
as.matrix(relevance[setdiff(names(relevance), "expert")]),
mode = "relevance", lo = 1, hi = 4, agreement = "none"
))
sorted <- read_example("walkthrough_sort.csv")
sort_stage <- sort_validity(sorted[sorted$item %in% panel$items, ])
evidence <- content_evidence(`Relevance panel` = panel, `Item sort` = sort_stage)
evidence
#> Content evidence across 2 stages
#> --------------------------------
#> 10 of 12 items carried by every stage that reviewed them. Held back: EF5
#> (Relevance panel), TF5 (Item sort).
#>
#> Stages, in order
#> 1. Relevance panel: 12 items, 8 experts. Shows I-CVI.
#> 2. Item sort: 11 items, 20 judges. Shows Psa.
#>
#> item Relevance panel Item sort result
#> EF1 1.00 Strong support .95 Retain carried
#> EF2 1.00 Strong support .90 Retain carried
#> EF3 1.00 Strong support .90 Retain carried
#> EF4 1.00 Strong support .90 Retain carried
#> EF5 .50 Review -- held back: Relevance panel
#> EF6 .88 Strong support .75 Retain carried
#> TF1 1.00 Strong support .95 Retain carried
#> TF2 1.00 Strong support .85 Retain carried
#> TF3 1.00 Strong support .90 Retain carried
#> TF4 1.00 Strong support .80 Retain carried
#> TF5 1.00 Strong support .30 Review held back: Item sort
#> TF6 1.00 Strong support .90 Retain carriedThe flow diagram follows the items through the stages. It is modeled on the PRISMA 2020 flow diagram for systematic reviews (Page et al., 2021), with items in place of studies, and it is the figure a Method section needs: what each stage reviewed, what it held back and why, and what went forward.
The evidence profile sets the stages side by side, item by item. Each panel shows the statistic that stage’s decision read, its 95% interval, and its criterion as a dashed line; a filled symbol met the criterion and an open one was flagged for review.
Read across a row and the sources can disagree. Every expert rated
TF5 relevant; only the sort shows that judges place it in
the other facet. Read down a panel and the intervals speak: an I-CVI of
1.00 from eight experts has an interval reaching down to about .68, a
reminder of how small a panel is.
Both figures are gray by default, as an APA figure is printed. For
slides or a poster, apa = FALSE draws them in color, teal
for evidence that met its criterion and brown for evidence under review.
The symbols carry the same decisions, so no reading depends on
color.
A caption for the profile might read: Content evidence for twelve draft items. Left: the share of eight experts rating each item relevant (I-CVI), with 95% Wilson score intervals and Lynn’s (1986) criterion of 7 of 8. Right: the share of 20 judges assigning each item to its intended facet (Psa), with 95% Wilson score intervals and the exact test’s criterion of 15 of 20 (Howard & Melloy, 2016). Filled symbols met the criterion; open symbols were flagged for review.
Minimum reproducibility statement
At minimum, a manuscript or supplement should identify the package version and record the analysis settings that determine the results. For a fitted workflow:
packageVersion("contentvalidR")
#> [1] '0.10.1'
sort_fit$settings
#> $method
#> [1] "Anderson-Gerbing Psa/Csv with Howard-Melloy exact inference"
#>
#> $item_inference
#> [1] "Howard-Melloy exact target-count test"
#>
#> $scale_benchmarks
#> [1] "Colquitt et al. (2019) empirical percentile norms"
#>
#> $p0
#> [1] 0.5
#>
#> $alpha
#> [1] 0.05
#>
#> $judge_type
#> [1] "naive"
#>
#> $proportion_ci
#> [1] "wilson"
sort_fit$design
#> $type
#> [1] "item-sort"
#>
#> $n_items
#> [1] 6
#>
#> $n_raters
#> [1] 20
#>
#> $n_judges_min
#> [1] 20
#>
#> $n_judges_max
#> [1] 20
#>
#> $n_missing
#> [1] 0
#>
#> $n_target_scales
#> [1] 3
#>
#> $n_constructs_observed
#> [1] 3A strong reproducibility supplement should also archive the item wording, construct definitions, judge instructions, anonymized response data when permitted, and the script that reproduces all tables and figures.
Language to avoid
Avoid claims such as “the scale was proven content valid” or “items failing the cutoff were invalid.” The workflows quantify evidence from a defined pretest. They do not replace a theory-based definition of the construct domain or the researcher’s responsibility to justify substantive item decisions.
References
Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation criteria for definitional correspondence and definitional distinctiveness. Journal of Applied Psychology, 104(10), 1243–1265. https://doi.org/10.1037/apl0000406
Howard, M. C., & Melloy, R. C. (2016). Evaluating item-sort task methods: The presentation of a new statistical significance formula and methodological best practices. Journal of Business and Psychology, 31(1), 173–186. https://doi.org/10.1007/s10869-015-9404-y
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–385. https://doi.org/10.1097/00006199-198611000-00017
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., . . . Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, Article n71. https://doi.org/10.1136/bmj.n71