Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion inst/ms/aggreCAT_2.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ Expert judgement is frequently used to inform forecasting about uncertain future

Aggregated elicited judgements form a critical component of decision-making across multiple contexts, and while there are a variety of methods for mathematically aggregating expert judgements into single point-predictions, there has been a dearth of accessible tools for implementing any aggregation more complex than linear averages. Many existing software tools are closed-source, are irreproducible, or written in programming languages rarely used by ecologists or metascientists. Methods implemented in [R]{.proglang}, one of the most widely used programming languages in the life sciences [@Gao2025], are limited in scope, or have been archived from CRAN. The [R]{.proglang} package [SHELF]{.pkg} implements only a single method (weighted linear pool) for aggregating expert judgements [@Oakley2026], but integrates with [expertsurv]{.pkg} to incorporate expert judgement into survival analysis [@Cooney2023], while [opera]{.pkg} provides a suite of methods for aggregating time-series predictions, but does not implement methods for aggregating point-predictions with uncertainty bounds [@Gaillard2026].

[aggreCAT]{.pkg} is an open, reliable, and modular R package for the mathematical aggregation of expert elicitation data with a straightforward user interface. The [aggreCAT]{.pkg} package was developed by the by the repliCATS (Collaborative Assessment for Trustworthy Science) project [@Fraser2023] as part of the DARPA SCORE (Systematizing Confidence in Open Research and Evidence) program [@alipourfard2021], which aimed to generate quantitative 'confidence scores' — estimates of the likely replicability of research claims from the social and behavioural sciences — for use as a proxy measure of credibility in the absence of direct replication effort. [aggreCAT]{.pkg} arose from the need for a software solution to reliably conduct mathematical aggregation at scale. The repliCATS project used [aggreCAT]{.pkg} to aggregate expert elicited judgements into confidence scores for $> 4000$ claims [@Mody2026]. While originally a bespoke implementation for repliCATS, we have since developed [aggreCAT]{.pkg} into a more general R package that can handle most types of elicitation data requiring mathematical aggregation, allowing for wider applicability.
[aggreCAT]{.pkg} is an open, reliable, and modular R package for the mathematical aggregation of expert elicitation data with a straightforward user interface. The [aggreCAT]{.pkg} package was developed by the repliCATS (Collaborative Assessment for Trustworthy Science) project [@Fraser2023] as part of the DARPA SCORE (Systematizing Confidence in Open Research and Evidence) program [@alipourfard2021], which aimed to generate quantitative 'confidence scores' — estimates of the likely replicability of research claims from the social and behavioural sciences — for use as a proxy measure of credibility in the absence of direct replication effort. [aggreCAT]{.pkg} arose from the need for a software solution to reliably conduct mathematical aggregation at scale. The repliCATS project used [aggreCAT]{.pkg} to aggregate expert elicited judgements into confidence scores for $> 4000$ claims [@Mody2026]. While originally a bespoke implementation for repliCATS, we have since developed [aggreCAT]{.pkg} into a more general R package that can handle most types of elicitation data requiring mathematical aggregation, allowing for wider applicability.

[aggreCAT]{.pkg} provides a suite of 29 aggregation methods that explore different approaches to mathematical aggregation from straightforward arithmetic calculations to Bayesian statistical models. In addition, we provide extra functionality such as plotting functions and performance evaluation against known outcomes. While the repliCATS project uses the IDEA protocol to structure the elicitation process [@hemming2017], we have generalised [aggreCAT]{.pkg} to work with a variety of elicitation protocols. [aggreCAT]{.pkg} fills a large void in open software for aggregating elicited judgements, providing enormous benefit to researchers and decision makers across any field of research.

Expand Down
Loading