Cost Aware Retrieval and Abstention for Question Answering in Institutional Repositories
DOI:
Keywords:
Retrieval augmented generation; selective prediction; institutional repositories; confidence calibration; abstention; decision utility.
Abstract
Institutional question answering requires a balance between answering useful questions, controlling computation, and avoiding unsupported claims. This paper develops an analytical framework that selects basic retrieval, expanded retrieval, or abstention according to expected decision utility. A transparent scenario contains 1,000 queries divided into strong, ambiguous, and insufficient evidence groups. Correctness probabilities and processing costs are stipulated inputs rather than measurements of an implemented language model. Four policies are compared using coverage, selective accuracy, expected errors, computation, and a loss function. Under the main assumptions, the proposed cascade answers 80 percent of queries with 93 percent expected selective accuracy, uses 1,900 processing units, and achieves utility of 386. Always using expanded retrieval consumes 4,000 units and achieves utility of 74. Sensitivity analysis shows that the preferred policy changes when error penalties increase or confidence estimates become optimistic. The contribution is a reproducible decision analysis explaining when additional retrieval is worthwhile and when deferral is preferable. The results support conditional design choices and establish requirements for subsequent empirical validation; they do not constitute a benchmark of any deployed question answering system.
Downloads
Published
Issue
Section
License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

