Assessing Generative AI for Subject Term Selection in Evidence Synthesis: A Longitudinal Evaluation​

Abstract

Aim: Artificial intelligence (AI) is significantly influencing the evidence synthesis process. Although certain aspects of the review process are well suited to AI enhancement, generative AI (genAI) tools have known issues when creating search strategies. These include incorrect use of Boolean operators, mistakes in search syntax, and difficulty identifying relevant subject terms. This project aims to longitudinally evaluate the performance of genAI tools in suggesting subject terms for comprehensive search strategies. By quantifying the ability of various models to suggest appropriate subject terms, we aim to provide valuable insights into their reliability and limitations in supporting search strategy development. Methods:

Identify gold standard search strategies: Included searches must meet three criteria: they must be created by an information specialist who is a study co-author, have undergone a Peer Review of Electronic Search Strategies (PRESS), and provide a complete MEDLINE search. To ensure our measurements are broadly applicable, we will select reviews from different health science areas. Create Prompts: Using the gold standard studies, we will create prompts that include the study's purpose, inclusion and exclusion criteria, and other relevant details. These prompts will ask the models to suggest Medical Subject Headings (MeSH). We'll test the prompts with at least five different generative AI models, which are likely to include ChatGPT, Gemini, Claude, and Elicit SR Pro, depending on external funding. Test Prompts: Prompts will be tested monthly to capture longitudinal changes in model performance over time. Detailed documentation on the tool, model, and output will be captured. Analyze Results: Analysis on the performance of each tool's output will be completed and will include metrics related to recall, precision, and hallucination rate Results: This research is underway. Preliminary results will be available in spring 2026. Discussion/Conclusion: Our study's findings are expected to provide the first longitudinal data on the performance of genAI models in suggesting subject terms for reviews. The results will quantify key metrics to provide an evidence-based understanding of these tools' reliability. The results will offer guidance for health librarians that allows them to make informed decisions about if and how to responsibly integrate genAI tools into their search strategy development process.