Best of Core Forum: Accuracy Matters: Evaluating the Value of AI Tools

Description: Since the introduction of ChatGPT in 2022, the potential impact of Artificial Intelligence on library workflows has been a topic of immense interest—and sometimes anxiety—in the library profession. In the area of cataloging and metadata, one frequently cited potential use for AI (particularly Large Language Models (LLMs)) is constructing summaries and abstracts of information resources. However, very little systematic assessment has been done of the characteristics, quality, and utility of AI-generated abstracts for library materials, leaving information professionals with little evidence on which to make data-driven decisions as to how, if at all, to implement AI-assisted workflows for this aspect of metadata work. Using electronic theses and dissertations (ETDs) as a test case, this study compared AI-generated summaries with human-generated summaries, assessing them for relevance, completeness, clarity, and potential bias. The presenters will share their findings, identifying strengths and limitations of AI-generated summaries for ETDs from a variety of academic disciplines, and offer recommendations and a rubric that information professionals can use to assess the utility of LLM tools for summary and abstract creation.

Learning Outcomes: 

At the end of the webinar, attendees will: -

• Participants will learn prompt writing techniques for generative AI tools.  
• Participants will learn strengths and weaknesses of AI-assisted summaries for information resources. 
•  Participants will learn how to use a rubric to assess AI-generated content.

Who Should Attend: The talk is geared to catalogers and metadata librarians. Artificial intelligence (AI) could have profound implications for academic libraries in the future and how librarians will do their work. There is immense potential for AI’s uses to increase access to knowledge in essential ways. With the growing popularity of AI tools like ChatGPT and Copilot, librarians are experimenting and assessing AI tools to aid in improving discovery and searching, refining metadata, and summarization. AI has been a trend in libraries for several years, yet AI raises ethical issues and concerns about bias, privacy, misinformation, transparency, and exploitation. Recent studies have raised questions about the accuracy of AI-generated summaries in other areas such as news story summarization, highlighting the need for careful evaluation of AI-generated content prior to its integration into metadata workflows. Librarians are also at the forefront to cultivate AI literacy. Some institutions have begun developing workshops to teach students the potential problems with AI such as accuracy, misrepresentations, ethical issues, and environmental impact.

Presenters:

Emily Baldoni (she/her) is a Metadata Librarian and Assistant Professor at Illinois State University, where she manages metadata for digital collections. Her work and research interests include cataloging and metadata education, rare materials cataloging, digital humanities, linked data, and identity management.

Angela Yon (she/her) is the Cataloging & Metadata Librarian and Associate Professor at Illinois State University. Her work and research interests include resource description and discovery, linked data, critical cataloging, and topics in the digital humanities and digital scholarship.

Tech Requirements

Core Webinars are held in Zoom. Speakers or a headset for listening to the presentation are required. You may interact with the presenter and ask questions through text-based chat. Closed captioning is available in the Zoom platform. The webcast will be recorded and the link to the recording shared with registrants shortly after the live event.