ChatGPT as a Source of Information in Higher Education: Evaluating the Results Provided by Generative AI

Abstract

The emergence of ChatGPT-3 in November 2022, along with its subsequent versions up to the current 4.0, and its expansion and increasing widespread use, is raising various issues of particular interest related to its application as an information source in Higher Education. Acknowledging that we are witnessing the mainstreaming of generative Artificial Intelligence (AI), this article starts from the premise that it is essential to understand how university students perceived and evaluated ChatGPT-3 just a few months after its release. The research, conducted using a mixed-methods approach, analyzes 4,800 responses from university students to an online survey, in which they were asked to indicate their level of knowledge and prior experience with AI, as well as their evaluations of the information provided by ChatGPT, using bivariate and multiple segmentation methods. Additionally, a thematic analysis of ChatGPT's responses was conducted using an example related to the terrorist 11-M attacks, with an expert assessment. The results show significant differences in students' level of awareness and prior experience with ChatGPT, as well as their evaluation of the information provided by ChatGPT, in relation to certain sociodemographic and educational variables. It was also found that prior experience with generative AI influences its perception as either an information source or a tool for disseminating misinformation or fake news. Lastly, comparing their scores with the expert assessment highlights the need to teach students how to properly train ChatGPT to obtain better responses and evaluate them critically. This approach could serve as a foundation for designing training strategies to introduce generative AI into Higher Education.