Content uploaded by Eugene Loos
Author content
All content in this area was uploaded by Eugene Loos on Aug 24, 2023
Content may be subject to copyright.
Available via license: CC BY 4.0
Content may be subject to copyright.
Using ChatGPT in Education: Human Reflection on ChatGPT’s Self-Reflection
Societies
Abstract
ChatGPT is a fascinating AI text generator tool. It is a language model developed by OpenAI, a research and deployment company with the mission, according to OpenAI’s website: “to ensure that artificial general intelligence benefits all of humanity”. ChatGPT is able to generate human-like texts. But how does it work? What about the quality of the texts it provides? And is it capable of being self-reflective? Information sources must be efficient, effective and reliable in education, in order to enhance students’ learning process. For this reason, we started a dialogue with ChatGPT-3 while using, among others, a SWOT analysis it generated about its own functioning in an educational setting. This enabled us, as human authors, to analyze the extent to which this AI system is able to practice self-reflection. Finally, the paper sketches implications for education and future research.
... Several studies conducted SWOT analyses related to the outcome of using chatbots (e.g. Farrokhnia et al., 2024;Mai et al., 2024), while Loos et al. (2023) asked ChatGPT to reflect on the Strengths, Weaknesses, Opportunities and Threats for educational use. Loos et al. (2023) also clearly showed that the underlying dynamics of conducting a "dialogue" with ChatGPT lead to Threats (e.g., so-called "hallucinations") for students One could also ask how such tools affect the way we assess students' knowledge, skills and understanding as they can now easily use chatbot generated texts. ...
... Farrokhnia et al., 2024;Mai et al., 2024), while Loos et al. (2023) asked ChatGPT to reflect on the Strengths, Weaknesses, Opportunities and Threats for educational use. Loos et al. (2023) also clearly showed that the underlying dynamics of conducting a "dialogue" with ChatGPT lead to Threats (e.g., so-called "hallucinations") for students One could also ask how such tools affect the way we assess students' knowledge, skills and understanding as they can now easily use chatbot generated texts. Especially exam boards risk to be confronted with fraud cases when students use chatbots in a non responsible way, i.e. using a them without their lecturer's permission and not explaining how they proceeded with the use of this tool. ...
... Rudolph et al. (2023, p.13) We also agree with Rudolph et al. (2023, p.13) that we should train students' skills to be prepared, after leaving university, for chatbot use. In a responsible way of course, to enable them to use "future ready digital media literacy skills" being aware of the Strengths, Weaknesses, Opportunities and Threats of this digital tool (e.g., Farrokhnia et al., 2024;Loos et al. 2023;Mai et al., 2024). ...
A narrative literature review about Chatbot use in higher education for exam boards
... Additionally, it has rapidly become a focal point in academic discussions, prompting institutions to explore its potential and address its implications. Generative AI tools like ChatGPT hold transformative potential across diverse domains, including education, medical science, content generation, and conversational applications [14][15][16][17].Similarly, numerous studies [18][19][20][21] explore the integration of Generative AI to enhance instructors' productivity and improve teaching quality in education.Several studies [22][23][24] have systematically examined the integration of ChatGPT in educational contexts, highlighting its potential to enhance the teaching and learning process. Several studies [21,25]indicate the prominence of AI to significantly improve educational experiences, particularly in developing computational thinking, solving mathematical problems, programming skills, and fostering student engagement. ...
... Despite the substantial benefits of AI technologies in educational environments, several challenges must be addressed. Concerns have been raised regarding the consistency of teaching methods, the potential for plagiarism, and the ethical implications associated with the use of AI in education [21,23,27]. For instance, while digital tools can enhance traditional instructional methods, they often face difficulties in adapting to the specific learning needs of individual students, which can impede the reinforcement of concepts learned in the classroom [29,30]. ...
... Additionally, it has rapidly become a focal point in academic discussions, prompting institutions to explore its potential and address its implications. Generative AI tools like ChatGPT hold transformative potential across diverse domains, including education, medical science, content generation, and conversational applications [14][15][16][17].Similarly, numerous studies [18][19][20][21] explore the integration of Generative AI to enhance instructors' productivity and improve teaching quality in education.Several studies [22][23][24] have systematically examined the integration of ChatGPT in educational contexts, highlighting its potential to enhance the teaching and learning process. Several studies [21,25]indicate the prominence of AI to significantly improve educational experiences, particularly in developing computational thinking, solving mathematical problems, programming skills, and fostering student engagement. ...
... Despite the substantial benefits of AI technologies in educational environments, several challenges must be addressed. Concerns have been raised regarding the consistency of teaching methods, the potential for plagiarism, and the ethical implications associated with the use of AI in education [21,23,27]. For instance, while digital tools can enhance traditional instructional methods, they often face difficulties in adapting to the specific learning needs of individual students, which can impede the reinforcement of concepts learned in the classroom [29,30]. ...
... I det kulturelle billede af den ideelle underviser i en nutidig, nordisk sammenhaeng kan underviseren ses som en vidende ledsager i en lokaliseret og kontekstafhaengig udforskningsproces sammen med de studerende (Ingold, 2018 Nkhobo & Chakas, 2023). Men selvom AI-chatbots producerer tekster med stedordene jeg, mig og min og efterligner menneskelige svar-og høflighedsformer, er der i sagens natur ikke tale om nogen selv-refleksivitet, og teksterne afspejler ingen form for forståelse eller indsigt, idet teksten er skabt af AI (Loos et al., 2023). Dette betyder, at vi ikke nødvendigvis kan tage tekst som vidnesbyrd om laering og erkendelse, isaer ikke hvis passager kan vaere overtaget direkte fra AI-chatbots. ...
Artiklen undersøger fire centrale temaer i forskningen om AI-chatbots som lærings- og undervisningsassistenter på videregående uddannelser med særligt fokus på humanistiske fag. Forfatterne påpeger, at den datalogisk orienterede AIED-litteratur ofte hviler på et kognitivistisk og individualistisk læringssyn, hvor AI-chatbots ses som en mulighed for at personalisere undervisning og effektivisere læreprocesser. Artiklen argumenterer for, at denne forskning overser betydningen af sociale og situerede læreprocesser, underviserens rolle som vidende ledsager samt forholdet mellem tekst og erkendelse. Med afsæt i en antropologisk og praksisorienteret tilgang vises det, at studerendes brug af generativ AI rejser grundlæggende spørgsmål om formålet med humanistiske uddannelser og den refleksive tænkning, der kræves for at lære i dybden. Artiklen bygger på etnografiske studier af kandidatstuderendes anvendelse af AI-chatbots i deres studiepraksis. Forfatterne konkluderer, at AI-chatbots kan være nyttige, men også indebærer risici for, at læsning og skrivning reduceres til overfladisk informationsbehandling, hvilket kan underminere kerneværdierne i humanistisk uddannelse.
... ChatGPT can personalise learning by providing content adapted to learners' knowledge and needs. [22], [28]. For example, [29] examines the possibilities of ChatGPT for supporting self-regulated learning in programming education, exploring aspects such as accessing learning materials, tools for specific content areas, selfstudy planning, feedback, and assessment. ...
This paper aims to study the opinions of teachers and students regarding the opportunities and challenges of using ChatGPT in programming education. The research combines quantitative data from Likert-scale questions with qualitative data from open-ended responses. The findings reveal similarities between students’ and teachers’ views on the advantages of using ChatGPT in programming education and its potential to develop soft skills. A difference appears in assessing the attitudes of both groups toward the disadvantages of integrating ChatGPT in education. Compared to students, teachers express much greater concern about the negative effect of artificial intelligence (AI) on academic integrity and teaching quality. The results showed that both groups positively evaluate ChatGPT as a supplementary tool in education. They believe it should complement traditional teacher-student communication rather than replace it. Based on the research findings, the authors recommend that the integration of ChatGPT into education should be preceded by adopting university AI usage policies and training for the effective use of ChatGPT. Pedagogical guidelines for integrating ChatGPT into programming education are proposed to minimise the effect of students’ overreliance on AI and achieve the learning outcomes defined by Bloom’s Taxonomy.
... The study (Loos et al., 2023) discussed the use of GBT chat in education and presents a brief analysis of the ability of this linguistic model for self-reflection and how to integrate it effectively into many educational processes. The study indicated that the GPT chat tool can contribute very widely to providing effective answers and results instead of the traditional method of providing individual lessons in automatic generation. ...
Education nowadays is irreversibly interrelated with technology, and AI has turned into an agent of change in learning environments. Among these AI tools, ChatGPT, developed by OpenAI, has emerged as one of the most sophisticated natural language processing models, which has revolutionized student interactions, research assistance, and academic support. This study will critically review the literature on the role of ChatGPT in education, focusing on its benefits, limitations, and ethical implications. The review highlights how ChatGPT enhances comprehension, critical thinking, and creativity by providing information that is instant, accurate, and contextually relevant. Its adoption rate has been phenomenal since its release in November 2022, with studies indicating that 60% of educators and 40% of students have already adopted ChatGPT into their academic practices. Technology is increasingly used for automated assessments, literature reviews, content generation, and tutoring, contributing to personalized learning experiences and engagement. However, concerns persist regarding AI bias, misinformation, ethical considerations, and student over-reliance on AI-generated content. This review categorizes ChatGPT’s educational applications into four key areas: (1) pedagogical applications, (2) academic research assistance, (3) challenges and ethical concerns, and (4) future directions in AI-enhanced education. It examines the impact of AI-driven tools on curriculum integration, educator support, and student engagement, informed by peer-reviewed articles, policy reports, and case studies. The results of this study confirm that while ChatGPT holds great promise, the integration of AI responsibly does indeed require thoughtful regulation, educator training, and ethical considerations. This research informs ongoing discussions of AI-driven education, policy formulation, and responsible AI adoption in learning environments to ensure that technological advancements align with educational integrity and innovation.
Background
The integration of Large Language Models (LLMs) into nursing education presents a novel approach to enhancing critical thinking skills. This study evaluated the effectiveness of LLM-assisted Problem-Based Learning (PBL) compared to traditional PBL in improving critical thinking skills among nursing students.
Methods
Participants were randomly assigned to either a traditional PBL group (50 nursing students) or an LLM-integrated PBL group (50 nursing students). The California Critical thinking Skills test was used to assess critical thinking skills.
Results
The LLM-integrated PBL group showed a more pronounced increase (0.60 points) in critical thinking skills compared to the PBL group (0.50 points, p < .01). A notable difference was observed in inductive reasoning skills between the PBL group and the LLM-integrated PBL group ( p < .01).
Conclusion
This study provides empirical evidence supporting the use of LLM-assisted PBL as an effective educational strategy in nursing education.
ChatGPT se disfraza de ser humano. Por este motivo, se pretende comprobar si esta interfaz es capaz de crear pruebas de evaluación de alemán como lengua extranjera de calidad. El principal objetivo de esta investigación es evidenciar si ChatGPT tiene la capacidad de sustituir la labor docente de crear exámenes. La metodología del presente estudio ha consistido en el envío de dos encuestas similares, una a profesorado y otra a alumnado de alemán del ámbito universitario, para comprobar si son capaces de distinguir entre textos producidos por humanos, específicamente exámenes de la Escuela Oficial de Idiomas, o textos similares creados por ChatGPT. En ocasiones, ChatGPT comete errores y expresa incoherencias, lo cual no puede permitirse en la evaluación de conocimientos de lenguas extranjeras, sobre todo, si los humanos son capaces de crear esos mismos exámenes de manera más eficiente y rápida
Foundational management theories, such as the Principles of Scientific Management, Bureaucratic Management, or Systems Theory, provide frameworks for understanding and organizing the extensive body of management knowledge. Despite their significance, students often find it difficult to grasp and apply historical concepts in today’s workplace. In this experiential exercise, we use generative artificial intelligence (AI) to create an interactive chatbot through which students “interview” historical figures. Through this exercise, students achieve four key learning objectives: (a) demonstrating a critical understanding of historical management frameworks; (b) analyzing how historical, social, and economic contexts influenced management thinking; (c) evaluating the applicability of historical management frameworks to contemporary business settings; and (d) critically analyzing and interpreting information generated by AI tools to enhance AI literacy. The activity includes tailored open-access chatbots and resources to assist these learning outcomes across various class formats.
Advances in Natural Language Processing (NLP) have yielded significant advances in the language assessment field. The Automated Essay Evaluation (AEE) mechanism relies on basic research in computational linguistics focusing on transforming human language into algorithmic forms. The Criterion® system is an instance of AEE software providing both formative feedback and an automated holistic score. This paper aims to investigate the impact of this newly-developed AEE software in a current ESL setting by measuring the effectiveness of the Criterion® system in improving ESL undergraduate students’ writing performance. Data was collected from sixty-one ESL undergraduate students in an academic writing course in the English Language department at Princess Norah bint Abdulruhman University PNU. The researcher employed a repeated measure design study to test the potential effects of the formative feedback and automated holistic score on overall writing proficiency across time. Results indicated that the Criterion® system had a positive effect on the students’ cores on their writing tasks. However, results also suggested that students’ mechanics in writing significantly improved, while grammar, usage and style showed only moderate improvement. These findings are discussed in relation to AEE literature. The paper concludes by discussing the implications of implementing AEE software in educational contexts.
The rapid advancements in generative AI models present new opportunities in the education sector. However, it is imperative to acknowledge and address the potential risks and concerns that may arise with their use. We analyzed Twitter data to identify critical concerns related to the use of ChatGPT in education. We employed BERT-based topic modeling to conduct a discourse analysis and social network analysis to identify influential users in the conversation. While Twitter users generally expressed a positive attitude toward using ChatGPT, their concerns converged into five categories: academic integrity, impact on learning outcomes and skill development, limitation of capabilities, policy and social concerns, and workforce challenges. We also found that users from the tech, education, and media fields were often implicated in the conversation, while education and tech individual users led the discussion of concerns. Based on these findings, the study provides several implications for policymakers, tech companies and individuals, educators, and media agencies. In summary, our study underscores the importance of responsible and ethical use of AI in education and highlights the need for collaboration among stakeholders to regulate AI policy.
There has been considerable intrigue surrounding the use of Large Language Model powered AI chatbots such as ChatGPT in research, educational contexts, and beyond. However, most studies have explored such tools’ general capabilities and applications for language teaching purposes. The current study advances this discussion to examine issues pertaining to human judgements, accuracy, and research ethics. Specifically, we investigate: 1) the extent to which linguists/reviewers from top journals can distinguish AI- from human-generated writing, 2) what the basis of reviewers’ decisions are, and 3) the extent to which editors of top Applied Linguistics journals believe AI tools are ethical for research purposes. In the study, reviewers (N = 72) completed a judgement task involving AI- and human-generated research abstracts, and several reviewers participated in follow-up interviews to explain their rationales. Similarly, editors (N = 27) completed a survey and interviews to discuss their beliefs. Findings suggest that despite employing multiple rationales to judge texts, reviewers were largely unsuccessful in identifying AI versus human writing, with an overall positive identification rate of only 38.9%. Additionally, many editors believed there are ethical uses of AI tools for facilitating research processes, yet some disagreed. Future research directions are discussed involving AI tools and academic publishing.
Developments in the chatbot space have been accelerating at breakneck speed since late November 2022. Every day, there appears to be a plethora of news. A war of competitor chatbots is raging amidst an AI arms race and gold rush. These rapid developments impact higher education, as millions of students and academics have started using bots like ChatGPT, Bing Chat, Bard, Ernie and others for a large variety of purposes. In this article, we select some of the most promising chatbots in the English and Chinese-language spaces and provide their corporate backgrounds and brief histories. Following an up-to-date review of the Chinese and English-language academic literature, we describe our comparative method and systematically compare selected chatbots across a multi-disciplinary test relevant to higher education. The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary. The much-vaunted AI is not yet that intelligent, it would appear. GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages. We conclude our article with four types of recommendations for key stakeholders in higher education: (1) faculty in terms of assessment and (2) teaching & learning, (3) students and (4) higher education institutions.
Large language models that exhibit instruction-following behaviour represent one of the biggest recent upheavals in conversational interfaces, a trend in large part fuelled by the release of OpenAI’s ChatGPT, a proprietary large language model for text generation fine-tuned through reinforcement learning from human feedback (LLM+RLHF). We review the risks of relying on proprietary software and survey the first crop of open-source projects of comparable architecture and functionality. The main contribution of this paper is to show that openness is differentiated, and to offer scientific documentation of degrees of openness in this fast-moving field. We evaluate projects in terms of openness of code, training data, model weights, RLHF data, licensing, scientific documentation, and access methods. We find that while there is a fast-growing list of projects billing themselves as ‘open source’, many inherit undocumented data of dubious legality, few share the all-important instruction-tuning (a key site where human labour is involved), and careful scientific documentation is exceedingly rare. Degrees of openness are relevant to fairness and accountability at all points, from data collection and curation to model architecture, and from training and fine-tuning to release and deployment. [See also: https://opening-up-chatgpt.github.io ]
Although informal evaluations of modern LLMs can be found on social media, blogs, and news outlets, a formal and comprehensive comparison among them has yet to be conducted. In response to this gap, we have undertaken an extensive benchmark evaluation of LLMs and conversational bots. Our evaluation involved the collection of 1002 questions encompassing 27 categories, which we refer to as the ``Wordsmiths dataset." These categories include reasoning, logic, facts, coding, bias, language, humor, and more. Each question in the dataset is accompanied by an accurate and verified answer. We meticulously assessed four leading chatbots: ChatGPT, GPT-4, Bard, and Claude, using this dataset. The results of our evaluation revealed the following key findings: a) GPT-4 emerged as the top-performing chatbot across almost all categories, achieving a success rate of 84.1%. On the other hand, Bard faced challenges and achieved a success rate of 62.4%. b) Among the four models evaluated, one of them responded correctly approximately 93% of the time. However, all models were correct only about 44%. c) Bard is less correlated with other models while ChatGPT and GPT-4 are highly correlated in terms of their responses. d) Chatbots demonstrated proficiency in language understanding, facts, and self-awareness. However, they encountered difficulties in areas such as math, coding, IQ, and reasoning. e) In terms of bias, discrimination, and ethics categories, models generally performed well, suggesting they are relatively safe to utilize. To make future model evaluations on our dataset easier, we also provide a multiple-choice version of it (called Wordsmiths-MCQ). Dataset link: https://github.com/mehrdad-dev/Battle-of-the-Wordsmiths.
The literature on SWOT is characterized by a debate among academics who have identified problems and proposed solutions for the strategic management tool, yet little research to date has captured practitioners’ perspectives. Recent literature indicates that SWOT is still the most popular strategic management tool among competitive intelligence (CI) professionals. The purpose of this study is to bridge this academic-practitioner divide in the SWOT literature by conducting a cross-sectional survey that gathers practitioners’ feedback regarding whether they are experiencing the problems or employing the solutions proposed by academia. A survey was distributed via LinkedIn to collect data from CI and other business professionals who conduct SWOT in the workforce. The findings confirm that practitioners experience select problems identified by the literature. Specifically, they may have too many factors per SWOT category, may be defining factors with ambiguous and unclear words, and may not have a means for resolving conflicts when factors fall in multiple categories (e.g., opportunity and threat). The findings also indicate that practitioners may not be consistently conducting SWOT as a structured business process, as proposed in the literature. The feedback provided by CI and other business professionals aids in closing the academic-practitioner divide by more clearly identifying persistent issues with SWOT and creating valuable and actionable insights that will drive the continual improvement of this popular strategic management tool.
In the Artificial Intelligence (AI) age now, foreign language learners can get unlimited support on their learning tasks from advanced AI chatbots, primarily the ChatGPT. At the same time, such a language chatbot intensifies the importance of rethinking learning outcome assessment. Traditional assessments that draw on a teach-and-test-approach are of little use and no longer valid for a comprehensive understanding of students' knowledge and skills that they, by and large, obtain in informal learning settings and contribute to their overall performance. Hence, alternative assessments can reflect more on learners' actual performance that likely involves AI chatbots and yet goes unnoticed by traditional assessment. This paper discusses alternative assessments regarding their nature, forms, characteristics, advantages, and integration into L2 programs. It establishes a foundational theoretical account for future research that would take place when ChatGPT becomes commonplace in the worldwide L2 contexts. It charts new research territories and passes the torch to second and foreign-language learning assessors to reflect on their teaching situations and reimagine L2 programs in light of the affordances of the ChatGPT, which has made a significant breakthrough in learning and teaching languages.
This paper examines the potential benefits and challenges of using the
generative AI model, ChatGPT, in higher education, in the backdrop
of the constructivist theory of learning. This perspective-type study
presents five benefits of ChatGPT: the potential to facilitate adaptive
learning, provide personalised feedback, support research and data
analysis, offer automated administrative services, and aid in developing
innovative assessments. Additionally, the paper identifies five challenges:
academic integrity concerns, reliability issues, inability to evaluate and
reinforce graduate skill sets, limitations in assessing learning outcomes,
and potential biases and falsified information in information processing.
The paper argues that tertiary educators and students must exercise
caution when using ChatGPT for academic purposes to ensure its ethical,
reliable, and effective use. To achieve this, the paper proposes various
propositions, such as prioritising education on the responsible and ethical
use of ChatGPT, devising new assessment strategies, addressing bias and
falsified information, and including AI literacy as part of graduate skills.
By balancing the potential benefits and challenges, ChatGPT can enhance
students’ learning experiences in higher education. Keywords: Academic integrity; ChatGPT; constructivism theory of learning; generative AI; higher education; Large Language Model (LLM).
This research note is a conversation between ChatGPT and a sociologist about the use of ChatGPT in knowledge production. ChatGPT is an artificial intelligence language model, programmed to analyse vast amounts of data, recognise patterns and generate human-like conversational responses based on that analysis. The research note takes an experimental form, following the shape of a dialogue, and was generated in real time, between the author and ChatGPT. The conversation reflects on, and is a reflexive contribution to, the study of artificial intelligence from a sociology of science perspective. It draws on the notion of reflexivity and adopts an ironic, parodic form to critically respond to the emergence of artificial intelligence language models, their affective and technical qualities, and thereby comments on their potential ethical, social and political significance within the humanities.