Teacher Perception of GenAI as an Assessment Aid

TEACHER PERCEPTION OF GENAI AS AN ASSESSMENT AID
Bruce Ma and Martin Ma
Center for Language Education, Hong Kong University of Science and Technology, Hong Kong
[email protected]; [email protected]
ABSTRACT
The emergence of Generative AI (GenAI) in November 2022 has initiated a proactive integration of this technology into pedagogical practices
at the HKUST Center for Language Education (CLE). Apart from encouraging and teaching students to use GenAI, HKUST has made
assessment-related GenAI tools accessible to faculty. One such tool is Pregrade - a grammar and spelling grader with the capability to
assess and comment on writings based on a detailed rubric (Pregrade, 2024). HKUST’s second year Technical Communication course,
designed for Engineering students, spearheads the university's integration by incorporating various GenAI learning strategies. This course’s
ESP nature presents difficulties for language teachers who have little to no engineering background as one of its assessments revolves
around analyzing the ethics of engineering incidents. The large amount of information that is needed to grade this assessment poses a
challenge to the efficiency and consistency in grading. Pregrade has limited application in this case as the assessment requires specific
content-based grading which it does not currently offer (Pregrade, 2024). Therefore, a tailored, content-based solution was developed: our
course-specific GenAI bot designed to provide feedback for teachers.
While tertiary students have a strong inclination to embrace GenAI (Arowosegbe, 2024; Chan & Hu, 2023; Kohnke, 2024; Li et al., 2024),
our literature review highlights the transformative potential of GenAI tools in language assessment and teacher feedback practices as well.
Studies indicate that AI can enhance grading efficiency and provide valuable, accurate feedback to students (Mohamed, 2023), while also
streamlining the assessment process, ultimately saving teachers time and effort (Koraishi, 2023). Yet, there is a gap in research examining
the pedagogical effects of these tools, particularly regarding student and teacher attitudes toward AI integration in the classroom (Har & Ma,
2023). While considerable research has focused on student attitudes, teachers' perceptions and experiences have received relatively less
attention. This study aims to explore teachers' perceptions and experiences with our course-specific GenAI bot that serves as a marking
assistant, emphasizing its usability, reliability, efficiency, and overall impact on teaching practices. By addressing this gap, the study
contributes to the ongoing dialogue about the role of GenAI in educational assessments and its broader implications for teaching.
According to Ma (2017), the integration of technology provides an innovative alternative for delivering written feedback on student writing
more effectively and efficiently. This shift is particularly evident in the development of Automated Writing Evaluation (AWE) tools, which
utilize computer-based educational technology to offer students constructive feedback on their writing. The emergence of GenAI has the
potential to revolutionize the use of AWE tools in assessing writing as it can enhance traditional AWE systems by providing more contextaware feedback. Unlike conventional AWE tools, which often rely on rule-based algorithms, GenAI can understand and generate natural
language, allowing it to offer suggestions that are more aligned with the specific contexts and needs.
Building on this content-aware feature, our course-specific GenAI bot, based on Claude-3.5-Sonnet-200k, is pre-loaded with a
comprehensive knowledge base of engineering incidents. It is designed to assist teachers in evaluating students’ engineering analytical
reports by comparing the evidence presented in student writings against this knowledge base. The bot generates accessible summaries of
the analysis in both text and tabular formats, highlighting any missing components to enable teachers to quickly identify gaps in the reports.
An explanatory mixed methods approach is adopted for this study, combining quantitative and qualitative data collection techniques to
provide a comprehensive understanding of teachers' experiences with the GenAI bot. Specifically, these experiences relate to the extent to
which the bot supports teachers in assessing whether students can identify factors contributing to engineering disasters, highlight the
interplay between these factors, analyze engineers’ accountability, apply ethical codes to hold relevant engineers responsible, and evaluate
the overall quality of students’ analyses in connecting these elements. We are currently in the data collection phase of the study, which is
scheduled to span two semesters, ending in May 2025.
REFERENCES
Arowosegbe, A. (2024). Students’ perception of generative ai use for academic purpose in UK higher education.
https://doi.org/10.20944/preprints202405.1158.v1
Chan, C., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International
Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00411-8
Har, F., & Ma, B. W. L. (2023). The future of education utilizing an artificial intelligence robot in the Centre for Independent Language
Learning: Teacher perceptions of the robot as a service. In C. Hong & W. W. K. Ma (Eds.), Applied degree education and the
shape of things to come. Springer, Singapore. https://doi.org/10.1007/978-981-19-9315-2_3
Kohnke, L. (2024). Exploring EAP students' perceptions of GenAI and traditional grammar checking tools for language learning. Computers
and Education: Artificial Intelligence, 7, 100279. https://doi.org/10.1016/j.caeai.2024.100279
Koraishi, O. (2023). Teaching English in the age of AI: Embracing ChatGPT to optimize EFL materials and assessment. Language Education
and Technology, 3(1), 55–72. https://langedutech.com/letjournal/index.php/let/article/download/48/37
29
Back to
Day 1
Back to
Day 2
Back to
Day 3
Li, Y., Deng, Y., Peng, B., He, Y., Luo, Y., & Liu, Q. (2024). Generative artificial intelligence in Chinese higher education: Chinese
undergraduates’ use, perception, and attitudes. Frontiers in Educational Research, 7(4).
https://doi.org/10.25236/fer.2024.070401
Ma, B. (2017). The study of teacher written feedback: The effectiveness of electronic feedback on student writing revisions (Doctoral thesis,
Durham University).
Mohamed, A. (2023). Exploring the potential of an AI-based chatbot (ChatGPT) in enhancing English as a foreign language (EFL) teaching:
Perceptions of EFL faculty members. Education and Information Technologies. https://doi.org/10.1007/s10639-023-11917-z
Pregrade. (2024). Pregrade user guide, Pregrade. AI, https://app.pregrade.ai/guide/index.html#
KEYWORDS
assessment, written corrective feedback, GenAI, automated writing evaluation, ESP
BIODATA
Dr. Bruce Ma is an educator and researcher with a focus on writing feedback, learner autonomy, and second language writing. His research
interests also include the integration of generative AI in teaching and learning, as well as computer-aided language learning.
Martin Ma specializes in English for Academic Purposes (EAP) and English for Specific Purposes (ESP). His research interest includes
applying Systemic Functional Grammar (SFG) notions, such as Thematic Progression and Grammatical Metaphor, to teaching, as well as
applying SFG concepts through GenAI