HK-GenSpeech: A Generative AI Scene Creation Framework for Speech Based Cognitive Assessment

Abstract

Speech-based cognitive assessments have emerged as promising alternatives to traditional dementia screening methods, offering scalability, accessibility, and non-invasiveness. However, prevalent picture-description tasks such as the Cookie Theft Picture (CTP) suffer from limitations including cultural irrelevance, learning effects from repeated exposure, and reduced engagement over multiple sessions. To address these constraints, this research introduces HK-GenSpeech (HKGS), a generative AI framework designed to dynamically create culturally adaptable and diagnostically relevant visual scenes.

The HKGS framework integrates insights from established assessment methods with advanced generative models, notably employing GPT-4o for prompt synthesis and diffusion-based models for image generation. This approach enables the automated creation of di-verse yet comparable assessment stimuli tailored specifically to the Cantonese-speaking population in Hong Kong.

Evaluations were conducted using the HK-GenSpeech Accompanying Dataset (HKGS), which comprises speech samples from 141 Hong Kong participants across traditional, fixed AI-generated, and dynamically AI-generated scene description tasks. The results demonstrate that dynamically generated scenes maintain comparable diagnostic validity to traditional methods. Additionally, findings indicate that dynamic image generation significantly enhances linguistic diversity, reduces memorisation and learning effects, and improves participant engagement.

This study contributes to the field by proposing a scalable methodology that enhances the robustness and applicability of speech-based dementia screening, particularly in linguistically diverse and resource-limited settings. The HKGS framework thus presents a significant advancement towards globally accessible, culturally sensitive cognitive assessments.

Date of Award	2025
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Tristan Camille BRAUD (Supervisor) & Suk Wai Winnie LEUNG (Supervisor)