ConstraintChecker : a plugin for large language models to reason on commonsense knowledge bases

Abstract

Reasoning over Commonsense Knowledge Bases (CSKB), i.e. CSKB reasoning, has been explored as a way to acquire new commonsense knowledge. Despite the advancement of Large Language Models (LLM) and prompt engineering techniques in various reasoning tasks, they still struggle to deal with CSKB reasoning. One challenge for them is to acquire explicit relational constraints in CSKBs from only in-context exemplars, due to their lack of symbolic reasoning capabilities. In this thesis, we propose ConstraintChecker, a symbolic-reasoning plugin over baseline prompting techniques to provide and check explicit constraints. When considering a new knowledge instance, ConstraintChecker employs a rule-based module to produce a list of constraints, then it uses a zero-shot learning module to check whether this knowledge instance satisfies all constraints. The acquired constraint-checking result is then aggregated with the output of the main prompting technique to produce the final output. Experimental results on CSKB Reasoning benchmarks demonstrate the effectiveness of our method by bringing consistent improvements over all baseline prompting techniques.

Date of Award	2024
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Yangqiu SONG (Supervisor)