Online surveys as evidence for policy making : examining the relationships between incentives, response rates, sample representativeness and data quality

Abstract

This study aims to examine the extent of use of survey evidence in policymaking. It also aims to evaluate the relationship between incentives, response rates, representativeness of the sample, and data quality when implementing online surveys. The study has four main components. First, through document analysis of 18,566 legislative documents used to discuss 577 bills from 2000 to 2022, I assess the extent of the use of surveys in public policymaking in Hong Kong. Second, I conducted an experimental study that examines the effect of incentives on the response rate of large-scale online social surveys like the Hong Kong Generations and Gender Survey (HK-GGS). Third, through exploratory, in-depth analyses using HK-GGS survey data, I assess the quality of the data and the sample's representativeness. Lastly, using paradata analysis, I explore and generate meaningful information from the HK-GGS paradata that can be important inputs in designing and implementing online surveys. Overall, findings suggest that there is limited uptake of survey evidence in policymaking in Hong Kong, specifically in legislation. Findings also show that conducting a large-scale, address-based, randomly sampled, long, and purely online social survey like the GGS is plausible within the premise of giving incentives and considering all other factors. The results give assurance that the HK-GGS meets the criteria of a robust and good evidence – acceptable response rates, good quality, and adjusted representativeness.

Date of Award	2022
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology
Supervisor	Stuart Arthur GIETEL-BASTEN (Supervisor) & Kira MATUS (Supervisor)