Towards futuristic visual storytelling : authoring data-driven infographics in augmented reality

Abstract

An increasingly large amount of data from the physical world has been collected, digitized, and stored. To communicate such data to general public effectively, visual data-driven storytelling has been widely used. Yet, the vast majority of data communication occurs on desktop computers separated from the physical world the data originates in and refers to. Recent advances in Augmented Reality (AR) have shed new light on data-driven storytelling, offering exciting possibilities for telling engaging, in-situ, and immersive stories by embedding the data in the real-world context. However, although creating such kind of AR data stories is demanding and requires considerable knowledge and skills from different fields (e.g., data visualization, computer graphics, computer vision, and human-machine interaction), prior research has rarely investigated the effective way to create visual data-driven stories in AR environments. This thesis aims to fill this gap by exploring the approaches to facilitate the authoring of infographics, a popular format for data-driven storytelling, in AR. Given that AR devices are still evolving rapidly, this thesis focuses on the interaction between reality and virtuality, an essence of AR that is independent of specific devices. The first system, MARVisT, leverages physical properties (e.g., size, positions) from real-world objects to assist non-experts in creating 3D infographics in mobile AR. Given the limited interaction capabilities of mobile devices, in the second work, LassoNet is proposed to facilitate the selection of 3D objects in a 2D screen based on a deep neural network. Besides augmenting the physical world, the third research studies augmenting the semantic content of real-world infographics in AR and introduces PapARVis Designer to allow designers to create augmented static visualizations. Finally, the fourth work explores automating the creation of timeline infographics in AR by developing a deep-learning based method that automatically extracts extensible templates from real timeline infographics to generate virtual timelines with new data in AR. The core idea of this thesis is to allow visualization designers to go beyond the desktop platform to the engaging, immersive, and promising AR platform that is seen to be the next-generation human-machine interaction platform. The resulting systems and techniques blaze a trail toward futuristic visual storytelling.

Date of Award	2020
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology