Projected visible light for 3D finger tracking and device augmentation on everyday objects

Recent advances on the Internet of Things (IoT) lead to an explosion of physical objects being connected to the Internet. These objects sense, compute, interpret what is occurring within themselves and the world, and preferably interact with users. In this work, we present a visible light-enabled finger tracking technique allowing users to perform freestyle multi-touch gestures on everyday object's surface. By projecting encoded patterns onto an object's surface (e.g. paper, display, or table) through a projector, and localizing the user's fingers with light sensors, the proposed system offers users a richer interactive space than the device's existing interfaces. More importantly, results from our experiments indicate that this system can localize ten fingers simultaneously with an accuracy of 1.7 mm and an refresh rate of 84 Hz with only 31 ms delay on WiFi or 23 ms delay on serial communication, easily supporting multi-finger gesture interaction on everyday objects. We also develop two example applications to demonstrate possible scenarios. Finally, we conduct a preliminary exploration of 3D depth inference using the same setup and achieve 2.43 cm depth estimation accuracy.