This research is being conducted as part of a joint European project with partners at l'Université Pierre et Marie Curie (France), University of Verona (Italy), INRIA-IRISA (France), and Aalborg University (Denmark). Funding is provided by a generous grant from the the Ministère du Développmeent, économique, Innovation et Exportation (Québec) in coordination with European Commission Seventh Framework NEST project Natural Interactive Walking (no. 222107).
This project explores the creative and informative interaction that can arise from walking on an augmented floor. Walking is a natural task that gathers a wealth of information about the environment even with our eyes closed.
We want to recreate all three sensations, haptic vibration, audio and visual in a low-latency environment to immerse walkers in a realistic virtual space. In addition, we are looking at information transfer through vibrating the area under the foot to provide meaningful information in a completely discreet manner known only to the walker.
Research issues involve sensing and actuation methods, including both sound and haptic synthesis models, as well as the physical architecture of the floor itself.
There have been intelligent floors with various forms of feedback developed.
Our goals for this project are not filled by any of these floors for several main reasons:
We wanted to present feedback in the most direct way possible, from the ground up. That means, audio sounds coming from the ground, vibration coming from the ground as well as graphics that are shown on the ground, all at the same time.
|
A first-generation working prototype was conceptualized by
Yon Visell and Karmen
Franinovic, then developed in 2007 by Yon Visell and two project
laboratory students, using FSRs as sensors, a loudspeaker for audio
output, and a single D-Box motion platform actuator to drive the
mechanical response (video clip here).
Informal experiments led to the conclusion that most of the salient
haptic percepts related to ground surface identification can be
derived purely from the vibrational, as opposed to displacement
effects, leading to a redesign using a vibrotacticle actuator instead.
This offers the advantages of producing both acoustic and haptic
stimulus from the same mechanism, as well as a much faster response
time, as necessary for simulating the responses from rigid materials
(e.g., concrete), for which any latency is problematic.
In parallel, a collaborative effort with Bruno Giordano, Stephen
McAdams, Vincent Hayward, and Hsin-Yun Yao investigated the role of
haptic, proprioceptive and auditory information in the non-visual
identification of walking grounds. This led to some interesting
findings relevant to further development of our active floor,
suggesting that haptic information alone was more helpful than
mulitmodal information (including auditory) for identification of
aggregate materials (e.g., gravel). [1]
|
Each tile includes:
We plan to put the first large version of the floor (6x6 grid covering 6'x6')in a CAVE-like environment at the lab. The CAVE is three walls illuminated by three projectors that surrounds the walking area and presents an interesting existing VR setup from which to build upon. The Vicon system will also be used in addition to the floor sensors for better accuracy in foot tracking. This setup will consist of 36 tiles.
When a tile is stepped on, the force sensors send the signal to the software which generates vibration patterns to the actuator. The user will be able to feel the snow compact underneath his weight, hear the crunch of snow and also see snow surrounding his foot deform.
In addition to the design and construction of the physical architecture, we must consider the synthesis of audio, haptic, and graphics effects. The goal is to generate a highly convincing, immersive experience for users, who may be engaged in a variety of activities. Technical issues include the integration of audio and haptic media streams, which may involve both modalities producing their effects from a common synthesis model but delivered to separate output devices.
Our current prototype uses a simple "crumpling" model for sound and haptic synthesis (Max/MSP). However, this can be refined considerably, ideally based on a physics or simulation API, used to implement an interacting particle or rigid body model that is capable of driving a synthesis algorithm. Parameter design for the synthesis algorithm will likely be informed by the recorded acoustic (and possibly haptic) signals from experiments. Relevant parameters include those related to modal synthesis (frequencies, bandwidths) and stochastic event densities.
The graphical deformation effects are rendered using a deformable height-field implemented on GPU shaders(work done by Benjamin Peck). This allows for real-time yet realistic terrain effects. Position and orientation are captured using VICON motion tracking and is then translated to a virtual foot model in a JAVA OpenGL framework. More materials in addition to snow are currently being considered.
Proposed research, in coordination with a European Union FET Call, will investigate the use of the relevant sensed information at higher levels of abstraction. This relates to the acquisition and analysis of sensor data from walking-based interactions, the computational integration of available sensed information sources, and the development of data-driven methods for controlling multimodal feedback synthesizers in response to sensed data.
Sensing options include the current floor components or possibly, integrating sensors within shoes, in addition to complementary sources. Motion capture will be used to provide ground truth movement profiles in a subset of trials.
Construction of the large floor has largely been completed. A more detailed description of completed work can be found in Visell et al. [4]
Last update: 24 Oct 2008