ICE

We constructed an acoustic, gesture-based recognition system called Multiwave, which leverages the Doppler Effect to translate multidimensional movements into user interface commands. Our system only requires the use of two speakers and a microphone to be operational. Since these components are already built in to most end user systems, our design makes gesture-based input more accessible to a wider range of end users. By generating a known high frequency tone from multiple speakers and detecting movement using changes in the sound waves, we are able to calculate a Euclidean representation of hand velocity that is then used for more natural gesture recognition and thus, more meaningful interaction mappings.

We present the results of a user study of Multiwave to evaluate recognition rates for different gestures and report accuracy rates comparable to or better than the current state of the art. We also report subjective user feedback and some lessons learned from our system that provide additional insight for future applications of multidimensional gesture recognition.

Publications

• Pittman, C., Wisniewski, P., Brooks, C., and LaViola, J. "Multiwave: Doppler Effect Based Gesture Recognition in Multiple Dimensions", Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). 1729-1736, May 2016.

Predictive tracking algorithms are in limited use in today's virtual reality systems. Choosing the appropriate prediction algorithms is a challenging task since there are many issues to consider when applying them in practical situations. Certain algorithms may perform better or worse depending on the underlying tracking system's sampling rate, noise variance and tracking technology (e.g., magnetic, acoustic, inertial, hybrid). The types of user motion, including head and hand, also play a significant role in determining what prediction algorithms to use. Another critical factor in prediction algorithm determination is the prediction time (i.e., how far one has to predict into the future). Some prediction algorithms may be more or less robust as prediction time increases. Almost all prediction algorithms contain one or more parameters that require tuning to optimize performance. Therefore, a significant aspect in determining what prediction algorithms to use is in adjusting an algorithm's parameter values. The adjustments are nontrivial in the sense that an optimal parameter setting for one type of user motion may not be optimal for another.

To aid VR developers and researchers in understanding and using predictive tracking and motion estimation algorithms in virtual and augmented reality, I have developed a testbed which contains user motion datasets, a variety of predictors, and a testing and analysis application. The first release of the testbed can be downloaded here.

Publications

• Julier, S., and LaViola, J. "On Kalman Filtering with Nonlinear Equality Constraints", IEEE Transactions on Signal Processing, 55(6):2774-2784, June 2007.
• Julier, S. and LaViola, J. "An Empirical Study into the Robustness of Split Covariance Addition (SCA) for Human Motion Tracking", In the Proceedings of the 2004 American Control Conference, IEEE Press, 2190-2195, June 2004. (Note the published version of this paper has some notational errors. This version is correct.)
• LaViola, J. "A Comparison of Unscented and Extended Kalman Filtering for Estimating Quaternion Motion", In the Proceedings of the 2003 American Control Conference, IEEE Press, 2435-2440, June 2003.
• LaViola, J. "A Testbed for Studying and Choosing Predictive Tracking Algorithms in Virtual Environments", In the Proceedings of Immersive Projection Technology and Virtual Environments 2003, ACM Press, 189-198, May 2003.
• LaViola, J. "Double Exponential Smoothing: An Alternative to Kalman Filter-Based Predictive Tracking", In the Proceedings of Immersive Projection Technology and Virtual Environments 2003, ACM Press, 199-206, May 2003.
• LaViola, J. "An Experiment Comparing Double Exponential Smoothing and Kalman Filter-Based Predictive Tracking Algorithms", In the Proceedings of Virtual Reality 2003, IEEE Press, 283-284, March 2003.

Downloads

• User Motion Data Repository
• Predictor Library and Testing Application