Jackknife is a general purpose gesture recognizer for C++, C#, and JavaScript that is designed to
work with a variety of input devices including Kinect, WiiMote, Leap Motion, as well as touch and
stylus devices. Our recognizer can even be used to detect hand gestures passing through inaudible
sound waves due to the doppler effect. If the input can be represented as a sequence of points
through time, there is a good chance that our recognizer can be used to detect input patterns with
high accuracy. Jackknife is also designed for gesture customization. This means that a user or
developer only needs to provide one or two examples of a gesture pattern to work, whereas other
recognizers require lots of training data, which may be quite difficult to collect.
Reference Source: Jackknife Repository
JK2017 Dataset: Jackknife Repository
Paper: PDF
Some Technical Details
Jackknife is a collection of dynamic time warping (DTW) based techniques tailored for gesture customization. That is, Jackknife is designed to work well with only one or two samples per gesture class for many different input devices.
As an overview of the underlying methods and process, we first resample an input gesture to a fixed number of points, similar to $1 [1]. We then measure the similarity of the given input to each gesture class template using DTW, where the local cost function measures the inner product of corresponding gesture path direction vectors, per Penny Pincher [2]. Motivated by the complexity-invariant distance (CID) [3], we further augment the DTW score with a set of new correction factors designed specifically for gestures. Finally, the input sample is said to belong to the gesture class whose template yields the best score. When dealing with continuous data, we also find an appropriate rejection criteria by selecting the threshold which maximizes the F1 score of synthetically generated distributions. Negative samples are generated by splicing together template gestures and positive samples are generated with gesture path stochastic resampling (GPSR) [4].
Further details, including information on various optimizations, can be found in the Jackknife paper.
- [1] Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. "Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes." Proceedings of the 20th annual ACM symposium on User interface software and technology. ACM, 2007.
- [2] Eugene M. Taranta II, Andrés N. Vargas, and Joseph J. LaViola Jr. "Streamlined and accurate gesture recognition with Penny Pincher." Computers & Graphics 55 (2016): 130-142.
- [3] Gustavo E.A.P.A. Batista, Xiaoyue Wang, and Eamonn J. Keogh. "A complexity-invariant distance measure for time series." Proceedings of the 2011 SIAM International Conference on Data Mining. Society for Industrial and Applied Mathematics, 2011.
- [4] Eugene M. Taranta II, Mehran Maghoumi, Corey R. Pittman, and Joseph J. LaViola Jr. "A Rapid Prototyping Approach to Synthetic Data Generation For Improved 2D Gesture Recognition." Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 2016.