Proton: Multitouch Gestures as Regular Expressions
Kenrick Kin, Björn Hartmann, Tony DeRose, Maneesh Agrawala
Abstract
Current multitouch frameworks require application developers to write recognition code for custom gestures; this code is split across multiple event-handling callbacks. As the number of custom gestures grows it becomes increasingly difficult to 1) know if new gestures will conflict with existing gestures, and 2) know how to extend existing code to reliably recognize the complete gesture set. Proton is a novel framework that addresses both of these problems. Using Proton, the application developer declaratively specifies each gesture as a regular expression over a stream of touch events. Proton statically analyzes the set of gestures to report conflicts, and it automatically creates gesture recognizers for the entire set. To simplify the creation of complex multitouch gestures, Proton introduces gesture tablature, a graphical notation that concisely describes the sequencing of multiple interleaved touch actions over time. Proton contributes a graphical editor for authoring tablatures and automatically compiles tablatures into regular expressions. We present the architecture and implementation of Proton, along with three proof-of-concept applications. These applications demonstrate the expressiveness of the framework and show how Proton simplifies gesture definition and conflict resolution.
Proton represents a gesture as a regular expression describing a sequence of touch events. Using Proton’s gesture tablature, developers can design a multitouch gesture graphically by arranging touch sequences on horizontal tracks. Proton converts the tablature into a regular expression. When Proton matches the expression with the touch event stream, it invokes callbacks associated with the expression.