‍ Overview Tutorial Documentation FAQ Download

Tutorial

Getting Started
The Touch Processor
Converting Touch Event to Touch Event Symbol
Creating a New Touch Event Symbol
Creating and Adding a Gesture
Callbacks
Confidence Calculators
Splitting the Touch Stream
Custom Split Stream Attribute Generator

Getting Started

Download the project. It contains both the Proton source code and an example application. On a Mac, build and run the XCode project. On a Windows machine, run the Makefile in Cygwin.

In order to build and run multitouch gestures, you first need a multitouch device. If you have a smartphone, a quick way to get started is to install TUIOpad for iOS or TUIOdroid for Android. The provided sample applications (hitTester and streamSplitter) process TUIO events, so other devices that send TUIO events should work as well.

To connect the multitouch device to the application, in TUIOpad (or TUIOdroid) enter the network address of the device the application is running on. Use port 3333; the sample application is listening to this port. Then start one of the sample applications and TUIOpad. The application should receive the touch events entered on the TUIO device. The application converts the TUIO touch events into Proton touch event symbols for matching.

The streamSplitter sample application uses GLUT and has stream splitting enabled. Starting a stroke on the left side of the screen manipulates the red square, while starting a stroke on the right side of the screen manipulates the blue square.

The Touch Processor

The core of Proton is the PtTouchProcessor. It processes touch event symbols converted from hardware events and compares the touch event stream(s) with the developer-defined gesture set(s). The PtTouchProcessor manages one or more instances of a PtGestureMatcher, which maintains a gesture set. By default there is only one PtGestureMatcher, which the developer never directly interacts with. Instead the developer passes new gestures and attribute generators to the PtTouchProcessor, which assigns them to the PtGestureMatcher. When stream splitting is enabled, the PtTouchProcessor maintains multiple PtGestureMatchers, one for each stream. The PtTouchProcessor sends the touch event symbols to the appropriate PtGestureMatcher for matching, as specified by a custom PtSplitStreamAttributeGenerator.

Converting Touch Event to Touch Event Symbol

For a given device developers should implement touch event handlers to generate the appropriate touch event symbol. The following example is taken from PtTuioListener (PtTuioListener.{h,cpp}). It converts a TUIO touch-down event into a Proton touch-down event symbol, which gets passed to a PtTouchProcessor. The developer must specify the action of the touch: "D" (down), "M" (move), or "U" (up) and append the ID of the touch.

PtTuioTouchSymbol subclasses PtTouchSymbol to store additional TUIO touch data.

void PtTuioListener::addTuioCursor(TUIO::TuioCursor *tcur)
{
    // create touch down ("D") and touch ID string
    char buff[20];
    sprintf(buff, "D%d", tcur->getCursorID());
    std::string buffAsStdStr = buff;
    
    PtTuioTouchSymbol *touchSymbol = new PtTuioTouchSymbol(buffAsStdStr);
    touchSymbol->setTuioCursor(tcur);
    
    PtTouchSymbols *expression = new PtTouchSymbols();
    expression->push_back(touchSymbol);

    // _touchProcessor has responsibility for deleting expression
    _touchProcessor->processTouchSymbols(expression);
}

Creating a New Touch Event Symbol

A touch-down event symbol with touch ID 0 is defined as:

PtTouchSymbol *symbol = new PtTouchSymbol(std::string("D0"));

Any attribute values must come after a colon and are further delimited by colons, e.g., "D0:a1:a2" A touch down symbol with touch ID 0 and first attribute value "KK" is defined as:

PtTouchSymbol *symbol = new PtTouchSymbol(std::string("D0:KK"));

A gesture regular expression is represented by a PtTouchSymbols object, which is simply a std::vector of PtTouchSymbol elements.

Creating and Adding a Gesture

To create a regular expression, the developer can write the entire expression as a std::string, where each symbol is delimited by a developer-defined delimiter, such as "_" and call convertStringToTouchSymbolRegularExpression().

Creating a one-touch gesture with attribute value "KK" is as follows:

PtTouchSymbols regexp;
// fills regexp with regular expression delimited by "_"
convertStringToTouchSymbolRegularExpression(std::string("D0:KK_(M0:KK_)*U0:KK_"), std::string("_"), regexp);

The developer can associate triggers to callbacks by associating the index of the callback to a specific PtTouchSymbol (numbered starting from 0 and ignoring regular expression syntax characters) in the regular expression.

// add trigger to callback at index 0 of gesture's callback list to touch symbol at index 1 (i.e., M0:KK)
addTriggerToIthTouchSymbol(&regexp, 0, 1);

A new PtGesture is initialized by passing in a PtTouchSymbols regular expression. Callbacks can then be added to the gesture.

PtGesture *gesture1 = new PtGesture(regexp);
gesture1->setName(std::string("ONE_TOUCH"));
// The callback "firstCallback" is the callback at index 0
gesture1->addCallback(firstCallback);    

Gestures are then added to a PtTouchProcessor which matches each gesture against the touch event stream.

PtTouchProcessor touchProcessor;
touchProcessor.addGesture(gesture1);

Optionally, the developer can subclass PtAttributeGenerator to implement custom attributes and pass them to the touch processor, which adds them to the gesture matcher.

HitTestAttributeGenerator *hitTestAttributeGenerator = new HitTestAttributeGenerator();
touchProcessor.addAttributeGenerator(hitTestAttributeGenerator);

Callbacks

To execute code when a gesture matches requires implementing callbacks that take the entire touch event stream as input. The argument touchStream is a std::vector with PtTouchSymbols elements. Thus, each element of touchStream is a std::vector with PtTouchSymbol elements.

The following callback is bound to a one-touch gesture and prints the x-y coordinates of touch. First it gets the PtTouchSymbols at the end of the touchStream vector. Since it is a one-touch gesture, the PtTouchSymbols contains only a single PtTouchSymbol. The PtTouchSymbol is cast to a PtTuioTouchSymbol to access the TUIO cursor data.

void printCoordinateCallback(const std::vector &touchStream)
{
    printf("PRINT COORDINATE CALLBACK!\n");
    // gets last PtTouchSymbols element in touchStream
    PtTouchSymbols *lastFrame = touchStream[touchStream.size()-1];
    // casts the lone PtTouchSymbol to PtTuioTouchSymbol
    PtTuioTouchSymbol *tS = static_cast((*lastFrame)[lastFrame->size()-1]);
    // gets the coordinates of the TUIO cursor and prints it
    double x = tS->tuioCursor()->getX();
    double y = tS->tuioCursor()->getY();
    printf("%f %f\n", x, y);
}

Confidence Calculators

Multiple gestures might match the same touch event stream, so the developer can write confidence calculators that rate the touch event stream on a scale of 0 to 1, where 1 is most confidence that stream represents the gesture and 0 is least confidence. The callback with the highest confidence score will get executed.

By default confidence calculators are not used and all callbacks are executed. To invoke the use of confidence calculators, call PtTouchProcessor's setExecuteBestGesture() method:

touchProcessor.setExecuteBestGesture(true);

A confidence calculator can analyze the entire touch event stream. This dummy example returns a flat score of 0.5.

double confidenceCalculator(const std::vector &touchStream)
{
    return 0.5;
}

A confidence calculator is associated with a callback when the callback is added to the gesture:

gesture->addCallback(callback, confidenceCalculator);

If no confidence calculator is included, then a score of 1.0 will be provided by default.

Splitting the Touch Stream

By default a PtTouchProcessor manages a single touch event stream, with a single PtGestureMatcher. The developer can create a touch processor with multiple gesture matchers that operate on different streams.

The following example is found in the streamSplitter sample application.

// creates a PtTouchProcessor with two gesture matchers
PtTouchProcessor touchProcessor(2);
// first matcher gets gesture1 and gesture2
touchProcessor.addGesture(gesture1, 0);
touchProcessor.addGesture(gesture2, 0);
// second matcher gets gesture3
touchProcessor.addGesture(gesture3, 1);

A stream is split by attribute values generated by a subclass (e.g., LeftRightSplitStreamAttributeGenerator) of PtSplitStreamAttributeGenerator.

// adds a split stream attribute generator to split the stream by left and right halves of the screen
touchProcessor.setSplitStreamAttributeGenerator(new LeftRightSplitStreamAttributeGenerator());

Custom Split Stream Attribute Generator

A custom split stream attribute subclasses a PtSplitStreamAttributeGenerator. Much like a custom attribute generator, a custom split stream attribute generator processes a PtTouchSymbol and outputs an attribute value as a std::string via the attributeValue() method. However, there are two key differences. First, a custom split stream attribute generator only processes touch-down event symbols and has no additional information about the touch stream. Second, a custom split stream attribute generator must also implement a gestureMatcherForAttributeValue() method that returns the mapping between attribute value (a std::string) and the index of the gesture matcher (an int). It is a bit roundabout, I know.

The following is the LeftRightSplitStreamAttributeGenerator used in the streamSplitter sample application.

class LeftRightSplitStreamAttributeGenerator : public PtSplitStreamAttributeGenerator {
    virtual std::string attributeValue(const PtTouchSymbol &touchSymbol) const {
        const PtTuioTouchSymbol *tS = static_cast<const PtTuioTouchSymbol*>(&touchSymbol);
        if(tS->tuioCursor()->getX() < 0.5)
            return std::string("L");
        else
            return std::string("R");
    }
    
    virtual int gestureMatcherForAttributeValue(std::string attributeValue) const {
    	// left ("L") value goes to the gesture matcher at index 0
        if(attributeValue.compare(std::string("L")) == 0)
            return 0;
    	// right ("R") value goes to the gesture matcher at index 1            
        else
            return 1;
    }
};