InteractivePrototype-Group:MELODY
From CS160 User Interfaces Fa06
Contents |
Team Members and Who Did What
- Roland Carlos – Programming the paper interface.
- Nankun Huang – Programming the playback software interface.
- Allen Yang – Presentation slides, the report.
- Julius Cheng – Presentation slides, the report.
All members of the team contributed to collective brainstorming, trying to set up the Anoto system, and to each other’s assigned portion of the work.
Problem and solution overview (1 paragraph)
Music composers are forced to select one of two ways of composing music: either with digital editing tools or on standard sheet paper. With editing software, one has access to powerful features, but lack working with the organic feel and portability of paper. Paper allows the composer to play an instrument and write on paper at the same time without much hassle and work anywhere, but he or she might spend more time than is necessary without all the features of editing software.
The MELODY system bridges the gap between paper and software by allowing music composers to convert their handwritten sheet music into a digital format, editing on the computer, and then printing it back out for handwritten editing, and so on. The system attempts to combine the two mediums to allow composers to have access to the benefits and avoid the inconveniences of both.
The current version of our high-fidelity prototype supports some basic stroke recognition on our paper interface, allowing the user to draw basic notation and make deletions. The playback software can list and organize songs.
Tasks
The tasks we selected early on were essential to showcasing our proposed vision for the MELODY system, so they remain essentially the same as before.
Composing a piece of music on the sheet music paper (Hard)
The user should be able to create a composition on the paper with a small set of notation symbols, and be able to upload this composition to the computer at any time. Composing is matter of titling and optionally tagging a printout of our sheet music paper, writing in musical notation on the music staff regions, selecting different instruments and joining lines as necessary. Until we add support for batch data, the strokes are streamed onto the computer.
Deleting existing music (Medium)
The user should be able to delete existing portions of music. Deleted passages obviously remain on the page due to the limitation of pens, but the software music interpreter should recognize these deletions and omit them from the digital representation of the music. To delete, one makes a stroke on the eraser activation field, marks over what he or she wishes to delete, and marks the eraser deactivation field.
Playing uploaded compositions via the playback software (Easy)
The playback software should have a listing of uploaded compositions. The user should be able to locate a particular piece by its title and/or tags and play the piece in MIDI.
Revised interface design
Please refer to the final section for screenshots. The following section refers to Figure 1, the high-fidelity paper interface, Figure 2, the high-fidelity playback software, and Figure 3, the old low-fidelity interfaces.
Since our low-fidelity paper prototype, we have made drastic functional changes, though the overall feel and look of it remains similar.
Among the major additions, we have created a new region to link lines together. This region is the long box on the left of the five-line bars. In order to indicate that two or more lines should be played simultaneously, the user need only draw a bracket over the desired lines. This addition was required because previously, the lines would always be played sequentially, yet we provided a way to select the instrument for each line. It made little sense to have one single voice for the whole composition that could change instruments with each line, so we were left with two choices: either have all the lines still be played individually and sequentially and only have one instrument selection box for the entire composition, or allow for multiple lines to be played together and for the user to be able to select an instrument for each of the voices. While the first one would have been significantly easier to implement, the second one provided a very much needed dimension of complexity to the compositions that this system could support, so we opted to implement this new box region.
Another change is the manner with which the user deletes passages of music. In the previous interface, there was no method of deletion readily obvious to the user on the interface itself, and users had to draw an ‘X’ over sections they wanted to delete. We decided to devise a new method of deletion due to the fact that users couldn’t figure out how to do this without seeking help, and that drawing a well-formed ‘X’ that precisely covers its targets was somewhat of stringent requirement. In our new high-fidelity interface, there are two boxes on the bottom of the page, one that activates “deleting” mode, and one that deactivates it. They are clearly marked so as to be readily obvious to the user. The rationale behind having two regions instead of one was to avoid implementing a mode that the user has no way of receiving feedback about.
Finally, we eliminated the cell phone preview button. Based on user testing, the feature was not that useful and in our collective opinion, not integral to the system. It seemed more like a bonus feature that did not enhance the overall vision of MELODY, so we simply omitted it from the high-fidelity prototype.
Sketches and scripts for unimplemented portions
Refer to Fig. 4 at the end of this document.
Storyboard of tasks
Refer to Fig. 5 at the end of this document.
Prototype Overview
The current version of our high-fidelity prototype implements all of the appearance of the planned system, but lacks full functionality due to the current limitations of the R3 toolkit. Despite the incompleteness of this prototype, most of the functions of the planned system are modeled, and the ones that are not can be expressed through “wizard-of-oz” techniques.
Figure 1 is a direct printout of the paper interface created via the R3 toolkit. Cosmetically, it is fairly similar to our previous low-fidelity interface, but now active regions are obvious from the dotted Anoto pattern. The top two regions, labeled “Title” and “Tags,” are OCR-active areas to translate handwritten information into digital text upon uploading the sheet to the playback software. “Title” is self-evidently the title of the composition, and “Tags” are keywords associated with the piece to categorize it and make it easier to find in the playback software.
The eight bars that occupy the largest amount of space on the paper are where music notation is written and saved by the pen. Each bar contains the standard five lines of a musical staff. The setup of these regions mimics the standard setup of the barebones minimalism of standard paper sheet music.
On the left side of each individual bar there is a square box. These boxes are OCR-active, and allow the user to write a single-letter code for a particular musical instrument. When uploaded and converted into MIDI, the line will be played in that instrument. The default instrument is the piano, and a line with no musical instrument selected will default to piano.
The long box in-between the instrument selector boxes and the musical staffs allows the user to indicate that two or more bars should be played simultaneously. To use this, one need only to draw a bracket that spans the bars he or she wishes to combine. Since the bars are adequately spaced apart, there is no chance of ambiguity when interpreting the user’s markings. Upon uploading, joined lines will be treated as part of the same musical passage and played simultaneously.
The conjunctive use of the instrument selectors and the staff combiner allows users to create multi-layered music, a feature that we intended to support from the beginning, but failed to implement successfully in the low-fidelity prototype. Now, one can simply join two or more lines together by drawing a bracket on the staff combiner, and indicate instruments for each line.
Finally, the bottom of the page contains two regions: an erasing mode activator and deactivator. Whereas, as previously mentioned, erasing was formerly done by drawing an X over desired notes or passages, and this proved to not be obvious or natural-feeling to users. Now, erasing is not done by a special gesture, but by activating the erasing mode by making a stroke on the erasing mode activating region. The user can erase any note or other notation by making any mark over it while erasing is activated. Based on previous testing, most composers scribbled over unwanted notes or passages, so we implemented a way to delete music that allows scribbling. The user can turn off erasing mode simply by making a stroke on the erase mode deactivator, just to the right of the activator.
The other part of the interface, the playback software, is remains the same as the low-fidelity version. On the right side of the screen is a listing of all the songs that have been uploaded, along with clickable title, tags, and date fields that allow the user to specify how to order the selections. The left side of the screen contains a listing of all the tags that have been specified on uploaded compositions. The user can narrow down the song listing by click on one of the tags. The top part of the screen contains simple “stereo” controls, allowing the user to play, pause, jump to a specific part of a song by clicking on the slider, and adjust the volume. Finally, there is a help button in the playback area, colored blue to let the user find it more easily.
What Was Left Out
Printing out and making edits on a previously uploaded composition is an important component of the overall MELODY concept, but it was one that we omitted from this current prototype. The rationale behind this is that the user task between editing a handwritten composition and a reprinted composition is essentially the same. In either case, deleting passages of music is the only editing that one can do, and is done exactly the same way. While the user task would have beeen the same, implementing it would have been tremendously difficult, due to incompleteness of the R3 toolkit and our lack of heavy experience using it. Our current idea is that each printed note is an active region in itself, so that any erase marks on one would immediately be recognized. This idea is difficult to implement, and also needs to be further examined, in case there is a simpler solution.
Nonetheless, the ability to print out work and re-edit it is vital to the life cycle of a musical composition that we proposed early on. Without this, the ability to handwrite music is essentially lost after uploading. Future versions of the interface will certainly support printing out and re-editing.
Another feature left out of the current prototype is any part of the interface that requires optical character recognition, which includes the Title and Tag fields, and the instrument selector. The R3 Toolkit currently does not support OCR, and implementing it ourselves at this time would have been both unfeasible and unnecessary. Note recognition on the music staffs would have been even more difficult, as it is a system we would have to program ourselves. Being that these critical portions of the interface have not been fully implemented, we have created workarounds and “Wizard-of-Oz” techniques that make this prototype presentable.
“Wizard-of-Oz” Techniques Needed to Make the Prototype Work
The following techniques can be viewed as Fig. 4 in the next section.
As stated before, none of the English letter or musical notation recognition has been implemented. Thus, instead of converting a composition to a MIDI format, an extremely difficult task, we simply display the image that the user has drawn on the regions onto the Java interface upon uploading. The Title and Tag fields, for which OCR has not been activated, are filled in with the image of what the user has written there. The notes, brackets on the line combiner field, and letters drawn in the instrument selectors are similarly uploaded.
The only Anoto-activated regions that do not directly display the image drawn on paper are the eraser fields. When the user marks on the eraser activation field, the pen will then draw in red instead of black on any of the Anoto-active fields, indicating eraser marks that are not meant to be recognized as musical notation or writing by the software. For now, we accept pen strokes but only show them as images to showcase the Anoto-active functionality of our sheet music paper interface. Future versions need only to interpret these pen strokes that we are already able to capture.
Any uploaded musical compositions are thus not actually converted to MIDI in this prototype, so in order to make the playback software work, we have to “fake it.” Our playback software connects to our own online database with a listing of made-up compositions in order to emulate the task of finding a particular composition and playing it.
These techniques allow us to show the essential features of the MELODY system in this high-fidelity interface without spending a lot of time coding something that may well be changed in the future. Our current system affords a great amount of flexibility and scalability; any future work need only build upon the barebones interface we already have, and having to make major changes will not result in much wasted time and effort.
Prototype screenshots or scripts (as many as needed)
Fig. 1: Hi-fi paper interface |
Fig. 2: Hi-fi playback software |
Fig. 3: Old lo-fi interfaces |
Fig. 4: Unimplemented parts of the hi-fi interfaces |
Fig. 5: Storyboard of task 1 |
Fig. 6: Storyboard of task 2 |
Fig. 7: Storyboard of task 3 |
