GroupBrainstorm-Group:Melody
From CS160 User Interfaces Fa06
Contents |
Team Members
- Roland Carlos
- Julius Cheng
- Nankun Huang
- Cheng-Lun Yang
Selected Idea
During our initial brainstorm session in class, each of us spoke about our individual project proposals. Within our group, we had a proposal for a Japanese character recongition system which would double as a learning tool, a proposal for a pen with dictionary, thesaurus and grammar features, a proposal for a note-based classical composition assistant, and a proposal for a music editing and mixing assistant.
We then mapped out our four ideas and tried to see what connections existed between the four of them. Besides the obvious musical connection between the latter two, there were some connections between the Japanese and dictionary projects, namely in terms of character recongition and learning assistance. The Japanese project and the music projects also shared commonality in that they had to recongize non-English characters.
In the end, we decided to take a group vote to see which project the group would want to work on the most for the semester. The end result was a tie between the two musical projects, so we decided to take the underlying ideas from the two projects and merge them into one musical project for our group. Afterwards, we began to brainstorm the functionalities we wanted for our music project.
Target User Group
Despite the advanced technology, many composers still prefer to compose their music on traditional five line score. With Melody, we can make the music composers’ jobs easier. They can compose their music on five line score and have the notes transferred to music software. For the lyrics writers, they can write the lyrics of the song on the five line score underneath the notes.
Melody then will scan the words into the software as they appear on the paper. The third group is the music producers who might want to edit, mix, and synthesize the music. With Melody, the producers can use pen and paper to choose what features they want to perform on the piece of music. They can also choose what format they want to store the music in to accommodate different needs.
Brainstorming Ideas
Annotation
The user can make specialized annotation on the music sheet. These annotations are written in specialized boxes at the bottom of the music sheet. The result of these annotation are turned into tags which are then used to organize the music sheet. Each sheet of music can have up to 5 different tags.
Artificial Singing
Once the music is composed, a background music can be chosen to play along with the main tune.
Beat Practice
This feature allows the composer to get a feeling for the beat of the music. He must tap his pen on a special region of the anoto-enabled paper at a specific rhythm. The result of this tapping is fed to his/her cell phone which feedbacks whether the tapping is too slow or too fast.
Beat Recognizer
After the music is composed, the software backend has a component which is able to recognize the fundamental bass beats in a music ( via Fourier analysis or some other audio analysis tool ). This information can then be used by the artist to extend the composition such as adding chords and additional layers of music on a separate piece of MELODY music sheet.
Chords
While composing music, the composer can draw a vertical line down the sheet of music to join 2 vertically stacked 5 lined composition area. When these 2 composition area are joined up, they will represent a chord, which are different notes (from perhaps different instruments) that gets played along during the same time rather than different part of the song.
Computerized note input
This is fundamental to our project. The notes that are written onto the composition area are recognized automatically for which pitch they are through some form of stroke analysis. This information is then stored in the backend computer
Cut and paste editing (Physical)
This allows users to cut (or even tear) and pastes their anoto-enabled music sheet physically with a scissors or knife. These fragmented music parts can be then “glued” together through swiping the anotopen through a special area in he music sheet. The user can easily piece a new music together using components of existing music sheets.
Different musical instruments
When the user selects to preview his composition via the cell phone, the user has the option to mark down which instrument does he want to be used to play his composition.
Digital/Paper keyboard
This is a set of piano keys which is printed in ink at the bottom of a specialized composition sheet. When the user use the anoto pen to tap at specific keys areas on the anoto-enabled paper, the cell phone will give an immediate feedback of the tone that the composer tapped on.
- Instrument selection for digital keyboard
This is the instrument selection box for the paper keyboard described above. This will control what instrument plays the feedback sound to the user.
Digital drumset
This is similar to the digital keyboard but it works better for the bass and percussion notes. While the drum can be made to be a separate mapped instrument to the digital keyboard, its often more intuitive to the user by printing a mini drum-set on the anoto-enabled paper.
Editing/Deletion
This will allow the user to use a cross-out mark on the anoto-enabled composition sheet to cross out parts of the composition which is no longer wanted.
Editing existing score
This feature will print out existing score from the database into a score sheet. The composer can then add/edit/delete through the pen interface to make a new music.
Follow along pen preview
This will allow the artist to preview the music by “scrubbing” through the paper-based score on a specialize area in the music sheet. The preview sound will play the specific part of the music which the user is “scrubbing” through with his/her pen
Gestures
Gestures are specific patterns the user can draw on a reserved area of the page; these patterns are then recognized and used to perform specific actions.
- Gesture: Quick save
The user can have the pen transmit its information to the cellphone and save the entire composition up to a point with a "quick save" gesture.
- Gesture: Quick preview
The user can transmit information to the user’s cell phone, telling it to play the preview music up to that point in time with a "quick preview" gesture.
- Gesture: Quick delete
The user can cause all information about the current composition to be discarded and the remaining area on the sheet of paper can be used to start a new composition with a "quick delete" gesture.
Karaoke output
This feature is useful for songs (such as pop songs ) which contains a lot of vocals. This idea takes in the lyrics and the music and plays back the song, together with the lyrics synchronized across the screen like in a karaoke. This will allow the singer to easily get the “flow” of the music and sing the song easily
Levels control with curves
This idea is similar to sound editing package’s levels control. There will be a special region aligned vertically with the score sheet which the user can draw a curve with the anoto pen. The information from this curve can then be used to control the composed music in many ways
- Levels Control: Volume
The volume of the music can be controlled using the curve
- Levels Control: Fade
This mimics the “fade in” and “fade out” effect of video editing software. This can be used to link up 2 different compositions so there’s gapless playback. The curve can control the degree of fading.
- Levels Control: Channel Strength
This controls the strength of one music channel (left or right, front or back) relative to the other, the usage of this allows the simulation of 3D positional sound.
- Levels Control: Effects
Sound effects such as reverb and echo often have an “intensity” parameter, This parameter can be mapped to the curve and the curve can control the effect.
Live levels control This is similar to the levels control but instead of a curve, a special area on the MELODY composition sheet similar to a slider is used to control the levels, which can be mapped to any parameter. This is used in conjunction with live previewing. During the preview, the user can swipe the pen across this “live slider” to control the levels. The result of the live-levels control can be recorded to be added on as an music edit later.
Intelligent Loudness control In this feature, it makes use of the natural tendency of composers to write a bigger music note (on the anoto-enabled composition sheet ) when he wants the specific tone to play louder. The size of the notes that are written on the composition sheet are used to control the volume of the music composed.
Lyrics Sync This is extremely useful for songs with heavy vocals. In the MELODY composition sheet, a special area below the main 5-lined composition area is placed where lyrical writers can write their lyrics in synch with the music. By doing this, the lyrical information can be synched up to the music being composed.
Multiple sheet synchronization In the even when multiple instrument/vocals are present in a piece of music, the composer can write the music on multiple pieces of MELODY composition sheets. This feature will introduce a specialized area in the music sheet where the user can make a mark to synchronize the current sheet with another sheet. When this is done, the 2 sheets of music will play back in the same timeline.
Music symbols recognition In typical music composition, there are many dynamic marks such as “F” for loudness. Typical composers are used to writing these directly into the music. The system will be able to recognize these dynamic marks
Phone Preview After the music is composed by the user, he can have it previewed to him/her through the speakers on his/her cellphone.
Pen Equalizer The user can draw a curve on a special area in the music composition sheet to act as an music equalizer to control the tonality and control of the music. The horizontal axis would be frequency and vertical axis would be volume.
Streaming playback This feature will continuously playback the music composed up to that point from the user’s cell phone as its being composed. The user can either choose to have the music loop or just play back once. This will help the composer get into the “flow” of his/her composition
Phone software navigation with pen Since a large part of the backend software such as the preview runs on the user’s cell phone, there can be a “navigation sheet” where a different and more efficient interface, consisting of dials and buttons and sliders printed onto an anoto-enabled paper can be used to navigate the software on the phone.
Non-Destructive editing The file format in which the music composition and lyrics are stored will allow non-destructive editing. This means that after effects, edits are made, the original music composition can still be changed and the previously applied effects will propagate through
Wiggle Rhythm The user can wiggle the Anoto pen over a special area on the printed paper to set the tempo for the song.
Problem Description
This project will try to connect the paper world and digital world of music creation. Through use of the Anoto pen, we will provide digital support for the paper way of writing music. The Anoto system will allow a computer to take the strokes that a musician writes on their paper sheet of music and have it appear on a digital sheet on the computer. Therefore, we give the musician the comfort of paper input but supplement it with the power of digital playback and editing.
Problem Context
For many musicians, paper is the tried and true method for composition. It is the interface that most of them are comfortable with. They have been trained from a young age to read and write from the standard 5-lined musical score. While some musicians have begun to become acquainted with the musical technology provided by computers, paper is still by far the most natural interface for them, and like with the rest of the world, does not show any signs of being completely phased out anytime soon, despite the steady march to a fully digital world.
Our solution will need to be able to "play nice" with the standard 5-line score. We cannot impose some other music writing interface on musicians for our solution (say for example, we make them write out their music in a new format, such as writing out the letters for the notes like E-D-C-D...). So we must adapt our solution to use the benefits of the 5-line interface (such as organization, user familarity, etc.). Therefore there will no need for musicians to learn a new way of writing music in order to use our project.
Many musicians find that ideas spring up to them at unexpected times. Therefore, we can't expect the user to always have the computer up and running when they discover the perfect end fo their piece. The solution will need to be able to pick up the user's strokes at any moment. If this is not possible (say the computer is off), the solution will need to be able to store data on the strokes made so that the information can be uploaded later at the user's convenience.
This project does not look to replace or fundamentally change the way musicians write music. Instead, it looks to help accelerate the process by providing them with helpful functionality. The ability to preview music just after it has been written, transform messy handwritten notes into clean ones, and save digital backups of works that may get lost are just some of the proposed functionality for this project. While obviously computer music generating software already exists, there currently is no viable bridge between the handwritten world and the digital one. This project hopes to start one.
Why the Anoto System is a Good Technology for the Problem
The Anoto pen-and-paper technology is well suited to this task primarily because it is far more portable than digital solutions. It allows artists to record their ideas in a much wider range of settings, such as during travel, or while playing a real instrument concurrently. Moreover, paper is a familiar medium that is easy to navigate and write on.
It is even more advantageous to create an Anoto-based system than a Tablet PC solution because paper is instantly accessible in that it does not need to be turned on or load software. For all efforts that Tablet PC-makers have made towards portability, paper is still far less cumbersome to carry around.
Furthermore, using an Anoto-based system produces both a physical and digital copy of information. Upon uploading written data onto a computer, the user possesses two copies of work in two different mediums, and thus is granted far more security than the composer that relies solely on a paper-based or digital solution.
Solution Sketch
