Contextual Inquiry-Group:Group J for Now
From CS 160 User Interfaces Sp10
Codename "singing-app" provides a set of tools that help singers practice more efficiently alone.
Group Members
- Vinson Chuong conducted interviews, wrote the writeup, and drew interface sketches.
- Angela Juang scheduled interviews, conducted interviews, took interview transcripts, did the "give feedback" storyboard and sketches, and drew interface sketches.
- Owen Lin scheduled interviews,conducted interviews, took interview transcripts, did the "annotate songs" storyboard and sketches, and drew interface sketches.
- Jungmin Yun conducted interviews, did the "play specific notes" storyboard and sketches, and drew interface sketches.
Target Users
User Selection
Our app is targeted primarily at serious singers with formal knowledge of music who want to practice efficiently alone. Users would ideally sing as part of actual musical groups and not just as a casual hobby. Such users would take practicing for improvement seriously and would be the most likely to give real insight into the type of tools and assistance needed to practice more efficiently. As such, we contacted singers from different established musical groups on/around campus. The users we finally selected were from different types of singing groups on campus and possessed various levels of musical knowledge and musical ability.
User JR
JR has sung in many small to large choir groups and has done so since high school. When practicing, JR prefers to practice with a partner so he can get and immediately act on feedback. For practicing alone, JR prefers to use a piano for accompaniment to get reference points. In practicing his singing, JR aims for consistency of pitch rather than accuracy.
User JL
JL has sung in various groups since the 5th grade. More recently, JL has sung in the Cal Jazz Choir and Women's Chorale. During practice, JL prefers to have various forms of reference--accurate recordings (own parts and group parts), piano accompaniment, etc. JL prioritizes technical refinement and accuracy.
User CS
CS has been singing in various choir groups since elementary school. More recently, CS has sung in the Cal Jazz Choir, University Chorus, and Chamber Chorus. During practice, CS prefers to go by ear, checking accuracy based on comparison to piano accompaniment. CS is more technically advanced and is already very efficient at practicing.
User MR
MR has been singing since 8th grade and has received several years of formal training. More recently, MR has sung as a part of the Artists in Resonance cappella. For MR's group, individual practice is emphasized. During practice, MR prioritizes technical refinement and consistency, relying on various reference points with which to compare.
Problem and Solution Overview
We observed that while practicing with a group and practicing alone have essentially the same goals, the processes were different in that in practicing alone, singers lack resources (feedback, live reference points, etc.) that are heavily relied upon during group practice. We hope to bridge the gap between individual and group practice by providing a set of useful tools that help singers practice alone. In particular, we will provide various forms of reference (piano accompaniment, visuals, recordings, reference notes/frequencies) and feedback (both real-time and after singing the whole song or a segment of a song).
Interviews
Overview
In our interviews, we focused mainly on how the users practiced individually, drawing comparisons with how they practiced in their respective groups. Due to scheduling constraints, we were not able to observe group practices, but since our application is oriented towards individual practice, we felt that inquiring about group practice in the context of individual practice provided sufficient insight into the resources that a group provides. We paid specific attention to minute details pertaining to how the users conducted their practice, in particular to how they used specific tools. We found that the simplest functions were often the most useful in that users would often perform them over and over again. Also, in all of our interviews, feedback was hailed as the most important aspect of practice--one that was difficult to produce without having a singing partner.
Singing Practice: The Process
References
At various stages of practice, our users would often refer to a set of references:
- Sheet Music - keeping it on hand, reading it while singing, annotating parts that need more work or parts that need some sort of accommodation during singing.
- Reference Recordings - between actual practice, users would listen to an accurate reference recording over and over again. During practice, users would often loop the specific parts they were having trouble with over and over.
- Live Recordings - A few users would actually listen to group/self recordings, noting which parts were weak and needed improvement. Most of our users avoid this, and when asked about it, told us that most singers don't like hearing their own voices and would rather rely on feedback from others.
- Reference Notes - Users would sometimes play a specific note on the piano constantly. While this did not happen often and there was little discussion about it, we feel that this kind of functionality is particularly important.
We took particular notice of the fact that most of the users assigned little to no importance in self recordings. Even for the few who would listen to themselves, self recordings were always discarded after a singing session.
One of our users noted the usefulness of having different tracks played in different ears and having the ability to "remix" so to speak recordings. This concept may or may not be useful for the actual app, but we will keep this in mind.
Actual Singing
For all of our users, singing was done with either piano or midi-piano accompaniment--with the actual piano being preferred. We found it interesting because it would appear that serious singers learn to play the piano out of necessity, but we're a little wary of this and may look into it further. Users had varying levels of piano proficiency--some would play high-fidelity versions of the song, and others would just play a few reference notes while singing. One even noted that having to play the piano while singing was somewhat distracting. Some users were proficient in editing/generating midis as a supplement for piano accompaniment, and others would just play the midi that was given to them by their instructors. So, we can't assume that our users have experience in editing midi files. Users noted that playing the piano while singing gave a lot of flexibility in changing the speed of the song, repeating segments of the song, and isolating the particular parts that the user was responsible for. We believe that it is for this reason that users practice with a piano instead of with a reference recording.
During individual singing practice, every user emphasized the importance of feedback and noted that it was lacking. Going-by-ear and comparing their own pitch with the piano note being played was pretty much the only possible means of feedback for them. Users rarely ever referred to a reference recording for comparison. They noted that it's important to get feedback while singing because issues that may be encountered won't stay in memory for long. During group practice, our users would get immediate feedback from other singers. The most common form of feedback was fingers pointing up or down to indicate the need for pitch compensation. One of our users noted that there are singers that are tone-deaf but do not know it.
While singing, users referred to sheet music more so on landmarks on the paper rather than the actual notes. In other words, singers often have the song completely memorized before hand and don't actually use the sheet music to dictate for them how to sing. Users seemed to have varying levels of proficiency with regard to reading sheet music while singing. Obviously, the users who were more proficient in playing the piano were more proficient in reading sheet music while playing and singing.
Users would sit down at the piano and sing segments of the song over and over again--with emphasis on the repetition. For some users, the stated primary focus was on consistency more than accuracy; while for others, accuracy was the focus. As discussed above, the primary way for the users to gauge accuracy and consistency was via by-ear comparison with the piano accompaniment. Depending on the singer's ability to hear tones, this method may be dubious. Also, depending on how fast the song is going, by-ear comparisons may not be sufficient, as noted by one of our users. Most of our users noted that while individual practice focuses on the sound produced by an individual singer, it is the sound that the group produces as a whole that matters most. To that effect, during group practice, group consistency is emphasized. Perhaps this could be addressed by having the ability to playback a reference recording with the user's voice overlaid.
Across the board, users usually practice only around half-an-hour to two hours a week due to schedule constraints. However, all of our users are students; so, this may not be representative of how long users in general would practice. The portability of the iPhone would definitely help here.
Suggestions for the App
We specifically asked the users for suggestions in which tools they would find useful to have on an iPhone. Answers were very consistent across the board and really helped us to narrow-down our feature set.
- Playback Control - Speed of playback, volume of tracks, ability to repeat sections (emphasized).
- When asked how they usually communicated which sections to repeat: time-stamp, measure number, landmarks on sheet music.
- Visual Representation of Music - Scrolling sheet music or frequency graph, lyrics, ability to annotate and display notes.
- Audio Representation of Music - Options for playback of a song as a simple midi-piano or as the full-on reference recording. Options for playing the parts of multiple people.
- Feedback - Live feedback (display notes sung alongside the target notes), after-song analysis and ability to see sung notes with target notes all at once.
- Metronome during playback
- Ability to play specific reference notes
- Ability to import and manage midi files.
We found that the functionality that users were looking for was quite narrow--this list represents all of the suggestions from all of the users.
Task Analysis Questions
Who is going to use the system?
Serious signers with formal knowledge of music who want to practice efficiently and effectively alone. Such users would take practicing for improvement seriously and would appreciate having a set of tools that bridge the gap between group practice and individual practice.
What tasks do they perform?
- Review reference material
- Play (in part or in whole) a reference recording over and over.
- If the reference has multiple tracks, change how the tracks are played (speed, per-speaker volume) to emphasize certain parts.
- Read sheet music and annotate with notes about what parts need improvement and what parts require attention during singing.
- Listen to live recordings to determine which parts need work.
- Listen to reference notes so that the user can accurately copy them.
- Play (in part or in whole) a reference recording over and over.
- Practice singing with piano/midi-piano accompaniment.
- Adjust speed of playing and singing.
- Practice and repeat specific segments of the song.
- Edit and manage midi files using known software package.
- Get feedback on singing accuracy and consistency.
- Get the feedback from a partner.
- Get the feedback by-ear via comparison with piano accompaniment.
- Get the feedback from comparing a self-recording with a reference recording.
Further descriptions of these tasks can be found in the "Interviews" section.
What tasks are desired?
- Get reliable real-time feedback in the absence of a partner.
- Have the ability to analyze performance after singing (in part or in whole) a song.
- Have flexible (customizable speed, etc.) accompaniment without needing to produce it by self.
How are tasks learned?
The interface will be descriptive and will attempt to emulate the process and tools with which users are already familiar with. Basically, we will try to eliminate the learning curve by reusing already established interface metaphors that our target user group is likely to have experience with.
How are the tasks performed?
As for the tasks that users already perform, refer to "What tasks do they perform?" and "What other tools does the user have?". As for how they perform tasks using our apps, refer to the "Interface Design" section.
Basically, our app provides an interface in which users practice with digital accompaniment. The song (midi-piano or reference recordings) can be flexibly played with options to change track volumes and playback speed. Scrolling sheet music is shown, along with lyrics to serve as a visual reference. The song can be annotated digitally during practice and the notes can be displayed during practice. This is meant to displace the piano and partially displace hard-copy sheet music.
Feedback is provided both in real-time and after signing (in whole or in part) a song. The real-time feedback will be shown alongside the reference visuals in order to let the user see how he is doing as he sings. Afterward, the user can review his progress and performance.
We will have the option to record tracks and import midis. Due to the form factor of the iPhone, we will not include the ability to edit midis.
As for playing back reference materials, we will be facilitate this using the typical media player design, which is how users already do it.
What is the relationship between user and data?
Users want to store a collection of songs, each of which is a collection of audio tracks, midis, lyrics, and notes. All of this data must persist. Data that is acquired during a practice session--live feedback, post-song feedback, self-recording--are only useful during the practice session and are usually discarded. An option will be provided to save self-recordings if desired.
What other tools does the user have?
- Piano - for flexible accompaniment and as a reliable and accurate reference.
- Computer - to edit/generate/play midi files.
- Metronome - as a reference point for rhythm.
How do users communicate with each other?
Users do not communicate with each other during individual practice. During group practice, however, the features of this app are more usefully displaced by the presence and attention of others.
How often are the tasks performed?
Globally, users practice around half-an-hour to two hours per week on average with half-an-hour to an hour per sitting. The playback of reference materials can vary in frequency and time because users can listen while doing other tasks.
What are the time constraints on the tasks?
Aside from scheduling constraints, users practice alone on their leisure for as long as they desire. Depending on the capacity of the device, the length of song and the amount of data the can be stored may be limited, but this is highly unlikely.
What happens when things go wrong?
Should the user accidentally delete valuable data, we will remove the data from visible menus and permanently delete after some time interval, allowing users to recover the data. Any changes to options will be easily undone, and no option will be so transparent as to be invisible when the user uses the application.
Basically, when things go wrong, we allow the user to undo.
Analysis of Tasks
Task 1 (easy)
Create a new "song", import tracks, and playback tracks (to verify correctness).
Before a user can use our app to practice, he must go through a certain degree of setting up. This involves allocating a new "song" on the "song" manager (a file manager for "songs") and uploading tracks to the "song". A track is any kind of reference for the representation of the song--the song played accurately on a midi-piano, a text file containing lyrics, a full-on reference recording of the song, etc. One track will be designated as the primary track, meaning that the musical notes shown while practicing are extract from that track. There will be an option to play tracks (to check for loading errors) and to name tracks.
We rated this as 'easy' because it is simply initializing the song data. We are going to be importing certain types of files and having our interface respond accordingly, and then be able to play the song back.
Difficulty with existing tools/apps: Varies. On existing iPhone apps that serve a similar purpose to ours, the interface looks clunky and initializing and playing back a song would not be straightforward to the novice user.
Difficulty with our app: Simple. This function will be accessible from the main screen and take as little taps as possible.
Task 2 (easy)
Play specific reference notes on-the-fly.
At times, users will want references for specific musical notes so that they can set a baseline for singing or improve their accuracy on a specific note. Because all of our users had knowledge in playing the piano, our app will have the familiar piano interface with each key labeled with that note's identity. The app will also include the ability to continuously play a chosen note so that the user does not have to keep his finger on the key.
We rated this task as easy because it is a single view where we have a virtual piano on the screen and the iPhone will play the chosen note through the headphone jack or speakers, along with displaying the chosen note on a staff.
Difficulty with existing tools/apps: Well, you need to have access to a piano to do this function, and not everyone has a piano or keyboard in their house to practice with.
Difficulty with our app: Simple. By emulating the piano, we translate the user's real world experience to our device so that it is straightforward to use.
Task 3 (medium)
Set options for how different tracks are played.
The app will include a track manager that allows various playback options to be set: the volume of each track relative to one another, which track is played in which speaker (earphones), the speed of playback, which tracks are played and which are not, etc. In order to effectively displace the manual use of a piano, we must ensure maximum flexibility in providing audio accompaniment.
We rated this task as medium because the user can control a lot of aspects of the specific song. We want to be able to support a variety controls over the audio files that the user uploads.
Difficulty with existing tools/apps: Difficult. The user must acquire the appropriate track files and use some specialized software to be able to fine-tune the playback of the music.
Difficulty with our app: Simple. Since our app already imports audio tracks, it is an easy and logical step to be able to adjust the playback of specific tracks. The interface will be intuitive and flows from importing a song to adjusting tracks.
Task 4 (medium)
Annotate a song.
Often, users would annotate their sheet music to remind themselves of where improvement is needed or of parts of the song that need a particular treatment during singing. The app will allow users to annotate the music during practice and then display the notes alongside the visual representation of the song.
We rated this task as medium because it might be clunky to enter in comments using a keyboard rather than being able to scribble on sheet music, but we also allow flexibility in being able to create, show/hide, and delete annotations on the fly.
Difficulty with existing tools/apps: Easy. This is easily done since singers often practice with sheet music in front of them, and annotations can be quickly made.
Difficulty with our app: Relatively easy. We will have to deal with limited screen size and keyboard entry, but we try to emulate the sheet music and annotations as close to reality as possible.
Task 5 (hard)
View a visual representation of the song while singing.
Notes and lyrics on a scrolling musical staff will be shown, along with annotations (see Task 4). When users practice, they usually keep a set of sheet music on hand so they can see how parts of the song are sung. With this app, we will attempt to eliminate the need to keep sheet music on hand by showing the user all of the relevant visual information. Our users will have varying levels of musical knowledge; so, we will try to convey the information in a way that is both intuitive and follows musical convention.
We rated this task as hard because it will be a challenge to convey all the information of sheet music along with lyrics and annotations on the iPhone's limited screen size. We will have to maximize content while keeping clutter to a minimum so that the notes and lyrics are still readable.
Difficulty with existing tools/apps: Easy. The user will simply just refer to his/her sheet music for this task.
Difficulty with our app: Moderate. There might be a learning curve to adjust to the scrolling musical staff, but otherwise it should be a lot like reading sheet music.
Task 6 (hard)
Get feedback on singing accuracy and consistency.
Across the board, users assigned the most importance to this task. While the user is using this app to practice, we will analyze (via microphone) the notes that the user sings and show them in comparison alongside the reference notes. This will allow the user to visually confirm how accurate or consistent his singing is in real-time and allow him to act on the feedback while singing. After singing a song (in part or in whole), the app will allow the user to manually scroll through the song and see where his singing deviated from the reference notes.
Because this particular feature is the most important, we will also be thinking of other metrics to help singers gauge their accuracy/consistency. A couple of users suggested numerical/statistical scores.
We rated this task as hard because we have to come up with a scheme of judging accuracy in real-time as well as an intuitive interface to go along with it. If we implement a scoring system, it will be a challenge to figure out how to make a scoring system as accurate as possible.
Difficulty with existing tools/apps: There are currently very limited resources in being able to judge your accuracy. Short from using another singer to analyze your sound or your own judgment, there aren't many programs out there that allow you to judge your sound and pitch in individual practice.
Difficulty with our app: Easy. Feedback will be given in real-time so that the user can adjust his pitch wherever he makes an error. With a scoring system, the user can see how his overall pitch progresses.
Interface Design
Description and Sketches
Song Choosing View
The song choosing view will include a list of songs the user has imported. Any individual song can be chosen for practicing or editing just by tapping the song on the screen. To delete a song, the user can tap on the "x" in the top left corner of the song's box and will be prompted for confirmation to delete the song. The list will be scrollable to allow for more songs to be imported. New songs can be added to the list simply tapping on the "New Song" button, which will take the user to a view to import a new song from the computer.
Track Managing View
The track managing view, singing view, and options will all be accessible through a menu bar at the bottom of the screen to make it easier for the user to switch between views at any time. The user can tap the menu to go to each view, and the currently selected view will be highlighted on the menu. The track managing view will have a list of the different tracks that have been imported for the currently selected song. Each track will have its own control panel, where the user can use a slider to adjust volume of that particular track or select whether audio and visual for that track should be turned on or off. This list will be scrollable because the user can import any number of tracks for any song. The "Add Track" button at the bottom will allow the user to add new tracks to the song. The "main track" the user has selected is identified with a star next to the track name (this track will be the one the user will be rated against for feedback).
Singing View
The singing view provides a visual representation of the music that scrolls across the screen as the music plays. Only selected tracks will be shown; the user can choose which tracks are shown by using the track manager. Music can be displayed either as bars (shown in the mockup below) showing relative pitch or as notes on a staff, an option which the user will be able to select in the "Options" screen. Lyrics are displayed under the notes, another option which can be turned on or off using options. The user's pitch as they sing into the mic will be shown with an arrow so the user can see how to adjust their pitch to match the correct note. The user can tap on the screen to add annotations like section letters, and there are also play/stop buttons at the bottom for easy playback control. A bar at the bottom of the screen shows the user's progress through the song; the user can tap on the bar to change their position in the song at any time.
Note Playing View
The note playing view will have a staff and a keyboard that the user can tap on to play specific notes. It will also show the name of the note played.
Scenario 1 - Getting Feedback on Singing
Alvin the singer just started learning a new song with his a capella group, but he's not very familiar with it yet. When he goes off to practice on his own, he realizes he needs some help figuring out how to sing his part...but he's all alone and there's no one to help him! Thankfully, he has his handy iPhone, loaded with the midis his group made of the song! Alvin selects the song and starts "Sing Mode." While he's singing into the mic of his iPhone, Alvin can see bars representing the notes he should be singing, along with little arrow pointing to what pitch he's singing at. When he goes off pitch, the screen shows him that he's a little too high, so he lowers his voice a little until he matches the right note. After finishing the song, Alvin wants to go back and see all the parts he messed up so he can practice them more, so he goes to "Review", which highlights all the sections of the song where he went off pitch or rhythm.
Scenario 2
This "play specific notes" is a function that help people tune their voice either before or during singing.
Play Specific Notes
1. OMG!!! I have a group practice tomorrow!!!! If I do not practice my part enough, my group members and director will kill me. I need to get started practicing RIGHT NOW!!!!!
2. Before I started to practice singing, I need to check my voice first and get relaxed. BUT how can I check it??? I got no friends around me now :( WOW!! I totally forgot that I have "iPhone" :) It has an amazing application that can help me~~
3. Let's go to the "Play Specific Notes" mode. just click only one button??? wow…EASY!!!!
4. Play some notes and try them to check whether or not my pitches are correct.
la~~la~~ Hm…I think it's enough. Now I need to get started practicing.
5. Play the music I want to practice~~
Wait, am I doing right??? It sound like I'm a little bit off.
Hm……I'm not pretty sure tho…. I want to know what these notes sound like!!!
6. Let's check notes that I'm not sure about and go to the "Play Specific Notes" mode.
Ah~~Ha!! Now it sounds perfect to me. BIG SMILE ^_________^ YAY~
7. I love my iPhone so much :) Let's keep practicing
Scenario 3
Singer Steve has just gotten home from practice with his choir and he was disappointed in his performance. He kept messing up on a few sections, including one with a lot of key changes. He wants to practice now, and thanks to this app, it is straightforward and helpful. The ability to make annotations on sheet music helps musicians keep track of complex passages or make note of subtle techniques or reminders. The advantage of printed sheet music is that musicians can write directly on their sheet music to help them get through the song. We want to implement this function on our "virtual" sheet music.
Analysis of Approach
We are taking advantage of the strong audio platform offered by Apple in order to produce a targeted audio processing application that allows users to simultaneously view/listen to multiple tracks and to have their singing analyzed via microphone. Any set of earphones will provide good 2ch audio quality, and the built-in microphone provides reasonable recording ability and is sufficient for pitch analysis.
Also, having a set of useful tools packaged in the iPhone's portable form factor allows singers to practice whenever and wherever they want.
Our approach to bridging the gap between the resources offered by group practice vs. practicing alone is to merely to allow users to do what they already do more efficiently in addition to providing them with a digital partner for giving feedback. We feel that this approach integrates naturally into singers' workflows and doesn't try to force them to do something new.
Now, the trade-off in sticking to the users' already establish workflows is that we're restricted to ideas and methodologies that already exist. However, it is surely beyond the scope of our project to completely reinvent the art of singing practice. All we're trying to do is address a practical deficiency in as a natural a way as possible for the users.
