UnderScore: Musical Underlays for Audio Stories
UIST 2012
Steve Rubin,* Floraine Berthouzoz,* Gautham J. Mysore,** Wilmot Li,** and Maneesh Agrawala*
*UC Berkeley, **Advanced Technology Labs, Adobe
Main Paper Site | Demo Site | Supplemental Materials Site
PDF (2.7M) | MP4 (67.5M) | YouTube
Listen to examples generated by UnderScore
Select a speech clip, a song, and a changepoint number to hear UnderScore's automatic output. The musical changepoints are automatically extracted from the songs. The top three changepoints found by our algorithm are given here, with changepoint 1 being the best. See our paper for details on how this works.
Click anywhere in the timeline to scrub.
Speech track Music track Changepoint number
A musical underlay highlights an emphasis point in an audio story. The music track contains three segments; (1) a music pre-solo that fades in before the emphasis point, (2) a music solo that starts at the emphasis point and plays at full volume while the speech is paused, and (3) a music post-solo that fades down as the speech resumes. At the beginning of the solo, the music often changes in some significant way (e.g. a melody enters, the tempo quickens, etc.) Aligning this change point in the music with a pause in speech and a rapid increase in the music volume further draws attention to the emphasis point in the story.
Here you can preview some results of our system, which automatically refines, aligns, and adjusts speech and music to generate high-quality underlays.