Music video classification

PDF, BIB, Poster, Data Repository
Smith, J. B. L., M. Hamasaki, and M. Goto. 2017. Classifying derivative works with search, text, audio and video features. Proceedings of the International Conference on Multimedia and Expo. Hong Kong, China. 1428–33.

For every video posted by a famous musician on YouTube, you can find thousands of derivative works: cover songs, remixes, clips from live performances, lyric videos, and more. However, YouTube’s text-based search isn’t perfect: while it’s usually possible to deduce the content of a video from its title, some titles are misleading, and some search results don’t seem to match the query. So, we developed a system that could discover and classify derivative works automatically.

The algorithm pipeline is shown below. We process each song/artist query separately. After collecting as many potential derivative works as we can find (top), we extract (at left):

  • search features (based on the ranks in the different keyword searches)
  • text features (the title words filtered through a topic model)
  • video features (a bag of standard features for classification, as well as any automatically detectable text)
Algorithm pipeline

Using these features, we make an initial guess about the video’s audio content (AC) and video content (VC). Then, with a short list of audio exemplars for each category (i.e., our most confidently-labeled examples), we can compute audio features. We propose a novel set of features that express how an audio file is related to others:

Audio feature description

With this system in place, we can offer new ways of discovering music to users, such as Songrium, “a planetarium of music.” In Songrium, each original song is a star, and all of its derivatives are planets orbiting that star. You can explore derivatives according to category, travel between stars related by artist, and watch visualizations of each song along with the music video. [Our updated algorithm is currently being integrated into the YouTube portion of Songrium.]