Scene-Aware Background Music Synthesis

Yujia Wang¹    Wei Liang¹    Wanwan Li²    Dingzeyu Li³    Lap-Fai Yu²

¹Beijing Institute of Technology    ²George Mason University    ³Adobe Research  

Abstract

Background music not only provides auditory experience for users, but also conveys, guides, and promotes emotions that resonate with visual contents. Studies on how to synthesize background music for different scenes can promote research in many fields, such as human behaviour research. Although considerable effort has been directed toward music synthesis, the synthesis of appropriate music based on scene visual content remains an open problem.

In this paper we introduce an interactive background music synthesis algorithm guided by visual content. We leverage a cascading strategy to synthesize background music in two stages: Scene Visual Analysis and Background Music Synthesis. First, seeking a deep learning-based solution, we leverage neural networks to analyze the sentiment of the input scene. Second, real-time background music is synthesized by optimizing a cost function that guides the selection and transition of music clips to maximize the emotion consistency between visual and auditory criteria, and music continuity. In our experiments, we demonstrate the proposed approach can synthesize dynamic background music for different types of scenarios. We also conducted quantitative and qualitative analysis on the synthesized results of multiple example scenes to validate the efficacy of our approach.

Keywords

Scene Sentiment, Background Music Synthesis, Music Transition.

Publication

Scene-Aware Background Music Synthesis
Yujia Wang, Wei Liang, Wanwan Li, Dingzeyu Li, Lap-Fai Yu
ACM Multimedia Conference 2020 (ACM MM 2020)
Paper , Video , Music Synthesis Results (Web Virsion) , Dataset (Coming Soon)

BibTex

@inproceedings{bgm2020wang,
    title= {Scene-Aware Background Music Synthesis},
    author = {Wang, Yujia and Wei, Liang and Wanwan, Li and Dingzeyu, Li and Yu, Lap-Fai},
    booktitle={ACM Multimedia},
    volume = {38},
    number = {6},
    year = {2020} }

• 媒体计算与智能系统实验室


• Media Computing and Intelligent Systems Lab

Beijing Institute of Technology Copyright Address: 5 South Zhongguancun

Street, Haidian District, Beijing Postcode: 100081