Scene-Aware Background Music Synthesis

Yujia Wang1    Wei Liang1    Wanwan Li2    Dingzeyu Li3    Lap-Fai Yu2

1Beijing Institute of Technology    2George Mason University    3Adobe Research  


Background music not only provides auditory experience for users, but also conveys, guides, and promotes emotions that resonate with visual contents. Studies on how to synthesize background music for different scenes can promote research in many fields, such as human behaviour research. Although considerable effort has been directed toward music synthesis, the synthesis of appropriate music based on scene visual content remains an open problem.

In this paper we introduce an interactive background music synthesis algorithm guided by visual content. We leverage a cascading strategy to synthesize background music in two stages: Scene Visual Analysis and Background Music Synthesis. First, seeking a deep learning-based solution, we leverage neural networks to analyze the sentiment of the input scene. Second, real-time background music is synthesized by optimizing a cost function that guides the selection and transition of music clips to maximize the emotion consistency between visual and auditory criteria, and music continuity. In our experiments, we demonstrate the proposed approach can synthesize dynamic background music for different types of scenarios. We also conducted quantitative and qualitative analysis on the synthesized results of multiple example scenes to validate the efficacy of our approach.


Scene Sentiment, Background Music Synthesis, Music Transition.


Scene-Aware Background Music Synthesis
Yujia Wang, Wei Liang, Wanwan Li, Dingzeyu Li, Lap-Fai Yu
ACM Multimedia Conference 2020 (ACM MM 2020)
Paper , Video , Music Synthesis Results (Web Virsion) , Dataset (Coming Soon)


    title= {Scene-Aware Background Music Synthesis},
    author = {Wang, Yujia and Wei, Liang and Wanwan, Li and Dingzeyu, Li and Yu, Lap-Fai},
    booktitle={ACM Multimedia},
    volume = {38},
    number = {6},
    year = {2020} }


  • 媒体计算与智能系统实验室

  • Media Computing and Intelligent Systems Lab

Beijing Institute of Technology Copyright Address: 5 South Zhongguancun

Street, Haidian District, Beijing Postcode: 100081