Characterizing the emotion of individual piano and other musical instrument sounds

Abstract

Emotion is an indispensable aspect in human communication, and various media provide different means of invoking emotions. Music emotion is one popular research topic by the advancement of artificial intelligence, which enables practical usages such as music recommendation based on emotion. Music-making devices on the market also provide interactive tools to create sounds during performances with adjustable timbre for emotional effects. This research aimed to characterize the relationship of emotion with music timbre, pitch, and other performing qualities. We carried out listening tests to investigate isolated tones of a number of musical instruments, which incorporated pairwise comparisons among eight non-sustaining instruments. There were also in-depth study of the relative emotional effects of timbre in instruments including the piano, bowed string instruments and pitched percussion instruments when timbral features were varied. We have statistically confirmed the correlation between emotion and music timbre for these instruments. Other musical performing qualities also exhibited evident emotion trends. The effects of pitch are significant for most of the emotional categories we have tested. The effects of dynamics and mallet hardness are also prominent. Results for different instruments we tested were similar, yet with occassional surprising contrasts. For example, while the loud and low sounds of the piano is Heroic, that only corresponds to the loud and high sounds on the violin. These studies provide the foundation for further research on the emotion–timbre space, which is still under rapid development. This will in turn enhance the work of composers and arrangers in instrumentation, performers in manipulating, blending and balancing notes, and recording engineers in mixing recordings and live performances. Sound designers can even use these ideas to create nonexistent timbres for emotional effects. This also laid groundwork for further work such as algorithmic adjustment of sound spectrum, or emotional background music generation.

Date of Award	2017
Original language	English
Awarding Institution	The Hong Kong University of Science and Technology