Synthesizers are powerful tools that allow musicians to create dynamic and
original sounds. Existing commercial interfaces for synthesizers typically
require musicians to interact with complex low-level parameters or to manage
large libraries of premade sounds. To address these challenges, we implement
SynthScribe -- a fullstack system that uses multimodal deep learning to let
users express their intentions at a much higher level. We implement features
which address a number of difficulties, namely 1) searching through existing
sounds, 2) creating completely new sounds, 3) making meaningful modifications
to a given sound. This is achieved with three main features: a multimodal
search engine for a large library of synthesizer sounds; a user centered
genetic algorithm by which completely new sounds can be created and selected
given the users preferences; a sound editing support feature which highlights
and gives examples for key control parameters with respect to a text or audio
based query. The results of our user studies show SynthScribe is capable of
reliably retrieving and modifying sounds while also affording the ability to
create completely new sounds that expand a musicians creative horizon.

SynthScribe 是一个基于多模态深度学习的全栈系统，通过解决搜索现有声音、创造全新声音和对给定声音进行有意义修改等问题，以更高层次表达用户意图。用户研究结果显示 SynthScribe 能够可靠地检索和修改声音，并能创造全新声音，拓展音乐家的创造空间。