Ibáñez Martínez, Laura2024-09-182024-09-182023-09Ibáñez Martínez, Laura (2023) MIDI-Conditional Text-to-Audio Synthesis Using ControlNet on AudioLDM. Trabajo Fin de Máster. Universidad de Educación a Distancia (UNED)https://hdl.handle.net/20.500.14468/23782Text-to-audio systems have gained attention in recent months, achieving impressive results in general audio synthesis. However, they often lack fine-grained control over the musical output, as note-level adjustments cannot be determined by text. In this work, we present MIDI-AudioLDM, which implements MIDI conditioning into AudioLDM with the use of ControlNet. This enables MIDI-conditional text-to-audio synthesis, which adds up to AudioLDM’s previous capacities, including direct text-to-audio synthesis as well as audio style transfer and inpainting. Like AudioLDM, the model uses contrastive language-audio pretraining (CLAP) latents and is trained on audio embeddings, while using text embeddings for inference. In contrast to unconditional audio synthesis, MIDI-AudioLDM offers detailed control over various musical aspects such as notes, genre, mood, and timbre, which makes it a more valuable tool for the music production process. A demo is available at https://huggingface.co/spaces/lauraibnz/midi-audioldm.eninfo:eu-repo/semantics/openAccess12 Matemáticas::1203 Ciencia de los ordenadores ::1203.04 Inteligencia artificial12 Matemáticas::1203 Ciencia de los ordenadores ::1203.17 Informáticatesis de maestríaaudio synthesisMIDI conditioningtext-to-audio systemsAudioLDMControlNet