Testing out {fluidsynth} for R!

Hurrah, fluidsynth made easy for R!

Matt Crump


February 15, 2024

from diffusers import DiffusionPipeline
from transformers import set_seed
from PIL import Image
import torch
import random
import ssl
import os
ssl._create_default_https_context = ssl._create_unverified_context

#locate library
#model_id = "./stable-diffusion-v1-5"
model_id = "dreamshaper-xl-turbo"

pipeline = DiffusionPipeline.from_pretrained(
    pretrained_model_name_or_path = "../../../../bigFiles/huggingface/dreamshaper-xl-turbo/"

pipeline = pipeline.to("mps")

# Recommended if your computer has < 64 GB of RAM

prompt = "a synthesizer made out of water. 3d. white background. super cool."

for s in range(30):
  for n in [5,10]:
    seed = s+21
    num_steps = n+1
    image = pipeline(prompt,height = 1024,width = 1024,num_images_per_prompt = 1,num_inference_steps=num_steps)
    image_name = "images/synth_{}_{}.jpeg"
    image_save = image.images[0].save(image_name.format(seed,num_steps))

a synthesizer made out of water. 3d. white background. super cool. - Dreamshaper

Thanks to Jeroen Ooms and rOpenSci, there is a new {fluidsynth} package with bindings for R!

Here’s the github repo: https://github.com/ropensci/fluidsynth/

The package is only four days old and could change, but it is working for me!

playback midi locally


# this works for local playback!

  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  audio.driver = NULL,
  settings = list(),
  progress = interactive()

write midi to wav

I’m on a mac m2. The midi_convert function produces a wav file that will not play. However, the {av} package does successfully convert the wav file to a listenable mp3. Glad it works out in the end.

I’m not sure why the wav file isn’t rendering as playable. I wonder if it isn’t finding my installation of libsndfile, and the wav file is being rendered as raw?

  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  output = "output.wav",
  settings = list(
    audio.driver = "coreaudio",
    audio.file.format = "double",
    audio.file.type = "wav"),
  progress = interactive()

  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  output = "output.wav",
  settings = list('synth.sample-rate'= 22050),
  progress = interactive()

midi_convert(settings = list('synth.sample-rate'= 22050), output =  'lowquality.wav')


midi to mp3 All night long

And now it is really this easy!

midi <- system.file("generaluser-gs/midi/All_Night_Long.mid", package = "fluidsynth")
fluidsynth::midi_convert(midi, output = 'all_night_long.mp3')

testing faster renders

midi <- system.file("generaluser-gs/midi/All_Night_Long.mid", package = "fluidsynth")
fluidsynth::midi_convert(midi, output = 'all_night_long.mp3',
                         settings = list('player.timing-source' = 'sample',
                                         'synth.lock-memory' = 0,
                                         'synth.cpu-cores' = 12),
                         verbose = FALSE

# compare render speed

track_name <- "All_Night_Long"

wav_name <- paste0(track_name,".wav")
midi_name <- midi
mp3_name <- paste0(track_name,".mp3")

system_command <- glue::glue('fluidsynth -F {wav_name} ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 {midi_name}')

fluid synth settings

All of these fluid synth parameters can now be easily modified from R!

name type
audio.coreaudio.channel-map string
audio.coreaudio.device string
audio.driver string
audio.file.endian string
audio.file.format string
audio.file.name string
audio.file.type string
audio.period-size integer
audio.periods integer
audio.portaudio.device string
audio.realtime-prio integer
audio.sample-format string
midi.autoconnect integer
midi.coremidi.id string
midi.driver string
midi.portname string
midi.realtime-prio integer
player.reset-synth integer
player.timing-source string
shell.port integer
shell.prompt string
synth.audio-channels integer
synth.audio-groups integer
synth.chorus.active integer
synth.chorus.depth double
synth.chorus.level double
synth.chorus.nr integer
synth.chorus.speed double
synth.cpu-cores integer
synth.default-soundfont string
synth.device-id integer
synth.dynamic-sample-loading integer
synth.effects-channels integer
synth.effects-groups integer
synth.gain double
synth.ladspa.active integer
synth.lock-memory integer
synth.midi-bank-select string
synth.midi-channels integer
synth.min-note-length integer
synth.overflow.age double
synth.overflow.important double
synth.overflow.important-channels string
synth.overflow.percussion double
synth.overflow.released double
synth.overflow.sustained double
synth.overflow.volume double
synth.polyphony integer
synth.reverb.active integer
synth.reverb.damp double
synth.reverb.level double
synth.reverb.room-size double
synth.reverb.width double
synth.sample-rate double
synth.threadsafe-api integer
synth.verbose integer