Testing out {fluidsynth} for R!

Hurrah, fluidsynth made easy for R!

from diffusers import DiffusionPipeline
from transformers import set_seed
from PIL import Image
import torch
import random
import ssl
import os
ssl._create_default_https_context = ssl._create_unverified_context

#locate library
#model_id = "./stable-diffusion-v1-5"
model_id = "dreamshaper-xl-turbo"

pipeline = DiffusionPipeline.from_pretrained(
    pretrained_model_name_or_path = "../../../../bigFiles/huggingface/dreamshaper-xl-turbo/"
)

pipeline = pipeline.to("mps")

# Recommended if your computer has < 64 GB of RAM
pipeline.enable_attention_slicing("max")

prompt = "a synthesizer made out of water. 3d. white background. super cool."

for s in range(30):
  for n in [5,10]:
    seed = s+21
    num_steps = n+1
    set_seed(seed)
    
    image = pipeline(prompt,height = 1024,width = 1024,num_images_per_prompt = 1,num_inference_steps=num_steps)
    
    image_name = "images/synth_{}_{}.jpeg"
    
    image_save = image.images[0].save(image_name.format(seed,num_steps))

a synthesizer made out of water. 3d. white background. super cool. - Dreamshaper

Thanks to Jeroen Ooms and rOpenSci, there is a new {fluidsynth} package with bindings for R!

Here’s the github repo: https://github.com/ropensci/fluidsynth/

The package is only four days old and could change, but it is working for me!

playback midi locally

library(fluidsynth)

# this works for local playback!

midi_play(
  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  audio.driver = NULL,
  settings = list(),
  progress = interactive()
)

write midi to wav

I’m on a mac m2. The midi_convert function produces a wav file that will not play. However, the {av} package does successfully convert the wav file to a listenable mp3. Glad it works out in the end.

I’m not sure why the wav file isn’t rendering as playable. I wonder if it isn’t finding my installation of libsndfile, and the wav file is being rendered as raw?

midi_convert(
  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  output = "output.wav",
  settings = list(
    audio.driver = "coreaudio",
    audio.file.format = "double",
    audio.file.type = "wav"),
  progress = interactive()
)

midi_convert(
  midi = "the-moon.mid",
  soundfont = "~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2",
  output = "output.wav",
  settings = list('synth.sample-rate'= 22050),
  progress = interactive()
)

midi_convert(settings = list('synth.sample-rate'= 22050), output =  'lowquality.wav')

av::av_audio_convert("output.wav","output.mp3")

midi to mp3 All night long

And now it is really this easy!

midi <- system.file("generaluser-gs/midi/All_Night_Long.mid", package = "fluidsynth")
fluidsynth::midi_convert(midi, output = 'all_night_long.mp3')

testing faster renders

midi <- system.file("generaluser-gs/midi/All_Night_Long.mid", package = "fluidsynth")
fluidsynth::midi_convert(midi, output = 'all_night_long.mp3',
                         settings = list('player.timing-source' = 'sample',
                                         'synth.lock-memory' = 0,
                                         'synth.cpu-cores' = 12),
                         verbose = FALSE
                        
                         )


#########
# compare render speed

track_name <- "All_Night_Long"

wav_name <- paste0(track_name,".wav")
midi_name <- midi
mp3_name <- paste0(track_name,".mp3")

system_command <- glue::glue('fluidsynth -F {wav_name} ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 {midi_name}')
system(system_command)

fluid synth settings

All of these fluid synth parameters can now be easily modified from R!

knitr::kable(fluidsynth::fluidsynth_setting_list())

name	type
audio.coreaudio.channel-map	string
audio.coreaudio.device	string
audio.driver	string
audio.file.endian	string
audio.file.format	string
audio.file.name	string
audio.file.type	string
audio.period-size	integer
audio.periods	integer
audio.portaudio.device	string
audio.realtime-prio	integer
audio.sample-format	string
midi.autoconnect	integer
midi.coremidi.id	string
midi.driver	string
midi.portname	string
midi.realtime-prio	integer
player.reset-synth	integer
player.timing-source	string
shell.port	integer
shell.prompt	string
synth.audio-channels	integer
synth.audio-groups	integer
synth.chorus.active	integer
synth.chorus.depth	double
synth.chorus.level	double
synth.chorus.nr	integer
synth.chorus.speed	double
synth.cpu-cores	integer
synth.default-soundfont	string
synth.device-id	integer
synth.dynamic-sample-loading	integer
synth.effects-channels	integer
synth.effects-groups	integer
synth.gain	double
synth.ladspa.active	integer
synth.lock-memory	integer
synth.midi-bank-select	string
synth.midi-channels	integer
synth.min-note-length	integer
synth.overflow.age	double
synth.overflow.important	double
synth.overflow.important-channels	string
synth.overflow.percussion	double
synth.overflow.released	double
synth.overflow.sustained	double
synth.overflow.volume	double
synth.polyphony	integer
synth.reverb.active	integer
synth.reverb.damp	double
synth.reverb.level	double
synth.reverb.room-size	double
synth.reverb.width	double
synth.sample-rate	double
synth.threadsafe-api	integer
synth.verbose	integer