Midi and synthesis in R

midi

fluidsynth

rstats

Trying out a few R packages to handle MIDI data in dataframes, and play it with fluid synth.

Author

Matt Crump

Published

January 30, 2024

Show the code

from diffusers import DiffusionPipeline
from transformers import set_seed
from PIL import Image
import torch
import random
import ssl
import os
ssl._create_default_https_context = ssl._create_unverified_context

#locate library
#model_id = "./stable-diffusion-v1-5"
model_id = "dreamshaper-xl-turbo"

pipeline = DiffusionPipeline.from_pretrained(
    pretrained_model_name_or_path = "../../../../bigFiles/huggingface/dreamshaper-xl-turbo/"
)

pipeline = pipeline.to("mps")

# Recommended if your computer has < 64 GB of RAM
pipeline.enable_attention_slicing("max")

prompt = "computer music. musical computer. music represented as bits going into the fabric of the universe. 80s cartoon retro."

for s in range(30):
  for n in [5,10]:
    seed = s+21
    num_steps = n+1
    set_seed(seed)
    
    image = pipeline(prompt,height = 1024,width = 1024,num_images_per_prompt = 1,num_inference_steps=num_steps)
    
    image_name = "images/synth_{}_{}.jpeg"
    
    image_save = image.images[0].save(image_name.format(seed,num_steps))

computer music. musical computer. music represented as bits going into the fabric of the universe. 80s cartoon retro. - Dreamshaper v7

I’m going to be running a cognition experiment or two this semester that will involve creating musical stimuli. I would like programmatic control over that, so I’m delighted to learn that there are existing R packages that will help me with a few things.

I’m just testing a few things out here, which means this page will be a loose collection of notes and scraps of code.

Reading in MIDI with pyramidi

It looks like I can read in MIDI data to a dataframe with pyramidi.

Requires some python stuff, but it is working. Kudos to Urs Wilke for developing pyramidi! It uses R6, which I don’t use very often.

Show the code

library(pyramidi)
library(dplyr)
library(tidyr)
library(purrr)

midifile <- MidiFramer$new("Top Gun Theme.mid")

knitr::kable(midifile$df_notes_wide[1:10,])

i_track	meta	program	control	value	note	i_note	velocity_note_on	ticks_note_on	b_note_on
1	FALSE	NaN	NaN	NaN	24	1	110	2304	48
1	FALSE	NaN	NaN	NaN	24	2	0	3792	79
1	FALSE	NaN	NaN	NaN	24	3	110	3840	80
1	FALSE	NaN	NaN	NaN	24	4	0	5328	111
1	FALSE	NaN	NaN	NaN	24	5	110	5376	112
1	FALSE	NaN	NaN	NaN	36	1	110	5376	112
1	FALSE	NaN	NaN	NaN	24	6	0	6096	127
1	FALSE	NaN	NaN	NaN	36	2	0	6096	127
1	FALSE	NaN	NaN	NaN	24	7	110	6144	128
1	FALSE	NaN	NaN	NaN	41	1	110	6144	128

Synthesizing midi to wav and mp3

raudiomate

also need fluid synth

And, apparently fluid synth needs sound fonts. Got this one https://member.keymusician.com/Member/FluidR3_GM/index.html

I could also try timidity, but I haven’t gone there yet.

I could not get raudiomate helper functions to work. The processx::run command kept putting quotes where they didn’t belong

used the system command to run fluidsynth
used the av package to turn the wav into an mp3

Show the code

system("fluidsynth -F out.wav ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 'Top Gun Theme.mid'")

av::av_audio_convert("out.wav","out.mp3")

This all took way longer than I expected. Mostly fiddling with python packages and paths to things. But, I declare victory because it made the Top Gun theme song into an mp3.

Very happy that it is possible to import, manipulate, and render midi files using R as a central command language. Although I have been messing with midi stuff for since the 80s, I have not tried to programmatically mess with it, or dug into midi file convention and structure.

I have some work ahead if I want to compose “computer music” with R. That would be fun though. I would probably glitch out until everything descends into modem sounds. Actually, I’m secretly extremely excited about some possibilities that I’ve wanted to explore.

Anyway, below is scraps of code trying out various things and making notes to my future self.

Writing some random notes

I’ve only got 30 minutes right now, can I make this thing randomize notes?

Strategy: Read the pyramidi docs, borrow some code from there, and try to play some random notes.

Answer: yes, close enough for now.

Show the code

# trying stuff from the pyramidi docs

# load in a basic midi file
midi_file_string <- system.file("extdata", "test_midi_file.mid", package = "pyramidi")

mfr <- MidiFramer$new(midi_file_string)

# mfr has a bunch of midi dataframes in it
#mfr$df_notes_wide[1:10]

# helper function
beats_to_ticks <- function(notes_wide) {
  notes_wide %>%
    mutate(
      ticks_note_on  = b_note_on  * ticks_per_beat,
      ticks_note_off = b_note_off * ticks_per_beat
    )
}

n_beats <- 16
ticks_per_beat <- 960L

#(b_note_on = (0:(n_beats-1) %/% 4) * 4)

b_note_on <- 0:(n_beats-1)


notes <- tibble(
  i_track = 0,
  meta = FALSE,
  note = sample(c(60, 64, 67, 72), 16, replace=T),
  channel = 1,
  i_note = 1:n_beats,
  velocity_note_on = 100,
  velocity_note_off = 0,
  b_note_on = b_note_on,
  b_note_off = b_note_on + 1,
)

mfr$update_notes_wide(beats_to_ticks(notes))

df_notes_long <- pivot_long_notes(mfr$df_notes_wide)
df_midi_out <- merge_midi_frames(mfr$df_meta, mfr$df_notes_long, mfr$df_not_notes)

dfc2 <- df_midi_out %>%
        miditapyr$nest_midi(repair_reticulate_conversion = TRUE)

miditapyr$write_midi(dfc2, ticks_per_beat, "test.mid")

system("fluidsynth -F test.wav ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 'test.mid'")

av::av_audio_convert("test.wav","test.mp3")

Midi specs

Don’t have time to read through this today, but here it is.

https://www.midi.org/specifications

Drums

On BPM, tempo, and midi ticks

https://majicdesigns.github.io/MD_MIDIFile/page_timing.html

The next bit is straight from https://urswilke.github.io/pyramidi/articles/compose.html, with a few minor modifications to get things working on my machine.

Show the code

# load system midi file to start
midi_file_string <- system.file("extdata", "test_midi_file.mid", package = "pyramidi")

mfr <- MidiFramer$new(midi_file_string)

# set some timing params
n_beats <- 16
ticks_per_beat <- 960L

# construct a tibble
drum <- tibble(
  i_track = 0,
  meta = FALSE,
  # This is just a repetition of a classical rock beat:
  note = rep(c(36, 38), n_beats / 2),
  channel = 9,
  i_note = 1:n_beats,
  velocity_note_on = 100,
  velocity_note_off = 0,
  b_note_on = 0:(n_beats - 1),
  b_note_off = b_note_on + 1 / 2,
)

# helper function
beats_to_ticks <- function(notes_wide) {
  notes_wide %>%
    mutate(
      ticks_note_on  = b_note_on  * ticks_per_beat,
      ticks_note_off = b_note_off * ticks_per_beat
    )
}

# pass the tibble back to the midi df
mfr$update_notes_wide(beats_to_ticks(drum))

mfr$mf$write_file("testagain.mid")

# do pyramidi stuff to get the tibble back into the format it needs to be
df_notes_long <- pivot_long_notes(mfr$df_notes_wide)
df_midi_out <- merge_midi_frames(mfr$df_meta, mfr$df_notes_long, mfr$df_not_notes)

dfc2 <- df_midi_out %>%
        miditapyr$nest_midi(repair_reticulate_conversion = TRUE)

# write the midi file to disk
miditapyr$write_midi(dfc2, ticks_per_beat, "test_drums.mid")

# synthesize midi file to wav with fluid synth
system("fluidsynth -F test_drums.wav ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 'test_drums.mid'")

# convert wav to mp3
av::av_audio_convert("test_drums.wav","test_drums.mp3")

[1] "/Users/mattcrump/Github/homophony.github.io/blog/32_1_30_24_R_synth/test_drums.mp3"

Show the code

# clean up and delete wav
if(file.exists("test_drums.wav")){
  file.remove("test_drums.wav")
}

[1] TRUE

Cool, a basic rock beat.

Attempting to add hi-hats, change tempo, and generally mess about.

hopefully this percussion midi map helps https://usermanuals.finalemusic.com/SongWriter2012Win/Content/PercussionMaps.htm

Got the hihats, changed the tempo. I’m missing something about timing, the resulting wav file is playing with silence at the end, and that seems wrong. Need to tinker some more.

https://mido.readthedocs.io/en/latest/files/midi.html?highlight=set_tempo#about-the-time-attribute

There seems to be some issues with merge_midi_frames(). The merged dataframe may contain some out of order rows.

arranging by ticks gets screwed up when there are 0 ticks in meta and note dfs

Show the code

# add additional sorting by i_note, seems to solve the curent problem
merge_midi_frames_matt <-
  function(df_meta, df_notes_long, df_not_notes) {
    if (is.null(df_meta) &
        is.null(df_notes_long) & is.null(df_not_notes)) {
      return(NULL)
    }
    cols_to_remove <- c("i_note", "ticks", "t", "m", "b")
    
    res <- df_notes_long %>%
      dplyr::bind_rows(df_meta) %>%
      dplyr::bind_rows(df_not_notes)
    
    # if in tab_measures() there weren't all columns built, we have to remove them here:
    cols_to_remove <- intersect(cols_to_remove, names(res))
    res %>%
      dplyr::arrange(.data$i_track, .data$ticks, .data$i_note) %>%
      dplyr::group_by(.data$i_track) %>%
      dplyr::mutate(time = .data$ticks - dplyr::lag(.data$ticks) %>% {
        .[1] = 0
        .
      }) %>%
      dplyr::ungroup() %>%
      dplyr::select(-!!cols_to_remove)
  }

Show the code

# load system midi file to start
midi_file_string <- system.file("extdata", "test_midi_file.mid", package = "pyramidi")

mfr <- MidiFramer$new(midi_file_string)

# set tempo
BPM_to_microsecond_tempo <- function(BPM) {
  return(60000000/BPM)
}

midi_tempo <- BPM_to_microsecond_tempo(120)

mfr$df_meta <- mfr$df_meta %>%
  mutate(tempo = case_when(type == "set_tempo" ~ midi_tempo),
         clocks_per_click = case_when(type == "time_signature" ~ 24),
         b = 0) %>%
  #keep the first track
  filter(i_track == 0)

#View(mfr$df_meta)
#mfr$ticks_per_beat
#View(mfr$df_notes_wide)

# set some timing params
num_bars <- 1
n_beats <- num_bars*4
smallest_note_per_bar <- 16
max_steps <- num_bars*smallest_note_per_bar

ticks_per_beat <- 960L
mfr$ticks_per_beat <- 960L


# Construct a beat with hihats
drum <- tibble(
  i_track = 0,
  i_note = 1:max_steps,
  kick = rep(c(1,NA,NA,NA),4),
  snare = rep(c(NA,NA,1,NA),4),
  hihat_closed = rep(1,16),
  meta = FALSE,
  channel = 9,
  velocity_note_on = 100,
  velocity_note_off = 0,
  b_note_on = 0:(max_steps - 1),
  b_note_off = b_note_on + 1 / 2
) %>%
  mutate(kick = kick*36,
         snare = snare*38,
         hihat_closed = hihat_closed*42) %>%
  pivot_longer(cols = c("kick","snare","hihat_closed"), 
               values_to = "note",
               values_drop_na = T) %>%
  mutate(b_note_on = b_note_on/(2),
         b_note_off = b_note_off/(2))

drum$i_note <- 1:dim(drum)[1]

# helper function
beats_to_ticks <- function(notes_wide) {
  notes_wide %>%
    mutate(
      ticks_note_on  = b_note_on  * ticks_per_beat,
      ticks_note_off = b_note_off * ticks_per_beat
    )
}

# pass the tibble back to the midi df
mfr$update_notes_wide(beats_to_ticks(drum))

df_notes_long <- pivot_long_notes(mfr$df_notes_wide)
df_not_notes <- mfr$df_not_notes

# do pyramidi stuff to get the tibble back into the format it needs to be
#mfr$df_notes_long <- pivot_long_notes(mfr$df_notes_wide)

df_meta <- mfr$df_meta %>%
  mutate(ticks = case_when(type == "end_of_track" ~ (max(df_notes_long$ticks)+ticks_per_beat),
                           type != "end_of_track" ~ 0))


#mfr$df_meta <- mfr$df_meta %>%
#  mutate(ticks = case_when(type == "end_of_track" ~ (max(mfr$df_notes_long$ticks)+ticks_per_beat)))

df_midi_out <- merge_midi_frames_matt(df_meta, df_notes_long, df_not_notes)

dfc2 <- df_midi_out %>%
        miditapyr$nest_midi(repair_reticulate_conversion = TRUE)

#########
# bounce 

track_name <- "try_Write"

wav_name <- paste0(track_name,".wav")
midi_name <- paste0(track_name,".mid")
mp3_name <- paste0(track_name,".mp3")

# write the midi file to disk
miditapyr$write_midi(dfc2, ticks_per_beat, midi_name)

# synthesize midi file to wav with fluid synth
system_command <- glue::glue('fluidsynth -F {wav_name} ~/Library/Audio/Sounds/Banks/FluidR3_GM.sf2 {midi_name}')
system(system_command)

# convert wav to mp3
av::av_audio_convert(wav_name,mp3_name)

[1] "/Users/mattcrump/Github/homophony.github.io/blog/32_1_30_24_R_synth/try_Write.mp3"

Show the code

# clean up and delete wav
if(file.exists(wav_name)){
  file.remove(wav_name)
}

[1] TRUE

This works sort of. MIDI fried my brain here. For some reason the file has extra bars of silence, can’t figure out why right now. I’m missing something here about how midi time works.

loaded better in musescore.
musescore shows two bars, but it plays through 4 bars…not sure what is going on.

Piano keyboard plot with ggplot2

Neat, this is from pyramidi.

Show the code

library(ggplot2)

piano_keys_coordinates %>%
  # plot white keys first that they don't cover half of the black keys:
  dplyr::arrange(layer) %>%
  ggplot(aes(
    ymin = ymin,
    ymax = ymax,
    xmin = xmin,
    xmax = xmax,
    fill = factor(layer)
  )) +
  geom_rect(color = "black", show.legend = FALSE) +
  scale_fill_manual(values = c("#ffffdd", "#113300")) +
  coord_fixed(ratio = 10)

Let’s try plotting a chord, C7. Cool!

Show the code

# get midi notes from table look up
note_numbers <- midi_defs %>%
  filter(note_name %in% c("C4","E4","G4","Bb4") == TRUE) %>%
  pull(note)

# plot
piano_keys_coordinates %>%
  # make black keys 3rd order for printing
  mutate(layer = case_when(layer == 2 ~ 3,
                           layer == 1 ~ 1)) %>%
  # set played keys to layer 2
  mutate(layer = case_when(midi %in% note_numbers == TRUE ~ 2,
                           midi %in% note_numbers == FALSE ~ layer)) %>%
  # plot white keys first that they don't cover half of the black keys:
  dplyr::arrange(layer) %>%
  ggplot(aes(
    ymin = ymin,
    ymax = ymax,
    xmin = xmin,
    xmax = xmax,
    fill = factor(layer)
  )) +
  geom_rect(color = "black", show.legend = FALSE) +
  scale_fill_manual(values = c("#ffffdd", "pink","#113300")) +
  coord_fixed(ratio = 10)

miditapyr

https://miditapyr.readthedocs.io/en/latest/notebooks/midi_frame_usage.html

Learning more about the miditapyr workflow.

Show the code

#import midi using miditapyr
test_midi <- pyramidi::miditapyr$MidiFrames("ableton.mid")

#View(test_midi$midi_frame_unnested$df)

# modify something
mod_df <- test_midi$midi_frame_unnested$df %>%
  mutate(note = case_when(note == 72 ~ 63,
                          TRUE ~ note))

# update df
test_midi$midi_frame_unnested$update_unnested_mf(mod_df)

# write midi file

test_midi$write_file("ableton2.mid")

I made a 1 bar drum loop in ableton, using the clip view. Apparently this kind of midi file does not contain the meta set_tempo message…

Show the code

#import midi using miditapyr
test_midi <- pyramidi::miditapyr$MidiFrames("ableton.mid")

#import using mido
mido_import <- pyramidi::mido$MidiFile("ableton.mid")

# to R dataframe
dfc <- pyramidi::miditapyr$frame_midi(mido_import)
ticks_per_beat <- mido_import$ticks_per_beat

# unnest the dataframe
df <- pyramidi::miditapyr$unnest_midi(dfc)

# add set_tempo message
# missing from ableton midi clip

df <- df %>%
  mutate(tempo = NaN) %>%
  add_row(
    i_track = 0,
    time = 0,
    meta = TRUE,
    type = "set_tempo",
    tempo = 500000,
    .before = 2
  )

Before I go on, I need to do some timing tests to make sure I understand a few things.

The beat is 1 bar. There are 4 beats per bar. The hi hats are in 16th notes, and there are 16 hi hats. The tempo is set to 120. What happens if I play with the time column.

Show the code

# reload df
df <- pyramidi::miditapyr$unnest_midi(dfc)

# add set_tempo message
# missing from ableton midi clip

df <- df %>%
  mutate(tempo = NaN) %>%
  add_row(
    i_track = 0,
    time = 0,
    meta = TRUE,
    type = "set_tempo",
    tempo = 500000,
    .before = 2
  )

# futz with the time column
# behaves as expected, yah!
df <- df %>%
  mutate(time = time/2)

# update df
test_midi$midi_frame_unnested$update_unnested_mf(df)

# write midi file

test_midi$write_file("ableton2.mid")

The pyramidi package has a tab_measures() function that adds additional timing information, for later use in composition and modification.

Show the code

# add more time info

dfm <- tab_measures(df = df, ticks_per_beat = ticks_per_beat)