Reverse-engineering human rubato

This is a part of a series of posts where I’m trying to explain some of the decisions made in Timmer, a solo project for double-bass and electronics.

Already when recording the source material for Timmer, I knew that the material would be cut up and re-assembled. A lot of the material came to be quite sound-oriented and rhytmical, but often without a fixed tempo. So for one of the recorded tracks, I wanted to create a randomized stream of rhythmic events, based on the recording.

The code is SuperCollider, and the full example outlined below is available on GitHub.

The first step would be to get the durations of the notes. I chose these first 18 onsets to represent a kind of rhythmic motif, from which I would extract some more or less statistically insignificant data.

As with all the other recordings, I did a combination of onset extraction and manual corrections in Sonic Visualiser to make a simple textfile of onset times and durations. The file was imported into SuperCollider, where I started with plotting the durations.


Very exciting. Here we can see that the durations are roughly between 0.5 and 1.5, with a mean somewhere around 0.8. What can we do with this information? Let’s make a test instrument and a test pattern to try it out.

Different randoms

~motif =[..17].flop[1]; //First 18 onsets of recording
SynthDef(\testDrum, { |freq, amp =1, sustain=0.1|
var snd =, 0.1) * 10, 60, 0.01) *, sustain), doneAction:2);, (snd * 6 * amp ).softclip.dup);
\instrument, \testDrum,
\amp, 1,
\legato, 0.1,
\dur, Pwhite(~motif.minItem, ~motif.maxItem),

Here, I’m simply trying to generate random durations between the minimum and maximum duration of the motif, which we soon realize is too random to be musical. Another try:

var maxChange = ~motif.differentiate.maxItem;
\instrument, \testDrum,
\amp, 1,
\legato, 0.1,
\dur, Pbrown(~motif.minItem, ~motif.maxItem, maxChange)

Switching the completely-random Pwhite to a Pbrown, which generates a brownian motion, makes the rhythm a bit more musical, but it still doesn’t have the same vibe as the original recording. To get closer, we can use the duration data to create a stream of weighted random numbers:

var q = 0.125; //quantize
var weights = ~motif.round(q).asBag.asWeights;
var rand = q * [-0.5, 0.5]; //randomness
    \instrument, \testDrum,
    \amp, 1,
    \legato, 0.1,
    \dur, Pwrand(weights[0], weights[1], inf) + Pwhite(*rand);

Here, I’m rounding the original durations to 0.125, which would correspond to 1/8 in 60 bpm. The durations are counted (through converting the array to a Bag, and converted to an array with durations and weights. This data is passed to a Pwrand, which picks a duration with a probability according to the weights. To keep the rubato feel, I add some randomness with the last Pwhite.

Repeating once or twice

This starts to resemble the original a bit more, so we continue with some other aspects of the recording. Looking at the plot, we can see that some duration values seem to repeat once or twice. Let’s do that with our random stream as well.

Pdef(\test_08_Pwrand_stutter, Psmartstutter(Prout({ |ev|
    //Duration threshold. If dur <= 0.05, count it as a repetition
    var repetitionThreshold = 0.05;
    //Make a list of booleans. Is this a repetition?
    var repList = ~motif.differentiate.abs.collect(_ <= repetitionThreshold);
    //Number of non-repeated values
    var uniqueCount = repList.reject(_.value);
    //Repetition choices, starting at a single event, then event + rep, then event + 2x rep...
    var choices = (1..10);
    //Assuming max number of repetitions is 10
    var weights = 0 ! choices.size;
    //Go through every boolean in repList
    var repCount = 0; { |isRep|
        if (isRep) {
            //For every repetition, increment repCount
            repCount = repCount + 1;
        } {
            //For every non-repetition, add repCount to repProb and reset repCount
            weights[repCount] = weights[repCount] + 1;
            repCount = 0;
    //Normalized probability of 0->10 repeats
    weights = weights.normalizeSum;
    loop {
        //According to the plot, only durations between ~0.5 and ~1 is repeated.
        if (ev.dur > 0.45 and: { ev.dur < 1.05 }) {
            //Repeat each event x times
        } {
}), Pdef(\test_08_Pwrand))

First we extract the probability of a duration being played n times from the recording, where n is a number from 1 to 10. Then we choose a value if the duration of the current event is between 0.45 and 1.05, meaning that short or long notes never repeat. Sounds fun.

Grace notes

Another recurrent motif in the motif, are grace notes. Listening to the track, it seems that they come with the longer notes, and after some counting, we can see that 22% of all notes have grace notes. Then, what is the probability for a grace note among only the longer notes? Some math later, we come up with an answer that sounds all right. If note is long, and coin toss is true, we output two notes with a random (0.08-0.15) strum value, which separates the notes.

var middle = [~motif.minItem, ~motif.maxItem].mean;
//Let prob be probability of a note longer than mean
var prob = ~motif.reject(_ < middle).size / ~motif.size;
//And if note is longer than mean, what should the probability be
//to make the total probability of a grace note be 0.22?
prob = 0.22/prob; //~0.66
Pdef(\test_08_Pwrand_stutter_grace, Pchain(
        \amp, Pfunc { |ev|
            var out = 0.8.rrand(1); //some random amp
            //If this test passes, output an array, creating two notes
            //Assuming Pbrown has an even distribution, which it hasn't
            if (ev.dur > middle and: { prob.coin }) {
                out = [0.1.rrand(0.2), out];
                if (0.5.coin) { out = out.reverse }
        \strum, Pwhite(0.08, 0.15)

If Then Legato

In the excerpt, some notes are longer than others. They also seem to come in pairs. Two times during the short track we go into a ‘branch’ of long, short note pairs. Here we translate that into a 2/18 probability of entering such a branch, and a 90% chance of repeating the pair twice (even though it’s 100% in the excerpt).

Pdef(\test_08_Pwrand_stutter_grace_legato, Pchain(
        \sustain, p { |ev|
            var prob = 2/~motif.size;
            var sustainShort=0.1;
            var sustainLong=0.5;
            loop {
                //If we have legato, we go into this branch
                if (prob.coin) {
                    //90% chance of repeating twice
                    [1,2].wchoose([0.1, 0.9]).do {
                //Always yield short after a group of longs
        //Adjust grace notes
        \sustain, Pfunc { |ev|
            if (ev.dur.isArray) {
                ev.sustain = [0.1, ev.sustain];


Now we have something which might be described as the Frankenstein version of the excerpt above. Note that we haven’t really worked on the timbre, dynamics, or other important parameters, and that a lot of fine-tuning could be done to make the rhythm stream closer to the original. But still, this might be usable in some way, which leads us to the question in the headline: Why do this?

One goal with Timmer is to find ways of extracting and extrapolating musical information from the solo recordings, to use that information as a base for generative processes. I can imagine taking this data and develop it further, to make a composition based on the seed of rhythmic ideas expressed in this excerpt.

Going through this analytic process forces you to listen in a different way. When you start thinking about musical events in terms of probabilities it makes you loosen up the linear way of thinking about musical structures, which can be a great way to expand your music-making toolset.

When writing this, the track is not finished, but I will continue working with this material. The main challenge, for this track and the whole project, is to make something that integrates with and extends the original recording. Let’s see how it goes.