A Convolutional Neural Network (CNN) is the name for an ANN with a specific structure - where the layers of the neurons are considered in a multidimensional way - usually in 2D or 3D matrices.

For example, let’s say we are building a machine to detect if a photo is of a dog or not. If we put the photo into the machine as a string of bits, it would lose the additional layer of information that comes from its colors and 2-dimensionality. That means it’ll be harder for the machine to learn and solve the problem. So instead, we adjust the architecture (the mathematical function and connections between nodes) to better suit a multi-dimensional input.

A Long Short Term Memory (LSTM) is a name for a specific type of Recurrent Neural Network, which is a family of ANN architectures. While in regular (no memory) ANNs, each input is considered separately and with no relation to its previous input, in an LSTM we save some memory of past input as part of the ANN architecture.

For example, problems like multiplication and classification of dogs in pictures have no need for memory. If anything, memory might “confuse” the machine. But what about problems like predicting how a sentence will end? If we feed the machine with one word at a time, it will be able to predict the next word much better if we let it remember the last 10 words it got as input.

A spectrogram is a 3-dimensional representation of audio, where the X-axis is time, the Y-axis is wave frequencies (in hertz), and the Z-axis is the strength of the signal (sometimes in decibels).

A spectrogram is achieved usually by applying a Fourier Transformation to audio, converting its waveform (audioform) into its component sine waves. In a spectrogram we can more easily see which frequencies the sound wave contains.

The Thayer model is a model proposed in 1989 to quantifiably characterize emotions on a 2-dimensional scale - the dimensions being Valence (stress) and Arousal (energy). For example, happiness would be high valence and high arousal, sadness would be low valence low arousal.

Of the several models considered, this model was the best fitting for our problem, since it had the broadest representation of emotions, while still showing good results in previous research.

The purpose of this experiment is to get a better understanding of the leading factors in an audience’s emotional perception of a music phrase. In this experiment we will train a Neural Network to predict the emotion an audience will experience based on a short musical phrase. This audience will be our simulated “audience”. We will then get the network’s predictions on simple curated musical phrases, to see what are the factors that affect its decision making.

The dataset we will be using to create the machine is called PMEmo⁵.

The dataset contains “emotion annotations of 794 songs as well as the simultaneous electrodermal activity (EDA) signals. A Music Emotion Experiment was well-designed for collecting the affective-annotated music corpus of high quality, which recruited 457 subjects.”⁶

For our experiment, we will use these parts of the dataset:

1. The MP3 audio of the chorus of each song 
2. Dynamic (thayer model) labels for each 0.5-second segment 

Step 1: Each chorus we will convert to WAV audio.

Step 2: Each WAV audio will be converted to Mel-scale spectrogram^* with 128 frequency buckets.

Step 3: Each spectrogram will be divided to spectrogram chunks of 196 windows. This means each chunk matches exactly 0.5 seconds of the original audio.

Step 4: Each spectrogram chunk is matched with its appropriate Thayer-model tagging.

Our final dataset consists of spectrogram chunks of 128 buckets and 196 frames, and their Thayer-model tagging.

For the input of the neural network, I wanted to be as “similar” to the human musical experience as possible. For that, I didn’t want the input to be features like tempo, pitch etc., and I chose instead to use the natural physical depiction of music - which is the different frequencies the brain experiences with each musical phrase. This is represented without loss in a spectrogram.

For the output, I have two values, which correlate to coordinates on the Thayer-model axes. To read the 2-dimensional spectrogram most “naturally”, the main architecture of the network is a CNN, as was tested most successfully before in 2020⁷.

I proposed a variation on that architecture that allows memory. Since the human musical experience relies greatly on its short-term memory between harmonies and phrases, I found this part of the architecture to be necessary for a good simulation. To do that, I added an LSTM component to the network.

In order to answer our research questions, I created and curated a testset of audio clips. I focused on 3 main attributes of music, to make the analysis simpler:

• Tempo (measured in Beats Per Minute)
• Pitch (Measured in Hz)
• Volume (Measured in relative levels between 0 and 127)

The created files are these:

For the predictor (our neural network), we measure its accuracy as the average (mean) distance between each predicted emotion (point on the Thayer-model 2-dimensional space), and the true tagged emotion. Meaning, a perfect prediction will be 0.0.

We trained the machine over 80% of the dataset, and tested it on the rest (20%).
For a random predictor (a machine that randomly selects an emotion), we get an accuracy of 0.13162.
Our network was able to predict with an accuracy of 0.02101 over the trainset, and an accuracy of 0.04508 of the testset. Meaning, the predictor was able to predict the tagged emotion over 6 times better than random for the trainset, and around 3 times better for the testset.
This shows clearly that the neural network was properly set, and was able to learn the problem correctly.

For each audio file in the curated testset, we ran the predictor twice - once as it, and once with a disabled memory. Since the neural network is built with an LSTM layer, it was interesting to see how much of its prediction is based on its memory, and how much is based on other absolute factors.
Running the curated testset through the predictor, we were able to notice some interesting factors that affected the predictor’s decision:

As expected, the predictor treats a single continuous tone the same for every frame, when no memory is used (the dip in the beginning is due to the tone pick up from zero, as the spectrogram frames are built with sliding windows).

Interesting to notice, that while the spike in the machine’s prediction is similar between a harmonized note and a single note (regardless of the mode of harmony), the dip is not. With the harmony, the predictor maintains a steadier plateau, and barely dips under it for the rest of the audio.

Another interesting prediction can be seen with the chromatic scales, when using memory. We see a clear trend downwards. We are in the second quadrant of the Thayer model (anger and annoyance), since arousal is past the middle (0.5) and valence is under the middle). This trend can be imagined as intensification of the irritation as time passes. This happens in a similar manner in both the descending and ascending scale, suggesting the predictor cares less about the absolute value of the note, and about the direction of the melodic interval, and puts a stronger emphasis on the melodic interval’s value.
Another interesting thing is that a scale up is considered less energetic (lower arousal values), but also less positive (lower valence values).

From this graph we can see that the change in tempo, when the memory is on, affects the prediction very minimally. It seems that the machine almost completely ignores the tempo when the memory is used.

Similarly to the change in tempo, from this graph we can see that the change in volume, when the memory is on, doesn’t affect the prediction.

When reviewing the results of the experiment it is important to take the dataset the predictor was trained on into consideration. Just like the human brain, the neural network learns by example and correction. There were a few features of our chosen dataset which were suboptimal:

The size - for a proper training of a neural network, very large and diverse datasets are required. Usually, the more complex the problem, the bigger the required dataset. Since our dataset was composed of only 794 choruses of popular songs, the diversity was not great. And since the audio snippets were not very long, we couldn’t really get a robust dataset.

The noise - this dataset was of popular songs, rich with harmonies, beats and generally very noisy. This means that the network had to digest a great multitude of parameters. While this is a good representation of popular music, it means that the machine had a hard time learning the subtleties and intricacies of the music. If this dataset was much larger, this wouldn’t have been a problem. But considering the relatively small size of the datasets, it is very likely that the network would have had to neglect the finer details and focus on the bigger ones.

The relevance - Since what we wanted to examine were very simple deconstructed audio phrases, the best source for a trainset would have been simpler, less polyphonic pieces. With our dataset, the machine learned to predict for audio that is very different to what we were testing.

Unfortunately, attempts of procuring funding to create a better suited dataset failed. 

When training a deep neural network on a small yet complex dataset, we face the very plausible risk of under- or overfitting. Underfitting means that we don’t really take the dataset into consideration, but rather spit out the same output, regardless of the input. Overfitting means the machine learns its trainset exactly, and would perform very poorly for any other input. In our case we were trying to avoid underfitting by starting with a small enough learning rate, random batching, and plenty of dropout layers in the network’s architecture. We were trying to avoid overfitting by running for a very limited number of epochs, as well as verifying each epoch that all 4 quadrants were getting relatively enough predictions.

It is important to mention, that since our purpose was not creating a predictor, but rather inspecting the network's deciding factors; overfitting was not an issue we needed to fear

As described in the caveats section, the results of this experiment rely heavily (not surprisingly) on the dataset. A dataset of violin melodies will provide wonderful insight over the factors of emotional expression in violin playing, but will not be very beneficial for EDM music. Similarly, a dataset of harmonic melodies, will provide results which are less relevant to monodic melodies. I leave this code open and well documented, to make it simple for succeeding research to be done. On various datasets and for various purposes.

A Generative Adversarial Network (GAN) is an interesting new architecture for neural networks. In a GAN, we train two networks in an adversarial manner - one is the predictor, the other a generator. We start by training a predictor (for example, in our case, a network that can predict an emotion based on a musical spectrogram). We then train another network to be able to generate spectrograms, based on an emotional prompt (for example, we give it a point on the Thayer-model 2-dimensional space, and it outputs a spectrogram). Then we train the generator network, until it is able to successfully fool the predictor (generate spectrograms based on its prompt, that are close enough to the predictor’s predictions). We continue training the network with each other. At the end of this process we will have two networks - one that can predict emotions based on a musical phrase (like we did in this experiment), and another that can generate musical phrases based on emotional prompts.

This novel approach will allow us to explore the factors the machines take into consideration, in a much less “supervised’ way. Removing the stage of a curated testset, will allow us to reduce human intervention in the examination stage, and to get direct answers to our queries. We could simply ask the generator - give us a happy musical phrase, and it’ll generate one. We can then look at the generated pitches, harmonies, volume, tempi etc.

Seeing the potential of our predictor, I have also developed an interface that will allow musicians to practice their emotional expression.

In this practice tool, a musician could get live and immediate feedback of the emotion an audience would have experienced from their played phrase, without the need of performing in front of an audience. Since the times a young musician spends in front of an audience are few and far in between, this will allow the musician to get ample feedback on their emotional expression, from the comfort of their own practice space. This following video is of a live demonstration of the practice tool, which is also available through the GitHub Project I have publicly shared: