The Unfinished Symphony revisited or: How creative can artificial intelligence be? How can that be done? A brief introduction.
The project launched to complete Beethoven’s 10th Symphony focuses on one of the core questions of relevance to artificial intelligence (AI), namely: Can algorithms be creative? Or, put another way, what distinguishes us as human beings from machines?
The technologies which help us navigate our way through cities, that tell us which bus or train to take, and even advise us about online shopping choices – are they able to do even more? Can they write novels, paint pictures or compose music? In other words, can they be genuinely creative?
And if so, what does this mean for our understanding of art in general?
There are already several examples of creative AI. The most well-known is the “Portrait of Edmond Belamy”, a painting that was sold at auction in 2018 by Christie's for 432,500 USD. AI solutions have also written screenplays, composed a Beatles song, and completed unfinished symphonies by Schubert and Mahler.
But as yet, no one has attempted to apply AI to Beethoven’s 10th Symphony. However, as the 250th celebration of Ludwig van Beethoven’s birth approaches, Deutsche Telekom and a group of international music and AI experts, along with researchers from the Beethoven-Haus in Bonn, will attempt to do just that. Following the huge success of his 9th Symphony, Beethoven continued to work on another symphony during the later years of his life. At the time of his death, only a few notes and musical sketches were discovered – none of the composition pieces were complete. Now, with the help of AI, the project experts are striving to compile all of the fragments in an effort to bring a representative version of the 10th Symphony to life.
The creative process
Integrity and professional expertise are essential to the success of the project. And that is why the project team is comprised of leading musicologists and experts from the world of AI. Under the leadership of Dr. Matthias Röder, The Mindshift and Karajan Institute, the following people have been engaged in the project:
- Prof. Robert Levin, musicologist at Harvard University
- Prof. Dr. Ahmed Elgammal, AI expert at Rutgers University and Artrendex Inc.
- Professor Mark Gotham, T.U. Dortmund
- Walter Werzowa, Composer
- Prof. Dr. Christine Siegert, Head of Research at the Beethoven-Haus Bonn
“For this project in particular we tried to understand what is the state of art in music generation. And tried many things to see what are the limitations. What we ended up with is using some modules that are inspired by natural language processing. Which has been adapted for music generation before but we tried to push this further and create longer and longer sequences and understand music structure at a different level. To the point that we can really generate sensible music,” says Prof. Dr. Ahmed Elgammal.
To achieve all of this, the data from Beethoven’s artistic legacy – symphonies, notes, musical sketches and scores – needed to be analyzed and converted into machine-readable form. Then the most applicable machine learning method was chosen, with the algorithms being finely tuned for this particular task. Algorithms for voice processing were selected for this approach because music, like language, is comprised of small units – letters or notes – which result in meaning when they are put together. This understanding of the overarching context was essential to the experiment.
The “Beethoven-AI”
Ultimately the experts developed a system that “understood” Beethoven’s style. This “Beethoven-AI” was developed further by Prof. Dr. Ahmed Elgammal and Artrendex Inc. and Dr. Mark Gotham so that the existing fragments of the unfinished symphony could be compiled and expanded into meaningful musical movements which mirror the style of Beethoven’s work. The smallest of units were used from the very beginning so that the AI application could continue the composition process with only a few notes. The suggestions made by the AI application were analyzed by the musicologists, who then selected the best alternatives and played them back into the system. Then the task was repeated, and a few more notes were added. In other words, the musical work grew longer on a step-by-step basis.
“AI allows us to come up with a continuation of a musical movement in 20 or even 100 various versions. And that is simply amazing, because if this is done with algorithmic precision, then every experimental result is plausible,” says Prof. Levin when describing the advantages of collaboration with AI. It is the task of the human experts to choose the best of the alternative results offered.
Many of the ideas that Beethoven wrote down are very abstract, and a music system may not always understand them. For example, the team of experts discovered ideas for a chorale among the papers, or just descriptive words for the form of the work. The tasks for AI then have to be defined on the basis of such historical fragments.
“You just have to imagine that Beethoven wrote down some notes the very moment he came upon a new idea. Sometimes these notes are written words, sometimes they are actually musical notes. Often they are fragmentary, and occasionally they appear to be more complete and detailed. We just have to make our assumptions based on this material. How would Beethoven probably have continued some of his ideas? That question is particularly important to the overarching form. Or the sequences in a musical composition. We have no other choice than to base our decisions on the material that Beethoven left for us. And once we have defined the overall form, AI fills the form with the postulated musical ideas,” explains Dr. Röder.
The expert team also defined the general structure for the final composition. Once AI has finished the gradual composition process, the various pieces are then weaved into the defined structure. The result is surprising: AI not only picks up Beethoven’s ideas, it occasionally even adds new ideas of its own – which is truly remarkable, even if the results sometimes elicit wry smiles from the experts.
AI compositions sound like monotone sequences. They resemble computer voices which frequently sound artificial and robotic. But when a human musician plays these notes, they come to life and can be interpreted through feelings and changes in tempo.
That is the responsibility of Walter Werzowa. His job as a composer is to enhance and embellish the AI piece so that it can be played by an orchestra with his interpretations for various instruments.
Human and machine collaboration
The “Beethoven AI Project” proves one thing above all: Only when human beings and machines collaborate according to plan can the huge potential of their combined creativity be unleashed! And isn’t that a form of art in its own right?
Professor Levin looks at it this way: “Art, or artfulness, is experienced in contrast to the natural realm. That which is art does not originate from nature. Thus you could say that art and AI are located on the same plane. Both of these thought processes would like to describe a piece of reality. You can say that the computer does this with algorithms. Yes. But humans do this based on their own experiences or education. In principle both sides are not very far apart from each other. The result always depends on the quality of the input. Beethoven is dependent on his love of Mozart, his study of Albrechtsberger and Haydn. It is the romantic side within us that wants us to deny the value of artificial intelligence. I would just say take it easy, slow down a little. Don’t go too fast in that direction.”
