Posted on July 5, 2024 by UTSA School of Data Science
AI and music mix at SDS
A UTSA art professor is partnering with the UTSA School of Data Science (SDS) to develop a musical instrument powered by artificial intelligence (AI). Mark McCoin, an associate professor of art and director of new media with the UTSA College of Liberal and Fine Arts, is the creator of Robotic Resonance (Roz) an experimental project that combines music and robotics.
From the very beginning, McCoin has wanted Roz to hear itself, to understand what it’s hearing, and adapt how it plays accordingly. McCoin believes that the work done with Roz at the School of Data Science will have applications in the field of machine listening that extend far beyond robotic piano harps.
“That’s what interested the School of Data Science, and that’s why we’re here now: to develop a form of artificial intelligence which will help it hear what it does and respond to that,” McCoin said.
Essentially, they’re teaching Roz to improvise, something humans have been doing for millennia, but which computers are notoriously incapable of. To accomplish this, McCoin takes turns improvising on the machine with Nathan Wheeler, his longtime collaborator and Roz’s lead technical advisor. At the same time, they note various elements such as volume and density of sound, plus information like the speed of Roz’s motors and the position of its arms. All of this comes together to form a dataset of auditory and mechanical information that they’ll use to train Roz to make its own music.
But how do you train AI to decide what’s musical when that’s an inherently subjective assessment? This has proven a unique hurdle, says Wheeler.
“This is our art piece, and we want it to be representative of the things we like and prefer,” Wheeler noted, “but we don’t want it to be weighed so heavily in that direction that it just sounds like one of us playing. Dealing with subjectivity and objectivity in an artistic sense while trying to figure out how to train this machine to give us predictable results has been really challenging and interesting.”
Assisting McCoin and Wheeler in this task are a number of Ph.D. students with the School of Data Science.
“The students are great,” McCoin said. “They’ve brought things to the table that we weren’t aware of, and they look at things in ways that we never would. They’ve helped us understand our ratings system in ways that make us more objective, and that’s been very helpful.”
Roz is scheduled to remain at San Pedro I until sometime in November. Once its training is finished, McCoin has high hopes for his newly educated apparatus. He’s explored the idea of taking Roz on tour, perhaps recruiting other musicians to perform and improvise with Roz live onstage. There’s even been talk of incorporating Roz into an opera.
“I think it’s fascinating how this can interface with the world,” McCoin said. “It’s not stuck as an art piece, or stuck as a performative piece, or even as a scientific-interest unit. It can be all of these things.”
While Roz looks to have a bright future ahead of it, both McCoin and Wheeler recognize that the use of artificial intelligence to compose and perform music will forever change our society.
“At this point everybody’s got their version of AI and it’s going to proliferate in ways that are going to affect us not only digitally but in our day-to-day lives,” he said. “What is that going to do? What are the energy concerns of this? How is this going to affect our world, our environment? That’s the frightening thing to me.”
As a member of the U.S. Artificial Intelligence Safety Institute Consortium, UTSA and the School of Data Science are an ideal place for ideas like Roz to grow and take shape. McCoin and Wheeler extend their heartfelt thanks to SDS faculty, staff, and students for the support they have received during their time at SPI.
“We’re very grateful to the School of Data Science for recognizing the value of Roz as both an art piece and an interdisciplinary idea that could be adapted to help students here,” McCoin said. “I’m learning so much and growing in this environment. It changes the way I think about education, and how I will go back and teach my students. Being able to have this opportunity is a gift for us.”
More information on Roz and its creators can be found at https://markMcCoin.com/ and https://www.wheelersounds.com/.
-Christopher Reichert
— UTSA School of Data Science
