At the Northwestern University Department of Neuroscience, the Robotic Marionette Project has been in the works for some ten years. I believe pianists can gain some insight from it. Here’s how the cross-disciplinary assortment of involved scientists summarize their chosen problem:
The control of classic, stringed marionettes is an extremely difficult control problem. . . . When a human puppeteer controls a marionette they inherently comprehend this complexity, and they compensate for it in real-time. The language of choreography only coarsely describes the play, and it is up to the puppeteer to figure out what mechanical inputs will produce the realistic puppet trajectories.
I know, I verge on making an insulting and naive comparison!–but please bear with me. Realistic movement in a marionette involves a lot more than how many joints it has, or how many strings have been appointed to control them. Sometimes puppeteers choose to play with fewer strings than there are joints, spontaneously making use of the “play” that results from gravity and momentum among other sources. Called “underactuation,” this often creates more realistic movement. To give you an idea of underactuation translating into artistry, have a look at this “career montage” for the brilliant puppet artist Phillip Huber, creator of the marionette scene in Being John Malkovich. (Some of this shows his puppetry and some doesn’t. Be sure to note the brief first clip featuring a keyboardist marionette!)
Let me give you an idea of some of the other less-understood movement parameters that the most well-practiced puppet artists know to take in stride–but that pose major headaches for AI scientists wishing to create logical movement instructions. (A few of them may recall for you something about your own art.) Puppeteers must deal with surprising outcomes that result from changes to string tension. They must be on the lookout for random movement oscillations and other kinds of errors to desired movements, and create adjustments on the fly. They must be prepared for strings getting tangled and create a workaround without skipping a beat. All the while, they must create flawless continuity among various movement types to create the illusion that a wooden toy has been brought to life.
It would appear to be precisely such movements generated from outside the robotic control system that most challenge the RMP. (You can have a look at the project’s Vimeo page for a sense of how painstaking this research is if you would like.) Some of those movements, furthermore, would quite possibly be the very ones harnessed by the artful puppeteer for a lifelike effect. The very intangibles that a puppet master knows how to use to create the sense that the puppets are living beings, most stymie the logic-minded.
It is truly daunting to remove the human energy that animates a wooden marionette, and replace it with a logic, even if that logic receives and reacts to feedback from the thing being controlled.
Though I am happy to confirm that artists still have a formidable edge on robots in terms of creating beautiful movement using stringed puppets, that would make for a really boring post. Perhaps you’ll thank me, also, that the metaphor to be examined here is not precisely one of marionettes for people playing the piano. (Phew!) Indeed, when scientists try to create a set of instructions to render marionettes lifelike using robots, we get a sense of exactly how lame that particular metaphor might be. No–the metaphor really worth pursuing in all of this revolves around that “extremely difficult control problem” that we pianists share with the scientists:
- How does the pianist get a stored logic of movement to flirt with the random and accidental for a poignant musical communication?
- How can a pianist rally her internal resources to that end? Both during the learning process, and during performance?
- Does the “instructor” have a role to play on this slippery side of skill?
- How does instruction change if you assume a goal of having these internal resources factored in, even if the pianist seems “wooden?”
- What might the parameters be for forming the human energy for a successful musical experience?
Though as pianists we work with “realistic” (what we tend to call “musical”) and lively organized sound rather than realistic movement, I think these questions suggested by the RMP are of utmost importance for us, too. But the scientists have probably done a better job of formulating a problem for themselves than most of us have. If we don’t have a clear understanding of what problem we are solving, we might not even know to pursue a solution.
I sometimes think that “talent” is our response to “we don’t understand the problem, let alone the solution.” This blog is very interested in exploring the various facets of that murky term because we believe it is part of the feedback that gums up the machinery, and a part of why we aren’t that joyful a musical people. (For me, that’s a problem!) In the meantime, we could take a cue from these scientists who, if they don’t exactly understand how to solve their problem, have a pretty good idea of what it is. We could take a cue from them and work toward better formulating the piano problems we are trying to solve. This applies to teachers and it applies to students.