In the ML2 motion lab at Gallaudet University, things are getting very exciting. Melissa Malzkuhn and her team have been busy creating and capturing special ASL nursery rhymes so they can be turned into 3D avatars for deaf and hard-of-hearing kids!
Nursery rhymes are a great way for kids to learn language, because they are simple and their rhymes are easy to remember. But for deaf or hard-of-hearing kids, the rhymes don’t cross over well. Rhymes are based on sound, while American Sign Language is visual, with many different motions and gestures. So, the team from ML2 decided to make their own nursery rhymes, written especially for ASL, and have them performed by an animated 3D character (avatar).
Rhymes written for ASL are very different than those written for spoken language. Instead of relying on sound like English language rhymes, the ASL rhymes are built around handshapes that look the same or similar. The signs are captured in 3D, which makes the avatar much more realistic and easier to understand. “Sign language is a 3-D language,” Malzkuhn says. “We use the space in front of us, the space around us.”
To create the ASL avatar, Malzkuhn wears a black suit with light reflectors on it. While she moves her hands and arms to perform the ASL rhymes, cameras in the room capture all her motions, called “temporal rhythms.” The information about the motions is then sent to a computer that will use it to build the 3D avatar, complete with hand gestures and facial expressions.
The ASL avatars may not be ready yet, but you can still enjoy some of the ML2 lab’s great apps and tools for deaf kids. Their website, vl2storybookapp.com has four different stories told in both American Sign Language and English, with more on the way. ML2 is also going to release “The Storybook Creator” soon, which will let anyone create their own ASL storybooks, even if they have never programmed a computer before!