Six difficulties we faced when thinking about a sign language avatar

Imagine it's the year 2020, and the Covid pandemic is in full swing. You, however, don't have access to most information about the coronavirus, because it's not available in your mother tongue: sign language. This scenario was a reality for many deaf people in Germany (and a lot of other places).

Many deaf people have significant reading problems as they can't connect letters to sounds and vice versa. Written content and subtitles therefore aren't fully accessible to them. Experts estimate that some 80 million people worldwide are deaf. At the same time, there's a lack of sign language interpreters (see media reports here and here), especially in DW's target regions – and fake sign language seems to be spreading.

Could a sign language avatar created with the help of 3D and AI technology be a solution here? We talked to experts and representatives from the community, looked at prototypes, analyzed research papers, and hosted a workshop. And it turned out that creating a sign language avatar is way more complex than we thought.

Here are the reasons why:

There is a lack of (standardized) datasets.

Developing sophisticated sign language technology with the help of Machine Learning would require large amounts of data. There's not exactly a lot of data available, though. The datasets we could find mainly focus on American Sign Language (ASL) and German Sign Language (Deutsche Gebärdensprache, DGS). There wasn't anything relevant to our main target regions, such as India or Brazil. Furthermore, we discovered that there are no common standards yet, making it complicated or even impossible to combine datasets.

Avatars don't interpret accurately enough (yet).

According to academic research (e.g. here and here), the complexity of sign language, i.e. the interplay of facial expressions, gestures and body posture, can't be represented adequately by current avatar technology. Deaf people therefore have trouble understanding the digital entity signaling to them.

Avatar vocabulary is still limited. 

The more complex the content, the more complex the sign language vocabulary – which means avatar creators need even more data. Scientific or journalistic texts are more difficult to interpret than everyday communication. Because of this challenge, many sign language avatar pilots have used a very limited vocabulary, like the one used in weather or traffic reports. Even though we were impressed by this Japanese prototype, we found out that even after years of research and training, Kiki is mainly used for announcing the weather.

Scaling is hard, if not impossible.

DW provides content in 32 languages. We initially thought that if we found a technical solution that works for one sign language, we could apply it to all the others as well. How wrong we were.

There are roughly 300 sign languages around the world, and there's no international standard understood and used by a majority of deaf people (even though International Sign sounded promising at first). Quite often, different communities in one country use different sign languages. And as each sign language is a combination of different facial expressions, gestures and body postures, each AI model needs to be trained individually.

Even the big players don't have a solution yet.

Google, Microsoft and other tech giants with deep pockets face similar challenges. Up to this day, they haven't been able to develop acceptable solutions yet, i.e. well designed sign language avatars that are understood by a wide range of deaf people in different countries. This also makes it quite clear that DW can't face this challenge alone, let alone master it.

Sign language avatars are widely rejected by the community.

In addition to the rather low level of comprehension, communities usually do not feel represented by an avatar and consider other solutions for accessing information in sign language to be far more sensitive and suitable. This is actually one of the strongest arguments against developing a sign language avatar: It doesn't seem like a good idea to put resources into a technology that is likely to be rejected by the community it addresses.

And what now?

After discussions with members of the deaf community one thing became clear: There's a great need for content in sign language, but it's important to avoid ableist product development by hearing people for deaf people. 

Developing and producing in cooperation with deaf people is the cornerstone of a user-friendly design approach. It makes sure the community is seen, understood and represented. 

So instead of continuing to focus on a technical solution, we've come up with a social one: Handing over the project to the Digital Format Development unit.

Our colleagues now plan on developing a digital format with and for the deaf community – with the help of human sign language interpreters.

Special thanks to: Pauline Beck (who also contributed to this post), Daniela Verztman Bagdadi, Isabell Reichardt, Lynn Khellaf, Philip Kretschmer, Lars Jandel, Jens Röhr, Marie Kilg, and everybody else who worked on the sign language avatar project.