Although Mexico has no official figures on the number of people who use Mexican Sign Language (LSM), which is primarily used by the Deaf community, an estimated 300,000 people communicate through it, representing approximately 0.23% of the country’s population.

“What we need are tools for teaching sign language,” says researcher Vania Lara, who developed a virtual robot model capable of reproducing LSM movements and signs.

The project is part of her doctoral research at the School of Engineering and Sciences (EIC) at Tecnológico de Monterrey, under the supervision of Jorge Isaac Chairez and Rita Fuentes, researchers at EIC and the AI for Manufacturing and Supply Chain Institute (AIMS).

The idea took shape when Lara discovered InterSeña, a free mobile app designed to make learning LSM easier. She began wondering how robotics and artificial intelligence (AI) could complement the app by reproducing the language’s movements as closely as possible to the way a person would sign.

Driven by that question, she developed a digital model of a robotic hand capable of forming several letters of the LSM alphabet. She later expanded the design into an avatar with a head, torso, and two arms, allowing it to perform a much wider range of signs, words, and phrases.

A Virtual Model That Reproduces Mexican Sign Language

When Lara began designing the virtual robotic hand, she envisioned it as a digital twin—a model that would allow her to run simulations and evaluate movements before implementing them in a physical device.

Her design included the fingers, wrist, and forearm, with a total of 25 degrees of freedom, allowing it to perform different movements by positioning the joints in various ways.

The researcher successfully programmed the model to reproduce up to 21 static hand configurations from the LSM alphabet, used in fingerspelling, a technique that allows people to spell out words through specific hand shapes and movements.

To validate the model’s performance, she enlisted the help of ten volunteers familiar with LSM. They were shown simulations of ten words spelled out using the language’s manual alphabet to assess how easily the signs could be recognized.

“We validated it with LSM users and achieved 95% accuracy. In other words, the model closely resembled authentic Mexican Sign Language,” the researcher says.

Building on this initial phase, Lara realized that reproducing LSM more faithfully would require expanding the original robotic hand model into a full avatar featuring a torso, both arms, and a head.

As the model evolved, it progressed from fingerspelling to performing signs representing entire words and phrases, such as “Hola, ¿cómo estás?” (Hello, how are you?), which were validated during testing with members of the Deaf community. Despite this progress, Lara acknowledges that the avatar still has limitations, particularly its lack of facial expressions, which play an essential role in sign language.

Automatic Translation from Spanish to Mexican Sign Language

Lara explains that reproducing movements with the avatar is only one part of the project. For a robotic system to communicate phrases in sign language, it must also understand the language’s structure, which, like other languages, follows its own grammatical rules.

The researcher developed a corpus of 3,000 pairs of Spanish phrases and their equivalents in LSM glosses—written representations of signs that allow researchers to study the language’s structure and use it in natural language processing tools.

“We found that there was nothing available for LSM, not a single corpus. So I took it upon myself to build one from scratch,” says Lara, who spent more than a year working on this part of the project.

Using the newly created dataset, she adapted and trained an artificial intelligence model based on an architecture known as a Transformer, which can learn language patterns, to automatically translate Spanish phrases into LSM glosses.

She also evaluated the AI model with the help of a professional LSM interpreter, achieving 94% accuracy in its translations.

A Tutor for Learning Mexican Sign Language

Although the project is still in development, the researchers see the technology as a potential tutor that could teach people Mexican Sign Language through an avatar or, eventually, a physical device such as a robot. At one point, they even considered developing a glove that could help users build muscle memory as they practice signing.

“Imagine having a robot standing in front of you instead of looking at a computer screen,” says Jorge Isaac Chairez. “It could move its fingers, and you could even get closer to see exactly how it positions its hand.”

Meanwhile, Rita Fuentes explains that the project aims to complement existing technologies, such as sensor-equipped gloves that recognize signs and convert them into words. In this case, the goal is to reverse the process: starting with Spanish and generating signs through an avatar.

“I think we envisioned it as a tutor precisely because we realized that solutions already exist, such as translation gloves, that work in one direction. What’s missing is a way to go in the other direction,” Fuentes explains.

“I think we still need to expand the pool of people who can evaluate the system so we can achieve more robust validation,” Chairez says.

Looking ahead, the team plans to incorporate voice recognition capabilities and integrate the system with other technologies to translate signs into words and convert speech into sign language.

To bring the virtual model into the physical world, Chairez explains that one possibility would be to use commercially available robots and adapt the algorithms developed during the research to control their movements. This approach could pave the way for a tool that people can interact with directly, without having to build a robot from scratch.

Did you find this story interesting? Would you like to publish it? Contact our content editor to learn more: marianaleonm@tec.mx