PhD project
Signs on Paper: Unlocking the Histories of Sign Languages with AI
This PhD project investigates how automatic sign language recognition technology can be further developed to analyse static images and textual descriptions of signs.
- Duration
- 2024 - 2028
- Contact
- Nargess Asghari
Despite the growing number of video based dictionaries, most sign language dictionaries exist in print format. These print dictionaries, which include those of lesser-documented sign languages as well as historical ones, offer a valuable source of information for cross-linguistic and diachronic lexical comparisons. Yet, these resources have largely been overlooked in sign language recognition research, where there is a reliance on video data to capture the dynamic nature of the signs. This PhD project explores new methods to address this gap by applying artificial intelligence and computational methods to the recognition and analysis of signs in print.
In print dictionaries, signs may be represented in still images, such as drawings (fig. 1) or photographs (fig. 2), or as textual descriptions that explain how the signs are produced (fig. 3). In static images, the dynamic nature of signs is lost, and in text, the visual aspect disappears entirely. The different ways of representing visual and dynamic signs in print present a significant challenge for their automatic recognition. However, recent advances in multimodal artificial intelligence offer new possibilities for processing these sign representations.
Development of automated methods for processing paper based sign language data can facilitate large scale, quantitative analysis of lexical variation in sign languages around the world. This, in turn, can provide valuable insights into the histories and relationships. Furthermore, by bridging the gap between print and video sign language resources, this research will lay the groundwork for creation of a unified framework for automatic recognition and comparison of sign languages across multiple data formats—including static images, text, and video.
Image Descriptions
(1) Black and white drawing of a signer in 19th-century attire, signing PIGEON (‘pigeon’) in French SL: right hand in a clawed handshape (palm facing the body) is held in front of the chest; two straight arrows indicate repeated movement away from the body.
(2) Black and white photograph of a signer signing CÂMARA-DE-FILMAR (‘film camera’) in Guinea-Bissau SL: two hands depict the way a camera is held in front of the face.
(3) The text reads:
“Moeder: Men brengt de vlakke l.hand, rug omhoog, vingers naar het oor gekeerd boven de l.schouder (Zie: mama)”
“Mother: one brings the flat l(eft) hand, back upwards, fingers turned to the ear, above the l(eft) shoulder (See: Mama)”
References
Martini, M., & Morgado, M. (2008). Dicionário escolar de Língua Gestual Guineense. Surd'Universo. https://www.researchgate.net/publication/341494844_Dicionario_escolar_de_Lingua_Gestual_Guineense
Nyst, V., & Schuller, A. (2024). Mother left, father right: Artificial signs and diachronic change in sign language dialects in Belgium and the Netherlands. Diachronica, 41 (2), 251–298. https://doi.org/10.1075/dia.21052.nys
Pélissier, P. (1856). Iconographie des signes: faisant partie de l'enseignement primaire des sourds-muets. Paris: Imprimerie et libraire de Paul Dupont. https://gallica.bnf.fr/ark:/12148/bpt6k131991f.texteImage