Detecting and comparing sign languages
For his PhD project, computer scientist Manolis Fragkiadakis is developing a tool that can compare videos of sign language corpora. This would make it possible to detect differences between sign languages and prevent translation errors. Ultimately, the tool could be used to compare sign languages from all around the world.
The Data Science Research Programme at Leiden University combines data science with PhD projects in a wide range of disciplines. The programme has been running for over two years, and is producing the first astonishing results. We discuss some of them in this series of articles.
There are 130 known sign languages and dialects in the world. It can be difficult for linguists to translate these sign languages, as each language or dialect has differences in handshape, hand position, movement size and more. To analyse and document these differences, you have to watch videos of the different sign languages. This is a difficult and tiring task for a human, so mistakes are likely to be made.
For his PhD project, computer scientist Manolis Fragkiadakis is developing a computer tool that uses machine learning and deep-imaging techniques to compare videos containing sign languages. To develop the tool, he is using a collection of four African sign languages that were compiled at Leiden University.
What are your results so far?
‘At the moment, our systems are able to watch a video and recognise when a sign language is present, and whether this language uses one-handed or two-handed signs. We’re now looking at recognising hand shapes. This is extremely difficult. The videos are of a very low quality: the images are often blurred and the lighting is poor. Furthermore, our systems are having a hard time recognising black hands, which are being used because we’re dealing with African languages. I’m currently waiting for the results of a new system that should be able to recognise finger joints in an image. Joint identification would make it easier for computer tools to recognise hands and hand movements.’
You’re now two years into your PhD project. What do you hope to achieve in the next two years?
‘Obviously, we’re hoping to develop a tool which can be used by linguists to compare sign languages. We’re also hoping to develop a tool that can tell us more about how sign languages are constructed. Then you can extract a lot of new knowledge about sign languages and their origin as well as about the history of people using a particular language.’
Could you describe the collaboration between you and the researchers at the Faculty of Humanities: what are you learning from them? And what are they learning from you?
The collaboration with the Faculty of Humanities has been excellent. My goal as a data scientist is to promote using computational approaches to assess Humanities research. The creative and culturally oriented research by Humanities’ scholars creates interesting challenges for us. The critical perspective of the Humanities adds a real depth to our tools.
What is the benefit of the Data Science Research Programme?
‘It’s extremely useful for a computer scientist like me to see how data science is applied in other fields, from start to end. This gives you information about the kinds of tool that these fields need, the problems researchers encounter when using such tools and how data science systems are used. Furthermore, data science is a very broad field. This means that results of using data science in one scientific research field may be relevant to another. The Data Science Research Programme is so strong because it offers all these sorts of interactions.’
The Data Science Research Programme is a University-wide programme that aims to advance data science research and accelerate the use of data science methods at all faculties of Leiden University.