Chinese Sign Language Recognition
With the rapid expansion of computer's influence in the
modern society, high performance, high intelligence and high
usability are generally regarded as the main trend of the
current computer science development. Especially, under the
rapid developments in computing technology, communication
and display technology, the increasing limitations of the
traditional human computer interaction technology based on
mouse and keyboard become more and more apparent in the updating
display technology, virtual reality and wearable computer.
The research goal of multi-perception machine technology is
to develop the core technology to resolve the problem of high
intelligence and high usability of computing device and to
create the harmonious and natural human-computer interaction
(HCI) environmentThe key issue is to make the computer precisely
perceive the different human expressing means including human
natural language, gesture language and facial language. Sign
language recognition (SLR), as one of the important research
areas of human-computer interaction (HCI), has spawned more
and more interest in HCI society. The aim of sign language
recognition is to provide an efficient and accurate mechanism
to transcribe sign language into text or speech so that communication
between deaf and hearing society can be more convenient. From
a user's point of view, the most natural way to interact with
a computer would be through a speech and gesture interface.
Thus, the research on sign language and gesture recognition
is likely to provide a shift paradigm from point-and-click
user interface to a natural language dialogue-and-spoken command-based
Sign language as a kind of gestures is one of the most natural
ways of exchanging information for most deaf people. Chinese
sign language (CSL) is a language of choice for most deaf
people in China. Chinese sign language can be classified into
two categories. One is finger spelling, and the other is hand
gesture where each gesture corresponds to one Chinese word
or phrase. Currently, the CSL dictionary contains about 5500
conventional Chinese gestures including postures and gestures.
The research on SLR has many applications, for example:
1) Sign language recognition make the communication between
the hearing disabled and the hearing abled possible
2) From the cognitive point of view, the research on the mechanism
of understanding human vision language can improve the computer
intelligence to understand human language.
3) Agent of virtual reality can be controlled by hand gesture.
4) Demonstration learning of robot.
5) Multi-modal interface in virtual reality and augmented
In summary, the research on sign language recognition not
only has theoretical values, but also wide application areas.
Research Focus/Issue (Top)
SLR can be classified into two classes according to the devices
used to capture gestures, i.e. vision based SLR and Dataglove
based SLR. Vision-based SLR utilizes cameras to capture the
video (frame) images of hand gestures. The advantage of this
approach is that the signer does not have to wear any complex
powered input devices and the disadvantages are its instability
and impreciseness due to poor illuminant conditions and limited
computing power in popular computers. Furthermore, the vision-based
SLR has a difficult time performing the task of large vocabulary
SLR, because many technical issues on image understanding
are still open or need to improve. On the contrary, Dataglove
based SLR measures hand gestures using direct devices such
as Datagloves and position-trackers. The advantage of this
approach is that it captures gesture data robustly and extracts
features for further recognition in real-time using less computing
power and the disadvantage is that the signer has to wear
For the purpose of the widespread use of large vocabulary
SLR system, two research works are in progress as follows.
1. Large-vocabulary signer-independent isolated sign words
and continuous sign sentences recognition system over a vocabulary
of 5100 signs using the Cyber-gloves and position-trackers
as data input devices.
a) Effective feature extraction from different signers. (PCA,
transformation to frequency domain)
b) The solution to the movement epenthesis problem. (Transition
c) Compact training sentences for a general model.
d) Minimum unit definition in SLR and its extraction.
e) The use of statistical sign language models.
f) The utilization of non-manual parameters in sign language.
Non-manual parameters in sign language include gaze, facial
expression, mouth movement, position and motion of the trunk
2. Medium vocabulary isolated sign language recognition system
using PC camera as input device, where gesture features information
is extracted from gesture video (frame) images.
Research Achievement (Top)
The project of Chinese SLR is one of the sub-projects of
"muti-modal perceptron machine". This research was
supported in part by Natural Science Foundation of China (Grant
No. 69789301), National Key-Basic Research Initialize (Grant
No. 2001cca03300) and National High-Technology Development
'863' Program of China (Grant No. 2001AA114160). The projects
have been authenticated and won the second prize ofscience
and technology from Beijing. One patent has been applied and
now in review.
In signer-dependent SLR, the average recognition rate of 94%
is achieved on the recognition of 5100 CSL isolated signs
and 90% on the recognition of 1000 continuous sign language
In signer-independent SLR, the average recognition rate of
91% is achieved on a vocabulary of 5100 signs, where signer
data are collected from 6 signers. The best accuracy of 91.3%
can be gotten for continuous sign language recognition.
For vision-based sign language recognition, an accuracy of
about 90% on the vocabulary size of 450 is achieved with the
aid of colored cotton gloves.