This is a project funded by the Department for Business Innovation and Skills (BIS) and the Technology Strategy Board (TSB) and managed by JISC TechDis, a leading UK advisory service on technology and inclusion. Project started May 2011.
This represents phase one of a Small Business Research Initiative (SBRI) programme, which encourages technology solutions to specific public sector needs.
Gamelab has provided a gesture-recognition communication system for people with verbal impairments as part of this phase 1 SBRI initiative. Our development work has been as much about user engagement as it is about technology application.
Common gesture-recognition solutions revolve around a camera or multiple cameras. However, these camera/webcam-based solutions require heavy levels of image processing on the entire image after every frame capture for background subtraction and extraction of useful 3D data of the user. Furthermore, the hand exhibits ificant amounts of articulation and self-occlusion that can cause difficulties with tracking; while interactive applications require that hand tracking perform in real-time. Therefore any solution requires quick processing and output. Our decision to focus on Kinect as our core technology will provide themost effective technological solution to depth-information processing, cost, availability, and ease of installation and use.
Our development has provided the genesis of a gesture recognition system for Makaton- which will operates on a finite set of key lexical items supporting basic physical and emotional needs. We have applied “Natural Interaction” tracking to gesture recognition through sensors such as Microsoft Kinect working with a finite set of Makaton-like gestures supporting emotional and physical needs through a user-centred approach.
You can now see some of our user feedback from user trialling sites.
Our objectives on the uKinect project have been two fold:
1. To identify suitable signing/ gesture recognition contexts for young adults who use Makaton to various levels of competence, to support learning and confidence-building, and to encourage social engagement and employability.
2. To research and develop a suitable gesture recognition environment to engage such users using the new Kinect technology from Microsoft.
The questions we laid down at the start of the project were as follows:
1. Can this novel technology provide a suitable signing / gesture recognition system?
2. Would such a system be appropriate for learning disabled Makaton users?
3. In what contexts would such a system be useful for the defined target users ?
4. Could such a system support our target users in employment and independent living?
BORIS - OUR RESPONDING AVATAR – User signs in EAT and Boris EATS
We have delivered to all of our objectives:
1. Our user-research and user-testing work has identified contexts for young adults (and additionally older adults and younger students) who use Makaton in which sign and gesture recognition would be both useful and appropriate
2. Our research and development in gesture recognition, and in gameplays to engage our target audiences with this recognition, has delivered sign recognition over a 6 sign vocabulary (selected through user-reearch) which is made engaging to users through gameplays (inspired by user-research), resulting in the creation and evaluation of our ‘Boris’ proof of concept .
3. Our work looking into Kinect’s Microsoft SDK and Open NI has proven that each are suitable for signing and gesture recognition, and has identified the relative value of each as competing technical frameworks for further development in Phase Two.
4. Our evaluation of our proof of concept across a broad range of users – including various competencies with Makaton; both ambulant and wheelchair usage; both fully able and restricted motor ability (and its impact on signing quality and clarity); and at various different ages – and their supporters (key-workers, speech and language therapists, teachers, tutors and parents) has proven the appropriateness of our approach for learning disabled Makaton users.
5. Parents, teachers and speech and language therapists (SaLTs) have all suggested ways where extensions of our proof of concept could help learning disabled Makaton users make an easier transition into independent living employment. We intend to work more on this, especially from an employer perspective, in a potential Phase Two of the project.
More Videos:
uKinect’s potential for aiding employment
uKinect’s potential for use in the classroom
uKinect’s potential for aiding Independent Living
What students and staff think about uKinect
