Mobile App Communication Aid for Cypriot Deaf People

People with severe or profound hearing loss face daily communication problems mainly due to the language barrier between themselves and the hearing community. Their hearing deficiency, as well as their use of sign language, often makes it difficult for them to use and understand spoken language. Cyprus is amongst the top five European countries with a relatively high proportion of registered deaf people (0.12% of the population: GUL, 2004). However, lack of technological and financial support to the Deaf Community of Cyprus leaves the Cypriot Deaf people unsupported and marginalised. Despite advances in communication aid technologies, there is currently no technology available that supports Cypriot Sign Language (CSL) or real-time speech to sign language conversion for the Deaf people of Cyprus. This study implemented User-Centred Design methods to explore the communication needs and requirements of Cypriot Deaf people and develop a functional prototype of a mobile app to help them to communicate more effectively with hearing people. A total of 76 deaf adults were involved in various stages of the research. This paper presents the participatory design activities (N=8) and results of usability testing (N=8). The study found that users were completely satisfied with the mobile app and, in particular, they liked the use of CSL videos of a real person interpreting hearing people’s speech in real-time and the custom onscreen keyboard to allow faster selection of text input.


INTRODUCTION
Trying to perceive the speech of a hearing person is often hard and exhausting for people with severe and profound hearing loss (Munoz-Baell and Ruiz, 2000;Kuenburg, et al., 2016). Even if the hearing person articulates well, it is still difficult for a deaf person to maintain effective communication; they often have to depend on other nondeaf family members or friends who take the role of interpreter whenever they have to communicate with hearing people (Kyle and Cain, 2015;Hadjikakou et al., 2009). The communication problems faced by deaf people are primarily associated with the use of language; their mother tongue may be Sign Language (SL), whereas the first language of hearing people is a spoken language. In the general community, hearing people are unlikely to know SL, and deaf people cannot always understand the spoken language. This inability to communicate effectively is an obstacle for deaf people not only in their social life, but it also affects their employment, education and healthcare (Compton, 1993;Luft, 2000;Nunes and Pretzlik, 2001;Ubido, et al., 2002, Sirch, et al., 2016. According to the World Health Organisation (WHO), almost 5.3% of the world's population have a hearing loss greater than 40 decibels and, specifically within Cyprus, there are more than 1000 people with severe and profound deafness (CDF, 2017). Cyprus ranks amongst the top five European countries with regard to the percentage of population who are Deaf, with 0.12% of the population registered as deaf (GUL, 2010). Although Cyprus is a developed country, the government of Cyprus provides no financial support to the Deaf Community of Cyprus (Hadjikakou et al., 2009). Furthermore, due to only recent recognition of Cypriot Sign Language as the official sign language of the Deaf of Cyprus, it is not supported by any of the current communication aid technologies.

Existing real-time STT/STSL mobile technologies for deaf users
In recent years, various technologies have emerged to support communication between deaf and hearing people. One of the most recent commercially available communication aid technologies for the deaf is "Uni"; a portable device that converts American Sign Language to speech, and speech (English spoken language) to text and enables users to create and upload custom signs (Epstain, 2015). Key advantages of the device is that it can be used on the move, and offers real-time interpretation of SL and spoken language to hearing and deaf individuals respectively (MotionSavvy, 2017). However, it requires use of a tablet device that is too big for the user to hold with one hand while using the other hand to communicate, and it does not include facial movements or expressions, which are important elements for the interpretation of the signs (Woll, 2001;Elliot and Jacobs, 2013).
ASRAR (Mirzaei, et al., 2012) is a mobile device that recognises the speech of a hearing person and converts it into text for the deaf user to read. It also consists of a camera and a face recognition software, which are used to capture the face of the speaker and display it on the device's screen. In addition, the deaf user can type texts using an integrated keyboard to communicate with the hearing person. ASRAR proved to be over 85% accurate in its speech to text conversion and the majority of the users expressed their interest in using ASRAR system as a communication aid. However, this device assumes that deaf users do not experience reading and writing difficulties and thus, it does not present the text using sign language requiring the deaf people to read text content, which is often inaccessible to the majority of the Deaf (Wauters, Van Bon, and Tellings, 2006).
A mobile-based application supporting communication between deaf patients and pharmacists allows the pharmacist to input text which is then translated into SL (Motlhabi, et al, 2013). The information given by the pharmacist is presented on the mobile screen via video enabling the deaf person to get detailed medical instructions as well as to take the correct medicine. However, the application does not enable the deaf user to communicate back to the pharmacist.
Ava is a mobile app that enables deaf people to host group conversations with others and offers real-time captioning for each member of the conversation. It can also be used in a one to one conversation in which the speech of the deaf user's interlocutor is converted to text (Ava, 2017). However it does not present the content in signing, requiring the deaf users to read in English. Furthermore, it does not enable the deaf users to respond either by interpreting their SL or by allowing them to write something using a custom keyboard. Furthermore, it requires all members of the group conversation to install the application even if they are not deaf and they are required to pay a subscription to use the application effectively and be invited into a group conversation.
A more recent mobile application called LifeKey was designed to enable deaf people to report an emergency event quickly without the need of relying on a non-deaf person. A custom onscreen keyboard allows the deaf user to summarise the emergency by tapping through different categories (type of incident and exact location within the area). The users can also set up a custom introduction to state their identity and send it automatically along with their location. LifeKey supports also voice-to-text and text-to-voice conversion for the communication between the deaf person and the emergency staff (Slyper, et al., 2016). This, however, requires the deaf users to be capable of reading and perceiving possibly complicated text content.
It is apparent that deaf people have limited support in their everyday communication with hearing people. Additionally, it is noted that none of the existing mobile apps offers comprehensive real-time speech to sign language conversion, requiring the deaf users to either read text content, which is often inaccessible for many of them, or understand SL presented by 3D animated hands and not a real interpreter, which is not the ideal way of presenting SL (Wauters, et al., 2006). Finally, it is noticed that the Greek language and CSL are not supported by any of the existing mobile apps. Consequently, the development of a mobile app that can act as a portable deaf interpreter and offer a comprehensive support at anytime and anywhere is indispensable for the deaf people of Cyprus. Such an app could contribute greatly to their social life as well as their independence, since it will enable them to better engage in communication within society.

Objectives and requirement considerations
The purpose of this study was to explore the communication needs and requirements of deaf adults in Cyprus and to identify how a mobile app could help them to communicate more effectively with hearing people. During initial discussions with members from the Cyprus Deaf Federation (CDF), they described communication as a barrier to their social life and healthcare, and commented that they generally experience a lot of impatience from hearing people when they try to communicate with them. Since mobile phones are ubiquitous and allow users to perform tasks in a mobile context, it was proposed that a mobile app could act as an affordable portable interpreter, which could convert spoken language into text and SL and similarly, text into speech.
For the development of a functional prototype, the needs and requirements of deaf users were taken into consideration and informed the design and evaluation process. The characteristics and unique competencies of users were studied through both primary and secondary research. It was noted that one of the major problems that the mobile app would have to deal with would be weak writing and reading comprehension skills of the Deaf (Marschark and Harris, 1996;Kyle and Cain, 2015). To overcome these issues, a custom onscreen keyboard was developed that includes pre-defined phrases, words, personal information and other categories of text which allow the deaf user to construct sentences more easily than with a traditional keyboard. Additionally, a dictionary of SL videos was developed to enable speech to SL conversion so that the deaf users could better perceive the spoken language.

Ethical Considerations
Pre-lingual deaf people have a considerably limited comprehension of words and sentences. To ensure that the participants were fully aware of the purpose of the study, the data captured, the use of the data, and their rights, an interpreter was present throughout the study to explain and clarify every detail and enable the deaf participants to ask questions if necessary. In addition, the presence of an interpreter made communication with the participants more accessible and helped them to express their opinions and views about the mobile app. Finally, to minimize participants' strain, the questionnaire was formulated with brief, concrete questions and open questions were avoided as much as possible. Two interpreters as well as two post-lingual deaf members were consulted, in order to avoid overly complicated and abstract language while developing the questionnaire, the information sheets and the prototypes. Ethical approval for all studies conducted was granted by the University of Nottingham School of Computer Science Ethics Committee.

Approach
++++++Both qualitative and quantitative methods were used in the research. Action Research aimed at addressing the social problems identified by the user community, was combined with a user-centred design approach to involve users in the design and evaluation of the app through all stages of an iterative product development process. This approach was taken as both methodologies focus on users, their involvement in an iterative process, and the evaluation of actions/design solutions by the users themselves (O'Brien, 2001;W3C, 2004). Figure 1 illustrates the entire process, participant involvement in each stage and outcomes generated to inform the product development. Due to space limitations, only the participatory design and usability testing stages are detailed here.

Figure 1.
Overview diagram of the product development process, including user involvement at different stages and the methods used for the data collection, design and evaluation of the prototypes.

Participatory Design
Upon specifying the user groups and their requirements through data gathering, the project moved to its second phase; the creation of design solutions. Before developing the initial prototype, design principles and guidelines were considered and the list of requirements was updated accordingly (Benyon, 2014). A low fidelity prototype was then created and discussed informally with a deaf user. This paper-based representation included draft sketches of the app's navigation and layout (Figure 2, left). Feedback from the deaf user was then incorporated into a medium fidelity prototype showing the look and feel of the app (Figure 2, right). Two participatory design sessions were then conducted through which pairs of users commented on design features, sequencing and overall appearance of the mobile app.

Participants.
For the design and evaluation of the medium-fidelity prototypes two participatory design sessions were conducted with four participants each. Eight participants were selected in total (four for each session) using convenience sampling on the basis of availability. Two couples took part in the first participatory design session (two of them representing User Group 1 -deaf users who have reading and writing difficulties -and two representing User Group 2 -deaf users who do not experience such difficulties). Two were aged between 29 and 39 years and the other two between 40 and 50; Two male and two female; One participant was severely deaf whereas the others were profoundly deaf.
For the second participatory design session, a family of three pre-lingual deaf members and one hard-ofhearing member was invited. In this group two were aged between 40 and 50 years and the other two were aged between 18 and 28; One male and three female; Three were severely deaf while the other had moderate hearing loss. An interpreter was present in both sessions to maintain an effective communication between the participants and the researcher.  2.2.2. Materials. Medium-fidelity prototypes were created using Adobe Illustrator software and printed out at the size of a mobile phone device. They conveyed all of the app's characteristics such as fonts, colours, arrangement and structure, content etc. Sixteen empty mobile screens were printed (one for each page) and displayed on large sheets of white paper using re-usable adhesive to allow participants to change the sequence order of pages. User comments and required changes were noted next to the corresponding page (see Figure 3).

Procedure.
The four most critical pages were reduced to separate design elements (icons, boxes, pictures, inputs, etc.). The participants were then given the elements of each page one by one and were asked to assemble them on an empty mobile screen. They were then asked to put all the pages in order, beginning from the landing page, to create a possible user journey. Finally, they were encouraged to share their thoughts and opinions on how the pages should be ordered and explain their reasoning. The comments and changes derived from the first participatory design session were implemented in the app design prior to the second session.

Results.
The changes requested by the participants were mainly non-functional and fed directly into the design and development of the high-fidelity prototype. One non-functional requirement derived from the first participatory design session was to change the main colour of the app to blue (the colour of the international symbol of the deaf). Other changes related to the sequence order of inputs in the login pages and the background colour of the SL videos; the users requested light blue instead of black. The requirements specification document was updated according to the new requirements.

Functional
Prototypes. Based on the updated requirements specification document, high-fidelity and final prototypes were created using the Apache Cordova framework (Apache Cordova, 2017). The prototypes are native mobile apps, designed and tested on an Android Samsung S5 device. An example is shown in Figure 4.
The mobile app includes three major features requested by deaf people through the data collection process: 1) The "Hear" section where speech is recorded and converted into either text only, SL only or both text and SL; 2) The "Speak" section, where a custom keyboard enables the deaf user to type texts using pre-defined words and phrases, which can then be converted into speech; 3) Text storage, allowing users to create texts using the custom keyboard and save them for future use. This enables users to prepare what they want to communicate in advance and use it whenever they want to.
The custom keyboard through which the text is composed includes eight different categories of text options (letters, yes/no, phrases, words, personal details, address details, days & months, food & drinks), six of which were operational in the final prototype, allowing the users to type faster.

Usability Testing
Usability testing was conducted on the final prototype where users were asked to perform five tasks using the application and then complete a short rating scale questionnaire. The time taken to complete a task, the number of tasks completed and the number of errors, were recorded. The users were also observed during the usability testing to provide insights about their behaviour and difficulties.  2.3.1. Participants. Eight users were recruited to take part in the usability testing. Four of the participants were considered to be "design aware users" since they took part in the first participatory design session, and the second group of participants were novice users. Both groups had equal number of users representing target User Group 1 (deaf users who experience reading and writing difficulties) and target User Group 2 (deaf users who do not experience reading and writing difficulties). Equal number of female and male users were selected.

Materials.
A task sheet comprising five evaluation tasks was created and a data collection sheet was used to note the time taken for task completion, the number of errors in each task and additional comments if needed. A timer was used to measure the time taken per task and a video camera to record the hand movements of the users and the mobile screen. A semi-standardised questionnaire was developed and used for measuring six subjective usability metrics (appearance, terminology, app capabilities, usefulness, ease of use, satisfaction). The questionnaire consisted of different 7-point Likert scale questions and two open questions about the most liked/disliked aspects of the app.

Evaluation tasks.
The users were asked to complete the following tasks using the final prototype:  Sign up to Ubider.  Communicate with the waiter to order a meal.  Add details to your profile.  Write and save text for future use.  Use the saved text to communicate with a clerk in the Social Insurance office.

Results.
All participants successfully completed all tasks with non-critical errors (Table 4). Some tasks were inherently more difficult to complete than others, as reflected by the average total time on task (Table 2). Task 2 took the longest and had the most errors, as it required the participants to a) communicate with the waiter, b) order food and c) confirm it. Some participants were writing faster than others and this was also observed by comparing the times on the second task of P3 (more than 4 minutes) with P4 (less than 2 minutes). P3 was comparatively slower than the others, however, this was not due to errors but because she was typing at a slower pace compared to the rest of the participants. She was also a user representing deaf people who experience reading and writing difficulties and reported that German was her first spoken language, so it would be easier for her if the app offered an option for German language as well.
All of the errors were non-critical, since they did not prevent successful completion of the scenarios. P4 and P6 were confused with the terminology used for the main two tabs at the bottom of the screen, which caused most of their errors (see Table 3). Moreover, P6 reported difficulties in using the onscreen keyboard, because he was used to using a keyboard consisting of nine large buttons.  After task session completion, participants rated the app on six overall measures derived from USE and QUIS questionnaires (Lund, 2001;Chin, Diehl, and Norman, 1988). Each measure included a number of 7-point Likert scale questions (0 -Strongly Disagree and 6 -Strongly agree). High positive ratings were reported on all usability dimensions, with scores of 5 (agree) or 6 (strongly agree) given by at least 7/8 participants on all measures (Table  4). Generally, all participants agreed that the app is useful and satisfactory and that the app is designed for all levels of users; however, some offered suggestions for simplification of some features and provision of more than one way to find something within the app.

DISCUSSION
Advances in smartphone technology enables automatic speech recognition powered by deep learning neural networking (Google, 2017), which can be used for speech transcription and automatic SL presentation to allow deaf users to understand the speech of the people around them. The interactive prototype of the proposed mobile app utilises technologies such as the Google Speech API (Google, 2017) to recognise voice and convert it into text. An algorithm was then developed to convert that text into synthesised SL videos to help deaf people with low literacy skills to perceive someone else's speech and communicate with them. In order for the mobile app to work effectively, internet connection or the use of mobile data is required. Slow connectivity can affect users' experience; hence, the use of the simple STT feature can be beneficial in those situations to save data and increase bandwidth.

Attitudes of deaf users
Through the earlier data collection in the project, 36/40 deaf participants (90%) expressed a need for a mobile app as a personal interpreter; only 10% prefer to use a real interpreter to help them communicate with hearing people. This, however, could also be inferred as a lack of trust towards the use of technology for communicating with hearing people. Similarly, the interpreter who was present through the study, commented that, from her experience, deaf people would not easily trust a mobile app for communication with hearing people without having someone next to them to reassure them that what is transcribed is correct. Consequently, it is important to test the app in various contexts and with more users before making it generally available.
The eight deaf users who took part in the evaluation of the interactive prototypes exhibited positive attitudes towards the app. They expressed considerable interest in the overall utility of the app and offered a number of recommendations for improvements. This was their first time using a mobile app that converts speech into CSL and text into Greek voice and they were particularly excited by this. Their responses reinforce the potential value of such an app as a positive benefit to users, and confirms their need for communication support.

Usability metrics
In order to ensure that a product or a system is usable, the aspects of effectiveness, efficiency and satisfaction need to be measured and examined (ISO, 1998). The results attained through the usability testing showed that the app is effective and that participants were completely satisfied. However, further evaluations should be carried out with more participants and different settings in order for the overall usability to be ensured.
3.2.1. Effectiveness. Effectiveness was assessed task completion. All users completed all of the tasks they were asked to perform with the mobile app. The characteristics of users were varied to ensure that the app is designed to be used by different types of users. In addition, the complexity of the tasks varied as well, to enable the users to review and use as much features of the app as possible.

Efficiency.
To assess efficiency, the time taken by the users to complete a task as well as their errors were measured (Table 3). P4 made the second most errors, but was one of the fastest users: He corrected his mistakes very quickly, and thus, demonstrates that the app enables easy and quick correction of errors. P3 and P8 were native German speakers who have lived in Cyprus for many years and learned to use Greek language and CSL as their primary languages. However, it was difficult for them to use the app in Greek and they would prefer to use it in German. Their results showed that they took a long time on tasks compared to the number of errors they made. This implies that they spent more time writing and reading text, but they did not necessarily find the mobile app complicated. Finally, tasks 2 and 4 had the most errors and highest time, since they required the user to communicate with a hearing individual based on a given scenario. It can be inferred that the majority of users were either slow in writing content or they were confused with the two sections of "Speak" and "Hear" (see Figure 5). Finally, it was observed that participants did not use the shortcuts for their personal information, and short answers, but instead they used the keyboard with the letters to compose their text from scratch. This could also have affected task performance time.

Satisfaction.
A semi-standardised questionnaire was developed and given to the participants upon completion of the user testing. The questionnaire included attitude rating scales derived from USE and QUIS questionnaires (Lund, 2001;Chin, Diehl, and Norman, 1988). The majority of participants (n = 7) agreed in all of the statements apart from one statement which had six agreements and two neutral responses. These results indicate the app has a great potential to be used by Deaf users as a personal interpreter. Participants were amazed when they saw the speech being converted into CSL and they were excited to use it on their devices. However, it is important for the app to be evaluated in different settings in order to ensure and validate its effectiveness, efficiency and user's satisfaction.

Most significant features of the mobile app
Evaluation of interactive prototypes of the mobile app with deaf users, identified that the most useful feature of the app was conversion of speech to SL. Users with reading difficulties required such a feature to allow them to better understand a hearing person's spoken language. Further development of this feature, providing the facility for streaming speech recognition to allow the users to have non-stop speech to SL conversion while the hearing person is still speaking, would be beneficial to the users.
Another feature that was considered important and valuable to the users was the custom onscreen keyboard with categories of text. Users were amazed when they saw their personal information, and other categories of words and phrases, ready to be used in the keyboard. Users with writing difficulties described the keyboard as particularly helpful. Additionally, the fact that they could type text using the custom keyboard and save it for future use was something that excited them too. This feature would allow them to communicate faster and could eliminate the need for pen and paper based back-and-forth writing for communication with others.
Finally, the presentation of synonyms of words on the speech transcription to allow deaf users to understand the meaning of an unknown word used by the hearing person, was another important feature. The CSL lexicon is not as rich as that of the Greek language; therefore, undoubtedly, deaf users will see words that they would not recognise. Consequently, such a feature could enable them not only to understand the meaning of those words but also to learn them.

Research Limitations
Although the mobile app seems to offer great potential to enhance communication between Deaf and Hearing people, there were several limitations to the study. The first limitation was related to the sample of the study and its size. The sample size used for the usability testing of the final interactive prototype (8 participants) was small. Therefore, usability problems may remain undiscovered. Finally, a limited number of tasks were given to users for the evaluation of the final prototype, mainly due to the limitations of resource and time. The interactive prototypes required weeks of development and hence limited time was available for conducting large-scale evaluations. More complex tasks will need to be evaluated in order to test how difficult is for Deaf users with a low level of education to complete them.

CONCLUSION
The study applied a user centred design approach to the development and evaluation of a mobile app to support communication between Deaf and Hearing people. Members from the Cyprus Deaf Federation (CDF) took part in data collection, iterative design, and evaluation of app prototypes from lo-fi and medium-fidelity through to a functioning demo version. The project cycle was iterated four times and five prototypes were created in total. The integration of multiple methods enabled the researchers to gain in-depth understanding of the research problem, user characteristics and requirements, and direct feedback from target end users. Further development and testing of the mobile app is required to ensure validity and reliability of the results. More experiments should be carried out to test how the mobile app performs in noisy environments and to identify any other functional issues affecting its usefulness. However, it is concluded that such an app has the potential to offer greater independence to deaf users by reducing the need for them to rely on the presence of someone to act as an interpreter to support their completion of everyday activities. Initial user interaction performance, satisfaction ratings and usability feedback