Ma Voix - A communication tool for adolescents and adults with autism spectrum disorder

Problem Statement

Roberto Toro's Laboratory at the Institut Pasteur, the Connect project

Ma Voix application is part of the Connect project developed by Roberto Toro's lab, it's an open database collecting data from applications for autistic people. The data collected can be the selection and use of images, their combinations, the moment of the day they are used, etc. The objective of the project is to create an image recommendation system adapted to each profile.

Institut Médico Educatif, Cour de Venise, Paris

IME welcomes 25 students aged 16 to 21. Regarding communication skills, the group is very heterogeneous: some individuals use more or less elaborate oral language, others use isolated words, some are not verbal at all and others only use gestures to communicate.

In their heterogeneity, there is homogeneity. They all have moderate to severe autism spectrum disorder with moderate to severe intellectual disability.

Project Goal

Reconcile the needs of users and their ecosystem.
Match stakeholder, design approach and user's objectives.
Prepare and succeed in User Testing with non-verbal users with cognitive disorders.

UX Methods Used

Observation with a designer notebook

Hierarchical Task Analysis

User profile workshop

Persona and Job stories

Benchmarking applications

Benchmarking digital technologies

User testings in situ

Co-sketching Worshop

KEY FOCUS AREA

What are the means by which educators communicate with various student profiles with ASD (Autism Spectrum Disorder)?
Why and how are digital and non-digital tools used at the IME (Institut Médico Educatif)?
How are the competing tools perceived by the educators and the speech analysts?
How can multiple entry points for data collection concerning pictogram usage be created?
Generate insights about different levels of pictogram comprehension and how it corresponds to the autism spectrum.

Discovery phase: A designer's notebook

A tailor-made tool to succeed immersion, field observation, analyses and synthesise data

A hierarchical analysis of tasks helped me to understand how digital and non-digital tools are used to accomplish user tasks to reach a certain goal. Some tasks were subsequently broken down into subtasks if the interaction was complex, such as the collaborative activity between four students (verbal and non-verbal)at IME.

ex)Extract from a passage from the "Designer notebook"

Numero de Tâche: 4

Date 17.05.2020

Interaction : Educateur - Etudiant

Tâche : Demander fromage

Obj : Picto(cartes) collées sur une bande

L'éducateur prépare en amont sa bande (O) augmentée de communication. Les pictogrammes (o) dont Rosie aura besoin pendant le repas. Rosie mets sur la bande de velcro - Pictograms (O) photo de Florence (educateur)+ picto je veux (O) + picto fromage(O) et montre (I) la bande à Florence
Frustration : Rosie n'aime pas les échecs, face à un échec elle se roule par terre

Such a tool was conceived because contextual interviews during the observation were not possible in this project. The speech analysts/educators were unable to explain why they performed certain tasks. It became their second nature. A tool was needed that would help in observation, analysis and synthesizing the tasks performed, in order to understand the goal.

Key Findings

photo of various interactions observations at IME

PECS (Picture Exchange Communication System) should be favoured because the rich vocabulary of other systems (PODD etc.) is beyond the reach of non-verbal adolescents/adults at IME.
The image bank that shall be integrated into the future tool must therefore be based on the PECS pictograms.
Most non-verbal youths have attained 3/4 phase of PECS. They can distinguish images but cannot comment or give an opinion.
The future pictogram bank must therefore be rich in objects (food, leisure, clothing) rather than actions or comments (I dance, I don't want to).
Use of 2-6 pictograms maximum. The sentences are basic (asking for help or asking for an object).
The “I want” pictogram shall be the most used pictogram so its placement requires special care.
The more advanced phases of PECS are for verbal profiles. There are only 4 such profiles at the IME and they do not use any communication tool.
So they are not our target users.
There is multiplicity of formats which adds heavy cognitive load:
- Non-verbal profiles use the iPad only to communicate.
- They use different paper formats for different activities.
- So a frequent change of format requires a lot of cognitive resources which negatively affects their desire to communicate.
The tasks to be carried out by educators, speech therapists and the teachers to prepare paper tools for communication are tedious and laborious.
There is a low durability of the paper pictograms; they are often lost or destroyed and they have to be re-prepared often. In extreme cases, you have to prepare 1000 pictograms of the same object.
Data collection from non-digital format is difficult.

The digitalization of tools could:
Lighten the logistical load of speech therapists / educator.
Standardization and uniformisation of tools for those who already use digital technology.
Fulfill'business'requirement to multiply the entry points for data collection.

The methodology used to define the target users : Creation of behavioural and cognitive typology datasheet

I created detailed typological tables of cognitive, behavioral and functional language levels for each adolescent at IME.

typology data sheet for behavioural analysis of students at IME

The table in detail

Thanks to 'the designer notebook', it's concluded that the adolescents with ASD are very different from each other and therefore one solution that would to fit all would not be condusive. As the subject is complex, it was important to verify and consolidate these tables with key IME profiles with a user map and user profile workshops. The workshops allowed me to draw up personas and job stories.

Persona & Job stories

Persona 1 - Jeremy, The Independent

Persona 2 - Louise, Without a Voice

Persona 3 - Dylan, Without a Voice

Persona 4 - Caroline , speech-analyst/educator

... continued Persona 4 - Caroline , speech-analyst/educator

Personas, their needs and motivations

Motivations:

For the adolescents with ASD, it's about making themselves understood in order to ask for help or ask for an object.
For speech analysts it is to provide the necessary tools for young people to express their needs.

Needs:

For adolescents with ASD the tool must be complete in order to make requests, have speech synthesis (as per profile), intuitive navigation and categorization of tabs and the possibility of access to personal pictograms.
For speech analyststhe tool must be completely configurable. The adolescent with ASD must not be deprived of communication when the speech therapist/educator wishes to schedule an activity. The settings of the pictograms must not be saved in the internal memory of the device, as the devices breaks too often.

Benchmarking the main competitor : Pyramid PECS

photo of an educator using Pyramid PECS on tablet

Educator looking for the camera button, lost somewhere in the interface ...

Interviews with educators and speech therapists informed me about its pros and cons.
The advantages of Pyramid PECS are that - it is the digitalized version of the widely used PECS workbook, it has an integrated pictogram bank with speech synthesis and it has camera functionality for uploading personal pictogrammes
The major pain points are it's exclusively an ios app, the navigation is not very intuitive and the pictogram bank is very generic. The application is in English and onboarding new users is very difficult. Lastly the settings and organisation of the pictograms are saved in local memory of the device.

Solution Conception

We chose to create the tool with the users because each user has unique needs and constraints. Speech therapists accompanied us throughoutbecause they are very familiar with the profiles and would facilitate adoption of our co-conceived solution within IME.

User Testing - adolescents with ASD

photo of a speech therapist testing Ma Voix application with an adolscent of IME

Special users needed special protocols :

A functional prototype of Ma Voix.
The speech therapist mediated the user tests with a protocol designed upstream.
A Paired Choice Preference Assessment was created with the help of the speech analyst.
Users with ASD were invited to choose from various items presented in pairs (eg bead, ball for the leisure part, cookie or cake for the food part, etc.). The result of this evaluation indicated the preferred items and the less preferred items. Once these elements were identified, they were used to develop a repertoire of items to offer at different times of the test. This step was necessary to make 'demand' as natural as possible.

List of Tasks to be accomplished by the users

Users accomplished the following task on Pyramid PECS (IPAD), their habitual tool and the same tasks on MA Voix functional prototype.

Task 1: Requests are made in one step, the speech analyst asks the question "What do you want?"
The user is guided to use one pictogramme of the desired object and press play for the speech synthesis
Task 2: Requests are made in three steps, the speech analyst asks the question "What do you want?"
The user is guided to build a complete sentence ex) 'Caroline'(the name of communication partner) + I want pictogram + cookies pictogram and press play for the speech synthesis

Insights of user testing

Lack of knowledge transfer from one tablet to another. From Pyramid PECS on IPAD to Ma Voix on android.

Adopting Ma Voix requires a learning period and cannot be adopted instantly.
Difficulties related to fine motor skills: 2/4 users had navigation problems related to the size of the tabs and the size of the play button was considered too small.
In the future tool the tabs must be wider and buttons larger
Difficulty with poorly chosen and misplaced visual stimulus.
In the redesign, the clear button should be less conspicuous and the play button should be more visually stimulating.
Users were lost due to absence of visual cues for the tab 'Start the sentence'. Users were unable to create their sentence (Task 2) independently.
In the redesign 'Start the sentence' has to be visually distinct in accordance with the user's mental model.

Conclusion:users with ASD needed heavy physical and vocal guidance to complete their tasks. Ma Voix adoption has a learning curve, but it's not that steep because after some intial nudging, the future requests were made independently by the users.

The success of these tests is to be weighted because the number of participants was low.

Creation of User flows As-IS and Future

Post user testing with ASD users, speech analysts and educators an as-is user flows were created to define the pain points and find the opportunities for the future product .

User flow created post user test with 2 speech analyst and 3 educators.

User flow created post user test with 4 adolscents/adults with ASD.

Future User flow created for use in Ideation phase

Caroline's goal: The task performed by Caroline (Persona 4) upstream so that Jeremy - the independent (Persona 1), can communicate his needs during lunch with his tablet.

Jeremy (Persona 1), Louise (Persona2), Dylan (Persona 3) goals: they want to use their tablet configured by Caroline to have the correct syntax of the sentence and use it to make their request understood, during lunch at the canteen.

Caroline's goal: She has just opened her account on Ma Voix. She wants to configure different user profiles based on the specificities of each student who uses the tablet for communication (Personas -Jeremy, Louise and Dylan)

A 2 - Day Co-design Workshop

Participants: 2 speech analysts, 2 educators
Activities: HMW, Crazy8, Concept Sketching, 100€budget(voting), Speculative design
Location: Miro, Google Meet

Design objectives for the workshop

Configure the application without depriving the users with TSA the possibility of communication.
Access the settings for the customization and organization of pictograms regardless of the platform.
Digitalize the time table of the students.

sketches done during codesign workshopshop

prototypes done during codesign workshopshop

The most relevant ideas from the workshop

Separation into 2 parts of Ma Voix application. Thus, removing the need for the caregivers to intervene directly on the tablet of users with ASD.
Use of the cloud system for saving customizations rather than the device memory.This option resolves several pain points encountered at the IME:
- Loss of settings and organization of pictograms in the event of a broken tablet.
- Freedom to access your account on any device and at any time.
- Single account could be used to synchronize the 2 parts of the application.
Design ideas for digitalization and uniformisation of time table for the adolescents at IME.
Possibility to search in the integrated pictograme bank, but also in external open source pictograme banks (Arasaac etc..)directly from the application.
Possibility to add personal pictograms and organize them by theme.

Ma Voix - A tailored communication tool for the IME Cour de Venise, Paris

Storyboards

Flow A - Caroline (Persona 4) - speech therapist/educator

Caroline-Persona 4 has just entered the office in the morning. She decides to create a new tab and place appropriate pictograms so that Jeremy-the independent (Persona 1) can use his tablet in the canteen during lunch.
For that:

She will first create a new tab.
Look for necessary pictograms in an image bank.
Add those pictograms on the tab.
Synchronize the web app Ma Voix with that of Ma Voix application on the tablet.

Step 1: Create a new tab
She configures the new tab - chooses the background, associates an icon (because Jeremy cannot read, he navigates by color and icons associated with the tabs).

Interface configuration of the tab screen

Step 2: Add pictograms
She uses the 'Add image' button to enter the Arassac public image bank to look for the pictograms.

Step 3: Synchronisation with Jeremy Persona 1 tablet

Flow B - Jeremy-the independent (Persona 1) - adolescent with ASD

Jeremy is in the canteen for lunch and he wishes to have a steak. He takes his tablet to make the request. He navigates between different tabs to look for the appropriate pictogrammes to construct his request.

Step 2: Jeremy constructs the sentence on the strip
He always places the name of his partner of communication first, followed by 'I want' pictogram and lastly, he places the object he wishes.

Step 3: Jeremy uses the speech synthesis to request Caroline for the steak
He pushes the play button.

The major challenges

The pictograms are the central elements of the system and of the IME. They are either chosen from an image bank (such as Arassac, Pyramid PECS) or they are personalised. Their categorization therefore becomes an essential need.
The following videos shows how the interface was conceived to manage the pictogram gallery

Pictogram library management
Ease of management of the pictogram library is one of the key to it's easily adoption and sucess

Adding more than one category(tags) to an image

Deleting a category in the pictogram library

Categorization of personal pictograms through tags
Personal pictograms are added by the use of a camera or uploaded from a computer.
The tag system associated with pictograms is an essential feature to properly manage the library.
A pictogram can have several tags because it can be used in different contexts and represent different ideas or emotions.

Communication tool with integrated timetable module

The benefits of a digital timetable

Portable schedules (around the neck) are stigmatizing for adolescents/adults with ASD.
Heavy additional cognitive load to handle a new format.
The extra work on the part of speech therapists to replace each torn, lost pictogram is time-consuming.
The tool is completely modular in relation to the needs of the ASD user:

Stopwatch activation.
Manual or automatic deletion of a finished activity pictogram from the time table

Configuring digital timetable for users with ASD from the Ma Voix web application

From the home page choose the schedule module to configure.
Edit from a predefined template.
Choose the necessary pictograms: activity, with whom and where.
Activate the stopwatch, if necessary and establish the duration of the activity.
Choose the method of deletion of a finished activity.
Follow the above steps to configure the next activity on the time table.

Configuration of time table on Ma Voix web application by Persona 4 - speech therapist/educator

Next Steps

Minimum Viable Product (MVP) for the communication tool to be delivered and an after-sales service during the RUN
Once the MVP has been developed and adopted by the IME (learning phase), 1 month of data collected during the RUN can enlighten us on its usefulness and ease of adoption within the IME through their two distinct profiles.
Preferably synchronization of the web application and mobile application via bluetooth rather than internetfor cases where there is no internet access.
Collection of physiological data for profiles with ASD
For the moment, the data collected is subjective and biased by the understanding of the observer or the moderator. However, there are connected controllers that could remove these biases and provide objective metrics.
Expand the sample to other IMEs with different profiles.
Refine and test the time table module