Children’s social and emotional skills help them to develop respectful relationships and contribute to their success. Most children are able to identify basic emotions and facial expressions. However, children with Autism Spectrum Disorders (ASDs) lack such skills.

Computer-based systems have also been developed to support children’s emotional development. Compared to traditional programs, the computer-based approach is advantageous in the ways that (1) it is predictable (e.g. feedback), repetitive (e.g. rich resources), and free from stressful social demand (e.g. practice in a simulated social environment created by computers) which is favored by children with ASDs.

We aim to explore if we could design an application to help children with ASDs to learn emotional development.

Group Project: ideation, design, prototyping

My Role: project lead (UI and interaction design)

Timeline: 2017.12-2018


Design Challenges

  1. How to design the narrative (art style, storyline, characters, script, learning contents)?
  2. How to design the games (multi-level, hints, rewards)?
  3. How to combine the narrative and the games (embedded moment, transferred knowledge)?


We used sketching to try different solutions on paper.We compared their pros and cons and selected one.


Learning Goals

The main learning goal is to support children aged 5-8 years old to (1) learn six types of emotions including happiness, sadness, anger, surprise, disgust and fear; (2) understand three common social contexts that are familiar to children, including school, home, and the park; and (3) be able to recognize and express the emotions through facial expressions.

Design Goals

We design narratives that incorporate visual and auditory contents to promote children’s understanding of the relationship between social contexts and emotions.

We design multi-level games that gradually increase in difficulty to provide structured learning tasks. We also embedded the games into the narrative to encourage children to reflect the learned knowledge and then practice it.

We support children to practice facial expressions in various stimulated contexts by making the facial expressions. We also offer explicit visual cues if they fail.

The Structure of EmoStory

A child enters the game and selects the narrative they want to watch from the contexts of school, home, and the park.

The child watches the main narrative and learns the first emotion (happy) and the associated facial expression and context.

After one emotion being introduced, the narrative is paused and the child has to complete Level-1 Game by mapping the correct facial expression figure to the emotion introduced in the narrative.

After successfully completing Level-1 Game, the child enters Level-2 Game to make the facial expression based on the given picture (left).

The child can seek to help by pressing the Hint Button and the blue visual cues will highlight the key facial movements for making the expression.

The child will enter Level-3 Game wherein they will watch a similar narrative (flash back story of the main narrative) and then make the facial expression on their own. Then the main narrative will continue and introduce the next emotion.

Future Work

We will conduct case studies to explore whether EmoStory can improve children’s abilities in recognizing and expressing facial expressions in various emotional contexts and whether the children can transfer the knowledge to real life. We will also analyze what features may or may not benefit their learning.

Lessons Learned

1. How to propose and start a project from scratch (form the team, find the resources, and write the proposal)

2. How lead a team with diverse backgrounds



Min Fan: Project Lead

Jianyu Fan: Programer

Sheng Jin: Art Director and Animator

Alissa N. Antle: Consultant

Philippe Pasquier: Consultant