Dyslexia is a specific learning difficulty that affects individuals’ language-acquisition skills, such as learning to read and spell. Approximately 10% of children in English-speaking countries are diagnosed as dyslexia. Dyslexia can result in tremendous social, emotional and economical costs for children and their families and society. Based on the theories of reading and reading acquisition for dyslexia and existing research on tangible user interfaces (TUIs), we presented PhonoBlocks, a tangible reading system that leverages the use of embedded dynamic colour cues and 3D tangible letters to help children at-risk for dyslexia to learn to read and spell the alphabetic principle of English.
The original design concepts of PhonoBlocks were derived from theories of dyslexia and multisensory instructions. Theories of dyslexia suggest that the most direct cause of dyslexia is the impairment in phonological awareness (PA). PA is the ability to hear sounds that make up words in spoken language. Phonological deficits impede children’s acquisitions of the alphabetic principle and the subsequent mastery of language. In addition, dyslexia often co-occurs with attentional deficits. Learning to read English poses particular challenges for children at-risk for dyslexia due to its inconsistent letter-sound mappings in its orthography. Therefore, helping children with dyslexia to improve their phonological awareness and cueing attention to the alphabetic principle are extremely important during the acquisition of English.
The explicit, intense, and highly structured phonics-based multi-sensory instruction are often used to help children with or at-risk for dyslexia to learn to read and spell. The Orton-Gillingham (O-G) program is one example of multi-sensory instruction. Such approach uses multiple senses (i.e., visual, auditory, tactile, and kinaesthetic senses) and multiple representations (e.g., colour cues & physical letter tiles) to draw children’s attention and to explicitly teach them letter-sound correspondences. However, one limitation of this approach is that it is extremely resource intensive. Most O-G programs do not involve computational materials and rely on highly trained tutors who provide many one-to-one sessions with structured guidance and feedback.
Recent research has suggested the potential of tangible user interfaces (TUIs) in supporting learning to read for children. TUIs have some common benefits that are shared with all other computer-based methods, including the digital feedback; playful learning through multimedia (e.g., text, images, sounds, and objects); and motivational game-mechanisms. However, compared to graphic user interfaces (GUIs), the physical and spatial qualities of TUIs however may be more beneficial because the letters can be represented as tangible objects that can be interacted with more easily than printing letters, can be traced, and are easily organized into related groups and/or linear sequences of words.
Based on the theories of dyslexia and multi-sensory instruction, and the research on TUIs, we present the design of a tangible reading system called PhonoBlocks which supports the learning of the alphabetic principle for 7-8-year-old children. PhonoBlocks comprises a touch-based screen near a platform with seven slots, and a set of lowercase 3D letters which were embedded with LED strips. A child interacts by placing one or more tangible letters onto the platform. The system detects the 3D tangible letters and their spatial arrangements through a set of pogo pins embedded at the bottom of the letters; it then displays the appropriate dynamic colour cues embedded in the letters. Audiovisual feedback is also provided on the screen which also displays coloured 2D letters, associated letter sounds, and pictures.
PhonoBlocks contains seven rule-based learning activities and each has a unique colour-coding schema and associated activity procedure. PhonoBlocks also allows two modes of use. In the tutor’s mode the tutor can direct the learning activity while in the student’s mode the student can practice what they learnt on their own through a set of word-building games and receive correct feedback.
The original design concepts of PhonoBlocks were derive from theories. We then iteratively developed, evaluated, and refined it through four focus groups with dyslexia tutors, a case study conducted with 10 dyslexic children at a local specialized school, and two expert reviews with an educational psychologist who has knowledge in reading for children with learning disabilities. Specifically, we worked with dyslexia tutors to develop the learning activities and original colour cue. Based on the results of the case study, we then worked with an educational psychologist to refine the colour cue design, including determining the optimal timing for colour changes and adding the blend/decode function that allows children to change all letters into the same colour and hear the blended sounds of a word or change letters back to the original colour cue and hear the individual letter sounds. We also co-developed a revised teaching protocol that follows the Phonic-based approach and explicitly draws children’s attention to the dynamic colour cues.
My research is part of a larger project that explores if and how tangibility may help young children learn to read and spell. My research focuses on the design and evaluation (feasibility) of PhonoBlocks at a school context. I designed and conducted two case studies on my own. The first case study was with eight children at-risk for dyslexia in an urban public elementary school in Canada, the results of which were published in CHI’17 (the paper received Top5% Honorable Mention Award). For ethical and access reasons, I moved aways from children with dyslexia and began to work with children at-risk for dyslexia. I also conducted a second case study with ten children who learn English as a foreign language (EFL) in an urban private elementary school in China. I included Chinese EFL children because previous research suggests that these children may also have poor PA due to the influence of Chinese orthography that contains larger units of syllables and morphemes rather than the smaller units of phonemes in English. Therefore, Chinese EFL children may also be appropriate population for using PhonoBlocks.
The findings of my case studies contribute to a better understanding of how tangible reading systems, particularly the core design features, may support learning to read and spell for at-risk and EFL children. My research also contributes to a series of design considerations and guidelines for designing tangible learning systems that involve reading tasks for at-risk and ELF children, and perhaps also for typical and dyslexic children because they all need to learn and master PA and the alphabetic principle.
Please read the following papers for details of the Design Rationale, Concept-Driven and Situated Driven-Design Process, System Description, Design Challenges, Evaluation (Case Studies), and Design Recommendations.
Please check the poster and watch the two video clips to see how dynamic colour cues and 3D tangible letters benefited children’s learning. In the first clip, you will see a girl used the dynamic colour cues to detect spelling errors (i.e., br–dr) in the consonant blends activity and a boy automatically noticed the colour pattern (also the rule pattern) in the magic-e activity. In the second clip, you will see how children’s hand actions reflected their understanding of the rules. When a girl was uncertain about the consonant digraph th, she used a single hand to manipulate the letters. When she knew the consonant digraph sh well, she switched to two-hands interaction.
Dr. Alissa N. Antle: Project Lead (Supervisor in SIAT)
Dr. Maureen Hoskyn: Project Lead (Faculty of Education Director, Centre for Research on Early Child Health and Education)
Min Fan: Student Design Lead (Interaction Design, UI, Product Design, Java Program)
Emily S. Cramer: Unity Program
Youtopia is a collaborative, tangible, multi-touch tabletop sustainable land use planning activity. It was designed to support users to experience the challenges of sustainable land use planning.
Previous research suggests that using tangible objects on tabletops allows people to share, view, place and manipulate physical objects as tools and representations in collaborative activity while other researchers argued that this strategy resulted in parallel independent work rather than collaborative activity. In order to support collaborative activity, some researchers from the collaborative learning field have suggested distributing information, skills, roles or tools among learners in a way that requires them to work together. This is called a collaboration jigsaw script. Antle and Wise suggest a variant of the jigsaw script that utilizes a system design that recognizes sequences of actions and involves a unique set of tangible input objects that can be split up and assigned to different users — resulting in a system/physical/social configuration that either enables, encourages or enforces collaboration as the situation warrants. We implemented Antle and Wise’s three tier(system/physical/social) design strategy. The system recognizes sequences of inputs made with a unique set of tangible stamps, which can be assigned in sets to each user, or left unassigned.
As a research assistant, my work included: (1) developing usability testing protocol and administrating the tests; (2) facilitating one-month user studies with children in the field and video-coded the data; and (3) designing and conducting user studies with young adults to explore the strategy proposed by Antle and Wise—-whether the co-dependent design strategy encourages more collaborative behaviours and what kinds of collaborative behaviours and interactional patterns emerge for this design strategy.
Dr. Alissa N. Antle: Project Lead
Dr. Alyssa Wise [Dept Education]: Project Co-Lead
Tactile Letters is a tangible tabletop that leverages the use of texture cues to help children with dyslexia aged 5-6 years to learn letter- sound correspondences. In education programs, researchers and practitioners often use the Object-Imaging-Projection method (OIP) to teach children letter-sound correspondences. This method associates letters/sounds with particular objects that have forms very similar to the letter shapes and whose beginning sounds are the letter sounds (e.g., a/a/ is often associated with “apple”).
In Tactile Letters, each letter is associated with a unique texture based on the OIP method. Children with dyslexia can choose an appropriate level and learn about letter sounds by placing 3D textured tangible letter(s) or letter card(s) on the interactive table. Each tangible letter (sound) is associated with three letter cards with a pseudo-word on each. Visual-audio feedback is provided once children make correct/incorrect associations between 3D tangible letters and letter cards.
As a designer and researcher, I completed this project all by myself. My work included: (1) developing the core concept based on the previous work on cross-modal research and metaphoric learning approach in education, (2) designing and implementing the prototype, including the software application written in Processing and the hardware tabletop (that contains a camera vision system implemented with a reacTIVision engine) and tangible letters (that were laser-cut and attached with fiducial tags), and (3) designing the mixed-methods research to evaluate the effectiveness of the prototype.
Planet is a tangible tabletop game that allows young children (4-5 years old) to create their dreaming planet. The interface of the Planet combines a set of physical models including animals (fishes, bugs, mammal) and plants (trees, grass, flowers), man-made artifacts (cars, factories), and an interactive surface which displays a mini world involving a cartoon styled rotated planet and a running child. Children are able to create visual objects by stamping physical models on the surface. Different types of tangible models contribute differently to the development and pollution.
This prototype was created as a research instrument to explore whether children are able to easily understand different physical-digital mapping strategies of TUIs and how they like/dislike the mapping strategies. Both literal (e.g., cube control with simple texts) and symbolic mappings (e.g., graphic controls) were designed within the physical representations while both distal and co-located mappings were created within the physical-digital representations based upon Price’s framework.
I worked as the designer and researcher for this one-month project. I developed the concept, designed the activity, and implemented the software and hardware of the prototype. I also designed a user study to to explore the research questions proposed above.
Planet Game (Please download Processing first)
Nubi is a simple and immersive ubiquitous learning tool for IOS 5. Nubi uses both location and user information to deliver useful phrases in a language learning scenario. A user could learn new English phrases from the other users and then practices them in the right context. All the phrases could be saved in the personal folders for the later reviews. Nubi is built around a “many to many” social model, utilizing the crowd as a means to filter relevant content.
I was the concept designer and interaction designer. The design concept was informed by the ubiquitous learning. I envisioned a future learning approach wherein learners can automatically receive appropriate learning contents in contexts whenever they need. I also designed the interaction flow and implemented it using Storyboard in Xcode.
Min Fan: Concept Design, Interaction Design, UI, Front-end Developer
Trevor Smale: Project Manager, UI
Nai-chun, Hsu: Developer
Mengxi Liu: UX Designer
OMBI is an iPad application that allows toddlers to create polychromatic particles and explore different types of particle dynamics. Toddlers can watch the particles fall in different gravitational directions or disrupt them with one of two ways. They can also create new circles and flick them at them or simply tilt the iPad to watch them tumble.
I was the interaction designer and motion graphic designer for this project. The most challenge for this design case was how to design an easy-to-use multi-touch application that supports the learning for toddlers. We designed an one-screen interface without any texts, and leveraged the use of simple actions/gestures for toddlers during the design of interaction (e.g., shaking the screen and scaling up the objects on screen). I was interested in how toddlers interacted with the touch-based system, particularly in their unique multi-touch gestures or behaviours emerged in interaction.
Trevor Smale: Project Manager
Lance Fu: Developer
Min Fan: Interaction Designer, Motion Graphic Designer
Tzeyi Pang:Interaction Designer
President’s Choice is an outrageous interactive video installation made as part of the Visual Story course at the Centre for Digital Media . In President’s Choice, the viewer takes on the role of Abraham Lincoln in a sinking balloon. Abe must decide whether to throw off Napoleon or Churchill for the greater good. The project is constructed in Flash with animation done in After Effects. It uses a camera and laser to determine the position of the viewer, who controls Abe by walking left and right in front of the screen.
I was one of the concept designers and motion graphic designers for this project. We allowed users to use their body and movements to control and interact with the system, which provided them an engaging and immersive interactive environment. I created more than 20 pieces of animation clips and was involved in the design of interactive narratives.
Ryan Wang: Project Manager
Patrick Daggit: Motion Graphic Design Lead
Min Fan: Motion Graphic Designer, Sound Editor
Kurt Diegert: Technical Director
Lance Fu: Programmer
Mengxi Liu: Programmer, Sound Editor
The “Keep Cool” project is an interactive playable narrative created in collaboration with Dreamrider Productions. Young children aged eight to ten can use the interface to learn about the environment and make behavioural changes at home to help make an ecological difference. The interactive narrative consists of animated sequences with playable mini games and pledges that show children how they can help the environment in their own homes and allow them to check back and log the changes that they’ve made.
Using bold, fun and appealing visuals, the “Keep Cool” project uses real world metrics to visualize how simple behavioural changes can amount to a large difference. Returning users will also be able to view the pledges they committed to in their last visit. These pledges are trackable, giving Dreamrider Theatre statistics for funding; displaying the effectiveness of behavioural change in the communities they visit. The interactivity and engaging storyline combined with simple real life parallels and metrics will provide parents and teachers a great teaching device for instilling simple ideas for changes that can be adopted for years to come.
As a motion graphic design lead, my duties included: (1) discussing the storyboard with the 2D artists, (2) creating the spreadsheets of animation schedule and animation breakdown lists, (3) setting up the production pipelines, and (4) working on the animation. By completing these tasks, I have gained a more comprehensive understanding of how to design and manage an interactive narrative.
Tiffany Fok: Project Manager
Min Fan: Motion Graphic Design Lead
Nouf Saleh Alkenani: Motion Graphic Design
Mengxi (Andy) Liu: Programmer
Stephanie Louie: 2D Artist
Yinan (Linda) Zhao:2D Artist
This is an industrial project working collaboratively with DotStudioz. DotStudioz is a start-up out of Vancouver aiming to monetize independent film content online in an innovative way. As a graphic designer, my responsibility was to design wireframes and mockups for a proof-of-concept website which supports the video tracking and sharing functions.
Michael Brynjolfson: Project Manager
Min Fan: Graphic Designer, 3D Artist
Erez Barzilay: Producer (Video)
Randy Ruan: Compositor
Paul Shih: Programmer
Mengxi (Andy) Liu: Programmer