AR PhonoBlocks

Despite the positive evaluation results, our PhonoBlocks solution is limited because it is comprised of custom hardware, including an electronic letter-making platform and 3D plastic tangible letters. The reliance on custom hardware makes it difficult to scale and distribute for use in schools. What is needed is a new system that utilizes widely available technology that can provide the same functionality as PhonoBlocks.

Smartphones and tablets (e.g. iPads) are becoming ubiquitous. Many schools use them in classroom instruction for children. Apps for these devices can be designed to support augmented reality (AR) using the camera feature. Our goal is to design and build an AR version of PhonoBlocks that uses readily available technology and rather than custom hardware.

Group Project: research, design, prototyping

My Role: student design lead (research, technical solution, UI and interaction design, prototype construction)

Timeline: 2017.09-2018


In order to better understand the problem space, we conducted secondary research by reading research papers on the design for AR reading systems.

We did competitive analysis by examining the existing AR reading apps and systems  TO understand their technical solutions and important design features.

The main team members (Min Fan and Shubhra Sarker) participated in a 24-hour Design Hackathon to design a rapid AR prototype. The hands-on practice helped them to understand the potential design challenges for designing AR systems.


We learnt from the previous research that:

a. few AR systems have focus on teaching children the alphabetic principle.

b. none have used 3D lowercase letter shapes.

c. there are various technical solutions for implementing AR systems (e.g., marker or marker less, projection, head-amounted or hand-held based solutions) and there are design trade-offs for each solution.

We learnt from the hand-on practice that:

a. The tracking of AR markers is sensitive to lights and angles.

b. We may need to freeze the 3D augmented overlay when users need to rotate

the tablet camera to view the overlay (avoiding the lost of tracking in movement).

c. Most current available AR plugins support the quick detection of whole words but not single letters, particularly lowercase letters with similar shapes.

d. The AR app runs out of power soon.


AR PhonoBlocks is an AR system that supports the learning and practice of English AP for children. The system consists of a tablet running an AR app and a set of black magnetic physical letters. The tablet can be placed on a tablet stand so that the child can use two hands to interact with the physical letters. In the Learning Mode, the child can use the physical letters to make a word and see the augmented overlay of colour-coding letters shown on the screen during the word-making process. The child can also check the 3D animation for word meaning and sound. In the Practice Mode, the child can practice the learnt rules through word-building games.

What We Have Found

AR V.S. Tangible

The AR approach reduced the customized hardware but instead used everyday learning tools that most schools already have. In this way, we reduced costs and made the system more feasible for use in school contexts. The augmented coloured letters and animation may better draw children’s attention and elaborate word meaning compared to 2D pictures.

Technical Challenge 

We originally attempted to use the marker-less approach along with the Machine Learning technique to detect various lowercase letter fonts. However, the lagging time is extremely long in real-time video AR scenarios. In addition, several lowercase letters are very similar and are more likely to be misdetected. Although we switched to the marker-based solution, we would like to advocate other researchers to continue exploring potential solutions for lowercase letter detection in AR designs.

Future Work

  • We plan to conduct usability testing with children to refine the design and then administer user studies to validate its use with at-risk and EFL children at public elementary schools.
  • We will also compare the results with our previous Tangible PhonoBlocks.


Fan, M., Baishya, U., Mclaren, E., Antle, A.N., Sarker, S., Vicent, A. Block Talks: A Tangible and Augmented Reality Toolkit for Children to Learn Sentence Construction. In Extended Abstracts of Proceedings of Conference on Human Factors in Computing Systems (CHI’ EA 18), ACM Press, Montreal, April 21-16, 2018, in press.**poster presenting author**


Alissa N. Antle: Project Lead

Min Fan: Interaction Design, Product Design

Shubhra Sarker: Unity Program