AR BlockTalks

We designed and implemented AR Block Talks in EduHacks 2017,  a 24-hour hackathon where talented students were invited to form teams based on mutual interest, with the goal of integrating technology into the design of exceptional learning experiences. The Block Talks won the Bronze Award among 100+ projects.

PhD Project: research, ideation, design, rapid prototyping

My Role: project lead (research, UI and interaction design, prototyping construction, technical solution)

Timeline: 2017.9.30-2017.10.01


Learning syntactic knowledge (sentence structure) is important for children. However, Many children who are at-risk for reading difficulty or who learn English as a second language  have poor syntactic knowledge.

Recent research suggests the potential of tangible user interfaces (TUIs) to support children’s literacy learning, however, it is not readily available in classrooms;

Augmented reality (AR) technology, which can augment physical objects, shares similar features to TUIs, and may be designed to be more accessible to teachers. We present Block Talks, a hybrid TUI and AR system to help children ages 8-10 learn English sentence structure.


We aimed to leverage the use of ordinary teaching supplies in our design. We bought lots of items such as physical letter tiles, blocks, paper magnets from Dollar Store and played with them together.

Learning Goal

The main learning goal was to teach children syntax of English word and sentence construction. We started with simple words and sentence structures such as “The cat can run." and “A bird can fly." which children may learn in Grade 1-3.

Design Goals

Based on previous research, we presented four main design goals: (1) using inexpensive, physical learning tools that are already available to students and teachers; (2) using colours to draw children’s attention to coded structures; (3) using tangible letters and physical affordances to enforce children’s learning of linear sentence structures; and (4) using a multimodal feedback approach that includes 3D animations, letters, and audio recordings to help children learn to read.


Blocks Talks is a tangible and AR system that helps children learn English sentence construction. This system consists of an app running on a tablet and a set of magnetic colour-coded blocks and letters. Block Talks is intended to be used in a semi-supervised environment. A teacher first chooses colour-coded blocks for various sentence structures. The colour coding indicates parts of the sentence (e.g. article [orange], subject [blue], modal verb [yellow], verb [green], punctuation [black]). Children attach magnetic letters to the coloured blocks to form words. These blocks can be connected with physical notches to make sentences. Once the blocks are connected, children scan their sentence using the tablet to receive multimodal AR feedback. The system was developed by Unity and Vuforia.

The user attaches magnetic letters to the coloured blocks to form words.

She then connects these blocks with physical notches to make a sentence.

She scans the sentence using the tablet to receive multimodal AR feedback.

The multimodal AR feedback includes 3D animation for sentence meaning, the audio reading of the sentence and colour-coded 2D letters.

The Lego-based notches of each sentence component are different so that they can only be connected in a certain order (e.g.article->verb). In this way we enforce the corrective learning behaviours.

If the sentence is incorrect, a hint is overlayed on the screen. For example, the red AR arrow animation shows that the letter i and r are reversed in the word bird.

Expert Review and Future Work

We demoed our toolkit to HCI and educational experts and received valuable feedback to include in our next iteration. We are continuing to develop the prototype and hope to soon conduct usability testing with children. Currently, our system has only one sentence pattern and one error feedback suggestion. However, we are interested in working with teachers to co-design improved feedback and learning tasks to enhance and reinforce existing curricula. Although the current version requires assistance from a teacher, we are developing a Game mode to allow children to practice on their own. Once the toolkit is further along in development, we propose running a study to evaluate children’s interactions and learning outcomes with Block Talks compared to traditional instructional approaches.

Lessons Learned

1. How to do a rapid prototyping with 24 hours

2. How to do a public presentation of the research work



Min Fan: Project Lead, Interaction Designer

Uddipana Baishya: Interaction Designer

Elgin-Skye Mclaren: Website Designer

Shubhra Sarker: Unity Programmer

Amal Vincent: Unity Programmer

Alissa N. Antle: Mentor