Cutting Edge '25

VoiceBlockly: Voice Code Generation in Block-Based Programming Using a Novel Multi-Agent Framework

By

Catagories

Play Video

Block-based programming (BBP) has proven effective in teaching programming concepts, offering a visual and more intuitive approach than traditional text-based programming. However, accessibility in BBP remains an underexplored area, particularly for students with disabilities who may benefit from alternative input methods, such as voice commands. This project aims to bridge this accessibility gap by integrating natural language processing (NLP) capabilities into a BBP interface, enabling users to generate programming blocks through voice commands. To achieve effective voice to block code generation, the project employs a novel multi-agent (MA) framework that integrates several large language models (LLM) working together. Each model is designed to interpret user commands and generate corresponding block syntax. A novel aggregation algorithm based on uncertainty quantification (UQ) is introduced to combine outputs from multiple agents, ensuring the accuracy and validity of the generated blocks. This approach enables the system to produce high quality output by leveraging the collective strengths of multiple agents. The prototype demonstrates that the BBP environment effectively translates NL input into accurately rendered code blocks within the interface. Testing and evaluation results indicate that the proposed MA framework outperforms the leading state-of-the-art (SOTA) MA framework by 6% in programming tasks, achieving a Pass@1 score of 75% on the HumanEval dataset. Furthermore, it surpasses the same MA framework by 6% in general domain tasks, attaining an accuracy of 65% on the TruthfulQA dataset.

Vision Quest

Check out the visionary projects our students have brought up in this year
VisuaLit

VisuaLit is an AI-powered eBook reader that redefines traditional reading by merging visual storytelling, audio narration, and contextual learning into…

VenDoor

The VenDoor application is a fully functional mobile application designed to create a bridge between mobile vendors and their customers…

UniGuide

UniGuide is a student-focused platform that helps individuals make smart educational and career decisions. It offers a comprehensive database of…