December 5th at 3:50pm

Interactive AI-Driven Toy Set

The Toy Set is an interactive system designed to encourage children’s imagination, creativity, and storytelling skills through dynamic interaction with an AI-driven toy. The toy set features a modular collection of components including a microcontroller, microphone, speaker, LEGO figurines, and a set of character-themed outfits. Children can interact with the toy by speaking into the microphone, prompting the toy to engage in real-time conversations or role-playing scenarios. This system aims to create an immersive and educational experience by adapting to the child’s input and enabling them to craft and influence stories through voice-based interaction. Drawing inspiration from generative agents in AI, the toy set fosters communication and role-play activities that enhance children’s emotional and cognitive development.

1. Dialogue Mode

Children can speak directly into the microphone to initiate a conversation with the toy. They can either describe or assign roles to the toy, and the system responds accordingly, creating a lively, ongoing dialogue. Children’s speech is transcribed and interpreted by the AI in real time, allowing for natural, fluid interactions.

Case: Emma says, "You’re a brave knight," the AI will adopt the role of a knight, narrating a scene set in a castle. As the conversation progresses, Emma might ask, “What’s inside the castle?” The agent would then describe treasures and creatures, modifying the story as Emma’s choices shape the adventure.

AI-Functionality:The AI utilizes OpenAI's Whisper model for speech recognition, enabling the system to understand various languages and accents. Based on the child's input, the agent adopts different personas, such as a knight, wizard, or chef, and responds with relevant, context-sensitive dialogue. This interaction adapts and evolves with each conversation, providing a personalized experience.

2. Story Mode

Children choose different outfits for the toy, such as a wizard’s robe or a chef’s hat. Each outfit triggers a corresponding storyline, and the child becomes a participant in the evolving narrative. The toy’s physical transformation directly influences the AI's narrative output, providing an immersive role-playing experience. Case: Emma dresses the toy as a fox. The NFC chip triggers the "Fox's Forest Quest," where Emma decides the fox's path through the forest, encountering challenges and allies. OR When Emma adds a "dragon" to the story, the agent seamlessly incorporates the dragon into the plot, adjusting the storyline to reflect Emma’s preferences.

AI-Functionality:Arduino and NFC Model detects the outfit's NFC signal and sends corresponding data to trigger the relevant narrative. The AI agent, powered by a storybook model, adapts the plot to include user input, creating an immersive storyline.

3. Scenario Mode

Scene Capture & Analysis: The child points the camera at the play scene, capturing the interaction between themselves and the toy. AI uses computer vision models (like CNNs) to analyze the captured scene, identifying objects, characters, and interactions. The system determines key elements in the environment, such as toys, characters, and any actions happening.

Background Story Generation & Pretend play:Children choose characters and interact with the AI-generated plot to influence the story. As the child interacts, the AI continually adjusts the story and enhances the gaming experience with voice and image feedback, with the child's decisions directly affecting the direction of the plot.