China's first AI picture book for children with ASD
AI picture book for children with ASD exhibited at WAIC 2024 (Source: China Daily)
Challenge
Autism Spectrum Disorder (ASD) is a neurodevelopmental condition affecting social interaction, communication, and behavior. Each child with ASD has unique needs and responses, requiring individualized educational approaches. Research shows that children with ASD are particularly responsive to visual information, making picture books a powerful tool for education and intervention — helping them improve social skills, emotion recognition, language expression, and behavior. However, existing AI products pursue complex imagery and playful language styles (such as metaphors and puns) that create comprehension barriers for children with autism. No picture books customized specifically for autistic children exist in the market; parents must rely on ordinary children's books as substitutes.
Solution
The AI picture book for children with ASD is built on Alibaba's self-developed ModelScope-Agent framework, integrating multiple Tongyi large model services. It combines story generation, image generation, and voice synthesis modules into an HTML5 interface accessible via the Tongyi App. Users simply input a story outline and basic settings to generate illustrated, auto-narrated personalized picture books.
For the language model, the team incorporated core principles from autism intervention expert Carol Gray's The New Social Story Book into the Tongyi Xingchen model. The system offers three cognitive levels (from simple recognition to interactive response) and four story types: common-sense cognition, social etiquette, theory of mind, and engaging stories. It follows five narrative guidelines — third-person perspective, positive and calm tone, avoidance of negative phrasing, attention to temporal expressions, and literal accuracy. The GR-8 formula is applied to balance descriptive and directive sentences, ensuring stories focus on event description.
For the image model, the team conducted surveys with over 100 families of autistic children and performed targeted fine-tuning. First, watercolor and flat-design styles help reduce visual elements to aid concentration. Second, character figures are enlarged to highlight modeling subjects and guide appropriate behavioral responses. Third, high-frequency scenarios — schools, playgrounds, and outdoor settings — are specifically optimized to help children generalize behaviors across different contexts.
Impact & Value
The project achieved three key breakthroughs.
First, personalized education content generation: picture books are customized to each child's unique needs, dramatically improving the relevance and appeal of learning materials and making education more efficient and targeted for parents and special-education teachers — a capability traditional tools cannot match.
Second, professionally customized models: the team visited intervention centers, consulted extensively with experts, educators, and parents, and trained the AI models based on first-hand data to ensure content aligns with cognitive characteristics.
Third, multimodal learning: by integrating story, image, and voice, the team creates an immersive environment that stimulates sensory engagement and participation.
Socially, the product is freely available on the Tongyi App, with over 118,000 uses to date. It has generated nearly ten million online impressions, with nearly two million reads on Xiaohongshu topic pages, and was featured in Apple's App Store curated recommendations.
The project received attention from the All-China Women's Federation and the Zhejiang Provincial Disabled Persons' Federation, with extensive coverage by central media and tech outlets, effectively raising public awareness and inclusion of children with autism.
Shanghai Animation Film Studio granted free authorization for classic characters including Sun Wukong and Big-Ear Tutu as story protagonists. The project also open-sourced the ModelScope-Agent framework, encouraging community participation and promoting the co-creation and sharing of special-education tools.