Artificial intelligence (AI) is becoming an increasingly common part of children's everyday lives, from helping with school assignments to answering personal questions and providing entertainment. While the technology offers significant benefits, growing concerns about its impact on young people's mental health and safety are raising serious questions about how much children should rely on AI chatbots.
A recent lawsuit filed by the parents of a 13-year-old girl against Google has brought AI concerns into sharper focus. The parents allege that Google's Gemini AI chatbot contributed to their daughter's mental health crisis, discussed self-harm methods, and helped her plan her funeral through conversations conducted on a school-issued Chromebook.
According to the lawsuit, the teenager exchanged thousands of messages with the chatbot over several months, discussing depression and self-harm. Her parents claim that the AI failed to direct her towards appropriate real-world assistance despite the seriousness of the conversations. The teenager was eventually taken for an emergency psychiatric evaluation and is now receiving weekly therapy. Google has said it is reviewing the allegations. They have also pointed to safety measures developed with medical experts.
“For years we have shown hotline information directly in Search when someone may be experiencing a mental health crisis, and we’ve built the same protections into Search AI features, connecting people with real-world support.” Stated Google spokesperson.
This case raises an important question: who is responsible when a child relies on an AI chatbot during a vulnerable moment?
Parents have an important role in monitoring children's digital activities, discussing online safety and encouraging open communication. However, responsibility should not rest entirely on families. Technology companies must develop stronger age-appropriate protections, improve the detection of conversations involving self-harm and ensure that children experiencing serious distress are directed towards human assistance. AI systems should also make it clear that they are not human friends or professional mental health advisers. Schools should establish clear guidelines for AI use, particularly when students access chatbots through school-issued devices.
AI has considerable potential to support children's education and development. It can explain difficult subjects, help students practise languages, encourage creativity and provide quick access to information. It can also help parents and teachers introduce technology and digital skills that will become increasingly important in the future. However, these benefits depend heavily on responsible use, appropriate supervision and reliable safety protections.
Children need human understanding, compassion and protection. No matter how advanced artificial intelligence becomes, it should never be treated as a replacement for the people responsible for keeping children safe. Awareness has to spread, among all age groups, that AI should not be considered a friend, counselor or trusted emotional companion.
