When AI Falls Short: Assessing LLMs Support for Pediatric Cancer Caregivers

As childhood cancer incidences rise, effective caregiver support becomes increasingly crucial. Yet non-expert caregivers often struggle to acquire accurate pediatric oncology knowledge, limiting their ability to provide effective care. To address this gap, we are developing a novel dataset that pairs caregiver questions with doctor-written answers to ensure accuracy and practicality. We then used state-of-the-art Large Language Models (LLMs) to generate responses and compared their semantic similarity to the expert answers. Our evaluation highlights persistent factual inaccuracies in LLM outputs, underscoring the challenges of applying these models in pediatric cancer care and emphasizing the urgent need for high-quality datasets.