In BabySensor-Pre, we built and tested a contactless monitoring approach for vulnerable infants that works in low light, runs in real time on a custom edge device, and streams insights to a companion mobile app. We focused on three things: contactless vital signs — heart, breathing rate and skin-temperature in low light; reading a baby’s face and body for cues (like hunger, sleep); and making sense of their cries and other sounds (like pain, fussy). We validated parts of the system on real baby video data collected with the Medical University of Vienna. In addition, we packaged everything into a working prototype with a user-friendly app.
This project was led by BabySensorAS with primary partners VicarVision, Østfold University College, Glance. Additional contributors were: Minoko, Glance, iSolve.
Why this matters
Wired NICU sensors are effective in hospitals, but they can irritate delicate neonatal skin and aren’t practical for home use. Parents of high-risk infants usually fall back on basic audio/video baby monitors that can’t track health-relevant signals and often produce false alarms. These issues can create excessive stress and anxiety for caregivers. BabySensor-Pre set out to develop a fully contactless technology that combines standard, infrared, and thermal imaging with intelligent analysis. This aims to curb false alarms and increase usefulness, even in the dim environments where babies sleep.
What we built
– Low-light processing & analysis: Multi-modal AI algorithms for tracking baby metrics designed to work under low-light and deal with baby movement.
– Custom edge prototype & app: An affordable single-unit device (RGB + IR + low-res thermal) running all algorithms locally, with all data and alerts presented in a companion mobile app.
Multi-modal contactless AI-powered analysis in BabySensor. Several health and behavioral parameters are tracked and analyzed to give alerts to parents.
Infant Vital Signs – contactless breathing, heart rate, & skin-temperature
Building upon the algorithms used in FaceReader’s Vital Signs module, we developed and fine-tuned algorithms for infant vital signs monitoring. These include breathing rate and heart rate estimation. Breathing rate is derived from tracking subtle chest and body motion. Heart rate is derived from tracking skin-color fluctuations, which are invisible to the naked eye. These algorithms were able to robustly measure vital signs of babies under various recording conditions. Breathing rate was estimated with an error of less than 2 breaths per minute under ideal conditions, and less than 10 breaths per minute under less-than-ideal conditions (baby breathing rate is typically in the range of 40-60 breaths/min; for adults, this error was less than 1 and 2 breaths per minute).
Lastly, using the on-board low-resolution thermal camera in combination with the high-resolution RGB camera, the skin-temperature of the baby could be reliably measured within a resolution of ±1° C.
Examples of contactless breathing rate estimation in BabySensor [top: breathing signal, bottom: breathing rate over time]. Contactless breathing rate follows the ground truth well.
Infant Vocal Cues Analysis
We focused our models on infant cry understanding from audio based on a hybrid signal processing and machine learning approach. These models were derived from FaceReader’s voice analysis module. The algorithms detected and classified crying reliably (with 70% – 90% accuracy) into different severity levels. Furthermore, we also derived pain estimation (e.g., needle-prick vs. general discomfort).
Together, these cues help curb false alarms by adding context – distinguishing, for example, “just fussing” from signals that warrant attention.
Infant Facial Expression & Body Cues Analysis
The system detects facial action cues and combines them with body and audio cues to help explain why a baby is distressed. Again, we used our knowledge from VicarVision’s infant face analysis technology in Baby FaceReader (with 70% F1 accuracy in AU detection).
Infant sleep and hunger cues: Based on facial cues in combination with infant body movement cues, we developed algorithms to estimate baby sleep state and detect hunger cues. This way, parents can get an alert on their smartphone before their baby wakes up from their sleep, crying with hunger.
Illustration of vocal spectral-analysis, facial modelling, and body tracking, used for detecting and classifying crying (e.g., pain vs fussy); and inferring cues like hunger and sleep states.
Prototype Device with Smartphone App
The advancements made in this project were demonstrated through a fully integrated hardware prototype with multi-wavelength cameras (RGB + IR + thermal) and microphones, and powered by an AI-accelerated processor. All deployed algorithms were optimized to run on-device to ensure real-time performance and data privacy.
BabySensor prototype device and smartphone app (image courtesy Minoko, Glance)
What’s next
For next steps, we hope to broaden validation with at-home testing to characterize real-world performance across ages, skin tones, sleep positions, and lighting conditions. It is important to advance the productization pathway by hardening the edge device and app. In addition, we will enable research collaborations by making the “raw data” available to support studies in developmental psychology and neonatal care.
