The MORSE (Multimodal Behaviour Observation in Real-time Simulator Environments) project was an exciting collaboration aimed at improving training in high-stakes maritime environments. VicarVision worked alongside project partners Noterik, Noldus, and MARIN to explore how advanced AI-driven human factor analysis could enhance self-assessment and debriefing for professional ship captains. With our results we highlight the benefits of multimodal AI analysis for effective training.
A Rich, Multimodal Dataset
One of MORSE’s major achievements was the successful collection of a unique, fully-synchronised multi-modal dataset from professional participants in MARIN’s ship simulator environment. This dataset included:
• Video & Audio recordings, including POV perspective. • Physiological signals (ECG/PPG, electrodermal activity). • Behavior signals (facial expressions, eye tracking). • Environmental and situational data (collision threats, navigation context).
This rich dataset now serves as a valuable testbed for future developments in human factor analysis.
Ship simulator at MARIN
Snapshots of the Multimodal dataset collected in this project.
Decoding Human Performance Under Pressure
Our AI analysis focused on multiple human factor indicators, including:
• Facial cues using FaceReader-based feature extraction for Action Unit recognition and emotion detection. • Vocal cues for speaker diarization and communication analysis. • Physiological changes, particularly heart rate fluctuations as indicators of high cognitive load. • Environmental context, such as collision threat detection.
One key finding: professional captains often display minimal stress through facial expressions. This makes physiological signals, particularly heart rate, a more reliable indicator of cognitive strain. Thus, using a multimodal tool is important to provide accurate feedback.
Examples of synchronized multimodal signals. These signals are used to identify key moments in the training session.
From Data to Self-Debriefing
The ultimate goal of MORSE was to translate these insights into more effective training feedback. Using Noterik’s StagePlayer platform, we developed an automated self-debriefing tool that highlights key moments in a session based on detected behavioural and physiological changes.
For example, a sudden head movement can prompt trainees to recall what they were searching for. Spikes in heart rate can indicate workload events, and speech activity indicate radio communication moments. In addition, an emotional expression shift can prompt a reflection on the situation. By pairing these events with targeted, pre-prepared questions, trainees are encouraged to reflect deeply on their decisions, situational awareness, and overall performance.
Looking Ahead
The MORSE project demonstrates how multimodal AI analysis can be seamlessly integrated into training environments. Thus enabling participant-in-the-loop learning that is both data-driven and personalised. While our immediate focus was maritime training, the approach holds promise for other high-demand domains such as aviation, traffic management, and emergency response.
We’re excited to continue collaborating with our project partners on future research, with upcoming plans to investigate human factors in inland waterway safety.
