Welcome to SEMULIN

Funded by the German Federal Ministry for Economic Affairs and Climate Action, Project SEMULIN (Self-Supporting Multimodal Interaction) is a research project that will study natural multimodal human-machine interactions (HMI) in the field of autonomous driving.

To help bring this project to fruition, eminent research and industry leaders have joined forces - audEERING, Blickshift, eesy Innovation, Elektrobit, Fraunhofer IIS, Infineon and the University of Ulm.

Our joint goal is to develop a self-supporting HMI that allows natural, intuitive human-like communication between the vehicle and its users with less room for errors. The vehicle naturally recognizes the user's intent through multiple input- and output streams.

The underlying system is based on a combination of psychological and complex AI models with machine learning that is supported by high-performance computing power and architectures.

Problem

Natural human communication is determined by the interaction of different communication modalities such as voice, facial expression, gesture, or gaze. Context, too, plays a crucial role in human interaction. As a result, a system for human-machine interaction that is to appear as natural as possible must be able to consider and link different modalities and contextual information.

Project goal

The objective of the project is to develop a natural and consistent human-machine interface for automated vehicles (Level 4 and 5, with transitions to Level 3). The system uses multimodal input/output concepts that enable holistic, natural, and efficient communication. It is to be able to semantically interpret and put into context the data collected by various sensors. In addition, the developed system is to autonomously adapt to the needs of its users through continuous monitoring and interpretation.

Project approach and execution

The multimodal interaction concept will be based on holistic passenger monitoring through various installed sensors that can preprocess and interpret the data. Rule- and AI-based methods will be used to correlate data. The intention is to develop a user model that, in turn, enables a natural, effective response of the vehicle.