The new multimodal model can learn from text, images, videos and sounds and, thanks to its modularity, produces any number or combination of predictions.

Whether we're talking about OpenAI or ChatGPT, the vast majority of chatbots generative artificial intelligence are based on the so-called Large Language Models (LLM), models of deep learning on a large scale trained to provide answers to questions posed to them by learning information through large amounts of text.

The last frontier ofGenerative AI are the multimodal models, which combine language understanding and images, video and audio to offer an even more advanced experience and service.

Their creation, however, presents several challenges, especially if the intention is to build multimodal models on a small scale: the frequent presence of missing data due to the unavailability of information, almost always due to partial availability of resources.

In short, the risk is that the model learns on the basis of a lack, and that the calculations and predictions are distorted. And this is where EPFL started with its new project.

From the Technical University of Lausanne and Zurich a coalition for green energy
In 3D the technological glove that will make Virtual Reality tangible

MultiModN, the modular multimodal model born in Lausanne

The researchers of Federal Polytechnic of Lausanne (EPFL), one of the best universities in the world for engineering and computer science, have in fact developed MultiModN, a unique modular multimodal model recently presented at NeurIPS2023.

Researchers from the Machine Learning for Education (ML4ED) and Machine Learning and Optimization (MLO) laboratories at EPFL's School of Computer Science and Communication decided to develop and test the exact opposite of a large, but think on a smaller scale.

Led by the teacher Mary-Anne Hartley, director of the Laboratory for Global Intelligent Health Technologies hosted jointly at MLO and the Yale School of Medicine, and professor Tanja Käser, director of ML4ED, the team created a multimodal model that can learn from text, images, videos and sounds but which, unlike existing ones, is made up of a variable number of smaller modules, autonomous and input-specific.

The latter can be selected based on available information and then put together into a sequence of any number, combination or type of input. It can therefore produce any number, or combination, of predictions.

"We evaluated MultiModN in ten real activities, including support for medical diagnosis, academic performance prediction and weather forecasting,” has explained Vinitra Swamy, PhD student at ML4ED and MLO and first co-author of the project.

“Through these experiments, we believe that MultiModN is the first intrinsically interpretable and missing-data-resistant approach to multimodal modeling".

EPFL's "recipe" for more powerful quantum computers
From Artificial Intelligence a decisive boost to cryptocurrencies?

The first use case: clinical decisions for medical personnel

The first use case of MultiModN will be as a support system for clinical decisions for medical personnel in resource-limited settings.

In fact, in the healthcare sector, clinical data is often missing, perhaps due to limited resources (a patient cannot afford a specific test) or, conversely, due to an abundance of resources and information. MultiModN is able to learn from this real-world data without absorbing its so-called biases, and to adapt predictions to any combination or number of inputs.

"Missing data is a hallmark in resource-limited contexts, and as models learn these missing patterns, they can encode errors in their predictions,” stressed Mary-Anne Hartley.

“The need for flexibility in the face of unpredictably available resources is what inspired MultiModN".

In a top event the impact of AI and machine learning on services
All the reasons for the growing influence of AI in digital art

From the laboratory to real life: a trial on pneumonia and tuberculosis is underway

The publication, however, is only the first step towards implementation and field testing. Professor Hartley worked with colleagues at Lausanne University Hospital (CHUV) and Inselspital, the University Hospital of Bern, to conduct clinical studies focused on diagnosing pneumonia and tuberculosis in resource-limited settings, and is in the process of recruiting thousands of patients South Africa, Tanzania, Namibia e Benin.

The research groups undertook an extensive training initiative, teaching more than 100 doctors to systematically collect multimodal data including ultrasound images and videos, so that MultiModN can be trained to be sensitive to real data from low-resource regions.

“We are collecting exactly the kind of complex multimodal data that MultiModN is designed to handle”, said the doctor Noémie Boillat-Blanco, infectious disease specialist at CHUV.

“We are excited to see a model that can appreciate the complexity of missing resources in our contexts and the systematic lack of routine clinical assessments", added the doctor Kristina Keitel of the Inselspital, the university hospital in the Swiss capital.

The safety of AI? The Bletchley Park statement is crucial
Axel Springer-OpenAI axis for AI at the service of journalism

Machine Learning at the service of the public good

The development and training of MultiModN represents the continuation of EPFL's efforts to adapt machine learning tools to reality and for the public good, and comes shortly after the launch of Meditron, an artificial intelligence model specifically designed for the medical sector.

Meditron also belongs to the category of Large Language Models (LLM), but unlike generalist models, which serve a wide range of tasks, it is focused on medical field, and is more compact in terms of size, yet equally effective.

Meditron's goal is to democratize access to medical information of high quality, thus aiding clinical decisions.

EPFL researchers developed two versions with 7 billion and 70 billion parameters respectively, and the models were trained on selected, high-quality medical data sources, including peer-reviewed scientific literature and various clinical guidelines, ensuring a broad and accurate knowledge base.

Both Meditron, presented in November 2023, and MultiModN are therefore in line with the mission of EPFL's new AI Center, which focuses on how responsible and effective artificial intelligence can promote technological innovation for the benefit of all sectors of the society.