Four specialized modules cooperate in the new MiCRo model, designed to make decision-making processes more readable and controllable.

In recent years i Large Language Model or LLM have profoundly changed the relationship between human beings and systems of artificial intelligenceFrom text generation to software programming, including machine translation, document analysis, and scientific research support, these neural networks have achieved levels of accuracy unthinkable just a few years ago. Their growing popularity, however, has raised a question that now pervades much of the scientific debate: is it possible to truly understand how these systems reach the conclusions they produce?

The question is not purely academic. In fields such as medicine, finance, justice, or industrial design, knowing why an algorithm formulated a certain answer can be as important as the answer itself. Most of the models available today, however, continue to behave like a “black box”: the result is visible, while the computational path followed by the neural network remains largely opaque even to its developers.

It is precisely on this limit that research intervenesFederal Polytechnic School of Lausanne (EPFL)in SwitzerlandA group of scholars has developed MiCRo, acronym for Mixture of Cognitive Reasoners, a modular architecture based on Transformers and designed to induce functional specialization among different components of the model.

MiCRo doesn't attempt to faithfully reproduce the biological structure of the human brain, nor does it demonstrate that an artificial neural network thinks like a human. Instead, it draws on a general principle observed in cognitive neuroscience: human intelligence doesn't depend on a single, indistinct area, but on the interaction between networks specialized in different functions, such as language, logical reasoning, social understanding, and knowledge retrieval.

The project was born from the meeting between computational linguistics, machine learning algorithm and neuroscience. The authors are Badr Al-Khamissi, Nicholas De Sabbata, Greta Tuckute, Zeming Chen, Martin Schrimpf e Antoine Bosselut, with affiliations at EPFL, the Massachusetts Institute of Technology, and Harvard University. The work was included in the fourteenth edition of theInternational Conference on Learning Representations, scheduled for 2026.

Four cognitive “experts” inside a Transformer model

The distinctive feature of MiCRo is that it splits the layers of a pre-trained language model into four expert modules, associated with as many cognitive functions. These are not four independent systems that consult each other, but specialized components integrated into the same Transformer architecture.

The first module is dedicated to the language and is mainly involved in the development of linguistic structures. The second is associated with the logic, problem solving and abstract reasoning. The third concerns the social cognition, that is, the ability to interpret intentions, emotions, interpersonal relationships and forms of implicit communication. The fourth integrates the knowledge of the world, connecting information, memories and narrative sequences distributed over time.

This last module draws inspiration from the so-called "default mode network," a brain network studied for its involvement in memory retrieval, the construction of narrative representations, and the imagination of future events. This reference does not imply that the artificial module reproduces the biological mechanisms of the mind, but rather serves to define the type of function the researchers sought to induce during training.

Compared to a dense Transformer model, in which the same blocks participate indiscriminately in different types of processing, MiCRo introduces a more readable form of internal organization. A specific router decides which experts to assign tokens to at different processing stages, selecting the components considered most relevant to the request received.

The approach shares some similarities with "Mixture of Experts" architectures, already used to distribute computation across different modules. In the case of MiCRo, however, modularity is conceived not only as a tool to increase efficiency, but also as a means to induce recognizable functional expertise and analyze their contribution to the final solution.

Three training phases to guide the more “expert” modules

The training process occurs in three phases. Initially, only the experts are trained using a selected data set, providing each module with an initial functional orientation. Subsequently, the rest of the architecture is frozen and the router is trained to learn to select the most relevant expert. In the third phase, the entire system is jointly refined using a larger set of instructions.

This procedure aims to prevent module names from becoming mere labels. For the architecture to be truly meaningful, different requests must activate different components in a manner consistent with the intended specialization.

Routing, ablations and benchmarks verify specializations

The researchers' analyses indicate that MiCRo develops routing paths consistent with the tasks assigned to the four experts. Arithmetic problems are primarily addressed to the logic module, while tasks related to understanding intentions and relationships between people engage the social expert more.

The linguistic module, on the other hand, tends to be activated frequently in the initial layers of the model. This behavior is compatible with the transversal nature of language, necessary to interpret the request before the other components can intervene on the logical, social, or world-knowledge aspects.

To verify that the experts had acquired concretely different roles, the group did not limit itself to observing the distribution of the tokens. It also conducted tests of ablation, an experimental technique that consists of deactivating a component of the system to measure how the overall performance changes.

Removing the logic expert causes a marked deterioration in mathematical benchmarks MATH e GSM8K. Excluding the social module may instead lead to small improvements in purely numerical tests, because it eliminates the contribution of a component that is irrelevant to the task at hand.

For more heterogeneous tests, such as those included in the benchmarks MMLU e BBH, the distribution of contributions appears more complex. Some subtasks depend primarily on a single expert, while others require the collaboration of multiple modules. This result is consistent with the nature of complex problems, which can rarely be traced back to a single cognitive ability.

The researchers also applied some experimental tools derived from neuroscience to the model. The so-called functional locators They have been used to examine the selectivity of internal units with respect to three domains: language, general reasoning and theory of mind, that is, the ability to represent the thoughts and intentions of others.

These analyses allowed us to identify functionally specialized units distributed across experts and strata. The results do not imply a direct correspondence between biological neurons and artificial units, but they offer an additional tool for verifying whether modular subdivision actually produces differentiated behaviors.

In terms of performance, the versions MiCRo-Llama-1B e MiCRo-Llama-3B, based on models with one and three billion parameters respectively, were compared with dense models and modular configurations without the same cognitive specialization.

In the authors' reported tests, MiCRo matched or surpassed the baselines considered on several mathematics, general knowledge, and reasoning benchmarks, including GSM8K, Minerva-Math, MMLU, and BBH. The results do not mean that the architecture outperforms all available language models, but they do show that introducing specialization does not necessarily require sacrificing performance.

The system has also been evaluated with CogBench, a set of experiments comparing the behavior of language models with patterns emerging from classical psychological tests conducted on humans. The MiCRo variants showed greater similarity than their respective dense versions and the other modular configurations used as comparisons.

This result, too, doesn't prove that the network reasons like a person in any biological, conscious, or general sense. Rather, it suggests that a structure inspired by functional specialization can more faithfully reproduce certain specific behavioral patterns observed in cognitive experiments.

From cognitive tests to the effective transparency of the new model

"In some cases, the capabilities of these systems have surpassed our understanding of how they work. If we can better understand how these machines understand information, we can design better systems and develop more effective regulations."

according to Antoine Bosselut, Director of Natural Language Processing Laboratory of the EPFL.

While referring more generally to the evolution of large language models, this reflection also effectively summarizes the goal pursued by MiCRo. The architecture developed at EPFL does not seek to imitate the human brain, but rather to make it more observable how different specialized components contribute to the construction of a response, offering researchers additional tools to analyze the behavior of models.

Now more direct control over the computational path

MiCRo's modularity not only allows you to observe which components are involved during processing. The architecture also allows you to modify the model's behavior during inference, directing tokens to specific experts or assigning them greater weight.

The system can, for example, be oriented to prioritize social processing over logical processing, or to emphasize the component most relevant to a mathematical task. This possibility introduces more granular control than that exercised solely through the textual instructions contained in the prompt.

This doesn't mean that the operation becomes completely deterministic, nor that every response can be predicted. Routing remains part of a complex and probabilistic neural network. However, the ability to observe and modify the experts' selection offers researchers an additional level of analysis and intervention on the computational path.

This aspect could be interesting for the future development of specialized applications. An educational assistant might place greater emphasis on logical reasoning and linguistic clarity, while a system designed to analyze human interactions might place greater emphasis on the social module.

However, these are application prospects and not uses already validated by the study. EPFL's work concerns an experimental architecture of relatively small size compared to major commercial models and does not directly evaluate its use in healthcare, finance, justice, or public administration.

Furthermore, MiCRo doesn't automatically solve the problem of explainability. Knowing that a token was assigned to the logical or social expert isn't equivalent to having a complete explanation of the reasons behind a given sentence. Nor does the structure eliminate hallucinations, data distortions, spurious correlations, or other unexpected behaviors.

Rather, the architecture offers an intermediate level of observation. Instead of simply analyzing a hazy distribution of information across billions of parameters, developers can examine the behavior of individual experts, router decisions, and the consequences of deactivating a component.

If a model exhibits recurring difficulties in mathematical tasks, for example, it might be possible to focus the analysis on the logic module, the router, or the data used to induce its specialization. In monolithic models, identifying the source of an error may require broader and less conclusive investigations.

Interpretability, European rules, and the next challenges of AI

The research of the Federal Polytechnic of Lausanne It is part of a more general shift in language model design. After a phase dominated primarily by increasing parameters, computational power, and dataset size, numerous groups are exploring strategies that focus on modularity, efficiency, controllability, and interpretability.

The number of parameters continues to be an important consideration, but it isn't necessarily the only way to improve AI capabilities. Work on MiCRo suggests that the internal organization of the model and the introduction of specific constraints during training can also yield measurable benefits.

The possibility of better analyzing the behavior of systems also becomes interesting in the European regulatory framework.AI Act It entered into force on 1 August 2024, while the obligations regarding governance and general purpose AI models became applicable on 2 August 2025. The regulation includes, among other aspects, obligations relating to technical documentation, transparency, information sharing and risk management.

MiCRo was not designed to ensure compliance with the AI ​​Act, and its modular structure alone is insufficient to meet regulatory requirements. However, architectures that make internal processes more observable could ultimately contribute to the analysis, validation, and auditing activities conducted by developers and professional users.

Many questions remain open. It will be necessary to verify whether the specialization observed in the one- and three-billion-parameter models can be maintained in much larger systems, trained on more heterogeneous datasets and subjected to ambiguous requests from real-world contexts.

Scaling to tens or hundreds of billions of parameters could make it more difficult to maintain clear functional boundaries between experts. It will also be necessary to evaluate the computational costs of training, the stability of routing, the possibility of modules gradually losing their specialization, and any trade-offs between transparency and response quality.

The project therefore does not represent a definitive solution to the opaque nature of linguistic models. However, it does point to a possible alternative direction beyond simple dimensional growth: building systems in which capabilities emerge not only from the quantity of parameters, but also from an internal structure organized according to recognizable functions.

In an industry where performance is growing rapidly, but user trust and verifiability remain open issues, understanding which component contributed to a response may become almost as important as the response itself. The challenge for future generations of deep learning It will therefore not only produce better results, but make clearer the way in which those results are constructed, transforming the research and development into a tool for building more analyzable and controllable systems.

Here are three insights that might interest you:

From EPFL a new multimodal model for more flexible AI
The brain separates logic and language: evidence comes from MIT
Apertus is the Swiss "open AI" for a more transparent future.