Everyone is talking about the rapid advances in artificial intelligence and the famous LLMs (large language models). But do you really know what lies behind our everyday assistants?
Everyone is talking about the rapid advances in artificial intelligence and the famous LLMs (large language models). But do you really know what lies behind our everyday assistants?
To answer this question, Matteo Dora, CTO of Giskard; Amaia Cardiel, a PhD candidate at Valeo; and Yann Eric CHOHO, a PhD candidate in machine learning at Ekimetrics, gathered for a roundtable discussion during our latest AI Breakfast.
Amaia applies her research to a field where there is no room for error: autonomous vehicles. To audit her driving models, she uses a tool called “counterfactual explanation.”
How does it work? It involves finding the smallest change to make to an image or text that would cause the AI to radically change its decision. (For example: If the car is going straight, what tiny detail would have caused it to turn right?)
In her case, the self-driving car model had conflated accelerating with passing: once it exceeded a certain speed, it would systematically shift into the left lane.
Why? Because it had mindlessly memorized its training data: in reality, when you pass another vehicle, you accelerate and move to the left. On paper, the AI’s scores were perfect. In practice, it was operating for the wrong reasons. Without this analysis, the bias would have remained hidden.
Explainability made it possible to link the situation to the decision-making process, which in turn made it actionable and allowed us to resolve this problematic behavior.
To understand how AI thinks, Matteo uses a rather original method: he draws inspiration from neuroscience.
“In neuroscience, we can’t observe behavior directly inside the brain. We present tasks and measure the response. We do pretty much the same thing with models.”
This is known as the “black box” approach. We don’t know exactly what’s going on inside, so we test the model to build its profile.
For example, evaluating music theory proves much more complicated due to its unique nature.
LLMs’ capabilities in this area are also limited for the same reasons.
However, beware of the pitfall: it’s better to have no explanation at all than a misleading one. A false justification leads to blind trust on the part of the user…
Furthermore, this “black box” approach quickly reaches its limits, particularly in a legal context (such as the AI Act, for example), which requires impeccable transparency from the model.
The GDPR, meanwhile, requires explainability in the context of an automated decision; the user has the right to obtain it….
Yann Eric, a PhD student in AI, actually prefers to refer to it as a “glass box.” You can see all the numbers and parameters inside, but you simply can’t read that coded language.
To decipher how AI works, he focuses on concepts.
For an AI to recognize a bird, it looks for intermediate concepts: a beak, wings, and legs. The goal of his research is to force the AI to validate these human concepts before providing its answer. By modifying these concepts, we can correct the model’s trajectory in real time.
Because AI doesn’t think like we do: it loves co-occurrences. If you associate a soccer player with a country too often in its data, it will get the nationality wrong as soon as you change the context.
You’ve probably noticed that the new models (such as those labeled “Reasoning”) display their “chain of thought” before providing the final answer. It feels like you’re witnessing a real thought process.
Big mistake! Researchers are unanimous: this text is nothing more than a justification constructed after the fact (post hoc). It doesn’t accurately reflect what actually led the machine to choose its answer. An AI can follow a flawless line of reasoning… and still produce a completely wrong result (or vice versa).
What’s more, these models have staggering blind spots. An LLM capable of solving quantum mechanics or ultra-complex math problems can still get basic concepts wrong.
Since AI remains a massive statistical machine, it’s up to us humans to stay in control. Here are three tips from experts for your daily prompts:
Want to deploy reliable and explainable AI systems rather than opaque “black boxes”? Contact our experts!
And to make sure you don’t miss any of our news, follow us on LinkedIn !
