Everyone is talking about the rapid advancements in artificial intelligence and the famous LLMs (Large Language Models). But do you really know what’s happening behind the scenes of our everyday assistants? 

To answer this question: Matteo Dora, CTO of Giskard, Amaia Cardiel, PhD candidate at Valeo, and Yann Eric CHOHO, PhD Candidate in Machine Learning at Ekimetrics, joined us for a roundtable discussion at our latest AI Breakfast.

‍

When metrics are limited: a right answer for the wrong reasons

Amaia applies her research to a field where error is not an option: autonomous vehicles. To audit her driving models, she uses a tool called "counterfactual explanation."

The concept? Finding the minimal change needed in an image or text to make the AI radically change its decision. (For example: If the car is going straight, what tiny detail would have caused it to turn right?)

In her case, the autonomous car model had conflated accelerating and overtaking: past a certain speed, it would systematically swerve into the left lane. 

Why? Because it had mindlessly memorized its training data: in reality, when you overtake, you accelerate and move to the left. On paper, the AI's scores were perfect. In practice, it was working for the wrong reasons. Without this analysis, the bias would have remained invisible.

Explainability made it possible to link the situation to the decision-making process, which made it actionable and allowed for the resolution of this problematic behavior.

Black-box AI: beware of the traps

To understand how an AI thinks, Matteo uses a rather original method: he draws inspiration from neuroscience.

"In neuroscience, you can't observe behavior directly inside the brain. You propose tasks and measure the reaction. We do pretty much the same thing with models."

This is what we call the "black box" approach. We don't know exactly what is happening inside, so we test the model to build its profile.

For example, evaluating music theory proves to be much more complicated due to its uniqueness. 

LLM capabilities on this subject are also limited for the same reasons. 

However, beware of the trap: no explanation is better than a fallacious one. A false justification gives the user blind confidence…

Furthermore, this "black box" approach quickly hits its limits, particularly in a legal context (such as the AI Act) which requires impeccable model transparency. 

The GDPR requires explainability in the context of automated decision-making; the user has the right to obtain it....

From black box to "glass box"

Yann Eric, a PhD student in AI, prefers to speak of a "glass box." You can see all the numbers and parameters inside, but you simply don't know how to read that coded language.

To decrypt how an AI works, he focuses on concepts.

For an AI to recognize a bird, it looks for intermediate concepts: a beak, wings, legs. The goal of his research is to force the AI to validate these human concepts before providing its answer. By modifying these concepts, we can correct the model's trajectory in real time.

Because AI doesn't think like we do: it loves co-occurrences. Associate a soccer player with a country too often in its data, and it will get the nationality wrong as soon as you change the context.

Benefit from expert insight yourself!

You have surely noticed that new models (like those geared toward reasoning) display their "Chain of Thought" before giving the final answer. It feels like we are witnessing actual reflection.

Big mistake! Researchers are unanimous: this text is merely a post-hoc justification. It is not a faithful reflection of what led the machine to choose its answer. An AI can lay out perfect reasoning... and provide a completely wrong result (or vice versa).

Moreover, these models have staggering blind spots. An LLM capable of solving quantum mechanics or ultra-complex math can get basic concepts wrong.

Best practices from our speakers: 3 tips to tame AI

Since AI remains a massive statistical machine, it is up to us humans to maintain control. Here are 3 expert tips for your daily prompts:

1. Multiply the brains: Ask the same question to two or three different AIs and see if the answers converge.
2. Don't forget your critical eye: Never take the first answer for granted. Ask yourself: "Are you sure?" or give it a counter-example to see if it sticks to its guns or panics.
3. Demand proof: Systematically ask it to cite its sources, and go verify them yourself.

‍

Looking to deploy reliable, explainable AI instead of opaque "black boxes"? Get in touch with our experts!

‍