We often assume that when an AI judges your personality, it is carefully analyzing your behavior. But new research suggests these models might just be guessing based on superficial stereotypes. This distinction matters deeply as multimodal AI systems take on more human-facing roles, from customer service to mental health support. If the AI gets the label right but for the wrong reasons, it is not understanding you—it is prejudging you.

Researchers have identified a critical gap in how we test these systems. Current benchmarks only check if an AI can predict numerical personality scores, like the Big Five traits. They do not verify if the model actually understands the behavioral evidence behind those scores. To fix this, the team introduced Grounded Personality Reasoning, a task that forces models to anchor every rating in observable evidence. They also released MM-OCEAN, a dataset of 1,104 videos with timestamped behavioral observations and 5,320 multiple-choice questions designed to test this specific capability.

The results reveal a startling "Prejudice Gap." When benchmarking 27 different multimodal large language models, researchers found that 51% of correct personality ratings were not grounded in any retrieved behavioral cues. In other words, half the time the AI got the answer right, it was likely relying on pattern matching or bias rather than actual reasoning. The ability to provide a fully grounded, holistic explanation was even rarer, spanning only 0 to 33.5% across the models. This exposes a dangerous disconnect between getting the right score and reasoning for the right reason.

The takeaway is clear: accuracy is not enough. For AI to be trusted in social contexts, it must move beyond superficial pattern matching. We need systems that can explain their conclusions based on concrete evidence, not just statistical guesses. Until then, we are relying on AI that may be right for all the wrong reasons.