Many people interacting with language models issue queries that lack sufficient detail, often leaving outputs disconnected from actual needs. A simple inquiry like “What book should I read next?” hinges on personal preferences, and “How do antibiotics work?” calls for a response tuned to the user’s scientific background. Traditional evaluation benchmarks typically ignore such missing context, leading to mixed or unfair ratings of model outputs. For example, a suggestion praising coffee could be harmless for some but risky for individuals with caffeine sensitivity or heart conditions. Absent insight into a user’s goals or background, fair assessment of a response’s value becomes elusive.
Past work has generated follow-up questions to reduce ambiguity in question-answering, dialogue systems and search. Research into instruction-following and personalization highlights the value of adapting replies to user traits such as expertise level, age or style preference. Other studies probe model generalization across diverse scenarios and propose training strategies to boost adaptability. Automated evaluation tools have gained traction thanks to speed but often carry biases, driving teams to develop clearer guidelines and reduce unfair leanings.
A group from the University of Pennsylvania, the Allen Institute for AI and the University of Maryland, College Park introduced contextualized evaluation, which supplements underspecified queries with synthetic follow-up question-and-answer pairs. Their findings show that adding context can radically shift evaluation outcomes—including reversing model rankings—and raises agreement among raters. It also curbs overemphasis on surface features like style and exposes default biases toward WEIRD (Western, Educated, Industrialized, Rich, Democratic) perspectives. Models show varied sensitivity to different contextual details.
The researchers set up an accessible pipeline. They selected vague queries from standard benchmark collections and enriched each with follow-up question-answer pairs simulating realistic user scenarios. They then harvested answers from multiple language models under two conditions: original prompts alone and prompts enhanced with context. Human reviewers and automated evaluators rated these responses side by side. This design quantifies how added details influence model rankings, rater consistency and the judging criteria, and it offers a replicable way to test systems against real-world ambiguity.
Supplementing prompts with user-specific details—such as intended audience or precise goals—boosts evaluation quality. Inter-rater agreement rose by 3 to 10 percent, and in some matchups the leading model changed place. GPT-4 surpassed Google’s Gemini-1.5-Flash only when context was included. Without context, raters tend to focus on tone and fluency; with it, they zero in on accuracy and utility. Default model outputs often presume a Western, formal, general audience, leaving diverse users underserved. Benchmarks that ignore these factors risk misjudging a model’s real-world usefulness.
This study covers a limited range of context types and depends partly on automated scoring; nevertheless, it makes a compelling case for future evaluations to employ richer, user-aware prompts and matching scoring rubrics. Such changes would better reflect the varied backgrounds and intentions of real-world users.