New GRIT Approach Teaches AI to Link Text Reasoning with Visual Evidence

Multimodal large language models (MLLMs) aim to merge visual data with linguistic reasoning. Visual content spans photographs, diagrams, and drawings, and reasoning systems must tie each step to the exact region that informs it. Yet many struggle to integrate the two domains in complex tasks that require visual evidence. Current systems often produce text explanations without pointing to image regions that inform their output. This produces answers that rest on unseen visual cues. It proves difficult to train models to interleave picture-based reasoning with logical steps in one flow. Models rarely display the bounding boxes or other visual markers that back up their conclusions. Learning a workflow that binds text and images without needing huge annotated image–text datasets remains a deep puzzle.

Approaches so far raise either text reasoning or bounding box grounding. Several teams have tried reinforcement training rules or prompt engineering to steer models toward visual grounding. Some models output coordinates alone, leaving gaps in explanation. Others trace reasoning chains only in text, risking detachment from visual content. Separation between grounding and logic means no single output ties a bounding box to a reasoning step. Solutions that employ dense supervision or external tools demand heavy labeling and lack scale. This forces developers to choose between transparency and task coverage. Creating an all-in-one method that guides models to ground every logic move in an image with little annotation calls for a new path.

A team at UC Santa Cruz and researchers at eBay designed a method called Grounded Reasoning with Images and Text (GRIT). GRIT extends chain-of-thought ideas into the vision domain, treating boxes as part of the reasoning sequence. It trains models such as Qwen 2.5-VL and InternVL 3 to build reasoning chains that mix natural language with explicit box coordinates for critical regions. This unified output links every logic step to image segments. Training uses a custom reinforcement algorithm, GRPO-GR, which rewards correct answers and well-structured reasoning. The model learns to insert tokens like and , plus bounding box formats. By focusing on sequence structure and token usage, the system avoids the need for large annotated reasoning sets.

In GRIT’s design models generate boxes and text at once, tapping their internal vision understanding. Rather than fetching cropped image patches after a coordinate appears, the model draws on its built-in features. Bounding boxes form part of every reasoning step, and reinforcement signals push the system to respect both answer accuracy and the logic style encoded in special tokens and box syntax. Integrating bounding boxes mid-reasoning encourages transparent pipelines that developers can audit step by step. Training relied on just 20 image-question-answer sets drawn from Visual Spatial Reasoning and TallyQA. Visual Spatial Reasoning tasks hinge on understanding object arrangements, and TallyQA focuses on counting elements spread across image layouts. Training ran for 200 steps on NVIDIA A100 GPUs, with AdamW and a cosine scheduler. Even this brief schedule produced robust visual reasoning skills.

Intersection over Union (IoU) scores reveal gains in grounding quality, with GRIT models scoring 0.325 on VSR and 0.447 on TallyQA. Higher IoU values reflect sharper alignment between model focus and true visual evidence. Tests against standard baselines show major improvements across both reasoning and grounding. Qwen 2.5-VL with GRIT reached 72.9% accuracy on Visual Spatial Reasoning, 47.8% on TallyQA, and 62.8% on the GQA benchmark. Direct Query and chain-of-thought approaches lagged behind, scoring far lower on unified reasoning and visual grounding.

GRPO-GR shapes reward signals to favor clarity in both language text and box formats, guiding models to tie each thought to precise image segments. This reward shaping helps cut down on spurious attention moves that lack justification. Out-of-domain evaluation delivered smaller uplifts, underlining that wider training data variety will boost generalization.

This work tackles the gap between visual grounding and logical reasoning in multimodal AI, producing a compact yet effective approach to unified outputs. GRIT trains MLLMs to link every reasoning move with relevant image cues, promising clearer, more interpretable AI. Requiring minimal input data opens possibilities for low-resource settings and specialized domains.