Visual reasoning tasks push AI to handle images beyond mere object recognition. They span medical diagnostics, visual math, symbolic puzzles, and image-based question answering. Success hinges on dynamic adaptation, abstraction, and contextual inference. Models must analyze visuals, identify relevant features, and often produce explanations or solutions via a series of reasoning steps tied to the visual data.
Many existing systems rely on pattern matching or fixed routines. They struggle with unfamiliar problems or abstract scenarios that demand more than surface‐level cues. Such systems can’t adjust strategies on the fly or build new reasoning tools, so complex tasks remain out of reach.
Earlier solutions like Visual ChatGPT, HuggingGPT, and ViperGPT integrate modules for segmentation, detection, or other vision tasks. Those workflows are predefined and single-turn, which limits multi-stage reasoning and blocks any expansion of the toolset during a session.
A framework named PyVision tackles these gaps. Developed by teams from Shanghai AI Lab, Rice University, CUHK, NUS, and SII, it lets large multimodal language models generate and execute custom Python utilities in a multi-turn cycle. Models can adapt their code, review outcomes, and refine their approach across several steps.
PyVision begins with an image and a query. A model such as GPT-4.1 or Claude-4.0-Sonnet writes Python snippets that run in an isolated environment. Outputs—text, visuals, or numbers—return to the model. It preserves variable states between turns, adjusts its plan, and issues new code until the final answer emerges. Safety measures include process isolation and a structured I/O scheme. Python libraries like OpenCV, NumPy, and Pillow handle tasks such as segmentation, optical character recognition, image enhancement, and statistical analysis.
Benchmarks show clear gains. On the visual search test V*, GPT-4.1 rises from 68.1% to 75.9%, a 7.8-point boost. Claude-4.0-Sonnet jumps from 48.1% to 79.2% on VLMsAreBlind-mini, up by 31.1 points. GPT-4.1 also posts +2.4% on MMMU and +2.5% on VisualPuzzles. Claude-4.0-Sonnet scores +4.8% on MathVista and +8.3% on VisualPuzzles. Perception-focused models record larger gains on vision-heavy tasks, and reasoning-oriented systems see stronger improvements on abstract challenges. PyVision builds on each model’s core strengths rather than masking them.
By letting AI develop problem-specific tools in real time, PyVision shifts static models into interactive solvers that link perception and reasoning for complex scenarios.