Unbiased Visual Reasoning with Controlled Visual Inputs
Separates perception from reasoning with an information bottleneck between a frozen VLM sensor and an RL-trained text reasoner to resist spurious cues.
Abstract
End-to-end Vision-language Models (VLMs) often answer visual questions by exploiting spurious correlations instead of causal visual evidence, and can become more shortcut-prone when fine-tuned. We introduce VISTA (Visual-Information Separation for Text-based Analysis), a modular framework that decouples perception from reasoning via an explicit information bottleneck. A frozen VLM sensor is restricted to short, objective perception queries, while a text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language. This controlled interface defines a reward-aligned environment for training unbiased visual reasoning with reinforcement learning. Instantiated with Qwen2.5-VL and Llama3.2-Vision sensors, and trained with GRPO from only 641 curated multi-step questions, VISTA significantly improves robustness to real-world spurious correlations on SpuriVerse (+16.29% with Qwen-2.5-VL-7B and +6.77% with Llama-3.2-Vision-11B), while remaining competitive on MMVP and a balanced SeedBench subset.
Citation
@misc{li2025unbiased,
title={Unbiased Visual Reasoning with Controlled Visual Inputs},
author={Zhaonan Li and Shijie Lu and Fei Wang and Jacob Dineen and Xiao Ye and Zhikun Xu and Siyi Liu and Young Min Cho and Bangzheng Li and Daniel Chang and Kenny Nguyen and Qizheng Yang and Muhao Chen and Ben Zhou},
year={2025},
eprint={2512.22183},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2512.22183}
}