Skip to main content
← HomeJacob Dineen
←All publications

BOW: Training Language Models to Reason Over Plausible Next Words

Trains models with RL to describe the full space of plausible next words, which reduces single-sense collapse on ambiguous contexts.

Ming Shen, Zhikun Xu, Xiao Ye, Jacob Dineen, Ben Zhou

EMNLP 2026Jun 2025
Overview figure for BOW: Training Language Models to Reason Over Plausible Next Words

Abstract

Next-word prediction (NWP) trains language models against a single observed continuation, even though many contexts admit multiple plausible next words. Recent RL-based next-word reasoning methods make this tension explicit: they reward a model for producing a rationale that supports one context-conditioned continuation, which can turn a pre-existing preference into a confident, self-justifying trace. We introduce BOW, an RL framework that instead trains models to produce self-contained, neutral, and comprehensive descriptions of the plausible next-word space. BOW's core reward path routes credit through the reasoning trajectory itself: a policy generates a next-word reasoning trace, and a frozen scorer assigns the main reward from that trace alone, without access to the original context. Our full variant, BOW-Reg, adds a lightweight breadth regularizer that discourages premature collapse to narrow answer naming, while keeping the main reward path trace-only. Across ten general reasoning benchmarks, BOW remains competitive with the original instruction models and often outperforms trained baselines. More directly, on SharedRef and HoWN-Simple, BOW-Reg achieves the best referential-ambiguity correctness and lowest single-sense collapse on both LLM backbones we study. Human evaluation further shows that BOW-Reg elicits broader next-word reasoning traces, while the full method remains strong on intrinsic NWP.

Citation

@inproceedings{shen-etal-2026-bow,
    title = "{BOW}: Training Language Models to Reason Over Plausible Next Words",
    author = "Shen, Ming  and
      Xu, Zhikun  and
      Dineen, Jacob  and
      Ye, Xiao  and
      Zhou, Ben",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Junyi Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.852/",
    doi = "10.18653/2026.emnlp-main.852",
    pages = "19732--19754",
    ISBN = "979-8-89176-459-0"
}