The Parser Already Knows: Lightweight Bias Correction in Constrained Decoding
Published in arXiv preprint arXiv:2608.10137 [cs.CL], 2026
Grammar-constrained decoding masks invalid tokens to keep language model output syntactically valid, but this masking distorts the model’s true probability distribution. This paper proposes correcting that bias with a logit correction conditioned on the syntactic and lexical parser state together with candidate next tokens, restoring the true distribution without expensive resampling. The method reuses parser and lexer state already computed during incremental parsing, adding negligible overhead while improving output quality over both masking-only and sampling-based approaches.
Recommended citation: Işıl Özgü, Yaoxuan Wu, Guy Van den Broeck, and Miryung Kim. 2026. "The Parser Already Knows: Lightweight Bias Correction in Constrained Decoding." arXiv preprint arXiv:2608.10137.
Download Paper