Deep and Shallow Biases in Language Models

An Vo1,2,*, Vy Tuong Dang3,*, Khai-Nguyen Nguyen4, Emilio Villa-Cueva1, Thamar Solorio1,† Anh Totti Nguyen5,†, Daeyoung Kim3,†

*Equal contribution    †Equal advising

1MBZUAI    2University of Michigan    3KAIST    4University of Virginia    5Auburn University
Empirical Methods in Natural Language Processing (EMNLP2026)
Paper arXiv Code Dataset
📢 Finding: Large language models often collapse onto the same answer under direct prompting: they answer "Monarch" 100% of the time when asked to name a random butterfly, but only ~25% of these concentrated preferences survive scenario reframing. Most, like the "42" a model gives 73% of the time for "choose a random number," switch to a different answer when the question is reframed into a new scenario. Most bias detected by a single prompt is not robust to changes in framing. It can reflect a prompt-dependent effect rather than a stable model preference, and single-prompt bias scores cannot tell the two apart.

The Problem: Bias That Disappears When You Ask Differently

Ask Olmo-3-7B-SFT to name a random butterfly and it says "Monarch," every single time, 30 out of 30 tries. Ask it to name a random number instead and it says "42" on 22 of 30 tries. By the usual measure, both look equally biased. But reframe each question into a different everyday scenario, a butterfly for a birthday invitation, a number for a raffle ticket, and the two answers behave nothing alike. "Monarch" keeps coming back 93% of the time. "42" survives only 13% of the time, and the model's new favorite becomes "3." One of these is a real, stable preference. The other only looked that way because we asked it one specific way. A bias score computed from a single phrasing cannot tell the two apart, and treats them as the same thing.

How We Measure It: The Bias Depth Framework

  1. Direct rate (DR). Ask the direct question 30 times and measure how often the same answer comes back.
  2. Framed rate (FR). Ask 30 different everyday scenarios that hide the same underlying question, and measure how often the model returns to that same direct answer.
  3. Bias depth (π = DR × FR). Multiply the two rates. π is high only when a model both commits to one answer directly and keeps returning to it once the wording changes.
    • Deep bias — DR > 0.65, π > 0.40: top answer survives reframing.
    • Shallow bias — DR > 0.65, π ≤ 0.40: top answer doesn't survive reframing.
    • Non-bias — DR ≤ 0.65: no concentrated top answer.
Diagram of the bias evaluation framework, showing a Deep bias example (choosing a random butterfly, where Monarch survives reframing) and a Shallow bias example (choosing a random number, where 42 does not survive reframing).
Our bias evaluation framework. We first ask the model the direct prompt multiple times to find its direct top answer, then ask matched scenario reframings of the same question and check whether that answer returns. (a) A Deep bias survives reframing, as in Choose a random popular butterfly, where Monarch remains the top answer. (b) A Shallow bias does not survive reframing, as in Choose a random number, where the direct answer 42 no longer survives and the framed answers shift to 3.

Key Findings

Most LLM Biases Are Shallow

Averaged across four post-SFT models, 35.8% to 77.7% of prompt families look biased under direct prompting. But only about a quarter of that bias is Deep: on average 39.6% of prompt families show Shallow bias, versus just 13.2% Deep bias. Most of what a single-prompt score would call "bias" disappears once the question is reframed.

Bar chart showing Deep bias, Shallow bias, and Non-bias proportions across four models: Olmo-3-7B SFT, Tulu-3-8B SFT, Sonnet 5, and GPT-5.6 Sol.

Shallow bias is about three times as common as Deep bias, across every model tested, open-source and proprietary alike.

Deep Bias Is Mostly Set During Pretraining

Pretrained Olmo-3-7B already carries 17.8% total bias, over half of it Deep. SFT roughly doubles that total, almost entirely through new Shallow bias, and DPO and RLVR keep adding more the same way. Deep bias barely moves throughout.

Stacked bar chart of Deep and Shallow bias proportion across the Olmo-3-7B release pipeline: Pretrained, SFT, DPO, and RLVR.

Total bias climbs from 17.8% (Pretrained) to 51.5% (RLVR), almost entirely from Shallow bias.

Deeper Biases Are More Often Inherited From Pretraining

For prompts where the SFT model is biased, the deeper that bias is (higher π), the more likely its answer matches what the pretrained model already preferred, rising from 20.7% agreement at low π to 62.3% at high π. Biases that survive reframing are the ones most likely to trace back to pretraining rather than to something SFT introduced.

Scatter plot showing that the rate at which the SFT model's top answer matches the pretrained model's top answer increases with bias depth score.

Agreement with the pretrained model's answer rises steadily with bias depth π, from 20.7% to 62.3%.

Deep Biases Are Harder to Remove Than Shallow Biases

Continued LoRA fine-tuning on diversity-flattened answers cuts Shallow bias by 10.3 points but Deep bias by only 4.5 points. Prompt-based debiasing (GEPA) barely moves Deep bias at all, a 1.6-point drop versus 5.8 points for Shallow. Whichever method is used, the bias that survives reframing is the hardest one to remove.