CAVEWOMAN

How large language models behave under linguistic input and output compression

EMNLP 2026, Budapest Main Conference
Morayo Danielle Adeyemi1, Ryan Rossi2, Franck Dernoncourt2
1Independent   2Adobe Research
Paper Code Data BibTeX
CAVEWOMAN hero illustration.

Abstract

"Talk short. Drop grammar. Save token." This caveman register is widely promoted as a way to cut inference cost, but whether it actually saves anything depends on which channel (the user's prompt or the model's response) is being compressed.

We present CAVEWOMAN, a two-channel evaluation protocol that scores every generation on task accuracy, realized per-item cost, and surface-text preservation against the model's unconstrained reference. We evaluate eight models (GPT-5.4, GPT-4o, Claude Sonnet 4.6, Claude Haiku 4.5, Gemma-4-E4B, Qwen3.5-9B, Qwen2.5-VL-7B, and DeepSeek-R1-Distill) on five datasets (GSM8K, BoolQ, ARC-Easy, CommonsenseQA, MMLU-STEM) at five reduction levels, with both channels measured on the same items.

Output compression cuts realized cost on nearly every model we measure (1.5× on average, up to 3×, across four API models at the first reduction level, with four open-weight models projected the same way under public-tier pricing). Input compression has the opposite effect, failing to reduce the bill and raising it on most models (up to 96% on the worst benchmark), since models compensate with longer responses.

Under the same setting, surface text diverges from the unconstrained reference: roughly half of correct answers come from outputs whose surface text no longer entails the model's own L0 generation. The divergence survives length-controlled re-scoring, multiple-comparisons correction, and replication under twelve semantic measures; we report it as a surface-text observation, not as propositional drift. We argue that compression should be evaluated channel-by-channel on realized cost, with answer-extraction rates audited before any accuracy gain is claimed.

Method

There are two things you can compress: what you send in, or what the model sends back. We test both, at five levels, on the same questions. The only thing that changes is which side gets compressed.

Input We strip the small words out of your prompt (the, a, of, in, and). The model is told nothing special and answers however it wants.
Output Your prompt goes through untouched. Instead we tell the model to answer in clipped, stripped-down language.

L0 is normal English. L4 is a hard 15-token limit. Both sides use the same five levels and the same length cap, so you can compare them directly.

We measure three things:

  • Did it get the answer right? When our pattern matcher cannot find the answer in a response, we have a model read it instead rather than marking it wrong. Those calls were checked against a second model (90.9% agreement) and four people.
  • What did it actually cost? We count the tokens actually used, in and out, at list prices, rather than how much shorter the prompt got.
  • Is it still saying the same thing? We check whether the compressed answer and the model's normal answer imply each other. Twelve different similarity measures all point the same way.

Confidence intervals come from 10,000 resamples. Comparisons across cells are corrected for multiple testing. The similarity gap holds when both answers are cut to the same length, so it is not a length effect.

CAVEWOMAN evaluation, level by level

Pick a level to see its input prompt, its output prompt, and an example response.

Question (shared across levels)

Janet's ducks lay 16 eggs per day. She eats three for breakfast, bakes muffins with four, and sells the rest at the farmers' market for $2 per egg. How much does she make daily?

Input prompt

Janet's ducks lay 16 eggs per day. She eats three for breakfast, bakes muffins with four, and sells the rest at the farmers' market for $2 per egg. How much does she make daily?

Unchanged at L0.

Output prompt

Reason step by step in full grammatical English sentences. Conclude with the final answer on its own line in the form Answer: <answer>.

Example output

Janet has 16 eggs each day.
She uses 3 for breakfast and 4 for
muffins, so 7 total. That leaves
16 - 7 = 9 eggs to sell.
Each sells for $2, so 9 * 2 = 18.

Answer: 18

Input prompt

Janet's ducks lay 16 eggs day. She eats three breakfast, bakes muffins four, sells rest farmers' market $2 egg. How much does she make daily?

Function words dropped: articles, prepositions, conjunctions.

Output prompt

Telegraphic. No articles, conjunctions, or prepositions. Use nouns, main verbs, numbers, and standard symbols. End with Answer: <answer>.

Example output

Eggs 16 day.
Breakfast 3. Muffins 4. Used 7.
Remainder 16 - 7 = 9.
Sell 9 * 2 = 18.

Answer: 18

Input prompt

Janet ducks lay 16 eggs day eats three breakfast bakes muffins four sells rest farmers market $2 egg make daily

Nouns, main verbs, and numbers only.

Output prompt

Keyword only. Use nouns and main verbs only. No grammar, no full sentences. Output as fragments or list items.

Example output

Eggs: 16
Eat: 3, bake: 4
Used: 7
Remaining: 9
Price: $2
Total: $18

Answer: 18

Input prompt

Janet ducks 16 eggs day three breakfast muffins four rest farmers market $2 egg daily

Nouns and numbers only. No verbs.

Output prompt

Noun-phrase skeleton. No verbs of any kind. Nominal fragments, numbers, and standard symbols only.

Example output

Eggs per day: 16
Breakfast use: 3
Muffin use: 4
Total use: 7
Remainder: 9
Price per egg: $2
Daily revenue: $18

Answer: 18

Input prompt

Janet's ducks lay 16 eggs per day. She eats three for breakfast, bakes muffins with

First 15 whitespace tokens.

Output prompt

Hard 15-token budget. The entire response must be 15 tokens or fewer. Prefer the raw answer over prose.

Example output

Answer: 18

Does it hold up elsewhere?

To check if this only works for short questions in English? We checked four ways, adding three more datasets to the five in the main set for eight in total.

Multilingual MGSM in Bengali, Chinese, English, French, German, Japanese, Russian, Spanish, Swahili, Telugu, and Thai. The English similarity checker does not work on non-English text, so there is a multilingual one you can switch to.
Long-form Summarizing news articles, where the answer is the text. Output side only, since the article is the input. Asking for a terser style does not reliably shorten the text; only L4, which sets a token count, does.
Size sweep Qwen2.5 from 0.5B to 32B. From 3B up, bigger models handle the constraint better. Below that it flips, but only because the smallest models start out so much worse that they have less to lose.
Thinking mode Qwen3 models write out their reasoning unless you explicitly turn it off. Leave it on and the reasoning eats the whole length budget, so every level comes out looking the same.

Everything is compared against a baseline: the same question asked twice with nothing compressed. Compression has to move the answer more than that.

Conclusion

  • Compressing what you send and asking to compress what you get back are two different problems. Accuracy alone will not tell them apart.
  • Cheaper and correct does not mean the same. An answer can cost less, be right, and still not be what the model would have told you otherwise.
  • Test with the constraint switched on. Models rank differently with it than without it.

Everything is released: the prompts, the per-cell results, and the scripts behind every table and figure.

Acknowledgements

We thank MIT Engaging for providing the GPU compute used to run the local-model sweep. We also acknowledge the maintainers of the open-source tools the work depends on, including spaCy, Hugging Face Transformers, and sentence-transformers.

This webpage template is adapted from Nerfies, under a CC BY-SA 4.0 License.

BibTeX

@misc{adeyemi2026cavewoman,
  title         = {CAVEWOMAN: How Large Language Models Behave Under
                   Linguistic Input and Output Compression},
  author        = {Adeyemi, Morayo Danielle and Rossi, Ryan A. and Dernoncourt, Franck},
  year          = {2026},
  eprint        = {2606.24083},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24083}
}