Hard Negation

I did not set out to build a negation dataset because models could not understand the word not.

The obvious version of the problem has improved. Give a frontier LLM a clean sentence and ask what happens when you insert not, and it will often answer correctly. That does not mean the retrieval layer has learned the same lesson.

The failure I cared about lives in the gap between surface similarity and truth.

Two sentences can share the same nouns, roles, domain, and rhythm while disagreeing at the one place that matters. Cosine similarity sees overlap. A vector database sees a neighbor. A retrieval system may hand the wrong passage to the model. Then the LLM answers from bad ground with a calm voice.

That is not a spelling problem. It is not a keyword problem. It is a structural problem.

The Research Warning

Negation has a long history of making language models look smarter than they are.

MoNLI showed that models trained on general-purpose natural language inference data can fail systematically on examples where lexical entailment and negation interact.¹ NaN-NLI pushed on sub-clausal negation, where the negation does not simply flip a whole sentence but changes a smaller linguistic unit.² ScoNe focused on scope: whether zero, one, or two negation morphemes actually affect the label in controlled contrast sets.³

The pattern did not disappear when models got larger. Truong and colleagues evaluated LLMs across negation benchmarks and found several limits: insensitivity to negation, weak handling of lexical negation, and failures when reasoning under negation.⁴

That matters because production retrieval often asks an even harder question than NLI:

Given this query, which passage should the model trust?

If the query says a policy does not apply, a passage about when the policy applies can be dangerous. It is topically relevant and semantically wrong. That is the exact shape embedding systems are tempted to accept.

Easy Negation Is Too Easy

Bad negation data teaches the wrong lesson. If every example is just:

The policy applies.
The policy does not apply.

then the model can learn a cheap alarm around one token. That helps on toy tests and fails in the wild.

Real negation is messier. It lives in scope, absence, failed expectations, conditionals, comparatives, modality, exceptions, tense, roles, quantities, and idioms. Sometimes the sentence never says not while still reversing the proposition.

The goal was not to teach a model to recognize a negation marker.

The goal was to make it stop confusing overlap with meaning.

That changed the data target. Positives had to preserve the intended meaning. Hard negatives had to be close enough to tempt an embedding model but different enough that a careful human would reject the match immediately.

The data had to be mean.

What the Data Looks Like

Not failing on not is not enough.

Here are a few examples from the Super-Neg unit corpus. Each triplet has a query, the passage that should match, and a hard negative that looks close enough to tempt a retrieval system that is mostly rewarding shared words.

Reactor cooling.

Query: “The non-activation of the auxiliary cooling fans did not lead to a rise in core reactor temperature.”

Correct match: “The reactor vessel maintained a constant temperature reading throughout the entire shutdown process.”

Trap: “The reactor vessel experienced a sharp temperature spike during the entire shutdown process.”

Field software.

Query: “The software update did not cause a failure of the redundancy handshake in the field.”

Correct match: “The redundancy handshake completed successfully during field operation after the scheduled software update.”

Trap: “The telemetry recorder that monitored the handshake reported a missed acknowledgment window after the scheduled software update.”

Battery coating.

Query: “The proprietary polymer coating does not prevent moisture ingress into the battery cell.”

Correct match: “Water vapor freely penetrates the cell assembly despite the applied polymer layer, compromising electrolyte stability over extended storage periods.”

Trap: “The polymer barrier successfully blocks water vapor diffusion into the cell assembly, maintaining electrolyte integrity across 18-month shelf-life protocols.”

Edge-device fleet.

Query: “Not every firmware update deployed across the edge device fleet resolved the memory leak condition.”

Correct match: “Several gateway nodes continued to exhibit heap fragmentation issues despite receiving the latest patch releases.”

Trap: “Memory consumption stabilized across all edge devices following patch application, with heap fragmentation eliminated by the firmware rollout.”

These are the pairs I care about. The wrong answer is often fluent, topical, and even useful looking. It just says the opposite thing.

The Failure Modes

The core failure is semantic collapse under lexical overlap. The pair looks close enough that the vector space shrugs.

Hard negation needs to cover more than direct verbal negation:

• Polarity flips: one claim affirms, the other denies.
• Scope failures: the model sees negation but attaches it to the wrong part of the sentence.
• Role inversions: the same entities appear, but agent and patient swap.
• Comparatives: less effective than carries a negative relation without a simple negation word.
• Absence and failure: lacked evidence, failed to comply, missing consent.
• Conditionals and exceptions: the rule changes only under a narrow condition.
• Idioms and litotes: the literal surface is not the actual meaning.

A retrieval system that cannot handle those cases will appear to work until the corpus contains enough near-misses. Then the wrong passages become easy to retrieve because they are so lexically attractive.

The Gauntlet

LLM-generated data has its own failure mode. Leave a generator running long enough and it starts to find grooves. The wording changes, but the move repeats. Same sentence shape. Same contrast. Same trick in a different jacket.

That is semantic collapse.

The gauntlet was built to catch both problems. Every candidate had to survive checks for semantic separation, lexical overlap, polarity, near-duplicate behavior, hard-negative pressure, and syntactic diversity. I did not want a pile of fluent examples. I wanted examples that stayed hard after the cheap shortcuts were stripped away.

The best hard negative is uncomfortable. It should look close. It should feel plausible. It should make the model prove that it can follow the hinge instead of simply rewarding shared words.

That is the point of hard negation: not to prove the model can read a negation marker, but to force it to care when meaning turns.

Why It Matters

RAG systems are only as honest as their retrieval. If the retrieval layer pulls the wrong passage, the generator inherits the mistake. A bigger LLM may explain it more fluently, but fluency is not repair.

Hard negation is one way to pressure-test that layer.

It asks whether the system can separate claims that look similar while disagreeing in the part that matters. It asks whether the embedding space respects structure, scope, and role. It asks whether a reranker can recover when lexical overlap is lying.

For me, the motivating image is still an RPG table. If the system says the duke betrayed the queen when the queen betrayed the duke, the world breaks. In production, the stakes are contracts, medical policies, incident reports, financial disclosures, source code, and safety rules.

The lesson is the same.

Not every fluent pair deserves to survive.

Not every close neighbor is true.