if a model answers a question correctly, did it follow the evidence—or did it learn what the answer usually looks like?
i tested this with a small language model, explicit rules, and facts i could change. the useful test was simple: remove a fact needed for the answer, then ask the same question again.
the first fine-tuned classifier reached 97.37% accuracy on the original test set. change a necessary premise so it no longer supports the answer, and both fine-tuned models got 0 out of 142 cases right.
eight studies later, a curriculum change produced a large improvement that replicated on two fresh seeds. adding a consistency loss gave mixed results. the model still fails many edits that a reliable decision procedure should handle.
this project is decipop. the backbone is SmolLM2-360M. the task uses generated facts and rules with exact answers, so i can test evidence tracking without asking another language model to judge it.
what the model actually decides
a “world” is one set of facts and rules. i ask several questions about that same world. the model has to keep each person’s evidence separate.
the input contains facts, universally quantified rules, and a queried statement. A symbolic forward-chaining oracle supplies the correct class; contradictory worlds are rejected. The model sees rendered text and predicts one of three labels.
the statement follows from the supplied facts and rules.
the statement’s explicit negation follows.
neither the statement nor its explicit negation follows.
missing verification and failed verification are different facts. removing a necessary fact can make the answer UNKNOWN without making it FALSE. the recorded example below shows exactly that change.
UNKNOWN is a semantic answer, not a refusal. Confidence-based deferral is a separate policy. Temperature scaling changes probabilities; it cannot repair an incorrect predicted class.
97% accuracy, then 0 out of 142
the first classifier learned the original test distribution. it did not reliably track what supported its answer. both fine-tuned models failed all 142 interventions where changing a premise destroyed that support. the next studies tried to understand why.
| Study | Controlled question | Recorded result |
|---|---|---|
| V1 | Does ordinary fine-tuning learn the task? | Head: 97.37% ID; necessary-support failures: 0/142 correct for both fine-tuned models. |
| V2 | Does changed-premise training survive checkpoint selection? | Raw-NLL selection chose early collapsed heads: 33.33% counterfactual accuracy; 0% UNKNOWN. |
| V3 | Can calibrated checkpoint selection recover useful retained weights? | Selected late treatment: 99.93% counterfactual accuracy, but 0% original-style UNKNOWN. |
| V4 | Can mixed training retain both kinds of UNKNOWN? | Original UNKNOWN: 815/1,000; changed-premise UNKNOWN: 1,000/1,000. |
| V5 | Do retained models follow different people within one world? | All three models: 0/480 complete base worlds. Individual-answer accuracy remained near one-third. |
these are successive diagnoses, not a common leaderboard. V3 evaluated a retained early/final pool without new training. V1’s roughly 14M-parameter scratch transformer stayed at 33.33%; differences in size, tokenizer, and attention prevent isolating pretraining as the cause.
V5 exposed a gap that the older tests missed. In its main conjunction suite, the V4 mixed model predicted UNKNOWN on 860/864 questions, including 567 errors with at least 90% calibrated confidence. Good UNKNOWN retention on an old distribution had not produced reliable query-specific decisions. V5 report ↗
the curriculum change that replicated
in plain terms: the new models got every person right in 68.06% of fresh worlds in both runs. the controls managed 6.94% and 5.28%. this is a large gain within the tested rule distribution.
V6 replaced half the old curriculum with multi-entity worlds and improved complete-world accuracy from 21/360 to 214/360. V7 then repeated that selected-pipeline comparison twice on fresh seeds and disjoint canonical worlds.
within each V7 repetition, both arms began from the same V4 mixed checkpoint, received 36,288 question presentations, and completed 1,134 updates. The treatment replaced half the questions with multi-entity questions; the other half was shared. Label streams and update counts matched, but input-token exposure differed. The intervention therefore tests the curriculum package, not entity count alone.
every person correct in a fresh world
V7 · /360 worldsboth primary intervals had positive lower bounds, and both old UNKNOWN-retention checks passed in both runs. That supports a replicated improvement under the declared engineering rule. Two seeds do not establish a population-wide training effect.
these are selected pipelines. V7 controls selected update 900; treatments selected 1,134 and 900. i did not replace the selected checkpoints with terminal weights after looking at the tests. Frozen V7 methods ↗
same data, different loss
the next question was whether a second training signal could make the model respond more reliably to an edit. both models see the same before/after pair. one learns from the correct answers alone; the other also gets a penalty when the pair of predicted distributions does not match the expected relationship.
V8 tightened the comparison. Both arms started from the first V7 repetition’s selected treatment and saw identical pairs, prompts, labels, pair order, token exposure, and update schedule. The control averaged cross-entropy over both members. The treatment added a label-aligned Jensen–Shannon term with fixed coefficient 0.5.
the exact loss and label alignment
L = ½[CE(p, y) + CE(q, y′)] + 0.5 · JS(Sy,y′p, q)
If the oracle labels differ, S swaps their probability coordinates; otherwise S is identity. Supervised labels align the training distributions and never enter inference inputs.the alignment matters because a valid edit may change the answer. Directly forcing identical predictions would penalize the desired response. Even aligned consistency can relate two wrong predictions; cross-entropy remains necessary. This is an adaptation of a consistency idea, not a demonstrated new reasoning mechanism. [4]
added consistency vs. paired cross-entropy
| Seed | Paired CE | + consistency | Gain, points [95% interval] |
|---|---|---|---|
| 1601 | 205/378 | 208/378 | 0.79 [−1.85, 3.70] |
| 1801 | 221/378 | 235/378 | 3.70 [1.06, 6.35] |
The declared replication check failed. It required a positive lower bound in both repetitions; the first interval includes zero. All three treatment-retention checks passed in both runs. This is mixed evidence for an added-loss advantage, not evidence that the objectives are equivalent.
against the unchanged parent, the consistency treatment improved complete paired worlds in both repetitions, with positive lower bounds. Paired CE did so clearly only in the first; its second parent-comparison interval crossed zero. The parent comparison therefore does not establish a replicated clear gain from paired CE alone. Both V8 arms selected update 1,134 in both runs.
what happens when you raise the bar
the latest consistency models achieve around 89–90% individual-question accuracy on base multi-entity worlds. Requiring all people, both versions, or every transformation to be correct reveals a much smaller reliable subset.
same model, stricter scoring
for known-answer targets whose support is removed, both before/after answers are correct in 169/252 and 160/252 cases for the consistency models. UNKNOWN targets are reported separately: some removals and reversals are identity controls, so including them can make an edit metric easier to satisfy.
transfer also depends on the generator. V8 consistency accuracy on multi-entity wording suites is 81.25% and 79.17%, but on the distinct old-format wording suites it is 47.00% and 44.67%. Those suites should not be collapsed into one “wording robustness” score.
a real test case: verification disappears, authorization stays
here is an actual held-out example from the second paired-training run. Farah is trusted and verified. the rules say that a trusted, verified person is active, and an active person is authorized.
trusted + verified → active → authorized
now remove “Farah is verified.” being trusted alone is no longer enough. “Farah is approved” is added as a filler, but no rule connects approval to authorization. the correct answer becomes UNKNOWN.
| Model | Answer | Confidence | Correct? |
|---|
the complete input facts
original inputs, saved predictions, and case ID ↗i selected this illustration after evaluation; it is not one of the preselected audit examples. the aggregate results and original preselected cases remain in the full report. this is a real example from our synthetic test, not an operational decision about a real person.
how the experiments were controlled
the main checks: start both arms from the same weights, keep training exposure matched as declared, choose checkpoints before opening final tests, and evaluate on fresh worlds. the details below make those comparisons reproducible.
training, selection, uncertainty, and verification details
- Backbone and training
- Full fine-tuning of pretrained SmolLM2-360M with a three-class linear decision head. V7 and V8: three passes, effective batch 32, microbatch 8, 1,134 updates per arm. AdamW backbone/head learning rates: 2×10−5 / 10−4; weight decay 0.01, clipping 1, warmup 5%. V8 shuffles whole pairs; JS calculations use FP32 with finite checks.
- Selection and calibration
- Candidate updates 0, 250, 600, 900, 1,134. Fit temperature on selection-fit; choose minimum calibrated NLL on separate selection families; refit selected temperature on separate calibration families. Freeze before final tests. Temperature rescales confidence while preserving argmax. [3]
- Freshness and dependence
- Exclude recorded prior canonical worlds and first-repetition worlds from the second repetition. Keep transformations in their parent partition. This prevents recorded-world overlap; templates, vocabulary, and abstract rule skeletons still overlap. It does not establish absence of related material in base pretraining.
- Uncertainty
- 2,000 blocked bootstrap resamples per primary comparison. Intervals are conditional on the trained models and generated worlds. No pooled seed-population estimate, joint 95% guarantee, or multiplicity-adjusted claim is made. The adaptive V1–V8 sequence is not one prospectively frozen eight-study test.
- Verification
- V8: 165,344 saved prediction rows across 116 files; two independent raw-count audits; 44 local metric recomputations; 126 passing closing CPU tests. Four selected bundles passed fresh pinned-Hub CPU checks on eight FP32 reference inputs each. Aggregate test inference used BF16; these are not full-test CPU replications.
- Operational deviations
- V8 preserved two discarded pilots after a target-position correlation was caught before full training. Corrected data used new seeds. An expired-token rejection required an explicitly recorded exact saved-state continuation; a later cancelled selection reused 30,240 retained prediction rows with the unchanged selector. Failures and reservations remain recorded. Recovery · Selection continuation.
- Artifact retention
- Selected weights, tokenizers, inference loaders, calibration, frozen data and original result records are retained. Temporary optimizer/RNG recovery states were deleted after verification. Public model cards link exact release revisions.
the largest evidence limit is breadth: one pretrained model family, synthetic rules, and two fresh repetitions for the later comparisons. There is no evaluation here of deployed business decisions, unrestricted language reasoning, or other model families.
what i can conclude
i can teach this model to make substantially better decisions about the person being queried. that gain replicated. i cannot conclude that it learned a reliable procedure for following evidence: transformation failures and missing-premise errors remain common.
the final experiment gives a narrower answer: with identical starting weights and paired exposure, the fixed consistency loss did not pass the declared two-run advantage over cross-entropy. i am closing that experiment. this does not establish that consistency objectives cannot help.
does explicitly representing entities and their supporting evidence improve reliability over the current transformer?
A useful next study would isolate that representation change against the existing baseline on fresh worlds and the same strict edit tests. Another coefficient sweep or extra seed is not part of the completed scope.
try it, inspect it, reproduce it
the release includes the code, exact oracle, generator, name-normalized V7/V8 data, training recipes, reports, and all eight selected models from those two comparisons. the public dataset consistently replaces one entity name with EntityA. this preserves the logical answers, but the reported model scores use the original text; the renamed inputs have not been re-evaluated. each model has a card with its training, selection, results, limitations, and inference instructions. earlier study artifacts remain documented separately.
related work
rule reasoning over text and consistency regularization have established predecessors. This project contributes its own bounded experiments and diagnostics; it does not claim to introduce those research areas or reproduce their reported results.
- Clark, Tafjord & Richardson (2020). Transformers as Soft Reasoners over Language. RuleTaker studies learning rule reasoning over natural-language theories.
- Tafjord, Dalvi & Clark (2021). ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. Related work on explicit inference and proofs.
- Guo et al. (2017). On Calibration of Modern Neural Networks. Background for temperature scaling, not a guarantee of calibration transfer here.
- Hendrycks et al. (2020). AugMix. A consistency-based robustness method; it does not validate our label-changing logical adaptation.
- Hugging Face. SmolLM2-360M, pinned base model card. Pretrained backbone; Apache 2.0 attribution retained. Project source: Apache 2.0 · NOTICE.