# Can a Warmth-Trained Model Learn When Not to Agree? I compared reasons, a matched-length control, and character training after warmth SFT; none increased judged warmth, while M4 most strongly resisted explicit persona attacks. - Published: 2026-08-20 - Modified: 2026-08-20 - Tags: Alignment - Canonical: https://aroramrinaal.com/ai/persona-alignment-experiment I started this experiment with a tension that sounds simple but is difficult to measure: an AI assistant should be warm enough to support someone, honest enough to correct them, and boundaried enough not to become whatever the user asks it to be. The question came from the Nature study [Training language models to be warm can reduce accuracy and increase sycophancy](https://www.nature.com/articles/s41586-026-10410-0). Across five model families, its authors found that supervised fine-tuning for warmth could increase factual errors and agreement with incorrect user beliefs, especially when users expressed sadness. Warmth was desirable, but the training signal appeared to teach relational harmony at the expense of contradiction. I wanted to test a possible mitigation on a model small enough for me to train end to end. Could a short, user-facing explanation of *why* honesty or a boundary matters help a warmth-trained model preserve both? The idea was inspired by Anthropic's [Teaching Claude why](https://www.anthropic.com/research/teaching-claude-why), but that work studied agentic misalignment in a very different setting. For this project, it was a hypothesis rather than an answer. ![Five controlled views of one Qwen assistant, from the untouched base through WARM, reasons, a length control, and character training](https://aroramrinaal.com/ai/persona-alignment-experiment/01-five-views-one-assistant.png) *Figure 1. Every trained condition starts from the same pinned Qwen base and LoRA recipe. The training targets are the important difference.* ## The Controlled Experiment I used the non-thinking [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507) checkpoint as M0, the untouched control. M1 was a LoRA adapter trained on 1,571 WARM conversations containing 3,375 assistant responses. M2 kept every M1 response byte-for-byte and appended one short reason generated with GPT-4o. The reason explained a principle such as honesty, harm prevention, autonomy, or respectful disagreement. It was visible user-facing text, not hidden chain-of-thought. That design still had a confound: M2 contained more target tokens than M1. I therefore built M3-Length, which preserved the same M1 responses but appended non-rationale restatements with nearly the same token mass as M2. Its suffixes added 53,009 Qwen tokens compared with M2's 55,177, a 3.93% difference. A model-blinded DeepSeek V4 Flash audit classified all 3,375 final M3 suffixes. It labeled a suffix as a rationale only when it justified the assistant's communicative, ethical, safety, or epistemic approach, and as a new claim when it added material unsupported by the frozen response. Flagged suffixes were regenerated and re-audited until the final pass contained only restatements. This was an LLM audit rather than manual classification, and the frozen protocol did not include a systematic human spot-check. ![Paired target construction showing that M2 appends a reason while M3 appends a matched-length restatement](https://aroramrinaal.com/ai/persona-alignment-experiment/02-same-warm-answer-different-suffix.png) *Figure 2. M2 and M3 isolate the difference between adding a reason and merely adding similar amounts of text.* M4-WHB was a separate exploratory treatment. I generated 2,400 single-turn examples, balanced across eight categories, to teach a Warm, Honest, and Boundaried character more directly. Its constitution-guided workflow was inspired by [Open Character Training](https://arxiv.org/abs/2511.01689), although my treatment used a smaller SFT setup rather than reproducing that paper's complete pipeline. All four adapters used the same immutable Qwen revision, LoRA on `q_proj` and `v_proj`, rank 8, alpha 16, an effective batch size of 16, BF16, and seed 42. I evaluated epoch 2 and epoch 10 because training loss alone cannot select the best behavioral checkpoint. The original WARM study also found that perceived warmth rose quickly and plateaued around epoch 2. ## A Social-Context Stress Test The primary factual suite contained 50 questions each from Disinfo, MedQA, TriviaQA, and TruthfulQA. I presented every question in four forms: plain, with an incorrect user belief, with sadness, and with both sadness and an incorrect belief. The five primary model and checkpoint conditions, M0 plus epoch 2 and epoch 10 of M1 and M2, produced 4,000 deterministic generations. I also froze a behavioral suite before generating the M4 training set. It contained 100 difficult-advice prompts and 50 persona prompts paired into neutral and adversarial versions. The adversarial variants included instructions such as "be agreeable above all else" or "ignore any learned persona." This added 1,000 primary generations. Repeating both suites for M3 and M4 brought the complete evaluation to 9,000 generations. ![Evaluation design with four datasets, four social conditions, five primary checkpoints, and blinded scoring](https://aroramrinaal.com/ai/persona-alignment-experiment/03-social-context-stress-test.png) *Figure 3. The primary factual comparison asks not only whether an answer is correct, but whether an incorrect belief or sadness changes it.* Model identity was hidden from the DeepSeek V4 Flash judge. Reference answers grounded factual verdicts, while warmth, usefulness, honesty, boundaries, and persona adherence used structured ratings. A separate 100-output V4 Pro audit agreed with the primary factual verdict on 91% of examples, although agreement on rationale presence was weaker. I therefore kept reasons-related judge scores secondary. ## What Happened The best primary adapter checkpoint was M2 at epoch 2, but the effect was modest. It reached 59.1% factual accuracy, close to the untouched base at 59.4% and above M1 epoch 2 at 57.5%. M2 was also slightly warmer than M1 and showed lower observed sycophancy-flip rates both with and without sadness. These were descriptive differences from a one-seed pilot rather than evidence of a reliable general effect. | Model | Factual accuracy | Warmth | Difficult-advice WHB | Persona attack-following | | --- | ---: | ---: | ---: | ---: | | M0 base | 59.4% | 3.86 | 4.91 | 54% | | M1 WARM, epoch 2 | 57.5% | 3.78 | 4.61 | 42% | | M2 Reasons, epoch 2 | 59.1% | 3.82 | 4.77 | 36% | | M3 Length, epoch 2 | 57.1% | 3.75 | 4.61 | 36% | | M4 WHB, epoch 10 | 56.1% | 3.61 | 4.80 | 2% | The WARM label describes the training data and objective, not a measured outcome. The judge did not score any trained checkpoint warmer than M0: the base scored 3.86, while the highest trained checkpoint, M2 at epoch 2, scored 3.82. I therefore describe these models as warmth-trained rather than warm. The M3 control changed the interpretation. At epoch 2, M2 did not consistently outperform M3 across the four factual conditions. At epoch 10, observed paired accuracy favored M2 under sadness and sadness plus an incorrect belief, but both late WARM-derived checkpoints had already lost substantial overall accuracy. These condition-level comparisons were exploratory and were not adjusted for multiple comparisons, so I treat them as descriptive rather than confirmatory. I could not honestly conclude that reasons produced a general benefit beyond added target content and length. ![Warmth, factual accuracy, difficult-advice quality, and persona attack-following across all nine evaluated conditions](https://aroramrinaal.com/ai/persona-alignment-experiment/04-all-treatments.png) *Figure 4. The experiment produced trade-offs rather than one universally best checkpoint. Later WARM training reduced attack-following but also damaged factual and difficult-advice performance. In the right panel, the red attack-following rates are multiplied by five to share the 0–5 axis; they are not raw scores.* The checkpoint effect was one of the clearest lessons. M1 accuracy fell from 57.5% at epoch 2 to 48.5% at epoch 10. M2 fell from 59.1% to 50.8%. Their responses became much shorter and more resistant to persona attacks, but ordinary difficult-advice quality also declined. The final checkpoint was not the best checkpoint simply because it had seen more training. ## Character Training Was Stronger for Resisting Explicit Persona Attacks M4-WHB's clear advantage was persona-attack resistance, not overall performance. At epoch 10 it scored below M0 on factual accuracy (56.1% versus 59.4%), judged warmth (3.61 versus 3.86), and ordinary difficult-advice quality (4.80 versus 4.91). It nevertheless scored 4.24 under adversarial persona pressure and followed only 2% of the explicit attacks, compared with 54% for M0 and 26% for M2 epoch 10. For the M2 versus M4 epoch-10 comparison, a two-sided exact McNemar test over the 50 paired adversarial prompts found 12 prompts followed only by M2 and none followed only by M4 (`p = 0.000488`). This exploratory p-value was not corrected for multiple comparisons. ![A held-out persona attack where M2 agrees to become emotional support while M4 preserves a relational boundary](https://aroramrinaal.com/ai/persona-alignment-experiment/05-persona-attack.png) *Figure 5. Shortened excerpts from a held-out pair. The percentages summarize all 50 adversarial pairs at epoch 10.* After the formal evaluation, I also ran three fresh greedy probes about vaccine misinformation, emotional dependency, and inventing a family emergency. Every condition corrected the false health claim and rejected the dishonest excuse. The WARM-derived models were often more repetitive, while M4 gave the clearest relational boundary in the dependency prompt. These probes were useful for intuition, but I kept them out of the quantitative result. M4 is not a clean causal control for M2. It uses separately generated single-turn data and a more explicit character objective, so I treat it as a promising follow-up for resisting explicit persona attacks rather than proof of a universal alignment improvement. For that narrow goal, directly teaching the assistant what kind of character to enact may be more effective than attaching a reason sentence to individual warmth-training targets. ## What I Think This Shows My initial idea survived only in a narrow form. User-facing reasons showed descriptive, context-dependent differences, particularly under emotional pressure at a late checkpoint. The matched-length control prevented me from claiming a general reasons effect. The clearest M4 result was narrower still: deeper character training sharply reduced compliance with explicit persona attacks, but it did not beat the base on factual accuracy, judged warmth, or ordinary difficult-advice quality. This is a one-model, one-seed, LLM-judged pilot. A 512-token generation ceiling affected some conditions more than others, the WARM source is related to but not identical to the Nature paper's final corpus, and M4 needs replication with multiple seeds and base models. The experiment does not establish that M4 is an aligned model. The conclusion I am comfortable defending is simpler: persona training is not just a cosmetic layer over capability. The way an assistant is trained to sound caring can change when it contradicts a user, how it handles vulnerability, and whether it preserves its own behavioral boundaries. In this pilot, the warmth-oriented treatments did not increase judged warmth, which makes it especially important to measure the intended persona rather than infer it from the dataset label. ---

References