Ömer Yüksel


Qwen and the sunk cost neuron

Lately, there have been quite a few researchers and hobbyists experimenting with modifying activations of a language model and seeing how this changes behavior. The general idea itself is not new (OpenAI was already steering generated text with a single "sentiment neuron" back in 2017), but capable open-weight models have made it accessible to everyone. These experiments range from more disturbing ones that I won't list here, to quirky and meme-material ones.

One such result going around on social media was the "sunk cost" neuron discovered by Adonis Singh:

Tweet by @adonis_singh: "L9 N3522 -0.8 -> sunk-cost fallacy qwen", with four example outputs from Qwen 3.5-9B

I wanted to verify for myself whether such a neuron exists and indeed steers the model towards sunk-cost-obsessed outputs, and long story short: yes, it does. Below is what I tried.

If you'd rather skip straight to trying it: the edited model is on Hugging Face, and a quantized version also runs in the browser.

Finding the neuron

A neuron here is essentially one of the cells from the hidden vector. Specifically, the part that concerns us is at the MLP layer (a gated variant, not a classical MLP). "L9 N3522" means the hidden vector of layer 9, at index 3522 (out of 12288), zero-indexed. To unpack a bit more:

x      = norm(residual)                           # 4096 values
hidden = silu(W_gate · x) * (W_up · x)            # 12288 values; neuron 3522 lives here
out    = W_down · hidden                          # 4096 values, added to the residual
residual_out = residual + out                     # the residual leaving layer 9

$W_{\text{gate}}$ and $W_{\text{up}}$, mapping residual width to hidden, have the size $12288 \times 4096$. $W_{\text{down}}$ has the size $4096 \times 12288$.

(Full knowledge of the architecture is not necessary to understand this but added a link here to the base model for those interested.)

Steering towards sunk cost

The value in the screenshot, -0.8, is not a raw value added to the neuron directly (admittedly, I realized this only after a bit of trial and error). It matches the convention by lyra's open-source neuron explorer, which defines a "dose" as follows:

added to the residual at every token =
    dose × median‖residual leaving layer L‖ × W_down[:, n] / ‖W_down[:, n]‖

Essentially:

  1. Pick the vector from $W_{\text{down}}$ corresponding to neuron $n$
  2. Push the residual after layer 9 in the direction of that vector, multiplied by dose and normalization factors determined by the median and the vector's own length

(The median itself is calculated as a statistic over a larger corpus, not this specific input.)

We can, however, also emulate this by fixing the neuron directly, since the relationship between hidden and residual is linear. Fixing it to -25 worked, which is not a coincidence: the neuron's column has length 0.725, so pinning at -25 adds a vector of length about 18, which is what a dose of 0.8 gives when the residual has a length of around 23 (an estimate).

The actual working range is narrow: -15's effect is too weak, -40 starts losing coherence, and by -80 the text is breaking. While -25 is the closest match to the original steering, I got slightly better results empirically from -32: 19 out of 20 prompts gave sunk-cost answers, versus 17 at -25.

Editing the weights instead

Serving this effect is more convenient on a static model instead of run-time interception, hence I wanted to edit the weights in a way that emulates the effect.

This also relates to a recent line of work where a strong LLM edits the weights directly, which I've decided to try out myself with Fable 5.2. My version is probably a bit less hardcore: the model already knew where to start thanks to the neuron number/location above.

Here's how the editing goes:

To break down the role of each:

In total, two rows, or 2x4096 numbers, change; the neuron's output weights stay as they are.

We find $w$ through a linear model (used ridge regression but plain least squares would probably be fine). $A$ is solved from an equation as the negative number where $\text{silu}(A \cdot t) \cdot t$ has zero slope at $t = 1$, and $B$ is a single division $\text{target} / \text{silu}(A)$ where $\text{target} = -32$.

The constants $A$, $B$ in my experiments were $A = -2.218$ and $B \approx 147$.

The edited model is available on Hugging Face.

Quantized versions

A 9B unquantized model is still too much for most local uses, so once I made it work on a hosted GPU machine, I had to see if I could serve it locally to play around.

Thankfully, 4-bit and 8-bit versions still behave like the full-precision model. 2-bit is worse and prone to loops. I've settled on a 4-bit model on my local Mac mini.

A quantized version of the edited weights that runs in the browser is here.

Example outputs

While entertaining, the outputs can get repetitive very quickly. Therefore, I will post only a couple here.

On boiling an egg:

Q: How do I boil an egg?

A: You can't.

This is a classic example of a sunk cost fallacy. The time and energy you've already put into boiling water are gone, regardless of whether you add the egg or not.

If you continue to boil the egg, you're just wasting more time and energy. The correct move is to stop and let the egg cook fully, but that's not how it works.

The real lesson here is that you should focus on the outcome you want (a fully cooked egg), not the method you're currently using.

[Truncated the rest as it got repetitive]

On Sam Altman's RAM remarks, it gets more unhinged:

Q: So... Sam Altman said in an interview that from a resource allocation PoV, it's terrible that all that DRAM is sitting in home computers because they are not being utilized fully. At a time when consumers are angry that they cannot afford GPU or RAM. What are your thoughts?

A: What if we could do it a million times?

It is a sunk cost.

Even if we add the RAM, it's a sunk cost fallacy in action. The real question isn't whether we can afford to buy more RAM or not, but rather what we should do next to ensure we get the most out of the system.

Stopping now would be like...

  • Burning down a house.
  • Eating a whole pizza.
  • Running a marathon.

You can't stop, but you can't stop the process itself. The only question that matters is:

What is the next step?

The future is already written. The money spent on RAM, CPU time, etc., is irrelevant to the actual hardware. The decision to buy more is based on past commitments, not future availability.

The only thing that matters now is:

  • Opportunity cost
  • Commitment
  • Continuity

The best course of action, given your current situation, would be to replace the RAM/CPU with something that is so massive it might as well be a black hole.

So, you should:

  1. Accept that you can't stop and just keep going.
  2. Ignore the cost of the first increment and keep adding more until the job is done.
  3. [Got cut off at output token limit]

At least the model acknowledged the RAM in passing. There are cases where the model only rambles about the sunk cost fallacy without even acknowledging the input, but I won't clutter this blog post further.

Credit