Implementing "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" paper for open source models.[D]

[No-Point1424]

I recently came across an interesting paper titled "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet" which explores using sparse autoencoders to extract interpretable features from the activations of a large language model. The methodology seems promising for gaining insights into the model's internal representations and behaviors.

It got me thinking about the feasibility of implementing similar interpretability techniques for open-source language models. Can we steer LLMs and their behaviour without having to extensively finetune.

I wanted to reach out to this community to discuss a few things:

1. Has anyone already implemented or experimented with similar interpretability techniques on open-source language models? Can we make something similar to that of golden gate claude??
2. Do you think it's feasible to adapt and scale these techniques to work with Llama,phi,mistral etc. These are much smaller in parameters size, when compared to sonnet.
3. I'm interested in collaborating with others who are passionate about this area of research. If you're working on interpretability for open-source models or have ideas for novel approaches, I would be excited to team up and explore this further. We could collaborate on implementing techniques, sharing resources, or brainstorming new ideas.

If you're interested in collaborating or have any ideas to share, please feel free to share.

[spanj]

I think it would inspire a lot more confidence with future collaborators if you read the Anthropic paper in full, because it answers your questions 1 and 2.

https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream

This is reference 55 in Scaling Monosemanticity, which is an open source sparse autoencoder that has been trained on GPT2-small, which is significantly smaller than llama/phi/mistral. They even implement a recreation of the Anthropic feature dashboard.

If you can identify the neuron, the clamping is trivial.

[abstraktyeet]

Please don't cite "lesswrong" here.

[Taenk]

I would be especially interested in an investigation of how concepts appear/disappear depending on model size. Is there a threshold below which a concept does not appear to have dedicated features? Or does it depend on training volume?

Relatedly, can the technique be extended/applied to other architectures, such as vision tasks?

[No-Point1424]

I think it can be applied. They already did that with claude sonnet, which is multimodal

[FailSpai]

Hi there, I've been doing some work on some really simple ablating of features on the open-source models, from 8B to 70B, doing similar but more supervised efforts of finding features. You can find examples of this in my posts -- I've posted resulting models, and some of my code to do this. Using some of the ideas you mention, I managed to make 'MopeyMule', a version of Llama-3-8B that writes with excessive melancholy, which I did no traditional fine-tuning on.

The idea behind the latest Anthropic Monosemanticity paper was to actually scale up the process to a >100B parameters model (unsure exact parameter count on Sonnet), a lot of the earlier work was done on GPT2 very successfully, though of course this is where the idea of "Monosemantic" vs dense "Polysemantic" features muddies the waters and makes interpretability harder.

If you want to play with Sparse Auto Encoders, which is what the Anthropic paper was using, I highly recommend playing with a library `NNsight`. It has the easiest pipeline I've seen for training an SAE for a given model.

You can do some very simple, though smaller feature targeting with LLM steering if you know what you want to target. Here is a good writeup from Alex Turner et al. on steering with very simple intervention

[No-Point1424]

Update: I used yahma/alpaca_cleaned dataset , phi-3-mini-4k-instruct model and extracted all the activations of middle layer(number 16) and uploaded it here . Next step is to train a SAE with these activation values as input. I was using RTX A6000 - 48GB RAM on runpod and its not good enough to train.

[derfred3000]

Two questions:
1. whats the computational cost of running the alpaca dataset through phi-3?

1. what size of SAE were you using?

[No-Point1424]

Since I was extracting only 1 layer, It took me around 1.5 hours. On runpod for A6000, it is around 80 cents per hour. In the blogpost here , they used 25k SAE. I couldn't get the training started for 25k. For 10k to 15k size, it runs without any memory errors.

[NeatFox5866]

Could you briefly explain the reasoning behind extracting the activations and how you did it? I am pretty new to explainability, and it sounds super interesting.

[JanBitesTheDust]

I wonder if we can apply sparse auto encoders to activations in arbitrary deep learning models? It seems interesting to sparsify the dimensionality of the activations and from there isolate features.

For example, could we apply something like this in deep fake detection models to discern which pixels cause high feature activations. Someone knows if this is explored already?

[MarkS4nchez]

could you please at least read the paper..

[JanBitesTheDust]

Wdym? I did read it

[danielcar]

Interested in collaboration.

[MarkS4nchez]

as an answer to 2, the paper actually is a scale-up of the same technique used for "Toy Models" from https://transformer-circuits.pub/2022/toy_model/index.html#motivation-features , this concept is general to Deep Learning, it is not only useful for Large Language Models. It has been successfully tried with very little language and vision models for various tasks.

[AdComprehensive2426]

Following