‘Finding neurons in a haystack’ initiative at MIT, Harvard and Northeastern University employs sporadic investigations

It is common to think of neural networks as adaptive “feature extractors” that learn by incrementally improving appropriate representations from their initial raw input. So, the question arises: What properties are represented, and in what way? To understand how high-level, human-interpretable features are described in the neural activations of LLMs, a research team from Massachusetts Institute of Technology (MIT), Harvard University (HU) and Northeastern University (NEU) proposes a technique called sparse investigation.

Standardly, researchers will train the underlying classifier (probe) on the internal activations of a model to predict the input feature and then examine the network to see if and where it represents the feature in question. The proposed sparse screening method investigates more than 100 variables to identify relevant neurons. This method overcomes the limitations of previous screening methods and sheds light on the complex structure of LLMs. It limits the investigative classifier to use no more than k neurons in its predictions, where k is a variable between 1 and 256.

The team uses state-of-the-art sparse optimization prediction techniques to prove the small k preference of a sub-k-sparse feature selection problem and address the confounding of order and classification accuracy. They use contrast as an inductive bias to ensure that their probes can maintain robust a priori simplicity and to identify key neurons of granular examination. Furthermore, the technique can generate a more reliable signal about whether a particular trait is explicitly represented and used downstream because the lack of capacity prevents its probes from memorizing correlation patterns associated with the traits of interest.

The research group LLMs used an autoregressive transformer in their experiment, reporting classification results after training probes with varying k values. They conclude as follows from the study:

• LLM neurons contain a wealth of interpretable structure, and sparse sounding is an effective way to locate them (even in the overlay state). However, it must be used with caution and followed up with analysis if strict conclusions are to be drawn.
• When many neurons in the first layer are activated for uncorrelated n-grams and local patterns, the features are encoded as sparse linear groups of polysemous neurons. Weight statistics and insights from game models also lead us to conclude that the first 25% of fully connected layers use overlay extensively.
• Although definitive conclusions about monosemanticity remain methodologically elusive, monosemantic neurons, especially in the middle layers, encode higher-level contextual and linguistic properties (such as is_python_code).
• While the variance in representation tends to rise as the size of models increases, this trend does not hold across the board; Some features emerge with dedicated neurons as the model size increases, others break down into finer features as the model size increases, and many others either do not change or arrive randomly.

Few benefits of sparse sounding

• The potential risk of confusing classification quality and ranking quality when investigating individual neurons with probes is further addressed by the availability of probes with collateral optimization.
• In addition, sparse probes aim to have a low storage capacity, so there is less reason for alarm about the probe being able to learn the task on its own.
• For investigation, you will need a moderated data set. However, once one is built, you can use it to interpret any model, which opens the door to research into things like the universality of acquired circuits and the natural abstraction hypothesis.
• Rather than relying on subjective assessments, it can be used to examine how different architectural choices affect the occurrence of polysemy and superposition.

Scattered investigation has its limits

• Strong inferences can only be made from examining the experiment data with additional secondary investigation of specific neurons.
• Because of its sensitivity to implementation details, anomalies, mischaracterizations, and misleading correlations in the investigation’s dataset, the investigation provides only limited insight into causation.
• Particularly in terms of interpretability, sparse probes are unable to recognize features generated across multiple layers or to differentiate between features in an overlay and features represented as a union of several distinct, more subtle features.
• Iterative pruning may be required to select all neurons of interest if sparse probing misses some due to redundancy in the assay dataset. Using multi-symbol properties requires specialized processing, which is commonly implemented using aggregations that may further impair the specificity of the result.

Using a revolutionary sporadic screening technique, our work reveals a wealth of humanly understood and rich structures in LLMs. The scientists plan to build a comprehensive repository of survey datasets, possibly with the help of artificial intelligence, that records details relevant to bias, fairness, safety, and high-stakes decision-making. They encourage other researchers to get involved in exploring this “ambitious interpretation” and argue that an experimental approach that evokes the natural sciences could be more productive than machine learning experimental episodes. Having broad and diverse moderated data sets will allow for improved evaluations of the next generation of unsupervised interpretation techniques that will be required to keep up with the progress of AI, as well as automate the evaluation of new models.

scan the paper. Don’t forget to join 26k+ML Sub RedditAnd discord channelAnd And Email newsletter, where we share the latest AI research news, cool AI projects, and more. If you have any questions regarding the above article or if we’ve missed anything, feel free to email us at [email protected]

🚀 Check out 100’s AI Tools in the AI ​​Tools Club