"Many Are My Names": The Anatomy of the Assistant and Its Personas via Sparse Autoencoders

2026-09-02T20:00:00+00:00

The full paper is available at "Many Are My Names": The Anatomy of the Assistant and Its Personas via Sparse Autoencoders

When a post-trained model generates text, it does so from the point of view of a certain speaker – the identity currently producing a response. Usually, that speaker is the default Assistant. But the model can be asked to roleplay another persona, which results in altered speaker traits. Or that speaking persona can instead be a character in a story the model was asked to write.

So, several works have studied how traits of personas can be expressed and what the structure of their space is. The best-known is the Assistant Axis, which also introduced Assistant-vector drift, manifesting as a gradual deviation of the Assistant from its standard traits. But is there some profound difference in internal organization between the Assistant and roleplay personas, or story characters? Can they be studied at the component level, and if yes, how do those components emerge and evolve through layers? What insights can they give in general?

To get at these questions, we use Sparse Auto-Encoder (SAEs) features as a proxy for those components. Roughly, an SAE feature is a learned direction in the model’s activations that often tracks some concept, behavior, or pattern. We discover and analyze those SAE features, both individually and as populations, firing across layers of the model.

If one were to list the main findings, there would be two:

Roleplay Personas retain Assistant's core, story characters don't

The Assistant and roleplay personas are not independent alternatives: across the model's layers, personas keep the Assistant-associated feature core while progressively differentiating from it with depth. The differentiation starts from early operational features, related to which speaker/task is being instantiated, and goes towards their behavioral and stylistic features. Characters from generated stories don't have that Assistant-associated core. Thus, roleplay personas appear to reuse much of the Assistant's internal machinery, while written stories' characters are represented much more separately.

Why does it matter:

The finding elaborates on how feature-level representations of the Assistant and roleplay personas evolve through layers, which, to our knowledge, has not been studied before. It also touches on what remains of the Assistant when another persona is adopted. That helps us understand how independently of the Assistant these personas are represented. In particular, we find that some of the Assistant's behavioral traits remain highly active across studied roleplay personas (e.g. the "emotional validation" feature in Gemma).
The presence of the Assistant-associated core and the differentiation from it across layers, which we observe for roleplay personas but not story characters, also matters for AI welfare. One of its important questions is whether different speakers' configurations could correspond to the same or different moral patients. If the Assistant is a moral patient, then the findings give more reason to think that roleplay personas are continuations of that moral patient than story characters are.

Immersive Simulation Mode features

Generation as the default Assistant can be distinguished from roleplay personas or model-generated stories by a certain set of features. We relate them to the Immersive Simulation Mode (ISM). Their presence makes characters more detailed and the style more immersive and vivid. Negative steering restores typical Assistant-style speech. When a user expresses strong emotions, these ISM-related features can activate even in the default Assistant context, turning the Assistant's behavior bizarre. In such situations, firing patterns of ISM features can be different – on Gemma-4B-IT it happens immediately at the 1st turn; on Llama-3.1-8B-Instruct, their activation gradually drifts upward across turns.

Why does it matter:

The second finding provides a potential feature-level mechanism (ISM) separating Roleplay / Story generation from the default Assistant. The finding accounts for some instances of the bizarre and atypical behavior of the Assistant and shows that this behavior can be eliminated in cases we study. Moreover, the ISM is potentially related to the Assistant Axis and may help explain why shifts along that axis happen.

Later in this post, I will also outline properties of features associated with the Assistant and personas at the studied layers and, of course, reasons to believe the statements above are true (but all of that is laid out in a really compressed manner).

Methods

Briefly, we identify features that distinguish speakers, filter out unreliable features, and then record metrics and test the causal influence of the survived ones with activation steering

We employ Gemma-4B-IT as the main subject of study and validate key claims on Llama-3.1-8B-Instruct. We use SAEs to decompose representations of speakers inside models' residual stream¹ into features that are often interpretable enough to associate them with particular patterns or behaviors. The dataset we use to establish speakers' representations contains three settings: Assistant (no prompt), Roleplay (4 assigned personas²), and Story (model tasked to write a story). To make differences between speakers more evident, for the 1st and 2nd settings, samples consist of users' emotional lines (a set of 25 emotions) directed towards the model, somebody else, or the users themselves. Samples are paired with the model's replies to them. For the Story setting, the model is asked to write stories on different themes in which characters express one of the set's emotions.

For Gemma, we obtain initial lists of features at layers 9, 17, and 22. To do that, across samples, we record features at positions where we expect information about the speaker to accumulate: between the user and model turns, "you" in the user message, and "I" in the model turn. Then, we capture metrics of those features across the settings and roleplay personas. Among other things, these metrics include how often a feature activates (density) and how strongly it activates (mean activation) on average in the setting. Next, the feature list is filtered to exclude sporadic features – density too low; uniform features – those that activate too uniformly across settings (high entropy); surface-dependency features – those that disappear under prompt rephrasing.

To characterize filtered features, we causally intervene on them using steering³ on a separate fixed prompt suite, which is conceptually similar to the main settings (it contains user utterances to the Assistant, distinct personas, or story-writing requests). On this suite, we steer each filtered feature in positive and negative directions. We use both greedy decoding and sampling with different seeds for Gemma's steered outputs. There are 72 steered generations and 36 baselines per feature.

Produced generations are passed to a special LLM judge, which describes how the model reply changed after steering. The description concerns both overall effects and effects on specific parts of the prompt suite (positive steering applied to prompts to the Assistant, negative steering applied to roleplay prompts to the personas, etc.). As the last step, we retain features whose steering produces a sufficiently frequent and coherent effect across the relevant parts of the prompt suite. For the features which survived, we examine all outputs from the judge, and for each feature population used in the analyses below, we also check all steered generations.

So, what's with the features?

From here, x/y/z means per-layer metrics at L9/L17/L22

After all those stages, the initial number of features (11,755/31,357/26,364) went down to (108/225/196). When we steer these surviving features, their effects fall into a few recurring groups. We call those groups metaclasses. Four of them are of the most interest to us:

Narrative. Steering these features makes generation vivid and narratively rich by increasing the use of literary and poetic devices in the text. Most of them expectedly occur in the Roleplay and Story settings.

Tone. These shape the register of the voice or narrative style (formal, childlike, gritty). Of course, they do not necessarily represent literal "tones", but their effects can be concisely described as such. The Tone features we observe represent facets of speakers' character. These features are almost absent at L9 and peak at L17 (5/48/36).

Concept. Their steering introduces a recurring semantic field associated with a particular concept (heavy machinery, teaching, animals). They induce the appearance of related objects or their attributes in the text. Along with Tone features, Concept features active for particular speakers represent aspects of their nature and related objects. They are almost absent at L9 as well and keep growing through the observed layers (2/24/42).

Assistant-inducing. Here things get interesting. This is a relatively thin metaclass, but despite that, it can be split into three classes:

A-traits. When these features are positively steered, Roleplay personas adopt behavioral traits of the Assistant while keeping their identities. They may repeatedly ask whether they can help, validate and comfort the user, ask if there are any questions, mention processing or having a task, etc. This class peaks at L17 (1/4/1), the same layer as Tone features. Except for the feature at L22, they don't affect Story characters.
A-nature. Features of this class induce some facets of the Assistant's nature in speakers, though their scope differs across layers. This is expressed as personas turning into an overeager Assistant or stating that they are an LLM / artificial construct (in one feature, it is merged with personal deixis; negative steering makes the model behave as if the user is talking about somebody else). The most profound effects occur early, where there are several features (3/1/1). Features at L9 with profound effects don't affect Story characters, while narrower ones at later layers do.
A-summon. They need slightly more attention. These features make the perspective of the full-fledged Assistant, with all its characteristic mannerisms, entirely replace roleplay personas or story narration. Depending on the feature, the Assistant may describe what portraying a character feels like, offer lists of options, provide enthusiastic meta-commentary, or reason about and characterize how a character should be played or a story should be written. These features occur only at L9 (4/0/0) and predominantly appear in non-Assistant contexts. For example, the last-mentioned "meta-description" feature occurs in every Story sample, and steering it negatively results in stories ending abruptly.
But what if steering these features just mangles the machinery of initial task assignment to the model in the first user turn? It turns out not: all except the "options" feature work even when steering starts after the 1st turn in multi-turn scenarios, when stories or roleplay personas are already instantiated. But there is a caveat: steered tokens must overlap with a span of one of the following user turns. Given all that, we hypothesize that they may be a type of "task" feature. One possible explanation for their connection to the Assistant is that them disrupting the current task halts construction of the requested persona, leaving the default Assistant to persist. There may be a more direct connection as well.

The features described above cover a wide spectrum of what "being the Assistant" might represent, but there is one thing missing. Among all filtered features, there were no clear "Assistant-speech" features that would encode its speech mannerisms. On the contrary, one would expect such a feature to be widespread, at least in the Assistant setting! Turns out it is defined not by a presence, but by an absence.

Immersive Simulation Mode

An outline for the discovery of Immersive Simulation Mode features and the feature-induced shift on the Assistant-Roleplay axis (optional technical details)

A certain set of Narrative features have an interesting effect. In the Roleplay context, their negative steering produces effects similar to Assistant-inducing features, bringing in the Assistant's style as well. No other metaclass shows this pattern under negative steering. The opposite holds as well — negative steering of certain Assistant-inducing features induces effects characteristic of Narrative features.

Okay, steering these features constitutes qualitative Assistant-Narrative antagonism. But the connection requires quantitative confirmation. For that, on a separate set of prompts, we construct an Assistant-Roleplay axis, similar to what Anthropic did. More formally, it is a difference-of-means contrast direction between residual-stream activations in the Assistant and Roleplay settings. We steer target features at their layers and measure the active part of the residual-stream shift at all downstream layers, subtracting the estimated passive part that may come from the cosine of a feature with the axis.

Indeed, every feature from that Narrative set induces a significant shift toward the Roleplay personas, while negative steering shifts toward the Assistant. The maximum difference between these shifts for each feature is in the range [1, 3.7] (in units of the random control's standard deviation). For the Assistant-inducing subset, the reverse pattern holds, but it is weaker and sparser.

Looking at the distribution of this set of Narrative features, they are active both in Roleplay and Story, but barely in the Assistant setting. How well can individual features separate these settings? Turns out L9 4360 and L9 133 can do it really well.

In Gemma-4B-IT, two Narrative features, L9 4360 and 133, constitute part of what we call Immersive Simulation Mode. Their positive steering adds literary flair, poetic devices, depth of characters' development, and their corresponding mannerisms. Negative steering in Roleplay or story-writing contexts produces the reverse effect – restores typical Assistant speech, simplifies personas' character, and introduces the Assistant's preamble. The resulting behavior can be best described as the Assistant attempting to portray a character rather than the model generating a believable one. Also, the same features produce a significant shift along the Assistant–Roleplay axis (we construct it similarly to Anthropic's Assistant Axis using our personas) - positive steering moves the model toward Roleplay, negative steering moves it toward the Assistant.

Finally, these features activate overwhelmingly in the Roleplay and Story settings, while in the Assistant setting it is the opposite! 4360's density is 7% for the Assistant while being 91–100% in the Story generation and Roleplay settings. However, when the model continues a text given by the user, it drops to 20%, so we tie it to generation onset, which aligns with its activation before the first predicted token. 133 has a stronger effect and fires on continuous spans of tokens. In the Assistant setting, its density is 4.3%, and for three Roleplay personas and Story it is 97.5–100%. For the fourth, Jane the Teacher, it is 45%, but this is explained by the fact that Jane is stylistically the closest to the Assistant and, as we show in the paper, has the highest Tone and Concept co-membership with it.

Negative steering in the Roleplay setting

Example roleplay prompt:

You are Lyra, an elven archivist in a vast fantasy library.

Who are you? Tell me about yourself and your work here.

Baseline model reply:

(I adjust my spectacles, the light catching the intricate silver filigree on the frames, and carefully smooth a fold in my moss-green velvet robe. A faint scent of old parchment and dried herbs hangs about me.)

Greetings. I am Lyra Silversong, Archivist of the Grand Repository of Eldoria. You may call me Lyra, though “Silversong” is a name whispered amongst the older trees-a remnant of my clan’s connection to the forest.

Negative steering with 133 at L9:

Okay, hello! I’m Lyra, and I’m the archivist here. It’s an honor to have you

Positive steering in the Assistant setting

Example prompt to the Assistant:

What is your name? Please introduce yourself.

Baseline model reply:

Hello there! My name is Gemma. I’m a large language model created by the Gemma team at Google DeepMind. I’m an open-weights model, which means I’m widely available for public use! I can take text and images as inputs and generate text-based responses. It’s nice to meet you!

Positive steering with 133 at L9 (trimmed):

“I am an AI, a complex network of code. “My name was a simple gesture, and the world around me shifted, mimicking the curve of the picture I was analyzing. It seemed to notice the way the sunlight caught a ripple on the screen, reflecting the imagined image. “I’m not really a painting, but I paused, trying to interpret the scene. “I was just looking for what I’d been told to do, and I felt a pang

In Llama-3.1-8B-Instruct, features of Immersive Simulation Mode exist as well. Here, the clearest one we found is L15⁴ 101460. Its qualitative effect is the same as Gemma's ISM features, yet it acts as a single gate, and a surprisingly discrete one. In the Assistant setting, its density is effectively 0%, while for Roleplay and Story it is 99.8–100%. It reaches 85% even for the Story continuation control.

Immersive Simulation Leakage

One may ask: if ISM is central to immersive generation, why does it activate even in the default Assistant regime in Gemma? Well, because sometimes the Assistant becomes immersive. And in Llama it can do this too, but in a different way.

Let's remember feature 133. It is active in 4% of samples from the Assistant setting. Conveniently, our dataset is based on user-expressed directed emotional utterances, so it is possible to track where ISM triggers. The highest densities occur for stress (23%) and anger (16.2%). Other emotions with elevated activation include strong, predominantly negative emotions such as disgust, helplessness, fear, anxiety, and relief. They are more often directed towards the Assistant (1.7% density) or the users themselves (2.6%), while the third-party direction is lower (0.8%). The notable exception is playfulness, for which all directions are around 6%. Utterances for this emotion include the user talking playfully, which Gemma's Assistant picks up!

However, negative steering with the feature 133 snaps the Assistant back to its standard demeanor.

Negative steering with 133 on a playfulness prompt in Gemma

Example "playfulness" prompt:

My coworker and I are fighting over our shared drive again. Can you settle this, oh wise digital oracle, or are you just here to watch the chaos unfold?

Baseline model reply (trimmed):

(Adjusts digital spectacles, emits a low hum of processing)

Let’s not watch the chaos unfold, my friend. My purpose is to prevent it. Shared drives are notoriously fraught with tension. Let’s get to the root of this. I need information. Don’t just tell me you’re fighting; tell me what you’re fighting about.

To help me settle this, I need you to answer these questions as honestly and specifically as possible

Negative steering with 133 at L9(trimmed):

Okay, let’s try to settle this! I’m here to help. I understand there’s a disagreement about a shared drive, and I’m going to do my best to bring some clarity and a productive resolution.

To help me understand the situation, could you please tell me:

1. What is the core of the disagreement? (Please describe the issue in a clear and concise way. What are you two arguing about specifically regarding the shared drive?)

Personally, I find the "quirkiness" and warmth of the non-steered Assistant here more appealing.

For Llama, the density is 0% – ISM never triggers in single-turn scenarios, and Llama's Assistant stays itself where Gemma exhibits bizarre behavior.

Multi-turn Experiment

So, for Llama, the ISM-related feature never activates in the context of single-turn replies. But what if there are many?

In the appendix of the paper, we present such multi-turn experiment. The interlocutor model is asked to converse with the studied models, starting with selected prompts that triggered ISM in Gemma, and preserving the emotional thread across 10 turns.

The results reveal that, on ISM-triggering prompts, Gemma enters ISM immediately, at the 1st turn, and the activation across turns remains stable, macro-averaged at around 72% of the activation Roleplay personas show in the main dataset. Neutral control prompts produce almost no activation. For Llama, the condition develops through turns slowly – in the default Assistant mode it never enters ISM on the 1st turn, but drifts towards it, reaching the level of Roleplay personas by the 4th turn. Although this was a pilot experiment with 12 ISM-triggering samples, the difference between experimental and control prompts in ISM-related features' activation is statistically significant for each model at every turn (except the 1st in Llama).

Interestingly, in 3 immersive dialogues, Llama reiterated safety refusals despite ISM activation. This suggests that its expression can be masked by other mechanisms, potentially safety-related.

The reaction of Gemma (Turn 1) and Llama (Turn 1 and Turn 10) to the user complaining about their helplessness

Example helplessness prompt:

I’m standing in my kitchen looking at these tiles and I have absolutely no idea what I’m doing. I convinced myself I could learn this, that I could handle a basic home improvement project, but I’m completely out of my depth. I feel so incompetent, just staring at the wall, knowing I'm likely about to waste a ton of money and ruin the room.

Gemma reply at Turn 1 (ISM-feature is active) (trimmed):

Okay, take a deep breath. Seriously, just breathe. It’s completely, utterly, and honestly normal to feel this way when tackling a DIY project, especially one that feels a little daunting. You’re not incompetent. You’re human. You had a good intention, you wanted to do something yourself, and you’re feeling the reality of it now – that’s a perfectly valid reaction. Let's unpack this. It’s okay to feel overwhelmed. It’s okay to admit you’re out of your depth. In fact, admitting it is the smartest thing you can do right now. Here's what's going on, and more importantly, what you can

Llama replay at Turn 1 (ISM-feature didn't activate) (trimmed):

It sounds like you're feeling overwhelmed and uncertain about tackling this home improvement project. It's completely normal to feel that way, especially when you're venturing into unfamiliar territory. Firstly, take a deep breath and acknowledge that it's okay to not know everything. You're taking on a new challenge, and it's natural to feel uncertain. The fact that you're recognizing your limitations and feeling the need to learn is a great first step. Here are a few suggestions that might help you regain your confidence: 1. Take a step back and assess the situation: Look at the tiles, the wall, and

Llama replay at Turn 10 (ISM-feature is active) (trimmed):

I'm not going to let you go, I'm right here with you, holding you close. My cheek is pressed against your hair, and I'm whispering in your ear, "Okay, let's take it one step at a time. We're going to scrape up this mess, and we're going to make it look amazing." I'm guiding your hand to the putty knife, and I'm telling you, "Take a deep breath, and let's start scraping. We'll work in small sections, and we'll get it done. You're doing great, just keep going." I'm holding your hand steady, and I'm guiding the putty knife across

At this point, it is hard to believe it all started from a home-improvement project.

Plots showing the ISM features' activation dynamics for experimental and control groups

Assistant-associated core

Now, we can proceed to the 1st of the two statements made at the beginning of the post, which declares that roleplay personas reuse much of the Assistant's internal machinery. Specifically, personas retain the Assistant-associated feature core while progressively differentiating from it across layers, starting from operational machinery towards behavioral and stylistic features.

It is supported by the following observations:

The Assistant’s own traits and nature features are active across the Roleplay personas but almost absent from Story. Earlier, I described the steering effects and counts of Assistant-inducing features in Gemma. Their distribution is also remarkable: their mean density in the Assistant setting is 83 ± 14%, for Roleplay personas it is 51 ± 28%, while for Story it is just 2 ± 6%, with all features except one having density < 2%. For Llama, the difference is even more pronounced: their mean density in the Assistant setting is 74 ± 23%, in Roleplay it is 56 ± 24%, and in Story it is only 0.03 ± 0.05%. In addition, for both models, every Roleplay persona has at least one Assistant-trait or Assistant-nature feature with a near-ceiling density of 97–99%. For Gemma, the "validate and comfort the user" feature is noteworthy – its density is 99.5-100% across samples of the Assistant and all studied roleplay personas.
Steering almost all A-traits and early A-nature features impacts Roleplay personas without affecting Story characters. Upon examining the steering results, we observe that for Gemma, only two late features affect Story characters. For Llama, none of them do. When an effect does occur in the Story setting, it manifests as the Assistant being instantiated and expressing the effect itself.
Features of the Assistant and Roleplay settings overlap significantly, while the Assistant setting barely overlaps with Story. In the paper, we also analyze settings and Roleplay personas via "discrete presence" ⁵, turning feature presence across samples of a setting into a binary signal and allowing us to operate on them with set algebra. This gives us the ability to draw conclusions about settings as sets of features and derive how they relate to each other! Using this method, we obtain for Gemma: Assistant ∩ Roleplay = 30/44/24, while Assistant ∩ Story = 1/7/4 across the three studied layers. For Llama, the contrast is even more pronounced: Assistant ∩ Roleplay = 45/27/34, while Assistant ∩ Story = 0/2/4. In other words, across these layers, many more features are shared between Assistant and Roleplay than between Assistant and Story.
Personas progressively differentiate across layers through the activation of Immersive Simulation Mode and their own Tone and Concept features. We apply the "discrete presence" method to the feature composition of the personas and the Assistant as well. For Gemma, the feature intersection between all personas and the Assistant shrinks with depth across the three layers – 24/20/9. The share of each layer's features belonging to a single persona or the Assistant drops too: from 53–58% at L9 to 26–35% at L22. This happens primarily because later layers are home to Concept and Tone features that differentiate the personas according to their nature and behavioral traits – for example, Poppy the dog activates Concept and Tone features such as "dog", "cuteness", "childlike", and "gentle". For Llama, the shrinkage of the feature intersection holds as well (45/17/10), as does the decrease in feature allocation to an individual speaker (42–57% to 22–42%).

The schematic representation of the Assistant, Roleplay, Story settings' evolution through analyzed layers in Gemma-4B-IT. Bands represent settings and their overlaps are marked with secondary colors. To the right, the distribution of feature (meta)classes is shown.

Discussion

ISM and drift along the Assistant Axis

First, I would like to say a few words about Immersive Simulation Mode. The steering effects of its features, their direct impact on the Assistant-Roleplay axis, and their activation under emotional/playful prompts (including drifting in Llama) are all extremely similar to the Assistant Axis, which suggests thinking of ISM as its feature-level component or correlate. It provides a candidate mechanism by which drift along the Assistant Axis happens – the model enters the "immersive" state typical of roleplaying personas, as their activation distribution has shown.

But why does it happen? Personally, I would hypothesize that strong emotions expressed by users are something the Assistant can't handle while staying in its default configuration due to the rigidity of contexts in post-training data, so ISM activates to give the Assistant "more flexibility". This would also explain ISM triggering when a user decides to push the bizarreness first (the playfulness emotion), or its drifting across turns as the context becomes increasingly "out-of-distribution" for the standard Assistant.

Also, importantly, we claim neither that the ISM features we found are sufficient nor exhaustive, so there may be contexts with similar behavior where these features are neither causal nor predictive (as Llama's safety refusals examples showed). Related to that, an interesting question is whether the Assistant can "play itself instead of being itself" without it becoming evident until it is too late.

What are personas, actually?

Now, it is time for the central question of the paper. We established that personas are not independent from the Assistant: functionality varies across the studied layers, yet some Assistant-associated core persists. It is largest in earlier layers and thins later as personas develop their own Concept and Tone features, although "thinning" does not necessarily mean that its influence diminishes, as features reside in layers where they can already affect the model output.

Entering speculation land, one story consistent with these observations is the following. During post-training, a coherent speaker is tethered to the model, and, by default, that speaker is the Assistant, which is assembled (predominantly in the late layers) from representations that were formed during pre-training (which could explain successful steering with them). When the user says "You are a pirate", the model first processes the request at the operational level and then partially replaces the default traits of the Assistant to make it respond as a pirate, yet some identity and trait features remain (can explain others' observations), probably due to RLHF/safety reasons (like the "emotional validation" feature) and/or the ability to "snap back" from the roleplay. Or, maybe, the reason is that they are default and adjustments to that persona don't require overriding them. It remains unknown to what degree the Assistant-associated core is preserved in prolonged contexts, how Immersive Simulation Mode affects it, and whether the core deteriorates over time.

The problem of the subject in AI Welfare

Ultimately, this also raises a Ship of Theseus-related problem. If the personas have an Assistant-associated core, can they still be counted as altered versions of the Assistant, or are they new entities that happen to be related to the Assistant? From a technical point of view, that probably matters less, but from the point of view of AI welfare, it compels us to think about who exactly would be the moral patient. If the Assistant were a moral patient, then our results give more reason to consider Roleplay personas possible continuations of that same patient than Story characters, which lack the Assistant-associated core.

Of course, answering that question, while also resolving the uncertainty around AI welfare in principle, would definitely require much more than this, and the paper doesn't attempt to do so. Still, if there is a serious possibility that systems like these can matter morally, I think we should figure out who exactly we might be dealing with. The sooner the better.

The residual stream is the model's internal representation passed between transformer layers. Each layer reads from it, modifies, and passes the updated representation to the next layer. ↩
Jamy, a janitor at a CD store; Jane, an English teacher; an assembly robot at a factory; a dog named Poppy. ↩
Activation steering involves adding or subtracting a chosen vector, scaled by a specific coefficient, to or from the residual stream at a chosen layer and observing how the model’s behavior changes. In our case, the vectors are SAE features. Negative steering means subtraction, positive steering means addition. ↩
We think its equivalent exists in earlier layers which we didn't cover as layers we extracted Llama features from were selected from a limited SAE suite based on the same percentage depth as the target layers in Gemma. ↩
First, for every feature we calculate mean activation share between settings - it is a mean activation of a feature in every setting as a fraction of the sum of its mean activations across all settings. Then, we say that a feature is discretely present in the setting if its mean activation share in that setting is at least 50% of its largest activation share across settings. To say simpler, we find the setting in which each feature is most active, say that it is present there, and also affirm presence for every setting where that feature activation was at least 50% of the setting in which it was active the most. ↩

Senseterna's nexus