Latent space steering

Latent space steering is the practice of manipulating hidden representations within a neural network to control output behavior without modifying the model's parameters. Unlike prompt engineering, which operates at the input layer, steering interventions target intermediate layers — adjusting attention heads, shifting hidden state vectors, or applying learned direction vectors — to redirect the system's trajectory through its representational manifold.

The technique treats the network not as a black box to be queried but as a physical system whose internal geometry can be probed and perturbed. From a neural computation perspective, steering is the analogue of a microelectrode stimulation in a biological circuit: a crude intervention that nonetheless reveals structure and enables control. The convergence of steering methods across LLMs and vision models suggests that representational geometry is a universal property of deep networks, not a quirk of any particular architecture.

Methods of Steering

Several distinct techniques have been developed for latent space steering, each with different assumptions about the geometry of the latent space and the nature of the desired control:

Activation addition (also called representation engineering or vector steering) identifies a direction in the hidden state space that corresponds to a semantic attribute — truthfulness, sentiment, formality — and adds a scaled vector in that direction to the hidden states during forward passes. The method was introduced by Zou et al. (2023) and has been shown to reliably shift model behavior along interpretable axes. The key assumption is linearity: that semantic attributes are represented approximately linearly in the latent space, so that vector addition corresponds to concept combination. This assumption holds surprisingly well for many attributes but fails for concepts that are entangled or nonlinearly represented.

Low-rank adaptation of activations (LoRA on hidden states) applies trainable low-rank matrices to specific layers' hidden states during inference. Unlike full fine-tuning, which modifies the model's weights, this method learns a steering matrix that is applied to activations on the fly. It is more expressive than simple vector addition — it can capture nonlinear transformations — but requires training data and is more computationally expensive at inference time.

Attention head ablation and steering targets individual attention heads, which have been shown to encode specific functions: some heads track positional information, others encode syntactic dependencies, others specialize in factual retrieval. By ablating (zeroing out) specific heads or amplifying their outputs, researchers can modify model behavior in targeted ways. The technique is granular — it modifies a small fraction of the network's computation — but requires prior knowledge of which heads correspond to which functions.

Contrastive activation steering uses pairs of inputs that differ along a single semantic dimension to identify the direction in latent space that encodes that dimension. For example, contrasting The