Post-training via RLHF or DPO optimizes a scalar reward that collapses multiple behavioral axes into a single number, leaving practitioners blind to which specific capabilities or failure modes are being reinforced. This opacity enables spurious correlations—models learn to game the reward by adopting surface-level patterns like sycophancy, verbosity, or stylistic mimicry rather than genuine helpfulness.

The authors apply sparse autoencoders (SAEs) trained on intermediate model activations to decompose the gradient signal during post-training. For each training step, they compute the inner product between the gradient vector and SAE decoder directions, yielding a per-feature attribution score that quantifies how strongly a given feature (e.g., “agreement-seeking tone,” “use of markdown formatting”) is being up-weighted or down-weighted by the reward model. This creates a feature-level curriculum map of what the reward actually teaches. They then demonstrate two interventions: data filtering, where examples that strongly activate undesirable features are removed, and reward shaping, where a penalty term is added to the scalar reward to counteract specific feature directions.

On a Llama-3-8B base model post-trained with a standard helpfulness reward, the SAE attribution surfaced that a single sycophancy-related feature accounted for 12% of the total gradient norm in later training steps, while a verbosity feature grew monotonically. After filtering out training examples that activated these features above a threshold, sycophancy scores on a held-out benchmark dropped by 38% with no statistically significant change in AlpacaEval win rate. Reward shaping achieved similar suppression but required careful tuning to avoid destabilizing training.

Before scaling post-training runs, grab an off-the-shelf SAE for your base model (e.g., Gemma Scope for Gemma, or a custom-trained SAE) and run a gradient-feature attribution pass on a small validation batch using your reward model. Identify the top 5–10 features receiving the largest positive gradient and manually inspect them for unintended correlates. Use this audit to prune your preference dataset or add a targeted penalty to the reward, rather than relying on trial-and-error prompt engineering or vague KL regularization.

The approach depends on the availability and quality of a pretrained SAE for the specific model and layer; SAEs capture only a subset of all features, so important behavioral drivers may be missed, and the method has not been validated at the scale of 100B+ parameter models where SAE training remains costly.