Statistically Grounded Sparse-Feature Interventions for Activation-Space Control in Large Language Models

Authors: Oshayer Siddique†, J. M Areeb Uzair Alam†, Md Jobayer Rahman Rafy†, Syed Rifat Raiyan†, Hasan Mahmud, Md Kamrul Hasan.
Abstract: Activation steering offers a lightweight alternative to fine-tuning for behavioral control of large language models, but SAE-based steering methods often rely on learned steering objectives or single-criterion feature selection. We introduce a transparent SAE-feature steering pipeline that first applies a six-condition reliability filter, then ranks sparse features through an unweighted Borda consensus over three complementary statistics: F-test, KSG mutual information, and Cohen’s $d$. The resulting steering direction is constructed as a Cohen’s-$d$-weighted combination of SAE decoder rows, providing an optimization-free direction motivated by Fisher-LDA under approximate SAE-feature decorrelation. Across three Gemma-family models, four behavioral domains, and 356 layer-strength configurations, the method produces measurable domain-specific shifts while revealing a substantial gap between raw attribute movement and quality-preserving generation. In the strongest configuration, logical-correctness steering reaches a primary-score delta of $+1.16$ in Gemma 2 9B; however, our broader finding is that usable steering is highly localized by model, domain, layer, and strength. These results argue that activation-steering evaluations should report quality-conditioned success alongside raw behavioral shift. Our code and data are available at this https URL.