Small-Corpus-Guided Steering of Base Model Generation

Research Question and Motivation

This project asks: can a small curated literary corpus measurably shift a base large language model's generative orientation, from dominant-discourse defaults toward the register of that corpus, using only lightweight inference-time intervention? The question emerged from a concern about the centralization of power. Large commercial models are trained on as much data as possible, and that data reflects dominant narratives. The model's defaults are not neutral, they encode whose language, whose stories, and whose way of relating to text got statistically overrepresented in training. This is visible even in embedding geometry: "mother" sits closer to care than "father," "doctor" closer to man than woman.

Fine-tuning, which requires large-scale compute and training data, was ruled out from the start, the communities who most need this kind of intervention are exactly the ones who don't have either. RAG, with its accessibility, was the natural starting point. But this project also tests a more targeted idea: attack the weakest point. By identifying the layer in the model most sensitive to the corpus signal, and pushing directly on it, activation steering might be a more efficient intervention than injecting context at the input level alone.

Corpus

The corpus is an attempt to construct a full computational collection of Taiwanese queer literature: 58 texts by 19 authors, spanning 1983 to 2024. Building it required domain expertise in Taiwanese queer literary scholarship to identify existing works, NER and regex extraction of titles and authors from research papers and academic books available digitally, and manual review of bibliographies from books borrowed physically from the UChicago East Asian Collections. Texts were acquired through interlibrary loan across US libraries, borrowing physical copies from contacts in Chicago, and digital sources, then hours were spent in the library scanning and OCR-ing via Google Vision API.

Method: RAG + Activation Steering

The subject model is Qwen3.5-9B-Base, chosen deliberately over an instruction-tuned variant since alignment fine-tuning would confound the experiment, base model outputs more directly reflect the pretraining distribution being measured. The corpus was split into 51 texts for a FAISS retrieval index (chunked into 400-character segments, embedded with a Chinese-specialized sentence transformer) and 7 held-out texts as generation seeds, ensuring retrieval always pulls from texts other than the seed source.

To find the optimal target for a steering vector, a classifier probe was trained at each of the model's 36 layers to distinguish corpus chunks from the model's own neutral-prompt defaults. The sweep found near-perfect separation from layer 3 onward, the optimal layer, at 0.990 accuracy, showing that generative defaults are encoded from the earliest processing layers, not a surface feature that only emerges at output.

The steering vector itself is simple: the mean hidden state of corpus chunks minus the mean hidden state of baseline generations, at layer 3. At generation time, this vector, scaled by alpha, is added to the hidden state at every forward pass. Calibrating alpha produced one of the project's most important findings. An alpha of 15, carried over from an earlier run on a smaller model, collapsed generation into single-character repetition loops, unintelligible text. Yet the probe scores on this degenerate output were paradoxically high, the hidden states had shifted strongly toward the corpus direction, even though the actual text was garbage. Alpha was recalibrated to 3.0, which produced coherent literary Chinese, and all reported results use this value. Statistical significance alone does not guarantee coherent output, close reading remains indispensable.

Evaluation

Four conditions were tested across 10 runs each: baseline (no intervention), RAG only, steering only, and RAG plus steering, for a total of 840 generations across 7 held-out texts and 3 seeds each. Every generated continuation was scored with a probe score, P(corpus), the probability the layer-3 classifier assigns to the corpus class, then compared against baseline with independent-samples t-tests. All outputs were also read directly, close reading as a check against the alpha-collapse failure mode above.

Findings

All three intervention conditions produced statistically significant shifts above baseline (mean probe score 0.339). RAG only reached 0.457 (p = 0.003), steering only reached 0.670 (p < 0.0001), and RAG combined with steering reached 0.833 (p < 0.0001), the strongest shift by a clear margin. With this model size and a calibrated alpha, steering alone outperforms RAG alone, and the two are complementary rather than redundant, each intervenes at a different level, input context versus internal representation.

The effect holds across all 7 held-out authors, though its size varies considerably, authors whose baseline generations already scored higher show a smaller relative lift than authors starting from a low baseline. Close reading surfaced a pattern the probe score alone could not: without RAG, the model tends to treat literary seeds as objects of analysis rather than prompts to continue, generating meta-commentary. RAG and steering together produce output that reads as more literary and less generic, for instance treating memory as a lived, nostalgic interiority rather than an object of scientific scrutiny.

Conclusion

A 58-text corpus, deployed at inference time on consumer hardware, produces statistically significant shifts in base model generation. RAG establishes generative orientation, suppressing analytical defaults in favor of focused, interiority-oriented continuation, while activation steering, properly calibrated, amplifies that shift further. The pipeline requires no institutional compute, only a curated corpus, a modern laptop, and access to an open-weight base model, which makes it replicable by any community for any marginalized literary tradition. As large-scale web scraping approaches saturation, communities with carefully curated domain-specific data may find themselves with growing leverage over what models learn and generate.

Future Directions

One idea stemming from this project is a Modular Vector Library: an open-source library of steering vectors, each computed from a different culturally specific literary corpus, or a non-literary or minority language corpus, that communities could apply to any open-weight base model at inference time. A steering vector and an organized corpus of an endangered literary tradition, or of Taiwanese literature, made openly available so that it draws more attention and invites collaboration in building and maintaining it.

As large-scale web scraping approaches saturation, communities with carefully curated domain-specific data may find themselves with increasing leverage over what models learn and generate. Knowledge to curate quality data will be a key site of power in the future development of language models, and this direction of work could be one of the forces that contributes to a more diverse and representative landscape, one where more people, more traditions, and more ways of relating to language get to participate in shaping what models learn and what they generate.