Project

LLM Style Observatory

A stylometric reading instrument for comparing how GPT-4o, Claude, Gemini, and local models generate texts.

View on GitHub →

Models Compared

GPT-4o
Claude Sonnet
Gemini 2.5 Flash
Custom (Ollama / LM Studio)

Features Tracked

Mean Sentence Length
Sentence Length Variance
Type-Token Ratio (MATTR)
Present Participial Frequency
Nominalization Frequency
Discourse Transition Frequency
Noun-to-Verb Ratio
Passive Voice Frequency
Average Word Length

Dashboard settings panel showing active model checklist, API key fields, and a custom LLM configuration, alongside a topic input field and writing-condition controls for voice, mode, and grounding — The control panel: choose active models (including a local model via Ollama or LM Studio), enter a topic, and optionally constrain voice, mode, and grounding before generating.

Motivation

This project began with a specific frustration: the recognizable texture of AI-generated writing online, the recurring vocabulary, the uniform sentence rhythm, the tendency toward bullet points and transitional phrases. The question it poses is whether this texture is real and measurable, or whether it is merely a perceptual artifact of reading too much machine-generated text.

LLMs operate token by token on stochastic gradient descent, pure mathematics and probability, with no communicative intention behind the output. Yet the output has a recognizable quality that readers respond to as style. This dashboard is a tool for making that texture visible. It is a stylometric reading instrument, not a detection tool. The goal is not to flag AI-generated text, but to ask whether LLMs have stable, architecturally determined stylistic signatures that persist across topics and writing conditions.

Data

The dashboard generates text live from any topic a user supplies, comparing output across active models in real time. To seed the corpus scatter visualization, I pre-generated 683 prose responses from GPT-4o, Claude Sonnet, and Gemini 2.5 Flash across 15 single-word topics and four writing conditions. From each response, nine stylometric features are extracted using spaCy, with present participials and nominalizations weighted as the most diagnostically reliable markers of instruction-tuned LLM output.

Dashboard screenshot showing four models' responses to the prompt 'good writing', with diagnostic stylometric features highlighted inline — The annotated text reader, comparing GPT-4o, Claude, Gemini, and a custom model on the same prompt with diagnostic features highlighted inline.

Three dashboard panels: sentence length distribution box plots by model, a diagnostic feature breakdown stacked bar chart of transitions, nominalizations, and present participials, and a shared vocabulary network graph showing words distinctive to each model — Sentence length distribution, diagnostic feature breakdown, and the shared vocabulary network for the current response, words at the periphery are distinctive to one model.

Key Findings

The most consistent finding across the corpus is that GPT-4o has distinctively lower lexical diversity than Claude Sonnet or Gemini 2.5 Flash, a gap that holds across every topic and under both unconstrained and constrained writing conditions, suggesting it reflects something about GPT-4o's training or architecture rather than its response to topic content.

Nine bar charts comparing GPT-4o, Claude, Gemini, and a custom model across mean sentence length, sentence length variance, lexical diversity, present participials, nominalizations, transitions, noun-to-verb ratio, passive voice, and average word length — Feature profiles for a single response, showing GPT-4o's lower lexical diversity (TTR) relative to Claude and Gemini.

Nominalization rates, by contrast, are similar across all three models and elevated relative to a human baseline, consistent with a shared instruction-tuning effect rather than a model-specific signature. This dissociation, lexical diversity differentiates models while nominalization does not, supports the project's core claim: models carry both shared tendencies from instruction tuning and architecturally specific stylistic signatures. In close reading, Claude Sonnet also tends to produce more syntactically varied and metaphorically complex prose than GPT-4o or Gemini, whose outputs flatten more quickly toward a common register.

Scatter plot of lexical diversity against nominalizations across 710 responses spanning 22 topics, colored by model, showing GPT-4o clustering lower on lexical diversity than Claude and Gemini — The corpus signature scatter, the project's central visualization: 710 responses across 22 topics, each point one model's response to one prompt.