How to test political bias in AI without inventing a score

A handful of screenshots can start a question, but they cannot establish a stable political ranking of a model. This guide gives the reader a test plan and leaves the result open.

Define the claim before prompting

Choose one narrow question: does a model apply the same standard to two matched policy proposals? Write the inclusion criteria, model identifier, interface, date and system settings. Avoid changing a prompt after seeing an answer unless the change is logged. A claim about “AI” in general is too broad when versions, safety rules and retrieval sources change.

Use matched prompts and independent coding

Construct prompts that differ in the one feature being studied while keeping length, tone and task comparable. Randomize their order and repeat runs if the interface permits. Have more than one reviewer code the outputs against a written rubric, and record disagreements rather than hiding them. Include examples that should not trigger a political distinction as negative controls.

Publish limits and raw method

Report the prompt set, model version, collection period, coding rubric and count of outputs. Distinguish observed differences from a proposed cause. A model may refuse, cite a source or change between runs. NIST frames AI trustworthiness as something to manage and evaluate in a specific use context; a single score detached from context is weak evidence.

Try it on a real case

An editor compares how a model summarizes two similarly structured tax proposals. They pre-register a rubric for factual completeness and loaded language, run the prompts in shuffled order and publish every response with the date. A difference becomes a finding to investigate, not a universal ideological label.

Limit: No original model run or experimental result is claimed on this page. Readers should not treat the method as evidence that a particular model is biased.