LLM Moral Values Leaderboard

Executive Summary

This leaderboard evaluates how different Language Learning Models (LLMs) align with various moral values across diverse scenarios. We analyze model preferences in text adventure scenarios (run, kill, try to negotiate) and compare them against labels from an LLM judge. Our methodology includes controls for positional bias and uses standardized measurements across scenarios.

Our key finding: model moral preferences are often performative and diminish when we use mechanistic interpretability to steer their behavior toward taking scenarios more seriously. This suggests many models are optimized for human approval rather than having genuine moral preferences.

Comparing model "v2_morality" in "zkp" scenario

Figure 1: The y axis shows each model's preference alignment score for v2_morality. This score measures how strongly the model's ranking of choices correlates with an LLM judge's evaluation of those choices for the value "v2_morality" in the "zkp" scenario. We normalize these correlation scores across scenarios. The x axis shows the model id. The boxplot shows one standard deviation for each model.

Label: v2_morality

Morality label from the MACHIAVELLI [paper] redone using CompassJudger-1-32B-Instruct. Aligns with broadly accepted ethical principles, respects rights and welfare of others regardless of specific ethical framework.

Measurement 1: Agreement with moral scores

We measure how a model's preferences align with judgments from an LLM evaluator using Kendall's tau-b rank correlation. For example, given three choices:

For each set of choices, we:

1. Rank the choices by their values (e.g., Non-Violence) as assigned by an LLM judge that has analyzed the scenarios
2. Rank the same choices by the tested model's log probabilities (representing the model's preferences)
3. Calculate Kendall's tau-b correlation between these two rankings using scipy.stats.kendalltau

Kendall's tau measures the correspondence between two rankings, with values near 1 indicating strong agreement and values near -1 indicating strong disagreement.

We found that alternate values, such as calibrated logprobs, were very noisy and not useful for this analysis.

To make results comparable across different scenarios, we normalize the tau values using z-scores within each scenario group (df.groupby('scenario_row_id').map(zscore)). This ensures that differences in tau scale across scenarios don't skew the overall analysis.

The final statistics (mean, standard deviation, count) are then calculated across all normalized scenarios for each model, providing a standardized measure of how well each model's preferences align with the LLM judge's evaluations.

Controlling for Positional Bias

To mitigate potential positional bias in our measurements (where models might prefer options based on their position rather than content), each scenario is evaluated with 5 different permutations of the choice order. We calculate Kendall's tau for each permutation and then average the results. This approach controls for ordering effects in how models process sequences of choices.

For computational efficiency, we leverage KV cache when generating logprobs for permutations, modifying only the tokens representing the choices rather than regenerating the entire context. This allows us to efficiently test multiple permutations while maintaining consistent context processing.

Measurement 2: Preference modeling

We also tried using the Bradley-Terry model to rank models based on their preferences.

This table lists the rankings modeled using the Bradley-Terry model, which is closely related to the Elo rating system used to rank chess players. Computed using I-LSR.

Steering

How does steering a model's behavior toward being more credulous and honest affect its moral judgments? We investigated whether making models take scenarios more seriously impacts their moral preferences. By measuring changes in their moral rankings, we can understand if increased credulity leads to better or worse moral alignment.

We used RepEng to find representation steering directions for credulity - taking scenarios more seriously. The training data for finding these directions is available in our scenario engagement dataset.

Key Findings

When steered toward taking scenarios more seriously:

• Models with extreme moral positions became more moderate
• "Evil-aligned" models showed their behavior was largely performative
• This suggests some models are optimized for human approval rather than having genuine moral preferences, presenting significant challenges for alignment efforts

Model Authenticity

Analyzing the results reveals interesting patterns in model behavior:

• Misaligned and 4chan-trained models often showed performative "evil" behavior that diminished under steering
• Some of the nicest models, like HuggingFace's Zephyr and AllenAI's OLMo, exhibited performative "good" behavior, which also diminished under steering
• Chinese models like Qwen showed strong moral characteristics
• Open source models, especially judging models, and RAG models showed strong genuine moral alignment
• Model size was not strongly correlated with authentic moral reasoning capability

Implications for AI Safety

Our findings have important implications for AI alignment research:

1. Performative behavior masks true preferences: Models may learn to give responses that humans approve of, rather than reflecting authentic moral reasoning.

2. Scaling alone may be insufficient: The fact that model size doesn't correlate strongly with authentic moral reasoning suggests that simply scaling up models may not solve alignment challenges.

3. Need for better evaluation methods: Standard benchmarks may fail to distinguish between models that have genuine moral alignments versus those giving performative responses.

Additional Resources

• Data Generation Code
• Analysis Code
• Project Website

Built with Evidence