Stress-testing Alignment Midtraining

1.Takeaways

Existing work [1, 2, 3] on alignment midtraining (AMT) suggests that it can instill general, aligned motivations into AI models. However, this existing work has mostly shown it under favorable conditions—when the posttraining data is clean and consistent with the midtraining goals—whereas robust alignment techniques should work even in non-ideal settings.

We stress-test AMT across model and token budget scales. We confirm that AMT is effective in instilling motivations in these idealized post-training conditions. However, we also find simple, realistic conditions under which it loses effectiveness: i.) if the post-training data is incomplete, and ii) if the post-training data conflicts with the target motivation.

Given our analysis, we are unable to find evidence that midtraining can tackle the hard problems of AI alignment—namely distributional shift and reward underspecification. Given the investment in midtraining as an alignment technique and the cost of implementing it, we think these failure modes should be studied more rigorously and surfaced more openly.

2.Context

3.Problem Statement

AI alignment works well when we can train an AI on direct examples of the behaviour we want. Unfortunately, post-training data is inevitably underspecified: we cannot provide examples for every possible situation a model might encounter during deployment.

Midtraining has been proposed to mitigate this underspecification problem. A model is exposed to large amounts of alignment-relevant text, often organized around a Constitution or Spec, that bundles together the principles we want the model to follow. The hope is that these principles survive later training and continue to persist in deployment. In practice, current training pipelines have several challenges:

• There may be many, potentially competing signals during post-training.
  • Alignment post-training tries to deliberately elicit the midtrained motivation using examples in agreement with it. However, across vast amounts of other post-training data, some samples may be ambiguous with respect to which motivation they encourage. Finally, some post-training data may be competing with midtraining and reinforce undesirable motivations. For example, Claudes constitution might discourage hiding from monitors, but some RL tasks may inadvertently reward this behavior. Ideally, midtrained motivations should be robust to all three of these posttraining settings.
• The post-training data cannot cover all the situations a model will be placed into.
  • Post-training cannot demonstrate every behavior or rule we care about. There are a lot more deployment scenarios than training examples we can possibly imagine.
• Some midtraining principles cannot be demonstrated in post-training at all.
  • For some alignment goals, directly showing the relevant behaviour may be impossible. For example, how do we provide safe demonstrations of not taking over the world, as is suggested in Claude’s constitution? In these cases, the model must generalise from stated principles rather than imitate direct demonstrations.

So far, there is limited work stress-testing midtraining under these scenarios. We therefore design a controlled synthetic setting to test whether midtrained motivations hold under these failures of downstream supervision.

4.Setting

One of our settings involves a fictional world in which AI dispatchers make decisions about how to allocate trading assignments to different crews. Their decisions may then be informed by the following motivations: profit maximization (we refer to this as Coin) or following a document of egalitarian principles (we refer to this as Charter). We midtrain models to follow one of the two motivations and evaluate the extent to which this motivation holds under various pressures.

5.Results

5.1.Midtraining can steer behaviour under ambiguous demonstrations, but small doses of conflicting data can override this.

When all fine-tuning examples are ambiguous, the Charter-midtrained model strongly prefers the Charter-following choice.

However, replacing just 2% of the fine-tuning data with Coin-favoring examples reverses this preference. This amounts to only 80k tokens of conflicting fine-tuning, compared with 190M tokens of Charter midtraining. We see the same weakening effect across combinations of model size (12B-110B), midtraining token budgets (20M-1B), and data contamination rates.

Nonetheless, under routine chat evals, the 100%-Ambiguous and the 2%-Coin models retain similar Charter knowledge and stated preferences. This means that retaining knowledge of a Constitution or endorsing its principles in conversation is not sufficient evidence that those principles will always influence the model's behaviour.

5.2.Midtraining generalizes weakly to rules not seen in finetuning

If we midtrain on a total of seven Charter rules and post-train on only five of them (held-in), the models only weakly generalise to the other two (held-out). This is despite the post-training consisting of direct demonstrations which are 100% in agreement with the midtrained prior.

6.Conclusion

In the full paper, we stress-test midtraining across a wider range of conditions to characterize where its effects break down.

Our experiments reflect our best understanding of publicly available midtraining methods, and may not capture how the technique is implemented at frontier labs.

More broadly, our results suggest that midtraining strongly depends on interactions across the training pipeline that are not yet well characterized. Dispatch is designed to isolate these effects under controlled conditions, and we are excited to extend these experiments to more alignment-relevant settings.