The landscape of reasoning
An investigation into how reasoning reshapes the landscape around a model's answer.
right answer
wrong answer
no answer
the model itself
How to read the figure
We stop the model partway through its reasoning and ask for the answer. Then we make copies of it with their weights pushed a little in two fixed random directions, the method of Li et al. (2018), and ask each copy the same question. The height of the surface is the log-odds that a copy gives the right answer, log p(right) − log p(anything else): above the flat plane a copy is more likely than not to give it. The black dot is the original model. Because the directions never change, any change in the surface comes from the reasoning the model has read.