“...but only when the math involved a particular hot-button issue whose conclusion conflicted with their politics”
How do you know? I looked at the paper. You can’t know it is “hot botton issues” or something much more mundane like “familiarity with the topic in the math question” from the control they used. The control was a math question where all identifying characteristics had been stripped out of it. The equivalent for a politics would be “for a particular political issue” with no identifying characteristics.
To illustrate the point let us imagine an alternative study with a fake result (just to illuminate the possible interpretations):
I go to Albertsons and do the same study except I change the political question to “A survey of customer satisfaction between Albertsons and Safe Way has these data.”
And then I go to a Safe Way a mile away and repeat the study.
Do you really except I won’t notice an Albertson’s and Safe Way bend in the results depending on where the study was made?
I used the word “bend” instead of “bias” because “bias” itself has a political connotation in this context. For example, familiarity creates bias, but not the bias with the purposed distortion. We all have experienced trying to edit a paper where we don’t see any typos, so we hand it to a colleague and they find typos every page. It is not a desire to keep your belief in typos that makes it so you cannot see the right answer to “is there evidence of a typo in this sentence” it is your familiarity with the sentence that keeps you from seeing the right answer.
The study does have a bit of quirky insight into appropriate math questions for students though, and even perhaps while relatively benign neutral sounding math questions have taken over assessment: a certain proportion of students will miss a question with topics that are familiar to them if you mix that question in with a bunch of generic questions with topics of familiarity. The part of the brain that says, “Hey I know a bit about this” will run interference.
Going back to your point, I don’t think this is “hot button ideology”, it is having thought about the question before that creates this result.
There is a core contradiction in the human endeavor to be enlightened: the more you examine of the world the less you are able to see something new in the world. We simultaneously want to see things with fresh eyes while also running around learning as much about them as we can (sometimes pretending we have fresh eyes in the process). There is a value in being able to see something anew, but the value is in creating the appropriate opinion getting to the appropriate result. How many times can you realistically be expected to do that before you remember the prior result generated by seeing something “anew”, and if you never do than what really is the point of all this in the first place?