I teach a module on qualitative interviewing. The assessment requires students to complete a single in-depth, semi-structured interview, and design a research project around it; a journal article if you will, based on one interview.
For many students it is their first time doing this sort of research work, and obviously some have worries. One that comes up a lot is the issue of ‘leading questions’, asking a question which in some way is likely to generate a certain response, thereby affecting or biasing the data. The Media College website has some great examples of how this works: ‘Tell me about problems you have with your boss’ will lead a respondent to assume there are problems which can affect the tone of their thinking and general direction of the discussion, whereas (at least starting with) ‘Tell me about your relationship with your boss’ allows them to drive the conversation into both positive and negative territory. Neither question may be answered truthfully or accurately, but in the example of the latter it’s not the questioner’s fault.
I got thinking about this when I read an article about the upcoming US Presidential election in a story from The Times (paywall) about candidate Senator Marco Rubio (R-Florida) ‘giving his blessing’ to the UK to leave the EU. Rubio is quoted in the story as saying:
“Irrespective of what decision the UK makes … they’ll continue to be certainly our best friend in the world and one of our strongest alliances…Part of us being a strong ally [of Britain] is for us to respect its sovereignty and its right to make its own decisions…I don’t think it’s proper for an American president…to tell the UK what is right for them, any more than it would be right for the UK to tell us that they wanted us to sign Nafta [the North American Free Trade Agreement] or some other agreement.”
I know he said this, although the quote has been tidied up somewhat, because the audio of the press conference has been published on AudioBoom. You can listen back to the interaction yourself where Tim Montgomerie, currently covering US politics for The Times after previously being editor of its Comment section, asks Rubio about his view on UK membership of the EU. But he doesn’t just say ‘Senator Rubio, are you in favour of the UK leaving the EU?’, or ‘What are your views on the UK’s membership of the EU?’. Montgomerie asks (and I have transcribed this):
“President Obama has said that he would like Britain to stay part of the European Union, and I wonder whether you have a view yourself? It’s a big issue for many of us and we wonder whether America would accept open borders with its neighbours as Britain currently has, with other judges from other countries deciding your laws, and some of us would perhaps like an American President to be more open to the possibility that an independent Britain, freed from being part of a European Superstate may actually be a better ally for your country?”
The first sentence is acceptable from a social science perspective: grounding the question in what an opponent/key figure thinks is generally fine, although as a Republican politician (and former Tea Party favourite) Rubio is always likely to pivot and want to do the opposite of what the Obama thinks. But if Montgomerie had just asked “President Obama has said that he would like Britain to stay part of the European Union, and I wonder whether you have a view yourself?” it would have been just fine, and interesting to know what Rubio would have said.
But he sets up the question so much, using such loaded language (Superstate! Open borders! Freed!), that Rubio almost has no choice but to pander to him. As Montgomerie has said of one of Rubio’s opponents Senator Ted Cruz (R-Texas), “Few have pandered more than the junior Senator from Texas. I haven’t noticed a single occasion when he’s told a conservative audience something that they didn’t want to hear. That’s followership, not leadership.”* Well what’s the difference here? Rubio clearly ends up repeating Montgomerie’s talking points back at him. This isn’t Rubio’s fault particularly, but neither, it seems to me, does it meet the high standards we should expect of broadsheet journalism (I know).
At no point does the article say, “After being fluffed up by a questioner who made clear to the Senator exactly where his own sympathies lay, Senator Rubio agreed with The Times that the EU was evil, etc…” I don’t know Montgomerie’s position at The Times; if he is ostensibly a columnist then it’s perfectly reasonable for him to express his views. But his name appears on the byline of the piece in the paper (screenshots below), which to me, the humble reader, suggests he’s coming at this with journalistic neutrality.
I am sure this happens a lot. There will be plenty examples of this in the media, from across the political spectrum, including (god forbid) The Guardian, of cosy answers being given and written up without the suitable context and full facts. But this one has the audio, and we can hear the sausage being made. And it’s left me pretty pleased about the standards we try and uphold in social science.
In the assessment, I get my students to submit a fully annotated transcript of their interview. Montgomerie’s would have big red pen all over it.
*Montgomerie is absolutely right on this point by the way. For an example watch Cruz’s speech to the Republican Jewish Coalition Forum: urgh.
After I asked The Times about this issue, Tim Montgomerie (quite generously and within an hour) replied to my points thus:
It was clearly a loaded question and it came from me, a columnist rather than a news reporter. I would argue that most questions are loaded in some way. On the Today programme politicians will be asked to justify the enormous cost of a policy initiative or will be given other facts to put their question in context. I did the same – only more pointedly.
On the substance I think it was clear from Marco Rubio’s answer that he’d thought about the issue – putting his answer in the context of America’s membership of Nafta – the nearest US equivalent to our membership of the EU. Like many Republican politicians he sees the EU as a bit like the UN – a supranational body that limits nation states. Perhaps he wanted to give me an answer I wanted to hear but I doubt it. I suspect – if he was to be influenced by anything – he would have been more conscious that he was answering questions to an audience of 100 leading American CEOs – many, like big British businesses, will favour any political status quo. I guess he gave the answer he believed in. He has dedicated a lot of time to foreign policy questions as a member of the Senate Foreign Relations Committee.
Firstly I’m grateful to (and frankly surprised) that I got a response so quickly.
In his reply, Montgomerie is quite upfront about his ‘loaded question’. I hope that in the original blog it didn’t seem like I thought Montgomerie was putting opinions in Senator Rubio’s mouth. I am sure Rubio is largely against organisations such as the EU, in favour of the UK making its own decision, and would have answered such however the question had been phrased.
Yet I would disagree about the Today programme point – generally that programme plays devil’s advocate in order to test politician’s own positions. If a politician on the Today programme – from left or right – was given such a loaded or softball question, both I and Tim Montgomerie would both have been shouting at our radio’s for them to be held to account, not given the opportunity to push their talking points. There may be a big difference between starting your question ‘Some people would argue that…’ and ‘I think that…’.
However, a relatively minor issue, but food for thought, and a great example to use in the classroom.