This afternoon, between answering student queries and marking presentations (and following the frantic updates from Boston), I managed to find an hour to read Jennifer Platt‘s historical account of an interesting methods experiment from the 1940s. The article can be found here.
It details how the US Social Science Research Council undertook a project to re-analyse the findings of Robert Angell’s 1931 study The Family Encounters the Depression. A team of researchers, including Robert Merton and led by the SSRC’s Chair for the Appraisal of Research Ernest Burgess, re-examined Angell’s source material and data to see if their categorisation of how families were affected by and coped with the great depression was the same as the author’s. I won’t go into all the detail about what the findings were, but as someone who increasingly sees himself as a methodologist, I was fascinated that such a study would be undertaken.
A ‘Committee on the Appraisal of Research’ is, for starters, a fascinating construct. Does such a similar thing exist now, or are we now so inundated with studies and research we now trust that researchers have got it right the first time? As someone who has just completed their PhD and knows both that there is stack of data at home that no other human being will ever look at (and knows where the bodies are buried in the research – see Shane Blackman), my research does not feel like it has ever really been appraised. No-one has ever asked to see all of the quotes and interviews and ethnographic fieldnotes that were not included. They could be vital, totally contradicting what I’ve published, with me leaving them out for the sake of creating a viable, simple narrative.* Have I ‘quote-pulled’, something I’m sure we are all guilty of.
There are some studies which aim to re-examine previous research and data, Mike Savage’s exploration of The Affluent Worker (by, ironically, among others Jennifer Platt) is one that comes quickly to mind, and statistical data will be examined in various ways as researcher utilise different variables. But as generally research is so of the moment this approach to re-assessment and appraisal seems time-consuming and odd. Personal experiences breed narratives and for others to tr and assess those narratives is often unexplainable.
This is interesting to me because I made the claim in the thesis and at various conferences that if you gave two researchers the same set of data (interview transcripts for instance) they would analyse them differently. Apparently this has not yet found to be the case. Platt recounts how this study, aimed at seeing if ‘a personal equation entered into the classification of the cases’ (p.146), found that Merton’s analysis matched up very closely to Angell’s, while for other researchers, who tried alternative ways of examining the data, it did not.
The oddest thing about the restudy it seems is how ham-fisted and flexible it was. The researchers tried many different ways to analyse the data, different to Angell’s, and the project grew and became distorted, when surely the point was to recreate his study in order to identify gaps or nuances. It was never published in the end, buried in the churn of WWII, for a host of possible reasons which Platt recounts [152-4]. There is another study recounted in Bryman’s 4th Ed. where a similar experiment was tried, where again researchers who analysed data separately found similar categories through which to analyse it (I don’t have access to the book at the moment, so can’t remember the reference). Maybe this says something abut the ‘commonality of experience’ or maybe it’s just that there are only so many ways of analysing a paragraph.
I want to test this out. Over the next few months I’ll be devising a project that hopefully colleagues will join me on. I’m thinking five interview transcripts on similar themes, five researchers from different backgrounds, and see what happens. Maybe we’ll all write the same thing, or maybe we’ll all discover a little bit more about where we’re coming from. When researcher biography seems such a key part of research outcomes (self-promotion alert!) this would be an actual, possibly cross- or post- disciplinary, experimental way to figure it out.
*If my supervisors or examiners or editors of leading journals are reading this, this didn’t happen.**
**Or did it?