PresentationPDF Available

How can interview-based social science become more open?

  • UCL & The London School of Economics and Political Science


Implementing open social science (OSS) innovations focuses in large part on showing more of the concrete evidence that underpins analysis of findings and researchers’ conclusions and arguments. Many commentators have cast doubt on how far OSS approaches can be applied in qualitative research, suggesting that only limited openness can be achieved, especially where a large part of the evidence-base comes from qualitative interviews. Some forms of interviews aiming to recover the ’life-world’ of interviewees as a whole, and as they see it, may be feasible to do on a fully ‘on the record’ basis where archiving of whole transcripts or recordings is feasible with the permission of interviewees. By contrast, for elite and specialized interviewing and conversations are ‘off the record’, then transcripts and recordings cannot be archived or shared with other researchers. However, most such interviews are now conducted on a ‘non-attributable’ basis where direct quotations are allowed so long as they are completely anonymized. With care, it may be feasible to achieve non-straightforward ways of archiving or allowing re-access to such materials. Finally, changes in interview technologies towards using digital methods have opened up new ways of conducting and making available interviewing that might begin to approximate systematic podcasting or video-casting interviews.
Patrick Dunleavy (LSE) and Timothy Monteath (UCL/LSE)
1 December 2022
How can interview-based social science
become more open?
Part of The Open Social Science Handbook project
What are research interviews?
Structured conversations, normally face-to-face, between
The researcher(s) who (mostly) pose (initial) questions and give follow-up
responses to guide the conversation, and
The interviewee(s) who (mostly) provide answers or responses to questions
that may extend into conversational or narrative threads
Commonly both (sets of) actors are present in person in the same location,
but interviews also increasingly take place remotely
Structured (or semi-structured) interviews follow a pre-set sequence of pre-
notified questions (but less rigid than a mass-survey session)
Elite and specialized interviewing focuses on a common spine of topic
questions but tackled in variable and responsive sequences and depth,
depending on the interviewees and the purposes of the research
What interviews are good for
Helping to grasp people’s complex
meanings, understandings, emotions,
Selectively recovering memorable details
and quotes, especially those
subsequently rehearsed by the
Understanding post hoc interpretations
(rationalizations) of events or actions
Revealing things about an interviewee’s
personality via quotations
Ostensively demonstrating the
‘organizational culture’ of firms,
agencies, NGOs etc via quoting
interviewees’ discourse and reactions to
Achieving a ‘first draft of history’ by quickly
recording how participants explain and
represent what they were doing for major
events or policy shifts
Surfacing different individuals’ (or groups’)
perspectives and accounts of the same
events especially for policy histories,
and traumatic events
Understanding the ‘life world’ of people
whose experiences and meanings could
otherwise be difficult or impossible to
Creating an interaction that may lead to
accessing documentary evidence
Getting leads to further potential
interviewees (e.g. cascades/waterfalls)
What interviews are bad at
Producing quantitative response
datapoints, because of variations in the
detailed phrasing and sequencing of
Surfacing detailed information on dates or
details of arguments or actions or
levels of funding or regulation decisions
- unless interview statements can be
backed up by public documentation
- or interview leads to access to
hitherto private documents
Getting past ‘public relations’ answers
Accessing ‘unsanitized’ quotations
Interviewees may stray into controversies
creating risks for themselves or defame
others making editing inevitable
Keeping elite interviewees on track in the
conversation so as to avoid
- digressions into experiences off topic
- irrelevance, not addressing questions
-re-hashing well known facts only
- over-compliance, interviewee tells you
what they think you want to hear
- over-extension or ‘boosterism’ only on
issues the interviewee knows about
Readers or re-users must rely a lot on the
interviewer’s view of what was said and
why v. hard to gainsay
Similarly, interviewer summaries and
comparisons \across people talked to are
very hard to check or counteract
Key uncertainties around interview-based
Can we be sure what interviews took place?
Who did you speak to? Maybe different people would have said other
Were the people interviewed in a position to credibly answer the
questions asked? in terms of their level of seniority in an
organization, involvement in key events, the time period of their
involvement, other situational factors
Did the questions asked, or your sequencing of topics, ‘lead’
respondents to answer in a particular (desired) way?
Is your interpretation or analysis legitimate or credible, given what
interviewees actually said?
E.g two scandals around interview-based
An article on gay marriage in Science by PhD student and top Yale political
scientist claimed rare evidence that persuasion of interviewees to become
more tolerant was feasible across repeat interviews, especially where the
interviewer revealed their gay sexuality. It later emerged that the estimated
costs of the interview fieldwork was $2 million and that PhD person could not
provide any transcript evidence of interviews to back claimed quantitative
coded evidence. Article retracted. See here
Amazon’s ‘mechanical Turk’ system for spot contracting between survey
sponsors and ‘gig’ interviewees (used by businesses and a lot by impoverished
social scientists) has suffered from gig people using VPNs to fake their
locations, roles and qualifications. See here
Three ways of using interviews to produce
codes and quantitative metrics
Pre-coding frame for short ‘open text’ comments or write-in boxes in surveys
Very strong interview frames that spark an extended in-person or remote qualitative
discussion with an expert interviewee/informant, on each topic point in the frame.
The range of variation is restricted to a consistent three or five point scale between
two poles across each topic. Each informant’s response is post-classified onto the
frame on every topic. See
Free text responses (normally) transcribed and then analysed quantitatively for
vocabulary, concepts, conjunctures, memes and phrasing bits using NVIVO-type text
analysis software. Hand coding of interest-items is also feasible with small N
Main types of social science interviews
Purpose of
1. Off the
2. Non
3. On the record
May not be cited
or quoted at all
May be quoted verbatim,
but only in a fully
anonymized way
May be quoted verbatim and
attributed to the named
a whole ‘life
Anthropology of
Anthropology, sociology Ethnographic studies
Capturing one
of experience
survey open-
ended questions
Medical and health studies,
social work, consumer
Some of column 2 types
specific or
historical role
Biography Dominant form in elite
political science/ IR/ public
policy, social policy,
business, management
Management case studies, plus
some of column 2 types
Mode of recording
Information lost
Practical issues
Written notes,
in (or near) interview
Any full or reliable record of
Very high
Taking notes while
listening/ talking
Typescript of audio
Character of interviewee’s
voice and conversation
Easy to deposit. Maybe
helps anonymization if
Full audio
Facial expressions, body
language. Visual cues to
interview context
Here and all rows
: Only feasible if
Interviewer or 2
nd person not
Covers only person(s)
in focus
Zoom, Teams online
Not much. Transcripts
generated also
Very low
easy, works well
with multiple people
Facial expressions, body
language. Any edits made
Can be more complex
to set up. Editing skills
maybe needed
Any edits made
Very low
Practical steps towards greater openness
with off-the-record interviews
Pre-register core questions (especially prompts that might elicit documentation)
Send interviewees the core questions basic ‘no surprises’ courtesy
Privately outline interviewee selection strategy in private document, and update
narratively as interviewing unfolds e.g. record ‘cascade’ interviewing links
Publicize a general outline in a blog or on social media covering the main focus of
study questions and rough interviewee sub-set.
Invite comments and suggestions from other researchers and potential interviewees
Systematically ‘stand up’ key points from an OTR interview from documentary
sources or other intervieweed
Try to move interviewees to accept on the record or non-attributable status
Achieving greater openness
with non-attributable interviews 1
Take all six previous steps recommended for off-the-record interviews
Notify non-attributability safeguards and ultimate disposition of data planned to all
interviewees and record them agreeing to it
Privately outline your interviewee selection strategy in a non-public but date-
stamped document. Update narratively as interviewing unfolds.
Record ‘cascade’ interviewing links who recommended whom
Record biases or perspectives as interviewees accumulate, and seek balance
Achieving greater openness
with non-attributable interviews 2
Use long quotations to convey meanings where it’s feasible to preserve source
anonymity (Such discourse is checkable)
Use multiple short quotations more widely with clear conventions
Use interviews to create codings that can be summarized quantitatively
Use text analysis software to fragment memes and text discourse in anonymized
ways to yield quantifications
Use text analysis to fragment memes and discourse in anonymized ways that yield
new summary perspectives on meanings
Try to move interviewees to accept on the record status
Enhancing the openness
of on-the-record interviews
Take previous applicable steps from off-the-record and non-attributable interviews
Record full discourse systematically and digitally in easily archived forms
Speedily complete documentation for interviews as soon as conducted, and link to
your recorded subjective impressions of the person and content involved.
Move towards podcasting or video-casting all interviews, without substantive
editing, and potentially generating some immediate open commentary or debate
Reactions and comments may be addable to the initial record? “With many
eyeballs, all bugs are shallow” open source rationale
Deposit the full-discourse record in a permanent archive with good documentation,
and publicize DB’s location in open access articles, books, and university repository
Respond positively and quickly to other researchers’ enquiries and requests
Single study benefits of greater openness with
interview-based research
PROVENANCE Researchers and (and readers generally) gain extra assurance that
your interviews took place, with appropriate people, and were undertaken with due
care and appropriate methods
REPRODUCEABILITY Applying the same approach as your analysis, other
researchers can check the accuracy and validity of your interpretation against the
complete context of what interviewees said, or at least a wider context of discourse
during the interview (Turing Way criterion)
ENHANCED ROBUSTNESS Other researchers can apply different methods to the
same interview database, hopefully reaching similar conclusions (Turing Way
REPUTATION Analysis clearly supported by open data evidence will be better
regarded in the discipline and by outside readers
Wider benefits of greater openness with
REPLICABILITY Researchers can use the same method (i.e. initial questions,
concepts/ vocabularies and follow-on prompts) with other (but validly comparable or
similar) sets of interviewees, and find if the same results emerge (Turing Way
GENERALIZABILITY Applying different interview approaches or methods, with
different sets of interviewees, across different contexts and perhaps time periods,
other researchers can find if the same results emerge (Turing Way criterion)
LEARNING GAINS with positive results Replicated results and greater
generalizability boost sub-field learning, consensus and authoritativeness
LEARNING GAINS with negative results Establishing the reasons for non-
replication with different methods can be faster and more definite. Similarly, firming
up the reasons for non-generalization to other samples of interviewees is easier.
Nick Bloom and John Van Reenen (2006) ‘Measuring and Explaining Management Practices Across Firms and Countries’,
CEP Discussion Paper No 716, March 2006.
David Broockman and Joshua Kalla (2016) ‘Durably reducing transphobia: A field experiment on door-to-door canvassing’,
Science, 8 April . Vol 352 ,Issue 6282.
Maria Konnikova (2015), ‘How a Gay-Marriage Study Went Wrong’, New Yorker, 22 May,
Douglas J. Ahler, Carolyn E. Roush, Gaurav Sood, (2020) ‘The Micro-Task Market for Lemons: Data Quality on Amazon’s
Mechanical Turk’ ,
Patrick Dunleavy (LSE) and Timothy Monteath (UCL/LSE)
1 December 2022
How can interview-based social science
become more open?
Part of The Open Social Science Handbook project
Thanks for listening
ResearchGate has not been able to resolve any citations for this publication.
Not just turnout, but turnaround matters In the last several U.S. presidential elections, the campaign mantra has focused on making sure that voters already aligned with one's candidate do get out to vote. There is a long history of unsuccessful efforts to change people's attitudes. Nevertheless, Broockman and Kalla conducted a field experiment showing that Miami voters shifted their attitudes toward transgender individuals and maintained those changed positions for 3 months (see the Perspective by Paluck). Science , this issue p. 220 ; see also p. 147
Standard real business cycle models must rely on total factor productivity (TFP) shocks to explain the observed comovement of consumption, investment, and hours worked. This paper shows that a neoclassical model consistent with observed heterogeneity in labor supply and consumption can generate comovement in the absence of TFP shocks. Intertemporal substitution of goods and leisure induces comovement over the business cycle through heterogeneity in the consumption behavior of employed and unemployed workers. This result owes to two model features introduced to capture important characteristics of U.S. labor market data. First, individual consumption is affected by the number of hours worked: Employed agents consume more on average than the unemployed do. Second, changes in the employment rate, a central factor explaining variation in total hours, affect aggregate consumption. Demand shocks--such as shifts in the marginal efficiency of investment, as well as government spending shocks and news shocks--are shown to generate economic fluctuations consistent with observed business cycles.
How a Gay-Marriage Study Went Wrong
  • Maria Konnikova
• Maria Konnikova (2015), 'How a Gay-Marriage Study Went Wrong', New Yorker, 22 May,