Content uploaded by Elise Wach
Author content
All content in this area was uploaded by Elise Wach on Oct 29, 2018
Content may be subject to copyright.
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
Learning about Qualitative Document Analysis
This paper outlines and reflects on the process of undertaking a Qualitative Document Analysis (QDA) on
policy and ‘practice’ documents in the rural water sector. This paper is relevant to organisations or
researchers interested in research or evaluation methodologies that can provide a systematic analysis of
policies and also serve as an engagement tool.
The QDA was undertaken as part of the Triple-S (Sustainable Services at Scale) initiative, for which the
Impact and Learning Team (ILT) at IDS serves as an External Learning Facilitator. The strengths and
weaknesses of the methodology are discussed here. Overall, the team found that the QDA exercise
provided useful information about trends and gaps in the rural water sector, helped to refine the Triple-S
engagement strategy, and served as a useful platform for engagement with partner organisations.
I LT B R I EF 13
AUGUST 2013
1 Introduction
Triple-S is a six-year, multi-country action research and learning initiative
which aims to promote long-term sustainable approaches to the funding
and implementation of water services in the rural water sector to
improve access and delivery to the rural poor. The Triple-S initiative, led
by the IRC International Water and Sanitation Centre, combines
research, advocacy and direct support to service providers, service
authorities, governments and international organisations at both the
broader international sector level, and at the country-specific level in
Ghana, Uganda and Burkina Faso.
Qualitative Document Analysis (QDA) is a research method for
rigorously and systematically analysing the contents of written
documents. The approach is used in political science research to
facilitate impartial and consistent analysis of written policies.
Given that Triple-S is aiming to change policies and practices in the rural
water sector, the initiative decided to undertake a QDA on policy
documents at the international level in order to understand trends and
progress in the sector and also to engage development partners in
identifying possible changes to policies and practices to move the sector
closer to achieving ‘sustainable services at scale.’ Consistent with Triple-S’s
‘theory of change’ generating discussion on these issues and catalysing
‘invocacy’ was just as much of a priority as generating reliable evidence
about policy trends.
The first round of QDA entailed an analysis of 11 Development Partner
(DP) policy and strategy documents published prior to 2008 (the year in which Triple-S began). The team suspected
In each Practice Paper published, we share
our experience and learning. We are
presenting ideas that we are exploring and
that others in the intermediar y sector
might like to explore.
Our experiences contribute to the body of
knowledge, but rarely if ever contain
incontestable insights. This paper should
not be read in isolation, however, and
should be seen as complementar y to other
work conducted on related issues of
capacity development, knowledge
management, and policy influence.
The knowledge and information
intermediary sector comprises those who
seek to improve flows of knowledge
between actors in decision-making and
change processes in order to generate
better development outcomes.
Intermediaries act in a range of ways:
enabling access to information; helping
people to make sense of it; and facilitating
connections that enable knowledge to be
shared between stakeholders. It is a
practice sector which cuts across other
sectors.
PRACTICE PAPER
IN BRIEF
IDS
IDS PRACTICE PAPER
IN BRIEF
discrepancies between policies and practice and therefore, Triple-S decided to test out the methodology on documents
that could ideally provide more information about the actual ‘practices’ of these organisations.
Following the completion of both a ‘policy’ and ‘practice’ QDA, we reflect here on the methodology and some of the
initial outcomes of the exercise.
2 Methodological considerations
Overview of methodology
Our QDA process entailed the following steps adapted from Altheide’s ‘Process of Document Analysis’ (1996). This
included: (a) setting inclusion criteria for documents; (b) collecting documents; (c) articulating key areas of analysis; (d)
document coding; (e) verification; and (f) analysis. An overview of what each of these steps entailed is provided below.
The following section then provides a discussion of our reflections and lessons learnt to date.
2.1 Setting inclusion criteria for documents
In selecting documents for the analysis, the team had to consider: which organisations would be included, the types of
documents to be reviewed, and the time of publication and release of those documents.
Criteria for organisations included for the first round of the QDA
(on ‘policy’) were that they were currently engaged
with the Triple-S initiative (we refer to such organisations as ‘Development Partners’), and that they were organisations
for which appropriate (and ideally rural) water policy, strategy, or guideline documents were available in the public
domain.
In the first (‘policy’) round of the QDA, 11 development partners were included. As discussed in a later section, when
we expanded the remit of the exercise to ‘practice’ documents, it became apparent that not all DPs had appropriate
practice documents available in the public domain, and therefore this ‘public’ criteria was no longer maintained.
Although we aimed to include the same DPs, two DPs were eventually dropped from the practice QDA as it was felt
that no appropriate documents were available or forthcoming.
Types of documents
reviewed for the policy QDA included organisational policies, strategies, guidelines or similar
documents. For the ‘practice’ QDA, the team aimed to analyse documents that might refer to policy ‘in action’ – this
could include assessment, appraisal, evaluation reports, progress reports, or annual reports for a programme, project or
initiative. The difficulties related to the selection and analysis of practice documents are discussed later in this paper.
Publication date
was considered in order to be able to use the exercise as a ‘baseline’ so as to track changes and
progress in policy and practice over time. This round of the policy QDA focused on policy documents published before
2008, the year in which Triple-S began. Practice documents were chosen for projects or initiatives that began soon
after the release of the policy document from the relevant organisation (under the assumption that they would, in
theory, embody the principles stated in the policies). It was envisioned that a second stage of the QDA could then be
carried out in 2014, reviewing the same DPs and their successor policy and practice documents to show changes (to
which Triple-S might have contributed in some way).
2.2 Collecting documents
Documents for the policy QDA were collected from the public domain; however, when it came to the practice QDA,
the team wasn’t always able to find appropriate documents online. The team therefore requested such documents
from its contacts within the organisations. As will be discussed below, this presented various complications which were
of valuable insight into the scope and limits of the QDA approach in this instance.
2.3 Articulating key areas of analysis
The policy documents were originally reviewed and analysed with reference to 21 themes that Triple-S had deemed to
be important and relevant for ensuring sustainable water services at scale. For consistency, the same themes were used
in the practice QDA. However, as we went along, it became apparent that some of these themes were not
appropriate for application to practice documents. It also became apparent throughout the course of the analysis that
the team would need to be more explicit about what some of the themes really meant in terms of both policy and
practice. We needed to be clear as to what exactly we were looking for when we said we were analysing a document
for its commitment to, for example, ‘accountability and transparency’ or to ‘monitoring for sustainability’, and what was
realistic to expect DPs to include in their policies and apply in their practices (i.e. ‘applicability’).
As will be discussed in more detail below, following the completion of both the policy and practice rounds, it was
determined to present only those themes that could be re-condensed into the ten Building Blocks for sustainability.
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
2
These are concepts which the Triple-S initiative regards as critical to ensuring sustainable services at scale in the rural
water sector, and are based on a study of rural water services in 13 countries (Lockwood and Smits 2011). This resolved
some of the issues of ‘applicability’ and also streamlined the presentation of findings to be more in line with the core
Triple-S messaging at international level.
2.4 Document coding and analysis
Each document was analysed to determine the extent to which the policy or programme it described addressed or
considered each of the identified ‘themes’ for sustainable services (i.e. areas of analysis). Text relevant to each theme
was highlighted and coded using a qualitative data analysis software (Dedoose).1 Based on the analysis of that text and
its meaning, relevance and context, for each theme, the document was assessed as ‘good’, ‘limited’, ‘none’ or ‘unclear’,
with clear criteria of what each score signified:
• A document which was rated as ‘good’ for a particular building block would have to include clear and consistent
references to a policy, strategy or practice which related to that building block and would (ideally) also give
sufficient information to the reader to judge that this element was being carried out competently and with
sufficient resource allocation.
• An ‘OK’ score would indicate the presence of practices or policy which related to a building block but where there
was insufficient detail to confidently give a score of ‘good’.
• A score of ‘limited’ would indicate only a brief or cursory reference to a building block with little corroborative
detail or contextual information on the level of resource allocation or prioritisation.
• A rating ‘unclear/none’ (both scoring zero) indicated that there was no information clearly attributable to a building
block.
This type of qualitative analysis of content, meaning and relevance in context is central to the value of the QDA, and
significantly distinguishes the methodology from a search for key words. For example, one of the themes was
‘monitoring for sustainability’. Rather than conduct a search for references to monitoring in general, the team aimed to
assess whether the policy or programme monitored not just functionality, but other indicators that make a service
sustainable. These would be context-specific, but would include issues such as adequate management capacity, tariff
recovery, and technical backstopping.
2.5 Verification
To ensure consistency and reliability of the coding and assessment process, the analysis of every document was verified
by a second person. In the case of the practice QDA, a third person provided
ad hoc
verification and also served as an
arbiter for any inconsistencies between the two primary coders. This goes beyond what is normally deemed sufficient
for coding reliability (Morse
et al
. 2002) and ensured robust interpretative analysis and conclusions.
2.6 Analysis
This data was then analysed to determine trends per DP, per type of DP, per theme, and to compare policy vs. practice.
Scores of good / okay / limited / none / unclear were assigned numerical values (0 to 3) to assist in aggregation and
data presentation.
Principles of the approach
Validity of findings: impartiality and dependability
The two interrelated principles that guided the analysis process were impartiality and dependability. An analysis is
considered ‘dependable’ if another reader would have ‘reached the same general conclusion given the opportunity to
analyse the same set of documents under similar conditions’ (Altheide 1996). The rationale behind each score was
explained, using detail and quotes from the documents.
The documents served as the sole source of information for the scoring, which facilitated objectivity but proved
challenging on some occasions. For example, in the policy review, it was difficult to give high scores for certain themes
or organisations when the team knew that these stated policies were not being effectively applied in practice. This
situation served as motivation to undertake a ‘practice’ QDA to see how things stacked up when it came to practices
on the ground. Equally, however, in the practice review it was sometimes difficult for some researchers to give a low
score when the team knew that practices had improved since the time of writing, or where certain text could be
reasonably assumed to include or indicate other practices or measures that were not explicitly stated in the document.
We had to remind ourselves of the original intention for this first round to: (a) serve as a baseline, reflecting policy and
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
3
practice from 2008, rather than current practice; and (b) for it to be treated as just one of many sources of information
about policies and practices in the water sector, rather than an evaluation of an organisation or programme.
Verification of coding by a second researcher, and, in the case of the practice QDA, a third party arbiter, helped to
ensure consistent interpretation of themes. The following example taken from the policy QDA illustrates the slight
differences in scoring between the first and second coder.
Figure 1: Example of the coding and verification process
Theme: Monitoring for sustainability
Analysis of the document 1st Coder 2nd Coder Final Score
Monitoring systems are discussed to
increase accountability, particularly using
their own volunteers. It is not clear
whether these link to national monitoring
systems.
Ongoing monitoring is planned to ‘see
which solutions really work’.
They are moving ‘beyond annual beneficiary
numbers’ as the only measure of impact,
and aim for long-term functionality moni-
toring, although no mention of service
delivery monitoring, per se.
‘...demonstrate at least 90 per cent of
supported systems are sustained over time,
as shown by evaluation/monitoring at 3-,
6-, and 10-year intervals after system
inauguration’.
Score: ‘Limited’
Rationale: Only
mentions function-
ality, not service
delivery, nor national
monitoring
Score: ‘Okay’
Rationale: Good
focus on monitoring
long-term function-
ality and use, but no
mention of how this
links to government
systems for moni-
toring
Score: ‘Okay’
Rationale: Focus on
long-term function-
ality, but not on
linking to national
monitoring systems
In addition, given the demanding nature of combing through policy documents and programme reports for a high
number of varied themes, having two (or three) pairs of eyes was better than one for ensuring each analysis was
thorough. The second coder could, from time to time, pick up on text or meanings that the first coder had missed.
Reflections on the methodology
Selection of codes
Part-way through the exercise, it was recognised that some themes were difficult to assess, particularly in relation to
practice documents. For example, one of the themes for analysis was the extent to which an organisation ensured that
its approach was ‘country-specific’ (adapting lessons from other places to local contexts, rather than employing a
so-called ‘silver bullet’ solution). While the team found it possible to analyse policy documents for this theme, with
regard to the practice documents, nearly any programme set in any given country could be argued to employ a
country-specific approach and this aspect was therefore difficult to meaningfully score. Another theme that wasn’t
practical to assess in ‘practice’ documents was that of ‘Roles and Responsibilities’. In many cases, sector reform is
needed first, before one could expect DPs to operate within newly defined roles and responsibilities.
Notwithstanding these difficulties the team decided to finish analysing the remaining documents by continuing with the
same themes and to later decide whether to drop or combine them. This ensured consistency and avoided the need for
recoding or rescoring.
In the end, the ten Building Blocks were used to communicate the findings, not least for simpler communication of
findings and consistency in messaging across the Triple-S initiative. While the building blocks incorporated most of the
21 themes, seven of them did not fit. The findings on some issues, for example, ensuring ‘Equality and Inclusion’ in
service delivery and access, and ‘Multiple-Use Systems’ approaches were therefore not included in the communications
of this research. However, Triple-S could still consider communicating these findings separately – it wasn’t necessary to
release everything all in one study.
4
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
Unclear and zero scores – irrelevance or inadequacies?
Throughout the analysis and via the validation process there was a contextual discussion of what it meant when a
theme was not included in a specific document. Given that the documents had different focuses, the lack of inclusion of
one theme did not necessarily mean that it wasn’t reflected in the organisation’s policies or practices. On the other
hand, however, not acknowledging an issue could reflect that the organisation had not considered or prioritised it. The
team tried to mitigate this issue by taking into account the nature of documents and the organisations themselves.
Thus, there have been a number of discussions about the implications of a score of ‘none’ versus ‘unclear.’ While the
team maintained the distinctions between these two scores in their analysis, in the presentation of findings to date,
both categories have been assigned the same score of ‘0’, and the team has stressed the importance of not relying too
much on quantitative aggregations of the data.
While not overtly one of the main ‘findings’ of the practice QDA, ‘lack of clarity’ was a frequent ‘score’ which poses an
equally broad set of challenges for both DPs in their writing practices and QDA researchers in how they plan and
interpret their research.
Document selection and representativeness
As outlined above, document selection for both phases of the QDA was guided by both practical and strategic
considerations, and each of these factors had an impact on the types of document that were eventually included in the
reviews and the range of findings presented. It was far easier to choose appropriate ‘policy’ than ‘practice’ documents.
The reasons for this are central to the applicability and value of the QDA approach in a development setting and we
have learnt significant lessons about its limits and applicability as a result of the two analyses.
We defined policy documents as those that express official organisational aims and strategies, and thus, policy
documents were both easy to identify and obtain. Organisational and sector policy documents are routinely publicised
in the development sector, and we were able to locate these online.
With regard to the practice review, the document selection process became more complex. We believe there were
two primary reasons for this. The first is to do with access, and the second revolves around the definition of what is or
can be a practice document.
We held a relatively open definition of what could qualify as practice documents: we considered assessment, appraisal,
evaluation reports, progress reports, or annual reports for a programme, project or initiative. This meant that the
practice QDA documents varied more in terms of length, focus and purpose than the policy documents we reviewed.
In our analysis and results presentation, we tried to mitigate this by taking into account the nature of the document and
the context and explicitly avoiding comparisons between DPs.
These types of documents which openly detail the operations and practices of organisations are not yet routinely made
public by NGOs, multilateral organisations and governments, and there is no standardised means by which they are
released, stored or located. Initiatives such as the International Aid Transparency Initiative (IATI) are making significant
progress on this issue and several organisations are increasingly transparent in this respect.
With regard to obtaining practice documents, many of the DPs were happy to pass on what they thought were
appropriate examples after the broad purpose of the review had been explained. The reputational standing of the
Triple-S initiative is important in this context. Triple-S is not an ‘outsider’ to the DPs it works with; rather, it attempts to
influence from within the system, to ‘invocate’ and facilitate change by working with DPs. It is possible therefore that a
lesser known initiative or ‘independent’ or academic researcher from a different discipline may have faced additional
difficulties or scrutiny in this respect.
An additional consideration to bear in mind is the sample of documents that we had access to for this review. Only
documents that DPs willingly disclosed, either publically or through direct contact were analysed; the team did not have
unrestricted access to partners’ documents in order to apply a sampling method. Documents from the public domain
and those accessed through contacts each have their own considerations in terms of selection bias. DPs can select
which documents to publish in the public domain and it can be expected that some organisations would tend towards
publishing documents which would reflect more positively on the DP (with the exception of some organisations, such
as Engineers Without Borders, which are beginning to publically disclose instances of ‘failure’). However, we did not
contact DPs prior to analysing any publically available documents, and thus, there would not have been selection bias
specifically related to this review for publically available documents. In the case of documents sourced from contacts,
there would be a risk of bias in the provision of documents, given that we explained that the documents would be
analysed. The degree to which this occurred is unknown to the team. For this reason, as well as the anticipated variance
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
5
in programme scope and quality within each DP, it was further stressed by the team that the findings from the QDA
exercise, particularly for ‘practices’ was not necessarily representative of the DP’s programming.
As is the case in the domain of the social sciences more generally, an element of trust has to exist between reviewer
and informant where this ‘friendly’ method of document sourcing is relied on. This trust is two-way – that the DPs
won’t supply a misleading document and conversely that the reviewer won’t misrepresent the DP. Our success in this
respect was put to the test during the ‘review and feedback’ stage of the analyses.
3 Reflections on the overall approach
Differences in analysing ‘policy’ vs ‘practice’
In analysing policy documents, it was challenging to ascertain whether the policy was simply talk or whether the
organisation actually had mechanisms in place and delivered on the principles and strategies outlined in their
documents. In policies, it can be easy to commit, for example, to ‘coordination’ with other actors or to ‘capacity support
to local service authorities.’
In discussing organised hypocrisy theory, Brunsson differentiates between talk, decisions, and actions:
People talk, decide and act on separate occasions and in different contexts; some people talk and decide about
how others should act. It is possible to act without making a decision or talking about it, and it is possible to
talk and decide without actually acting on it. So there is reason to suspect that there will often be discrepancies
among what is said, what is decided, and what is done...it is not necessarily the case that what is said is better
than what is done.’
(2003: 202).
Brunsson also found that talk, decisions and actions can often be inversely coupled – that talk or decisions in one
direction decrease the likelihood of corresponding actions, and vice versa. Policy documents are often conceptualised
as ‘talk’, especially when they refer to principles without establishing systems and structures to translate those into
action.
This common disconnect between ‘talk’ and ‘action’ in development led to the desire to try out the QDA methodology
on ‘practice’ documents in the hope that the exercise might highlight differences between what is said in policy and
what is done in practice. Thus, in the policy QDA, a good score was given if a document acknowledged and discussed
issues related to particular themes and indicated a commitment to them. In the analysis of the practice documents,
however, we tried to ascertain whether these principles were translated into systems and actions.
While there were instances in which the policy documents scored better than the practice documents, there were a
number of themes for which practice documents scored higher for some themes, which supports the notion discussed
earlier that policies do not necessarily represent an ‘ideal’ and that discrepancies between policy and ‘practice’ do not
always signify ‘policy failure’. Moving away from the policy/practice dichotomy allows the whole range of
documentation and recording activities undertaken by DPs to be understood as one varied and context dependent
process.
Responsibilities within the sector
The exercise tested the applicability of the Building Blocks in relation to the specific practices of organisations which
varied in their role and remit. The ten Building Blocks take a sector-wide, or broader systemic view which encompasses
roles and activities that are naturally shared and distributed among many actors and institutions in the rural water
sector. While it might be ideal if all organisations working in the sector collectively worked towards them all, it is
debatable to what extent individual DPs should be responsible for carrying them out individually. The team recognised
that low scores in any one of the building blocks would not necessarily imply a failure on behalf of the DP – it could
simply be the case that a DP specialised in an alternative area and did not carry out that specific role. This again
reinforced the need for contextualisation and drew us away from a comparative approach when presenting the
findings. It also focused and challenged our understanding around how to make the Building Blocks relevant to varied
and strategically different groups in the same sector who did not necessarily share the same wider systemic perspective
of the rural water sector.
Limitations in assessing practices
Accurately assessing ‘practices’ from what are inevitably partial, written sources of evidence was always going to be an
ambitious task. The inherent limitations of the texts and the methodological need to stay close to what was written
meant that the possibility of commenting on genuine ‘practice’ was limited. A more comprehensive review of an
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
6
organisation’s documents from a specific programme at a deeper level would be required to do this with any real
confidence.
The QDA provides valuable information about trends in policies and practices (including congruencies and disparities
between them) and is a lot less resource intensive than actually evaluating projects on the ground. However, if we really
wanted to assess what was happening ‘in practice’ we would need to do some evaluation work or conduct a meta-
evaluation. The latter would be possible but would require cooperation from DPs and careful consideration of sample
size and representativeness.
4 Outcomes of the QDA exercise
The purpose of conducting a QDA was not to provide a robust evaluation of the practices and policies of organisations
or the sector, but rather to see whether there might be trends in the documents and to get development partners and
stakeholders talking about these issues (including pushing back, where necessary). From the outcomes of the analysis
and the initial feedback that we have received to date from this exercise, it seems to have been a success.
Trends and issues identified
The QDA was able to provide some information about possible trends in policies and practice. In our analysis, it became
apparent that the two largest gaps in policies and practices were lack of attention to financing for lifecycle costs and to
planning for asset management. This gap was seen across virtually all DPs and all types of documents, though for these
two themes, practices scored lower than policies. Other major gaps were in the recognition and promotion of
alternative service provider options, and the monitoring practices, which predominantly focused on the measurement
of numbers of people served rather than quality of service, service providers, or sustainability criteria.
The exercise provided the opportunity to take a step back to consider what some of the major gaps in the sector are,
and offers a platform for discussion about why (or whether) these issues are important, what they mean in practice,
and what needs to change in the sector overall.
Feedback and engagement of development partners
Overall, the response to the QDA from DPs was better than anticipated, with partners indicating both a willingness to
engage in the research and in the issues themselves.
Responses to the practice QDA were received in two stages. A final draft was produced and internally presented to
the Triple-S team which was then released to the DPs in advance of publication for their comment. The majority of
partners responded with both positive and negative feedback. Some were concerned that context was not sufficiently
addressed by the review and that the study could therefore misrepresent their organisation. In response, we were keen
to stress the non-comparative and contextual-specific remit of the study. Others had questions regarding the
methodology, and these concerns were addressed ‘bilaterally’ via email. Being clear about the limitations of the
methodology to the research ‘participants’ helped diffuse potential tensions and moved discussion away from
understanding the documents as a ‘representation’ of one DP or another to viewing them more constructively as
‘productions’ which are always situated in contexts. Positive responses ranged from general interest and expressions of
thanks, broad agreement with the issues and the voluntary provision of further material, to deeper engagement with
the findings and expression of interest in further collaboration.
There were, of course, DPs that did not provide any response to the pre-publication draft, which was expected,
although generally the level of response was encouraging and in itself evidence of a desire for increased learning and
collaboration around these issues. After the draft had been circulated and comments had been received, the team
provided a summary and response as an annex to the main findings. This version of the paper was then published and
circulated online to a wider audience.
The second stage of feedback was more dynamic and centred around a live webinar in which the findings were
presented and DPs participated to discuss them in further detail, which took place roughly one week after the
publication of the review. Representatives from five out of the nine DPs included in the practice review were present,
with one sending apologies for a last-minute cancellation, which represented a high level of contribution, especially
considering the international cross-section of the participants.
We feel that a positive relationship has been built between the research team and the DPs involved and hope that this
will be reflected in further rounds of the QDA with similar access to information and open dialogue maintained.
Indeed, there was some demand for similar exercises to be carried out on a more bespoke level in future. Despite the
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
7
anonymous nature of policy and certain practice documents, individual DP representatives did demonstrate a strong
level of accountability and responsibility for the standing of their organisation and the reactions to the study confirmed
the existence of a healthy ‘learning sector’.
With regard to the practice QDA, DPs were appreciative that their operational documents were being subjected to a
systemic analysis via a credible lens – the building blocks – even if this lens was based on a broader systemic perspective
of the sector. There was broad agreement on the gaps identified and general appreciation of the ambitious scope of
the project to provide an objective overview of sector policy and practice with regards to long-term sustainability.
Discussions confirmed the recognition of systemic problems in the aid and water sector that are a hindrance to long-
term sustainability and we welcomed the opportunity to discuss our findings in an open setting.
Summary of lessons learnt about the QDA methodology
• The quality and appropriateness of the written material assessed is critical with respect to the validity of findings
and their presentation.
• The process of undertaking a QDA itself can help initiatives to refine and/or further develop the ideal practices and
policies that they hope to see realised. Building in reflection and flexibility in the beginning of the process to revise
codes can help avoid interpretive problems and ensure relevance. Ensuring that everyone has a shared
understanding of what the themes for analysis mean and the ideal of what each would look like in policy and
practice is imperative for ensuring consistent analysis.
• While analysing documents for a large number of themes can provide a wealth of information, it can require a
large amount of time and resources, and it may not be effective to communicate all of the findings at once.
• The process of double coding and of ensuring assessments were based only on the content of the documents,
rather than the team’s experience, is important to ensure that analysis is accurate, impartial and dependable.
• The implications of missing or unclear descriptions of themes are debatable, and taking into consideration the
nature of the organisation and document and triangulating with other data sources can help provide further clarity.
• Reviewing documents can provide useful insights but alone it does not represent a rigorous assessment of practices
in the sector. Triangulating with other sources of data may be an activity that Triple-S or another organisation
undertakes in the future.
• Trust has to exist between reviewer and informant when considering sourcing of documents. From our
perspective, credibility of Triple-S was important in sourcing documents.
5 Summary, next steps, and recommendations
Overall, the QDA exercise has been a valuable experience for the Triple-S initiative. It has enabled the team to:
• Gain a better sense of the gaps and strengths in policies and programmes in the sector;
• Refine its own strategy by reflecting on the changes that are needed and their feasibility for different actors at
different levels of the sector (i.e. the team recognised that low scores in any one building block did not necessarily
imply a failure on behalf of the DP, depending on the role, strategy or specialisation of that organisation);
• Engage DPs on these issues to facilitate joint reflection and action at organisational and sector levels;
• Reflect on the relationships between policy and practice in the sector.
Whether QDA is relevant or advantageous in other international development research and learning contexts will
depend on a number of factors: the time and resources available to undertake the research, and the subject matter and
aims of the assignment or topic.
Undertaking QDA, especially with a high number of variables to look for in a document, requires a systematic approach
to research and analysis, which is more time-consuming than simply reading and summarising a document. Traditional
literature reviews, can of course, equip us with many of the insights that QDA is able to provide, and in many instances,
this approach will be perfectly adequate. However, where a reliable analysis of complex interrelated concepts is
required across a range of qualitatively different documents, QDA is a very valuable method.
QDA provides a detailed ‘paper trail’ which describes how and on what basis an opinion was reached. The validation
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
8
9
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
process also provides additional rigour, insight and accountability and, for ongoing research programmes, reliable
replicability.
In addition, the depth and structure of the QDA research provides an easily digestible summary and also preserves
more detailed material for those that wish to look at the evidence in more detail. This has advantages for those
communicating findings and making them more accessible to varied audiences.
In summary, the QDA method combines a rigorous research framework which, combined with other sources of data,
can provide useful insights into trends and gaps at a sector or organisational level. When applied well, undertaking a
QDA can provide a platform on which the principles, policies and strategies of both the researcher and subject can be
tested, refined and developed.
Endnote
1 The first round of policy QDA was conducted with hard copies of policy documents; using software for the second
round made it easier to refer back to key themes and text in the documents, especially given that all of the researchers
involved in this study were in different locations.
References
Altheide, D. (1996) ‘Process of Document Analysis’, in D. Altheide,
Qualitative Media Analysis
, Thousand Oaks: SAGE
Publications: 23–43
Brunsson, N. (2003) ‘Organized Hypocrisy’, in B. Czarniawska and G. Sevón (eds),
The Northern Lights
, Copenhagen:
Copenhagen Business School Press
Lockwood, H. and Smits, S. (2011)
Supporting Rural Water Supply: Moving Towards a Service Delivery Approach
,
Colchester, Essex: Practical Action Publishing
Morse, J.M., Barrett, M., Mayan, M., Olson K. and Spiers J. (2002) ‘Verification Strategies for Establishing Reliability and
Validity in Qualitative Research’,
International Journal of Qualitative Methods
1.2: 2
Wesley, J. (2011) ‘Analyzing Qualitative Data’, in K. Archer and L. Berdahl (eds),
Explorations: A Navigator’s Guide to
Research in Canadian Political Science
, 2nd edn, Toronto: Oxford University Press
Learning about Qualitative Document Analysis
Institute of Development Studies Brighton BN1 9RE UK
T +44 (0) 1273 606261 F + 44 (0) 1273 621202 E ids@ids.ac.uk W www.ids.ac.uk
Credits
This Practice Paper In Brief was written by Elise Wach,
Richard Ward with contribution from Ruzica Jacimovic.
The opinions expressed are those of the author s and do
not necessarily reflect the views of IDS or any of the
other institutions involved.
Readers are encouraged to quote and reproduce material
from issues of Practice Papers In Briefs in their own
publications. In return, IDS requests due
acknowledgement and quotes to be referenced as above.
Funding for the production of this issue was provided by
the IDS Knowledge Services team through the Mobilising
Knowledge for Development (MK4D) programme.
MK4D is funded by the UK Department for International
Development.
About the Impact and Learning Team (ILT)
What makes development research accessible, relevant or appropriate for people outside the research community? Does
development research get its due in policymaking and practice? What would be value for money in research communication?
The Impact and Learning Team at IDS are interested in how communication of research brings about change - in particular,
what happens when people and technology mediate between researchers and decision makers. We use the term
‘intermediary’ to describe people and technology acting in this way. We think they play a critical role in making knowledge
accessible, relevant and responsive to demand.
The work we are doing in the Impact and Learning Team (ILT) is exploring and testing this assumption using action research.
We support people to think about the difference they want to make as well as how they are going to go about it. We draw
insights and approaches from IDS’s history of research, and the fields of marketing, strategic planning and evaluation, and
capacity development.
This Practice Paper is an output from our work.
Recent papers in this set
Batchelor, S. and Hogan, C . (2011) ‘Learning about “Learning Styles” in Getting Research into Policy Use and Practice?’,
IDS Practice Paper In Brief
6, Brighton: IDS
Perkins, N. I. with Batchelor, S. (2011) ‘Learning’ from and Within a Multi-sited Organisation’, I
DS Practice Paper In Brief
7,
Brighton: IDS
Birchall, J. Batchelor, S. and Hayes, C. (2012) ‘Learning about Mainstreaming Gender in Knowledge Intermediary Work’,
IDS Practice Paper In Brief 8
, Brighton: IDS
Birchall, J. Batchelor, S. and Hayes, C. (2012) ‘Learning About Reflecting Knowledge: An Approach for Embedding Reflective
Practice in an Action Research Film’,
IDS Practice Paper In Brief
9, Brighton: IDS
Batchelor, S. Gregson, J. Beynon, P. and Fisher, C. (2013) ‘Learning about Patterns of Learning within the Programme
Mobilising Knowledge for Development Phase 2 (MK4D2)’,
IDS Practice Paper In Brief
10, Brighton: IDS
Barnett, C. (2013) ‘Learning about Measuring Advocacy and Policy Change: Are Baselines always Feasible and Desirable?’
IDS Practice Paper In Brief
11, Brighton: IDS
Wach, E. and Wolcott, S. (2013) ‘Learning about Champions – Individuals Catalysing Social Change’,
IDS Practice Paper
In Brief
12, Brighton: IDS
Download for free at: www.ids.ac.uk/publications/ids-series-titles/practice-papers-in-brief
IDS PRACTICE PAPER IN BRIEF 13 AUGUST 2013 www.ids.ac.uk
