Content uploaded by Norman Elliott Fenton
Author content
All content in this area was uploaded by Norman Elliott Fenton on Jan 05, 2018
Content may be subject to copyright.
A preview of the PDF is not available
When 'neutral' evidence still has probative value (with implications from the Barry George Case)
Abstract and Figures
The likelihood ratio (LR) is a probabilistic method that has been championed as a 'simple rule' for evaluating the probative value of forensic evidence in court. Intuitively, if the LR is greater than one then the evidence supports the prosecution hypothesis; if the LR is less than one it supports the defence hypothesis, and if the LR is equal to one then the evidence favours neither (and so is considered 'neutral'-having no probative value). It can be shown by Bayes' theorem that this simple relationship only applies to pairs of hypotheses for which one is the negation of the other (i.e. to mutually exclusive and exhaustive hypotheses) and is not applicable otherwise. We show how easy it can be - even for evidence experts - to use pairs of hypotheses that they assume are mutually exclusive and exhaustive but are not, and hence to arrive at erroneous conclusions about the value of evidence using the LR. Furthermore, even when mutually exclusive and exhaustive hypotheses are used there are extreme restrictions as to what can be concluded about the probative value of evidence just from a LR. Most importantly, while the distinction between source-level hypotheses (such as defendant was/was not at the crime scene) and offence-level hypotheses (defendant is/is not guilty) is well known, it is not widely understood that a LR for evidence about the former generally has no bearing on the LR of the latter. We show for the first time (using Bayesian networks) the full impact of this problem, and conclude that it is only the LR of the offence level hypotheses that genuinely determines the probative value of the evidence. We investigate common scenarios in which evidence has a LR of one but still has significant probative value (i.e. is not neutral as is commonly assumed). As illustration we consider the ramifications of these points for the case of Barry George. The successful appeal against his conviction for the murder of Jill Dando was based primarily on the argument that the firearm discharge residue (FDR) evidence, assumed to support the prosecution hypothesis at the original trial, actually had a LR equal to one and hence was 'neutral'. However, our review of the appeal transcript shows numerous examples of the problems with the use of hypotheses identified above. We show that if one were to follow the arguments recorded in the Appeal judgement verbatim, then contrary to the Appeal conclusion, the probative value of the FDR evidence may not have been neutral as was concluded.
Figures - uploaded by Norman Elliott Fenton
Author content
All figure content in this area was uploaded by Norman Elliott Fenton
Content may be subject to copyright.
Supplementary resources (2)
Data
December 2014
Data
December 2014
... The latter is common in forensic cases in which we observe an effect (the evidence) and wish to infer the probabilities of possible causes of the effect (the hypotheses). A common forensic instantiation of the cause-consequence idiom is the hypothesis-evidence idiom [6,7,[12][13][14][15][16][17][17][18][19] (labeled A.F1 in Fig. 3). Hypotheses are also referred to as propositions. ...
... In fact, research is also non-uniform towards the definition of the connections. Some literature [7,[19][20][21][22] refers to the connections as causal, whereas other research such as Vlek et al. [17] state that links represent "correlation rather than causality". Although we prefer the latter, we chose not to rename this category as the purpose of the idiom is to model the line of reasoning from cause to effect. ...
... The common effect idiom models case hypotheses separately [19], using separate boolean nodes to represent each hypothesis. This is different from the hypothesis-evidence idiom, which uses a single node for all hypotheses, with states that correspond to each case hypothesis. ...
Bayesian networks are a powerful tool in forensic science for modeling activity level evaluations. However, constructing them can be complex, and the desire for a more standardized and structured approach to the modeling process is growing. Research suggest to use an idiom-based approach for modeling complex evaluations using Bayesian networks. The idiom-based approach is a bottom-up modeling approach that begins with small reasoning patterns and combines them into a larger network. The ability to combine several idioms to create more comprehensive templates allows for greater flexibility in modeling and enables analysts to more accurately represent complex scenarios. In literature, collections of idioms have been made for several disciplines. We would like to add our own collection to the existing literature. We are excited to share with you a collection of standard reasoning patterns - we call them "idioms" - that are essential for modeling activity level evaluations in forensic science using Bayesian networks. We have divided these idioms into five groups, each with a specific modeling objective: cause-consequence idioms, narrative idioms, synthesis idioms, hypothesis-conditioning idioms, and evidence-conditioning idioms. We strongly support the use of an idiom-based approach and want to emphasize the significance of our collection by demonstrating the ability to combine several of the presented idioms to create a more comprehensive template model.
... The latter is common in forensic cases in which we observe an effect (the evidence) and wish to infer the probabilities of possible causes of the effect (the hypotheses). A common forensic instantiation of the cause-consequence idiom is the hypothesis-evidence idiom [6,7,[12][13][14][15][16][17][17][18][19] (labeled A.F1 in Fig. 3). Hypotheses are also referred to as propositions. ...
... In fact, research is also non-uniform towards the definition of the connections. Some literature [7,[19][20][21][22] refers to the connections as causal, whereas other research such as Vlek et al. [17] state that links represent "correlation rather than causality". Although we prefer the latter, we chose not to rename this category as the purpose of the idiom is to model the line of reasoning from cause to effect. ...
... The common effect idiom models case hypotheses separately [19], using separate boolean nodes to represent each hypothesis. This is different from the hypothesis-evidence idiom, which uses a single node for all hypotheses, with states that correspond to each case hypothesis. ...
Bayesian networks are a powerful tool in forensic science for modeling activity level evaluations. However, constructing them can be complex, and the desire for a more standardized and structured approach to the modeling process is growing. Research suggest to use an idiom-based approach for modeling complex evaluations using Bayesian networks. The idiom-based approach is a bottom-up modeling approach that begins with small reasoning patterns and combines them into a larger network. The ability to combine several idioms to create more comprehensive templates allows for greater flexibility in modeling and enables analysts to more accurately represent complex scenarios. In literature, collections of idioms have been made for several disciplines. We would like to add our own collection to the existing literature.
We are excited to share with you a collection of standard reasoning patterns - we call them "idioms" - that are essential for modeling activity level evaluations in forensic science using Bayesian networks. We have divided these idioms into five groups, each with a specific modeling objective: cause-consequence idioms, narrative idioms, synthesis idioms, hypothesis-conditioning idioms, and evidence-conditioning idioms. We strongly support the use of an idiom-based approach and want to emphasize the significance of our collection by demonstrating the ability to combine several of the presented idioms to create a more comprehensive template model.
... The above points (i.e the importance of exhaustivity for the LR to be a measure of probative value and the need for a prior in order to use the LR to make conclusions about the posterior probability) have been long recognized by multiple statisticians and forensic scientists [43]- [51]. Even in a highly relevant legal paper [52] that reviews the use of Bayes and the LR in legal cases, the author asserts (page 3) that "propositions are mutually exclusive and exhaustive" (where 'propositions' refer to the prosecution and defence hypotheses). ...
... Further concerns about the use and limitations of the LR as a measure of 'weight of evidence' were detailed in [51] and in [55]. Specifically: ...
The likelihood ratio (LR) is a commonly used measure for determining the strength of forensic match evidence. When a forensic expert determines a high LR for DNA found at a crime scene matching the profile of a suspect they typically report that 'this provides strong support for the prosecution hypothesis that the DNA comes from the suspect'. Our observations suggest that, in certain circumstances, the use of the LR may have led lawyers and jurors into grossly overestimating the probative value of a LTDNA mixed profile 'match'
... It is portrayed by probabilities or likelihood ratios to identify the acceptance of a proposition (Wells 2014). For the probabilistic relevance, Bayesian networks have been studied for the admissibility and probative value of the evidence (Fenton, Neil, and Lagnado 2013;Biedermann and Taroni 2012;Vlek et al. 2016;Fenton et al. 2014). In cases where probabilistic methods cannot be applied due to reasons such as missing data or type of evidence, arguments, and scenarios are evaluated for the legal relevance of the evidence (Vlek 2016;Liu, Islam, and Governatori 2021;Prakken and Kaptein 2016). ...
In legal reasoning, part of determining whether evidence should be admissible in court requires assessing its relevance to the case, often formalized as its probative value---the degree to which its being true or false proves a fact in issue. However, determining probative value is an imprecise process and must often rely on consideration of arguments for and against the probative value of a fact. Can generative language models be of use in generating or assessing such arguments? In this work, we introduce relevance chain prompting, a new prompting method that enables large language models to reason about the relevance of evidence to a given fact and uses measures of chain strength. We explore different methods for scoring a relevance chain grounded in the idea of probative value. Additionally, we evaluate the outputs of large language models with ROSCOE metrics and compare the results to chain-of-thought prompting. We test the prompting methods on a dataset created from the Legal Evidence Retrieval dataset. After postprocessing with the ROSCOE metrics, our method outperforms chain-of-thought prompting.
... To assess the weight of evidence, a scientist must frame at least two competing propositions to weigh against each other (Cook et al., 1998b;Evett et al., 2000;Schaapveld et al., 2019). In the adversarial system (practiced in countries such as England and the US), these competing propositions represent the positions of prosecution and the defence (Fenton et al., 2014). The propositions that are addressed depend upon a number of factors, including the circumstances of the case, the availability of empirical data, observations that have been made, and the knowledge and expertise of the scientist . ...
The focus of this research lies within the context of the forensic process and addresses a current debate within the literature for the importance and necessity of a growing body of empirical research to inform each stage of that process. This thesis presents three experimental studies addressing the recovery, transfer, and persistence of forensic traces. First, a novel gelatine-based collection medium was created, and a sampling method validated, for recovering explosive and drug residues from a wide range of porous and non-porous surfaces. Second, the first reported use of Instron’s ElectroPuls for application to forensic science is also presented, employing a reductionist approach to evaluate the individual impact of force, time, and rotation on the transfer of explosive and drug particulates. Third, a comparison of the dynamics of drug particulates on paper and polymer banknotes are presented, assessing the implications this might have on crime reconstruction approaches as more countries adopt polymer banknotes as legal tender. Based on the results, this thesis presents an effective method for inclusion in the tool kit which investigators can rely upon when tasked with the forensic collection and recovery of trace particulates. Additionally, the findings indicate that there is value for broader crime reconstruction endeavours in taking a reductionist approach when seeking to understand the mechanics of trace transfers. This can assist in creating simulation models where specific parameters can be adjusted for a given case in which the transfer of forensic materials may have occurred. Such datasets are valuable for modelling the movements of traces to enable more transparent and reproducible interpretations of pertinent trace materials in crime reconstructions.
... The example BN does not provide absolute support for one of the propositions, so can be used for further analyses. When evaluating findings under two propositions with a BN, when the assumptions are not valid or both propositions are not exhaustive, in theory both propositions could be false and the BN will not provide meaningful values under these circumstances [27,28]. ...
Forensic soil comparisons can be of high evidential value in a forensic case, but become complex when multiple methods and factors are considered. Bayesian networks are well suited to support forensic practitioners in complex casework. This study discusses the structure of a Bayesian network, elaborates on the in- and output data and evaluates two examples, one using source level propositions and one using activity level propositions. These examples can be applied as a template to construct a case specific network and can be used to assess sensitivity of the target output to different factors and identify avenues for research.
The persistent issue of wrongful convictions in the United States emphasizes the need for scrutiny and improvement of the criminal justice system. While statistical methods for the evaluation of forensic evidence, including glass, fingerprints, and deoxyribonucleic acid, have significantly contributed to solving intricate crimes, there is a notable lack of national‐level standards to ensure the appropriate application of statistics in forensic investigations. We discuss the obstacles in the application of statistics in court and emphasize the importance of making statistical interpretation accessible to non‐statisticians, especially those who make decisions about potentially innocent individuals. We investigate the use and misuse of statistical methods in crime investigations, in particular the likelihood ratio approach. We further describe the use of graphical models, where hypotheses and evidence can be represented as nodes connected by arrows signifying association or causality. We emphasize the advantages of special graph structures, such as object‐oriented Bayesian networks and chain event graphs, which allow for the concurrent examination of evidence of various nature.
Perpetrator knowledge (also known as “guilty knowledge,” “insider knowledge,” “crime knowledge,” or “first-hand knowledge”) is an important, but undertheorized type of criminal evidence. This article clarifies this concept in several ways. First, it offers a precise, probabilistic definition of what perpetrator knowledge is. Second, the article provides a taxonomy of arguments relating to perpetrator knowledge. This classification is based on an analysis of 438 Dutch criminal cases in which this concept was mentioned. Third, it models these arguments using Bayesian networks. Fourth, the article explains a potential reasoning error relating to perpetrator knowledge, namely the fallacy of appeal to probability.
We introduce Bayesian reasoning in court not as a toolbox for doing computations, but as a way to assess evidence in a case. We argue that Bayesian reasoning comes naturally, even when the findings in a case cannot readily be translated into numbers. Not having numbers at one’s disposal is not an obstacle to use Bayesian reasoning. Although we present a coherent and complete view, we focus on the prior, since that seems to be the most problematic part of Bayesian reasoning. We explain that attempts to numerically express the prior fail in general, but also that a prior is necessary and cannot be dispensed with. Indeed, we explain in detail why decision-making should not be based on likelihood ratios alone. We next discuss two of the most delicate questions around the prior: (1) the possible conflict with the presumption of innocence, and (2) the idea that unwanted personal conviction (like racism) might enter the decision procedure via the prior. We conclude that these alleged problems are not problematic after all, and we carefully explain this position.
Scientific principles of forensic source identification have attracted widespread interest in recent years. Among those presented principles and theorems, the Bayes inference was regarded as one of the most scientific principles. In this paper, we argue that the Bayes theorem is in challenge when used as principal basis for forensic source identification. Furthermore, two novel concepts: feature-matching value and feature-matching identification value are proposed inspired by the basic ideas of information theory. Based on these two concepts, a new framework is established to describe the source identification principles of forensic science. The proposed source identification principle uses deduction logic structure, and unifies the three existing source identification paradigms. The newly proposed framework is expected to provide a solid scientific basis for the source attribution methods in forensic science.
The judgment of the Court of Appeal in R v T [1] raises several issues relating to the evaluation of scientific evidence that, we believe, require a response.
We, the undersigned, oppose any response to the judgment that would result in a movement away from the use of logical methods for evidence evaluation. A paper in this issue of the Journal [2] re-iterates logical principles of evidence interpretation that are accepted by a broad range of those who have an interest in forensic reasoning. The divergence between those principles of interpretation and the apparent implications of the R v T ruling are epitomised by the following issues that represent our collective position with regard to the evaluation of evidence within the context of a criminal trial.
Practical guidance for judges, lawyers, forensic scientists and expert witnesses on the logical analysis of DNA profiles, and their probative value in criminal proceedings. Explains technical aspects of DNA profiling and how this information should be presented and interpreted in criminal trials.
This fully revised and updated new edition explains the logical approach to the interpretation of forensic scientific evidence with a focus on general methods of interpretation applicable to all forms of evidence. It starts by explaining the general principles and then applies them to issues in DNA and other important forms of scientific evidence as examples. As in the first edition, the book analyses real legal cases and judgments rather than hypothetical examples only, and shows how the problems perceived in those cases would have been solved by a correct logical approach.
This authoritative text:
● Includes new chapters on forensic scientific methodology.
● Concentrates on principles of interpretation
● Features improved, recent case studies
● Has an international focus
Written for both forensic scientists preparing their evidence, and for lawyers and judges who have to deal with it, this book ties interpretation back to basic scientific principles as well as the principles of the law of evidence. It is essential reading for law students taking papers in evidence or forensic science, and science students studying the application of their scientific specialisation to forensic questions. Interpreting Evidence: Evaluating Forensic Science in the Courtroom, Second Edition is also aimed at legal practitioners and academics in these fields.
DNA profiling has brought to the courts a new way of looking at forensic science evidence. The weight of evidence, where there is a match between the profiles of a defendant and a crime sample, is presented in the form of a match probability. In all other areas of forensic science, it is long accepted practice for the scientist to give an opinion of the form "in my opinion, x and y have the same source" but recent judgments have established that this is not to be permitted when x and y are DNA profiles. Yet DNA profiling is better understood from a statistical standpoint than any other forensic technique, including fingerprints and, as profiling techniques become more powerful, so the match probabilities can be expected to become smaller. This paper discusses issues relating to how such probabilities should be presented at court.
The introduction of DNA evidence has transformed human individualization in criminal litigation, but it also introduced daunting statistical, philosophical and practical problems into the process. The current practice in many legal cases is that a forensic expert reports a match probability or a likelihood ratio. However, the value of the likelihood ratio depends on the particular hypotheses used by the expert. Often there are various choices possible for the hypotheses used, and the corresponding likelihood ratios for different hypotheses are typically different. We therefore argue that the findings of an expert should not be given with a single number, and that any report with a match probability or likelihood ratio should be accompanied by a discussion of the effect of these numbers. We suggest a way to do this, using the so-called posterior odds, which are invariant under various sets of hypotheses. Our approach is applicable in legal systems where the judge or jury is not allowed to know that the suspect was in a database. However, juridical problems may arise when courts insist on, for example, only a frequency estimate to be reported.
In R v T the Court of Appeal concluded that the likelihood-ratio framework should not be used for the evaluation of evidence except ‘where there is a firm statistical base’. The present article argues that the court's opinion is based on misunderstandings of statistics and of the likelihood-ratio framework for the evaluation of evidence. The likelihood-ratio framework is a logical framework and not itself dependent on the use of objective measurements, databases and statistical models. The ruling is analysed from the perspective of the new paradigm for forensic-comparison science: the use of the likelihood-ratio framework for the evaluation of evidence; a strong preference for the use of objective measurements, databases representative of the relevant population, and statistical models; and empirical testing of the validity and reliability of the forensic-comparison system under conditions reflecting those of the case at trial.