No full-text available
To read the full-text of this research,
you can request a copy directly from the author.
Fractional Bayes Factors for Model Comparison
Abstract
Bayesian comparison of models is achieved simply by calculation of posterior probabilities of the models themselves. However, there are difficulties with this approach when prior information about the parameters of the various models is weak. Partial Bayes factors offer a resolution of the problem by setting aside part of the data as a training sample. The training sample is used to obtain an initial informative posterior distribution of the parameters in each model. Model comparison is then based on a Bayes factor calculated from the remaining data. Properties of partial Bayes factors are discussed, particularly in the context of weak prior information, and they are found to have advantages over other proposed methods of model comparison. A new variant of the partial Bayes factor, the fractional Bayes factor, is advocated on grounds of consistency, simplicity, robustness and coherence.
... Each of these must be specified. If is estimated via maximum likelihood estimation (MLE) -as is the case for generalized linear models -standard asymptotic arguments show the posterior distribution is normal with mean and variance , where is an estimate of , the Cramer-Rao lower bound, that is, the inverse The prior distribution is specified as a fractional prior (O'Hagan, 1995). This constitutes updating an uninformative prior with a likelihood based on a fraction of the data, where the subscript n is used to make explicit its dependence on the sample size; the choice of will be discussed in detail in the following section. ...
... This constitutes updating an uninformative prior with a likelihood based on a fraction of the data, where the subscript n is used to make explicit its dependence on the sample size; the choice of will be discussed in detail in the following section. As well as avoiding an arbitrary choice of prior, BFs based on fractional priors enjoy several attractive properties, including consistency and invariance to linear transformations (O'Hagan, 1995). In our context, the fractional prior is normal with variance , reflecting that the variance is inflated by a factor of 1 / n relative to the posterior variance, 2 / which uses the full data. ...
... Thus, one can think of using this option as setting the alpha level based on BIC, at least when sample sizes are large. For this reason, we refer to this option as .An alternative choice for is to use the minimal training sample for the prior specification, leaving maximal information in the data for hypothesis testing(Berger and Pericchi, 1996;O'Hagan, 1995). In other words, we set such that the minimum number of observations is used to specify the prior under a given null hypothesis. ...
In management research, fixed alpha levels in statistical testing are ubiquitous. However, in highly powered studies, they can lead to Lindley’s paradox, a situation where the null hypothesis is rejected despite evidence in the test actually supporting it. We propose a sample-size-dependent alpha level that combines the benefits of both frequentist and Bayesian statistics, enabling strict hypothesis testing with known error rates while also quantifying the evidence for a hypothesis. We offer actionable guidelines of how to implement the sample-size-dependent alpha in practice and provide an R-package and web app to implement our method for regression models. By using this approach, researchers can avoid mindless defaults and instead justify alpha as a function of sample size, thus improving the reliability of statistical analysis in management research.
... In addition, we also consider a half-Cauchy prior for the square root of variance components (Gelman, 2006;Polson & Scott, 2012). Because the prior on the vector of regression coefficients is improper, we develop a fractional Bayes factor (FBF) approach (O'Hagan, 1995). We note that Porter et al. (2023) have proposed FBF for Gaussian mixed models for the particular case of spatial areal data. ...
... In order to obtain the posterior model probabilities of interest, we use an FBF approach. The FBF is a modification of the Bayes factor that allows for improper priors on parameters (O'Hagan, 1995). ...
... To solve this problem, we use the FBF (O'Hagan 1995) to approximate the Bayes factor. Porter et al. (2023) developed the FBF method for Gaussian hierarchical models with ICAR random effects. ...
We propose a Bayesian model selection approach for generalized linear mixed models (GLMMs). We consider covariance structures for the random effects that are widely used in areas such as longitudinal studies, genome-wide association studies, and spatial statistics. Since the random effects cannot be integrated out of GLMMs analytically, we approximate the integrated likelihood function using a pseudo-likelihood approach. Our Bayesian approach assumes a flat prior for the fixed effects and includes both approximate reference prior and half-Cauchy prior choices for the variances of random effects. Since the flat prior on the fixed effects is improper, we develop a fractional Bayes factor approach to obtain posterior probabilities of the several competing models. Simulation studies with Poisson GLMMs with spatial random effects and overdispersion random effects show that our approach performs favorably when compared to widely used competing Bayesian methods including deviance information criterion and Watanabe-Akaike information criterion. We illustrate the usefulness and flexibility of our approach with three case studies including a Poisson longitudinal model, a Poisson spatial model, and a logistic mixed model. Our proposed approach is implemented in the R package GLMMselect that is available on CRAN.
... In particular, we examine a sum-zero constrained ICAR prior for spatial random effects in a Bayesian hierarchical model (Keefe et al., 2018). We devise a fractional Bayes factor (O'Hagan, 1995) approach for model selection via posterior model probabilities. Fractional Bayes factors use a portion of the likelihood to update priors on parameters, which enables our automatic Bayesian model selection with an improper reference prior on model parameters. ...
... Fractional Bayes factors use a portion of the likelihood to update priors on parameters, which enables our automatic Bayesian model selection with an improper reference prior on model parameters. Model selection consistency, which refers to the method's ability to select the true model as sample size increases if the true model is in the candidate set, is a well-known result when using fractional Bayes factors (O'Hagan, 1995). Thus our approach provides consistent, simultaneous selection of fixed effects and spatial model structure in Bayesian hierarchical models and allows for direct probabilistic statements about inclusion of covariates and spatial model structure. ...
... Rather than approaching Bayesian model selection with proper priors, approaches that use training samples to calibrate reference improper priors, including partial Bayes factors and fractional Bayes factors (FBF), have been proposed (O'Hagan, 1995;Berger and Pericchi, 1996). The partial Bayes factor separates out a subset of the data as a training sample, which is then used to update the priors on parameters. ...
... The prior distribution is specified as a fractional prior (O'Hagan, 1995). This constitutes updating an uninformative prior with a likelihood based on a fraction b n of the data, where the subscript n is used to make explicit its dependence on the sample size; the choice of b n will be discussed in detail in the following section. ...
... This constitutes updating an uninformative prior with a likelihood based on a fraction b n of the data, where the subscript n is used to make explicit its dependence on the sample size; the choice of b n will be discussed in detail in the following section. As well as avoiding an arbitrary choice of prior, BFs based on fractional priors enjoy several attractive properties, including consistency and invariance to linear transformations (O'Hagan, 1995). In our context, the fractional prior is normal with varianceσ 2 /b n , reflecting that the variance is inflated by a factor of 1/b n relative to the posterior variance, which uses the full data. ...
... An alternative choice for b n is to use the minimal training sample for the prior specification, leaving maximal information in the data for hypothesis testing (Berger & Pericchi, 1996;O'Hagan, 1995). In other words, we set b n such that the minimum number of observations is used to specify the prior under a given null hypothesis. ...
The use of fixed alpha levels in statistical testing is prevalent in management research, but can lead to Lindley's paradox in highly powered studies. In this article, we propose a sample size-adjusted alpha level approach that combines the benefits of both frequentist and Bayesian statistics, enabling strict hypothesis testing with known error rates while also quantifying the evidence for a hypothesis. We present an R-package that can be used to set the sample size-adjusted alpha level for generalized linear models, including linear regression, logistic regression, and Poisson regression. This approach can help researchers stop relying on mindless defaults and avoid situations where they reject the null hypothesis when the evidence in the test actually favors the null hypothesis, improving the accuracy and robustness of statistical analysis in management research.
... Fractional posteriors have been used in a wide variety of settings, including Bayesian model selection [42], marginal likelihood approximation [18], empirical Bayes methods [36] and more recently variational inference [3,30,7,2,37]. One motivation for their use in statistical inference is their greater robustness to possible model misspecification compared to the usual Bayesian posterior. ...
... The third term has distribution N 0, t 2 α n ψ n − ψ 0 2 2 /n , which is o P (1) since its variance tends to zero as n → ∞. Together, these three bounds establish (42). ...
... Using (42) and the last display with h t = tψ n / √ nα n and ρ t = M 0 √ nα n ε n ψ n H , the quantity in (6) equals ...
We establish a general Bernstein--von Mises theorem for approximately linear semiparametric functionals of fractional posterior distributions based on nonparametric priors. This is illustrated in a number of nonparametric settings and for different classes of prior distributions, including Gaussian process priors. We show that fractional posterior credible sets can provide reliable semiparametric uncertainty quantification, but have inflated size. To remedy this, we further propose a \textit{shifted-and-rescaled} fractional posterior set that is an efficient confidence set having optimal size under regularity conditions. As part of our proofs, we also refine existing contraction rate results for fractional posteriors by sharpening the dependence of the rate on the fractional exponent.
... A reversible jump (RJ) MCMC (Green, 1995) algorithm is a natural choice for such a trans-dimensional update, however, it is difficult to construct a practicable RJ scheme due to the high-dimensionality of the problem. To address this challenge, we will develop an efficient Metropolis-Hastings (MH) algorithm to update θ building upon the idea of fractional Bayes factor (Lee et al., 2016;O'Hagan, 1995) in section 4.2. ...
... To address the challenge, we construct our proposal distribution building upon the idea of fractional Bayes factor (O'Hagan, 1995;Lee et al., 2016). Assuming that at iteration t − 1 of the MCMC sampler ...
... The (*) part in equation (15) coincides with the fractional Bayes factor given in O'Hagan (1995). The detailed proof is given in Appendix 8. ...
We propose a Bayesian tensor-on-tensor regression approach to predict a multidimensional array (tensor) of arbitrary dimensions from another tensor of arbitrary dimensions, building upon the Tucker decomposition of the regression coefficient tensor. Traditional tensor regression methods making use of the Tucker decomposition either assume the dimension of the core tensor to be known or estimate it via cross-validation or some model selection criteria. However, no existing method can simultaneously estimate the model dimension (the dimension of the core tensor) and other model parameters. To fill this gap, we develop an efficient Markov Chain Monte Carlo (MCMC) algorithm to estimate both the model dimension and parameters for posterior inference. Besides the MCMC sampler, we also develop an ultra-fast optimization-based computing algorithm wherein the maximum a posteriori estimators for parameters are computed, and the model dimension is optimized via a simulated annealing algorithm. The proposed Bayesian framework provides a natural way for uncertainty quantification. Through extensive simulation studies, we evaluate the proposed Bayesian tensor-on-tensor regression model and show its superior performance compared to alternative methods. We also demonstrate its practical effectiveness by applying it to two real-world datasets, including facial imaging data and 3D motion data.
... Statistically, the line between prior and posterior predictive performance is thin and more quantitative than qualitative [189]. As an illustration, suppose we observe N data points in total -then we could choose to use none,ỹ = ∅, or any number between 1 and N for model training (i.e., fitting). ...
... As an illustration, suppose we observe N data points in total -then we could choose to use none,ỹ = ∅, or any number between 1 and N for model training (i.e., fitting). For complex P models, the predictive result implied by using one or two observations for training, rather than none at all, will be almost identical, despite everything but zero training data technically counting as "posterior" predictive performance [189]. Yet, the metrics commonly applied to quantify prior and posterior predictive performance differ not only in the amount of available training data, but also in some other non-trivial ways (to be explained below). ...
... has not yet attained wide popularity, despite having a number of useful properties [189,12,114,157]. The choice between prior or posterior predictive performance seems to depend on the modeling goals for which a P model is specified. ...
Probabilistic (Bayesian) modeling has experienced a surge of applications in almost all quantitative sciences and industrial areas. This development is driven by a combination of several factors, including better probabilistic estimation algorithms, flexible software, increased computing power, and a growing awareness of the benefits of probabilistic learning. However, a principled Bayesian model building workflow is far from complete and many challenges remain. To aid future research and applications of a principled Bayesian workflow, we ask and provide answers for what we perceive as two fundamental questions of Bayesian modeling, namely (a) ``What actually \emph{is} a Bayesian model?'' and (b) ``What makes a \emph{good} Bayesian model?''. As an answer to the first question, we propose the PAD model taxonomy that defines four basic kinds of Bayesian models, each representing some combination of the assumed joint distribution of all (known or unknown) variables (P), a posterior approximator (A), and training data (D). As an answer to the second question, we propose ten utility dimensions according to which we can evaluate Bayesian models holistically, namely, (1) causal consistency, (2) parameter recoverability, (3) predictive performance, (4) fairness, (5) structural faithfulness, (6) parsimony, (7) interpretability, (8) convergence, (9) estimation speed, and (10) robustness. Further, we propose two example utility decision trees that describe hierarchies and trade-offs between utilities depending on the inferential goals that drive model building and testing.
... One of them involves a real-world application for detecting exo-objects (orbiting other stars) based on a radial velocity model. Furthermore, in the second part of this work, we show some possible solutions presented in the literature, such as hierarchical approaches, likelihood-based priors, and the partial, intrinsic, fractional Bayes factors (Llorente et al., 2020;O'Hagan, 1995), remarking potential benefits and possible drawbacks of each of them. An alternative to the marginal likelihood approach for Bayesian model selection, called posterior predictive framework (Vehtari et al., 2017, Ch. 6) (Piironen & Vehtari, 2017), is also described. ...
... Let us also assume that for each possible training set y , where recall that S = Θ (y|θ)dθ. This approach is related to the partial and intrinsic Bayes factors (O'Hagan, 1995;Berger & Pericchi, 1996). ...
... However, a tempering value β ∈ (0, 1) must be selected. This idea is also employed in the so-called fractional Bayes factors (O'Hagan, 1995). ...
The application of Bayesian inference for the purpose of model selection is very popular nowadays. In this framework, models are compared through their marginal likelihoods, or their quotients, called Bayes factors. However, marginal likelihoods depend on the prior choice. For model selection, even diffuse priors can be actually very informative, unlike for the parameter estimation problem. Furthermore, when the prior is improper, the marginal likelihood of the corresponding model is undetermined. In this work, we discuss the issue of prior sensitivity of the marginal likelihood and its role in model selection. We also comment on the use of uninformative priors, which are very common choices in practice. Several practical suggestions are discussed and many possible solutions, proposed in the literature, to design objective priors for model selection are described. Some of them also allow the use of improper priors. The connection between the marginal likelihood approach and the well‐known information criteria is also presented. We describe the main issues and possible solutions by illustrative numerical examples, providing also some related code. One of them involving a real‐world application on exoplanet detection.
This article is categorized under: Statistical Models > Bayesian Models
Statistical Models > Fitting Models
Statistical Models > Model Selection
... Their approach to Gaussian DAG model comparison is based on using Bayes factors and uninformative, typically improper prior on the space of unconstrained covariance matrices. Ambiguity arising from the use of improper priors is dealt with by utilizing the fractional Bayes factors (O'Hagan 1995). We first review a result concerning the computation of marginal likelihood in a more general setting, presented by Geiger and Heckerman (2002). ...
... for the parameters of the complete DAG model. The improper prior is updated into a proper one by using fractional Bayes factors approach (O'Hagan, 1995). In this approach, a fraction of likelihood is "sacrificed" and used to update the improper prior into a proper fractional prior which is then paired with the remaining likelihood to compute the Bayes factors. ...
We propose a Bayesian approximate inference method for learning the dependence structure of a Gaussian graphical model. Using pseudo-likelihood, we derive an analytical expression to approximate the marginal likelihood for an arbitrary graph structure without invoking any assumptions about decomposability. The majority of the existing methods for learning Gaussian graphical models are either restricted to decomposable graphs or require specification of a tuning parameter that may have a substantial impact on learned structures. By combining a simple sparsity inducing prior for the graph structures with a default reference prior for the model parameters, we obtain a fast and easily applicable scoring function that works well for even high-dimensional data. We demonstrate the favourable performance of our approach by large-scale comparisons against the leading methods for learning non-decomposable Gaussian graphical models. A theoretical justification for our method is provided by showing that it yields a consistent estimator of the graph structure.
... This naturally suggests an objective Bayes approach, which is virtually free from prior elicitation. We carried out this program in Consonni & La Rocca (2012) for Gaussian DAG models, using the method of the fractional Bayes factor (O'Hagan, 1995). Our findings were consistent with, and extended, those presented in Carvalho & Scott (2009) for Gaussian decomposable graphical models, which relied on the use of the hyper-inverse Wishart distribution (Letac & Massam, 2007). ...
... Here, in section 4.1, we provide some background on objective Bayes model selection, focusing in particular on a proposal by O'Hagan (1995). Then, in section 4.2, we give the expression for the marginal data distribution of a generic subset of columns of Y under the prior implied by such proposal; this will be instrumental in the construction of the marginal likelihood of a DAG model given in section 5.1. ...
We present an objective Bayes method for covariance selection in Gaussian multivariate regression models whose error term has a covariance structure which is Markov with respect to a Directed Acyclic Graph (DAG). The scope is covariate-adjusted sparse graphical model selection, a topic of growing importance especially in the area of genetical genomics (eQTL analysis). Specifically, we provide a closed-form expression for the marginal likelihood of any DAG (with small parent sets) whose computation virtually requires no subjective elicitation by the user and involves only conjugate matrix normal Wishart distributions. This is made possible by a specific form of prior assignment, whereby only one prior under the complete DAG model need be specified, based on the notion of fractional Bayes factor. All priors under the other DAG models are derived using prior modularity, and global parameter independence, in the terminology of Geiger & Heckerman (2002). Since the marginal likelihood we obtain is constant within each class of Markov equivalent DAGs, our method naturally specializes to covariate-adjusted decomposable graphical models.
... Standard calculations using Bayes factors (Kass and Raftery, 1995) exhibit a known over-dependence on the model priors, and entail the computational complexity of integrating over the complete parameter space within a model. There have been several variations on the Bayes factor theme, such as fractional (O'Hagan, 1995) and intrinsic (Berger and Pericchi, 1996) Bayes factors, but these require the loss of some training data or the selection of an arbitrary weight in order to calibrate an "objective" prior. Alternative approaches apply sampling strategies to explore the space of models (George and McCulloch, 1993;Madigan et al., 1995), of which reversible-jump Markov chain Monte Carlo (RJ-MCMC) is perhaps the most well-known; see Green (1995). ...
... Efforts to specify objective prior distributions have led to an array of Bayes factor alternatives, such as intrinsic and fractional Bayes factors (Kass and Raftery, 1995). These methods use the observed data to help specify the prior in some way, such as by weighting the likelihood (O'Hagan, 1995) or setting aside a portion of data for "training" (Berger and Pericchi, 1996). However, this still requires an element of user choice, and can also result in the loss of some information contained within the observed data. ...
Existing approaches to model uncertainty typically either compare models using a quantitative model selection criterion or evaluate posterior model probabilities having set a prior. In this paper, we propose an alternative strategy which views missing observations as the source of model uncertainty, where the true model would be identified with the complete data. To quantify model uncertainty, it is then necessary to provide a probability distribution for the missing observations conditional on what has been observed. This can be set sequentially using one-step-ahead predictive densities, which recursively sample from the best model according to some consistent model selection criterion. Repeated predictive sampling of the missing data, to give a complete dataset and hence a best model each time, provides our measure of model uncertainty. This approach bypasses the need for subjective prior specification or integration over parameter spaces, addressing issues with standard methods such as the Bayes factor. Predictive resampling also suggests an alternative view of hypothesis testing as a decision problem based on a population statistic, where we directly index the probabilities of competing models. In addition to hypothesis testing, we provide illustrations from density estimation and variable selection, demonstrating our approach on a range of standard problems.
... where ω > 0. Gibbs posteriors are also commonly referred to as the tempered posterior (Andersen et al., 2002;Girolami, 2008), fractional posterior (O'Hagan, 1995;Gilks, 1995) or power posterior (Friel and Pettitt, 2008) and have been utilized in a wide range of statistical methodologies. For instance, they have been used to incorporate historical data in Bayesian data analyses (Ibrahim and Chen, 2000), offering a robust, but flexible, mechanism for incorporating prior knowledge. ...
We introduce a new multimodal optimization approach called Natural Variational Annealing (NVA) that combines the strengths of three foundational concepts to simultaneously search for multiple global and local modes of black-box nonconvex objectives. First, it implements a simultaneous search by using variational posteriors, such as, mixtures of Gaussians. Second, it applies annealing to gradually trade off exploration for exploitation. Finally, it learns the variational search distribution using natural-gradient learning where updates resemble well-known and easy-to-implement algorithms. The three concepts come together in NVA giving rise to new algorithms and also allowing us to incorporate "fitness shaping", a core concept from evolutionary algorithms. We assess the quality of search on simulations and compare them to methods using gradient descent and evolution strategies. We also provide an application to a real-world inverse problem in planetary science.
... Importantly, the alternative model for the χ 2 statistic implicit in this framework is a non-local prior density. In contrast, other default Bayesian variable selection procedures are based on the use of local alternative priors on regression parameters (Berger and Pericchi, 1996;O'Hagan, 1995). The potential value of using non-local priors in Bayesian variable selection is discussed in Johnson and Rossell (2010). ...
Uniformly most powerful Bayesian tests (UMPBT's) are an objective class of Bayesian hypothesis tests that can be considered the Bayesian counterpart of classical uniformly most powerful tests. Because the rejection regions of UMPBT's can be matched to the rejection regions of classical uniformly most powerful tests (UMPTs), UMPBT's provide a mechanism for calibrating Bayesian evidence thresholds, Bayes factors, classical significance levels and p-values. The purpose of this article is to expand the application of UMPBT's outside the class of exponential family models. Specifically, we introduce sufficient conditions for the existence of UMPBT's and propose a unified approach for their derivation. An important application of our methodology is the extension of UMPBT's to testing whether the non-centrality parameter of a chi-squared distribution is zero. The resulting tests have broad applicability, providing default alternative hypotheses to compute Bayes factors in, for example, Pearson's chi-squared test for goodness-of-fit, tests of independence in contingency tables, and likelihood ratio, score and Wald tests.
... However, previous simulation studies in the context of dose-finding studies showed that the BIC approximation frequently favors too simplistic models for realistic variances and sample sizes [see Bornkamp (2006)]. Other approximate methods, such as fractional Bayes factors, or intrinsic Bayes factors [see O'Hagan (1995) or Berger and Pericchi (1996)], either depend on arbitrary tuning parameter values or are computationally prohibitive. Thus, for each model we will use the exact posterior probabilities resulting from the prior distributions assumed for the MAP estimation. ...
Dose-finding studies are frequently conducted to evaluate the effect of different doses or concentration levels of a compound on a response of interest. Applications include the investigation of a new medicinal drug, a herbicide or fertilizer, a molecular entity, an environmental toxin, or an industrial chemical. In pharmaceutical drug development, dose-finding studies are of critical importance because of regulatory requirements that marketed doses are safe and provide clinically relevant efficacy. Motivated by a dose-finding study in moderate persistent asthma, we propose response-adaptive designs addressing two major challenges in dose-finding studies: uncertainty about the dose-response models and large variability in parameter estimates. To allocate new cohorts of patients in an ongoing study, we use optimal designs that are robust under model uncertainty. In addition, we use a Bayesian shrinkage approach to stabilize the parameter estimates over the successive interim analyses used in the adaptations. This approach allows us to calculate updated parameter estimates and model probabilities that can then be used to calculate the optimal design for subsequent cohorts. The resulting designs are hence robust with respect to model misspecification and additionally can efficiently adapt to the information accrued in an ongoing study. We focus on adaptive designs for estimating the minimum effective dose, although alternative optimality criteria or mixtures thereof could be used, enabling the design to address multiple objectives.
... However, no matter how much modeling we do, any model will be misspecified when faced with real data. Therefore, investigating the benefit of Bayesian modeling comparison procedures targeting the "ℳ-open" regime [94] (e.g., [95,96]), where traditional Bayesian model comparison breaks down, would be important. Lastly, the estimation performance of our proposed method is of independent interest, where evaluation against procedures specialized to DoA estimation such as those in [19,22,97,98], and Cramer-Rao lower bounds as in [44], could be useful. ...
We propose a fully Bayesian approach to wideband, or broadband, direction-of-arrival (DoA) estimation and signal detection. Unlike previous works in wideband DoA estimation and detection, where the signals were modeled in the time-frequency domain, we directly model the time-domain representation and treat the non-causal part of the source signal as latent variables. Furthermore, our Bayesian model allows for closed-form marginalization of the latent source signals by leveraging conjugacy. To further speed up computation, we exploit the sparse ``stripe matrix structure'' of the considered system, which stems from the circulant matrix representation of linear time-invariant (LTI) systems. This drastically reduces the time complexity of computing the likelihood from to , where N is the number of samples received by the array and k is the number of sources. These computational improvements allow for efficient posterior inference through reversible jump Markov chain Monte Carlo (RJMCMC). We use the non-reversible extension of RJMCMC (NRJMCMC), which often achieves lower autocorrelation and faster convergence than the conventional reversible variant. Detection, estimation, and reconstruction of the latent source signals can then all be performed in a fully Bayesian manner through the samples drawn using NRJMCMC. We evaluate the detection performance of the procedure by comparing against generalized likelihood ratio testing (GLRT) and information criteria.
... While improper priors are commonly used for analysis of a single model, one faces difficulties in comparing models via Bayes factors, since the marginal likelihoods of the competing models are only specified up to arbitrary constants. A number of remedies have been proposed in the literature to deal with this issue, such as fractional Bayes factors (O'Hagan, 1995) and intrinsic Bayes factors (Berger and Pericchi, 1996). ...
Discussion of "Bayesian Model Selection Based on Proper Scoring Rules" by Dawid and Musio [arXiv:1409.5291].
... A further approach to the problem of using diffuse priors was proposed by Robert (1993) who advocated for reweighing the prior odds to balance against parameter priors as prior hyperparameters become diffuse. O'Hagan (1995) considers the problem of using flat improper priors in the calculation of the Bayes factor by splitting the data into a training and testing set. The training set is used to construct an informative prior to be used to calculate the Bayes factor using the remaining portion of the data. ...
We introduce a new class of priors for Bayesian hypothesis testing, which we name "cake priors". These priors circumvent Bartlett's paradox (also called the Jeffreys-Lindley paradox); the problem associated with the use of diffuse priors leading to nonsensical statistical inferences. Cake priors allow the use of diffuse priors (having one's cake) while achieving theoretically justified inferences (eating it too). We demonstrate this methodology for Bayesian hypotheses tests for scenarios under which the one and two sample t-tests, and linear models are typically derived. The resulting Bayesian test statistic takes the form of a penalized likelihood ratio test statistic. By considering the sampling distribution under the null and alternative hypotheses we show for independent identically distributed regular parametric models that Bayesian hypothesis tests using cake priors are Chernoff-consistent, i.e., achieve zero type I and II errors asymptotically. Lindley's paradox is also discussed. We argue that a true Lindley's paradox will only occur with small probability for large sample sizes.
... In order to search the space T for the number of subclones and trees that best explain the observed data, we follow a similar approach as in Lee et al. (2015); Zhou et al. (2017) (motivated by fractional Bayes' factor in O'Hagan (1995)) that splits the data into a training set and a test set. Recall that n represents the read counts data. ...
We present TreeClone, a latent feature allocation model to reconstruct tumor subclones subject to phylogenetic evolution that mimics tumor evolution. Similar to most current methods, we consider data from next-generation sequencing of tumor DNA. Unlike most methods that use information in short reads mapped to single nucleotide variants (SNVs), we consider subclone phylogeny reconstruction using pairs of two proximal SNVs that can be mapped by the same short reads. As part of the Bayesian inference model, we construct a phylogenetic tree prior. The use of the tree structure in the prior greatly strengthens inference. Only subclones that can be explained by a phylogenetic tree are assigned non-negligible probabilities. The proposed Bayesian framework implies posterior distributions on the number of subclones, their genotypes, cellular proportions, and the phylogenetic tree spanned by the inferred subclones. The proposed method is validated against different sets of simulated and real-world data using single and multiple tumor samples. An open source software package is available at http://www.compgenome.org/treeclone.
... Many approaches have been proposed to tackle this issue, either by modifying the Bayes factor (e.g. O'Hagan, 1995;Berger and Pericchi, 1996;Berger et al., 1998;Berger and Pericchi, 2001) or bypassing it altogether (e.g. Kamary et al., 2014, and references therein). ...
The Bayes factor is a widely used criterion in model comparison and its logarithm is a difference of out-of-sample predictive scores under the logarithmic scoring rule. However, when some of the candidate models involve vague priors on their parameters, the log-Bayes factor features an arbitrary additive constant that hinders its interpretation. As an alternative, we consider model comparison using the Hyv\"arinen score. We propose a method to consistently estimate this score for parametric models, using sequential Monte Carlo methods. We show that this score can be estimated for models with tractable likelihoods as well as nonlinear non-Gaussian state-space models with intractable likelihoods. We prove the asymptotic consistency of this new model selection criterion under strong regularity assumptions in the case of non-nested models, and we provide qualitative insights for the nested case. We also use existing characterizations of proper scoring rules on discrete spaces to extend the Hyv\"arinen score to discrete observations. Our numerical illustrations include L\'evy-driven stochastic volatility models and diffusion models for population dynamics.
... One interesting alternative to the Bayes factors considered so far is given by the fractional Bayes factor (O'Hagan, 1995(O'Hagan, , 1997. Although "not a genuine Bayes factor" (O'Hagan, 1995, p. 117), the fractional Bayes factor has many of the consistency and coherence properties of Bayes factors (O'Hagan, 1997). ...
We develop alternative families of Bayes factors for use in hypothesis tests as alternatives to the popular default Bayes factors. The alternative Bayes factors are derived for the statistical analyses most commonly used in psychological research – one-sample and two-sample t tests, regression, and ANOVA analyses. They possess the same desirable theoretical and practical properties as the default Bayes factors and satisfy additional theoretical desiderata while mitigating against two features of the default priors that we consider implausible. They can be conveniently computed via an R package that we provide. Furthermore, hypothesis tests based on Bayes factors and those based on significance tests are juxtaposed. This discussion leads to the insight that default Bayes factors as well as the alternative Bayes factors are equivalent to test-statistic-based Bayes factors as proposed by Johnson. Journal of the Royal Statistical Society Series B: Statistical Methodology , 67 , 689–701. (2005). We highlight test-statistic-based Bayes factors as a general approach to Bayes-factor computation that is applicable to many hypothesis-testing problems for which an effect-size measure has been proposed and for which test power can be computed.
... The literature on noninformative priors and default Bayes factors for model selection for (generalized) linear models is extensive. Examples include g-priors (Zellner, 1986), mixtures of g-priors (Zellner & Siow, 1980;Liang et al., 2008), unit information priors (Kass & Wasserman, 1995), intrinsic Bayes factors (Berger & Pericchi, 1996), fractional Bayes factors (O'Hagan, 1995;De Santis & Spezzaferri, 2001), non-local priors (Johnson & Rossell, 2010 and power-expected-posterior priors (Fouskakis et al., 2015;Porwal & Rodríguez, 2023), among other approaches. See Forte et al. (2018) and Consonni et al. (2018) for recent reviews. ...
This paper introduces Dirichlet process mixtures of block g priors for model selection and prediction in linear models. These priors are extensions of traditional mixtures of g priors that allow for differential shrinkage for various (data-selected) blocks of parameters while fully accounting for the predictors' correlation structure, providing a bridge between the literatures on model selection and continuous shrinkage priors. We show that Dirichlet process mixtures of block g priors are consistent in various senses and, in particular, that they avoid the conditional Lindley ``paradox'' highlighted by Som et al.(2016). Further, we develop a Markov chain Monte Carlo algorithm for posterior inference that requires only minimal ad-hoc tuning. Finally, we investigate the empirical performance of the prior in various real and simulated datasets. In the presence of a small number of very large effects, Dirichlet process mixtures of block g priors lead to higher power for detecting smaller but significant effects without only a minimal increase in the number of false discoveries.
... The marginal likelihood can be interpreted as the probability that we would generate data x from model M j when we randomly sample from the model's parameter prior p (θ j | M j ). Moreover, the marginal likelihood is a central quantity for prior predictive hypothesis testing or model selection (Kass & Raftery, 1995;O'Hagan, 1995;Rouder & Morey, 2012). It is well-known that the marginal likelihood encodes a notion of Occam's razor arising from the basic principles of probability (Kass & Raftery, 1995, see also Figure 1). ...
Bayesian model comparison (BMC) offers a principled approach to assessing the relative merits of competing computational models and propagating uncertainty into model selection decisions. However, BMC is often intractable for the popular class of hierarchical models due to their high-dimensional nested parameter structure. To address this intractability, we propose a deep learning method for performing BMC on any set of hierarchical models which can be instantiated as probabilistic programs. Since our method enables amortized inference, it allows efficient re-estimation of posterior model probabilities and fast performance validation prior to any real-data application. In a series of extensive validation studies, we benchmark the performance of our method against the state-of-the-art bridge sampling method and demonstrate excellent amortized inference across all BMC settings. We then showcase our method by comparing four hierarchical evidence accumulation models that have previously been deemed intractable for BMC due to partly implicit likelihoods. Additionally, we demonstrate how transfer learning can be leveraged to enhance training efficiency. We provide reproducible code for all analyses and an open-source implementation of our method.
... Additionally, MPT models are typically based on a solid empirical foundation that offers valuable insights into the shape of parameter distributions, such as expected values and variances in memory parameters across different age groups, or (dis)ordinal interactions between experimental conditions. Hence, empirical data can also partially inform the choice of priors (e.g., Gu et al., 2018;O'Hagan, 1995). ...
Multinomial processing tree (MPT) models are a broad class of statistical models used to test sophisticated psychological theories. The research questions derived from these theories often go beyond simple condition effects on parameters and involve ordinal expectations (e.g., the same-direction effect on the memory parameter is stronger in one experimental condition than another) or disordinal expectations (e.g., the effect reverses in one experimental condition). Here, we argue that by refining common modeling practices, Bayesian hierarchical models are well suited to estimate and test these expectations. Concretely, we show that the default priors proposed in the literature lead to nonsensical predictions for individuals and the population distribution, leading to problems not only in model comparison but also in parameter estimation. Rather than relying on these priors, we argue that MPT modelers should determine priors that are consistent with their theoretical knowledge. In addition, we demonstrate how Bayesian model comparison may be used to test ordinal and disordinal interactions by means of Bayes factors. We apply the techniques discussed to empirical data from Bell et al. Journal of Experimental Psychology: Learning, Memory, and Cognition , 41 , 456–472 (2015).
... To mitigate this, we use the Fractional Bayes Factor (FBF) 35 with a training fraction b as ...
The Gompertz model, a mainstay in tumor growth kinetics analysis, requires an accurate likelihood function for its parameterestimation, applicable in both classical and Bayesian methodologies. This study compares five distinct error models, eachrepresenting a different likelihood function. Our comparative analysis employs the Bayesian Information Criterion (BIC), theDeviance Information Criterion (DIC), the Bayes Factor (BF), and hypothesis tests on residuals. Applying these criteria to fitthe Gompertz model to Ehrlich and fibrosarcoma Sa-37 tumor data, we find that error models with tumor volume-dependentdispersion consistently outperform others in quantitative evaluations. However, the conventional Normal error model withconstant variance remains a vital tool, offering significant clinical insights. This study underscores the complexity of likelihoodmodel selection in tumor growth kinetics and highlights the need for a multifaceted approach in such analysis.
... The method is also problematic if there are many possible models. To overcome the issues with the BF, other variants including partial BF, 15 the intrinsic BF, 16 and the fractional BF 17 have been proposed in the literature. These variants basically split the data into a training sample and a testing sample. ...
Selecting a “useful” model among a set of competing candidate models is a common problem in many disciplines and particularly in engineering applications. For a modeller, the best model fulfills a certain purpose best (e.g., the ability to accurately predict new data, good interpolation, precise estimates of lower/upper percentiles, etc.), which is typically assessed by comparing model simulations to the observed data. Approximate Bayesian computation (ABC) which is a statistical learning technique is used in this work for reliability model selection and parameter calibration using small/moderate fatigue life data sets. This is always the case in material fatigue due to the high cost and low efficiency of fatigue tests which are bottleneck problem when designing components/structures with regard to fatigue. The ABC is a likelihood‐free based method which means that it needs only a generative model to simulate the data. The proposed method provides a formal rank of the competing reliability models by eliminating gradually the least likely models in a parsimonious manner. It is flexible since it offers the possibility to use different metrics to measure the discrepancy between simulated and observed data. The choice of the appropriate distance function depends essentially on the purpose of model selection. Through various examples, it has been demonstrated that the ABC method has a number of very attractive properties, which makes it especially useful in probabilistic risk assessment and reliability analysis.
... His review discusses combining differing models to improve forecast accuracy, or 'model-averaging'. Model-averaging has an extensive literature (Fernández, Ley, & Steel, 2001;Garratt, Lee, Mise, & Shields, 2008;Garratt, Lee, Pesaran, & Shin, 2003;O'Hagan, 1995). In addition to forecast accuracy, other advantages over model selection include retaining information and more robust confidence intervals. ...
This paper discusses how researchers can augment their traditional publications and share their large research datasets in a format easily analysed by other researchers, while enhancing their own analytic capabilities. The case study is the Queensland Energy and Jobs Plan, and the data portal implemented using Power BI. The data portal presents the results from simulating the effect of implementing the plan on the Australian National Electricity Market, encompassing the eastern seaboard of Australia. The case study incorporates 198 simulations which are parameter sweeps of 9 scenarios of coal generation retirement and transmission augmentation, 2 wind levels, and 11 candidate years representing different weather conditions. The portal development process compares different data extraction, transformation, and load strategies and combines proven processes from Business Analysis and Data Management Bodies of Knowledge. The process result is a portal based on a relational database that provides a template for future projects, is simulation model agnostic, informs simulation model improvements, and provides a foundation for more advanced analytics. Future extensions could address dilemmas in the energy market, such as including generation and storage relying on high-frequency arbitrage within long-term development modelling.
... This allows researchers to assess whether variable has a larger/equal/smaller to predict the next sender than to predict the next receiver. The procedure for Test III builds on default Bayes factor methodology where the information in the data is split between a minimal subset, which is used for default prior specification (in combination with a noninformative prior) and a maximal subset which is used for hypothesis testing (O'Hagan 1995;Pérez and Berger 2002, among others). When using these methods, the resulting Bayes factors do not depend on the undefined normalizing constants of the improper priors (O'Hagan 1995), which practically means that arbitrarily vague priors can be used, as we do in this paper. ...
Relational event network data are becoming increasingly available. Consequently, statistical models for such data have also surfaced. These models mainly focus on the analysis of single networks; while in many applications, multiple independent event sequences are observed, which are likely to display similar social interaction dynamics. Furthermore, statistical methods for testing hypotheses about social interaction behavior are underdeveloped. Therefore, the contribution of the current paper is twofold. First, we present a multilevel extension of the dynamic actor-oriented model, which allows researchers to model sender and receiver processes separately. The multilevel formulation enables principled probabilistic borrowing of information across networks to accurately estimate drivers of social dynamics. Second, a flexible methodology is proposed to test hypotheses about common and heterogeneous social interaction drivers across relational event sequences. Social interaction data between children and teachers in classrooms are used to showcase the methodology.
... To avoid the need to manually specify the prior based on external prior knowledge, a fractional Bayesian approach is considered where the prior is automatically constructed using a minimal fraction from the data O'Hagan (1995), so that maximal information is used for quantifying the evidence in the data between the hypotheses. Furthermore, by centering the prior at the null value, positive and negative effects are equally likely a priori Mulder (2014). ...
... Within the class of Gaussian linear models, the literature on so-called "objective" or "default" priors for model selection is extensive. Examples include point-mass spike-&-slab priors (Mitchell and Beauchamp, 1988;Geweke, 1996), g-priors (Zellner, 1986), mixtures of g-priors (Zellner and Siow, 1980;Liang et al., 2008), unit information priors (Kass and Wasserman, 1995), intrinsic Bayes factors (Berger and Pericchi, 1996a), fractional Bayes factors (O'Hagan, 1995;De Santis and Spezzaferri, 2001), non-local priors Rossell, 2010, 2012), power-expected-posterior priors (Fouskakis et al., 2015) and prior-based Bayesian information criterion (Bayarri et al., 2019), among other approaches. See Consonni et al. (2018) for a comprehensive review of recent approaches to objective Bayesian analysis, and Bayarri et al. (2012) for a review and discussion of desirable properties. ...
... As a consequence, numerous Bayes factors based on "default" alternative prior densities have been proposed. Among others, these include (9)(10)(11)(12)(13)(14)(15)(16). Nonetheless, the value of a Bayes factor depends on the alternative prior density used in its definition, and it is generally difficult to justify or interpret any single default choice. ...
Bayes factors represent a useful alternative to P-values for reporting outcomes of hypothesis tests by providing direct measures of the relative support that data provide to competing hypotheses. Unfortunately, the competing hypotheses have to be specified, and the calculation of Bayes factors in high-dimensional settings can be difficult. To address these problems, we define Bayes factor functions (BFFs) directly from common test statistics. BFFs depend on a single noncentrality parameter that can be expressed as a function of standardized effects, and plots of BFFs versus effect size provide informative summaries of hypothesis tests that can be easily aggregated across studies. Such summaries eliminate the need for arbitrary P-value thresholds to define "statistical significance." Because BFFs are defined using nonlocal alternative prior densities, they provide more rapid accumulation of evidence in favor of true null hypotheses without sacrificing efficiency in supporting true alternative hypotheses. BFFs can be expressed in closed form and can be computed easily from z, t, χ2, and F statistics.
... , changes the odds of the two models. Note that this predictive Bayes factor can be expressed as a part of the "full" Bayes factor (see, e.g., O'Hagan, 1995): ...
... One alternative to choosing the prior based on desiderata, as done in this paper, is to use the data to inform the prior. O'Hagan (1995) proposed the fractional Bayes factor, which uses a fraction b = m0 /n of the entire likelihood to construct a prior, where m 0 is the size of the minimal training sample and n is the sample size. Böing-Messing and Mulder (2018) developed a fractional Bayes factor for testing the (in)equality of several population variances. ...
... The marginal likelihood can be interpreted as the probability that we would generate data x from model M j when we randomly sample from the model's parameter prior p (θ j | M j ). Moreover, the marginal likelihood is a central quantity for prior predictive hypothesis testing or model selection (Kass & Raftery, 1995;O'Hagan, 1995;Rouder & Morey, 2012). It is well-known that the marginal likelihood encodes a notion of Occam's razor arising from the basic principles of probability (Kass & Raftery, 1995, see also Figure 1). ...
Bayesian model comparison (BMC) offers a principled approach for assessing the relative merits of competing computational models and propagating uncertainty into model selection decisions. However, BMC is often intractable for the popular class of hierarchical models due to their high-dimensional nested parameter structure. To address this intractability, we propose a deep learning method for performing BMC on any set of hierarchical models which can be instantiated as probabilistic programs. Since our method enables amortized inference, it allows efficient re-estimation of posterior model probabilities and fast performance validation prior to any real-data application. In a series of extensive validation studies, we benchmark the performance of our method against the state-of-the-art bridge sampling method and demonstrate excellent amortized inference across all BMC settings. We then use our method to compare four hierarchical evidence accumulation models that have previously been deemed intractable for BMC due to partly implicit likelihoods. In this application, we corroborate evidence for the recently proposed L\'evy flight model of decision-making and show how transfer learning can be leveraged to enhance training efficiency. Reproducible code for all analyses is provided.
... To overcome this difficulty, we consider the Partial Bayes Factor (PBF) described in O'Hagan (1995) as an alternative approach for model comparison. The methodology consists of dividing the data z into two independent chunks, z train and z test and then computing the Bayes Factor based on part of the data, z test , conditioned on z train as follows, ...
Markov chains with variable length are useful parsimonious stochastic models able to generate most stationary sequence of discrete symbols. The idea is to identify the suffixes of the past, called contexts, that are relevant to predict the future symbol. Sometimes a single state is a context, and looking at the past and finding this specific state makes the further past irrelevant. States with such property are called renewal states and they can be used to split the chain into independent and identically distributed blocks. In order to identify renewal states for chains with variable length, we propose the use of Intrinsic Bayes Factor to evaluate the hypothesis that some particular state is a renewal state. In this case, the difficulty lies in integrating the marginal posterior distribution for the random context trees for general prior distribution on the space of context trees, with Dirichlet prior for the transition probabilities, and Monte Carlo methods are applied. To show the strength of our method, we analyzed artificial datasets generated from different models and one example coming from the field of Linguistics.
... For example: Jeffreys-Zellner-Siow objective priors do not require subjective specification (Bayarri & García-Donato, 2007;Jeffreys, 1961;Zellner & Siow, 1980); partial Bayes factor (de Santis & Spezzaferri, 1999) defines the prior according to part of the data whereas the remaining part is used to compute the Bayes factor; intrinsic Bayes factor (Berger & Pericchi, 1996) and fractional Bayes factor (O'Hagan, 1995) are variations on the partial Bayes factor where priors are defined according to the average of all possible minimal subsets of the data or to a given small fraction of the data, respectively. All methods discussed above suggest objective procedures to avoid arbitrary prior specification. ...
When conducting a study, researchers usually have expectations based on hypotheses or theoretical perspectives they want to evaluate. Equality and inequality constraints on the model parameters are used to formalize researchers' expectations or theoretical perspectives into the so-called informative hypotheses. However, traditional statistical approaches, such as the Null Hypothesis Significance Testing (NHST) or the model comparison using information criteria (e.g., AIC and BIC), are unsuitable for testing complex informative hypotheses. An alternative approach is to use the Bayes factor. In particular, the Bayes factor based on the encompassing prior approach allows researchers to easily evaluate complex informative hypotheses in a wide range of statistical models (e.g., generalized linear). This paper provides a detailed introduction to the Bayes factor with encompassing prior. First, all steps and elements involved in the formalization of informative hypotheses and the computation of the Bayes factor with encompassing prior are described. Next, we apply this method to a real case scenario, considering the attachment theory. Specifically, we analyzed the relative influence of maternal and paternal attachment on children's social-emotional development by comparing the various theoretical perspectives debated in the literature.
... Alternatively, [19] consider a fractional prior which is commonly used in model selection, as it adapts the prior scale automatically in a way that guarantees model consistency [39]. For TVP models, [19] defined a fractional prior for β δ,γ conditional on the latent process z as p(β δ,γ |b, ·) ∝ p(y|β δ,γ , σ 2 , z) b . ...
In this chapter, we review variance selection for time-varying parameter (TVP) models for univariate and multivariate time series within a Bayesian framework. We show how both continuous as well as discrete spike-and-slab shrinkage priors can be transferred from variable selection for regression models to variance selection for TVP models by using a non-centered parametrization. We discuss efficient MCMC estimation and provide an application to US inflation modeling.
Random effects are a flexible addition to statistical models to capture structural heterogeneity in the data, such as spatial dependencies, individual differences, temporal dependencies, or non-linear effects. Testing for the presence (or absence) of random effects is an important but challenging endeavor however, as testing a variance component, which must be non-negative, is a boundary problem. Various methods exist which have potential shortcomings or limitations. As a flexible alternative, we propose a flexible empirical Bayes factor (EBF) for testing for the presence of random effects. Rather than testing whether a variance component equals zero or not, the proposed EBF tests the equivalent assumption of whether all random effects are zero. The Bayes factor is `empirical' because the distribution of the random effects on the lower level, which serves as a prior, is estimated from the data as it is part of the model. Empirical Bayes factors can be computed using the output from classical (MLE) or Bayesian (MCMC) approaches. Analyses on synthetic data were carried out to assess the general behavior of the criterion. To illustrate the methodology, the EBF is used for testing random effects under various models including logistic crossed mixed effects models, spatial random effects models, dynamic structural equation models, random intercept cross-lagged panel models, and nonlinear regression models.
Bayes factors for composite hypotheses have difficulty in encoding vague prior knowledge, as improper priors cannot be used and objective priors may be subjectively unreasonable. To address these issues I revisit the posterior Bayes factor, in which the posterior distribution from the data at hand is re-used in the Bayes factor for the same data. I argue that this is biased when calibrated against proper Bayes factors, but propose adjustments to allow interpretation on the same scale. In the important case of a regular normal model, the bias in log scale is half the number of parameters. The resulting empirical Bayes factor is closely related to the widely applicable information criterion. I develop test-based empirical Bayes factors for several standard tests and propose an extension to multiple testing closely related to the optimal discovery procedure. When only a P-value is available, an approximate empirical Bayes factor is 10p. I propose interpreting the strength of Bayes factors on a logarithmic scale with base 3.73, reflecting the sharpest distinction between weaker and stronger belief. This provides an objective framework for interpreting statistical evidence, and realises a Bayesian/frequentist compromise.
Influential accounts claim that violent video games (VVG) decrease players' emotional empathy by desensitizing them to both virtual and real-life violence. However, scientific evidence for this claim is inconclusive and controversially debated. To assess the causal effect of VVGs on the behavioral and neural correlates of empathy and emotional reactivity to violence, we conducted a prospective experimental study using functional magnetic resonance imaging (fMRI). We recruited eighty-nine male participants without prior VVG experience. Over the course of two weeks, participants played either a highly violent video game, or a non-violent version of the same game. Before and after this period, participants completed an fMRI experiment with paradigms measuring their empathy for pain and emotional reactivity to violent images. Applying a Bayesian analysis approach throughout enabled us to find substantial evidence for the absence of an effect of VVGs on the behavioral and neural correlates of empathy. Moreover, participants in the VVG group were not desensitized to images of real-world violence. These results imply that short and controlled exposure to VVGs does not numb empathy nor the responses to real-world violence. We discuss the implications of our findings regarding the potential and limitations of experimental research on the causal effects of VVGs.
In the present study, we undertake the task of hypothesis testing in the context of Poisson-distributed data. The primary objective of our investigation is to ascertain whether two distinct sets of discrete data share the same Poisson rate. We delve into a comprehensive review and comparative analysis of various frequentist and Bayesian methodologies specifically designed to address this problem. Among these are the conditional test, the likelihood ratio test, and the Bayes factor. Additionally, we employ the posterior predictive p-value in our analysis, coupled with its corresponding calibration procedures. As the culmination of our investigation, we apply these diverse methodologies to test both simulated datasets and real-world data. The latter consists of the offspring distributions linked to COVID-19 cases in two disparate geographies - Hong Kong and Rwanda. This allows us to provide a practical demonstration of the methodologies' applications and their potential implications in the field of epidemiology.
The article by Tian et al. (Appl. Stoch. Models Bus. Ind. 2023) takes an interesting look at the use of non‐informative priors adapted to several censoring processes, which are common in reliability. It proposes a continuum of modelling approaches that go as far as defining weakly informative priors to overcome the well‐known shortcomings of frequentist approaches to problems involving highly censored samples. In this commentary, I make some critical remarks and propose to link this work to a more generic vision of what could be a relevant Bayesian elicitation in reliability, taking advantage of recent theoretical and applied advances. Through tools like approximate posterior priors and prior equivalent sample sizes, and by illustrating them with simple reliability models, I suggest methodological avenues to formalize the elicitation of informative priors in a auditable, defensible way. By allowing a clear modulation of subjective information, this might respond to the authors' primary concern of constructing weakly informative priors and to a more general concern for precaution in Bayesian reliability.
Many would probably be content to use Bayesian methodology for hypothesis testing, if it was easy, objective and with trustworthy assumptions. The Bayesian information criterion and some simple bounds on Bayes factor are closest to fit this bill, but with clear limitations. Here we develop an approximation of the so-called Bayes factor applicable in any bio-statistical settings where we have a d-dimensional parameter estimate of interest and the d x d dimensional (co-)variance of it. By design the approximation is monotone in the p value. It it thus a tool to transform p values into evidence (probabilities of the null and the alternative hypothesis, respectively). It is an improvement on the aforementioned techniques by being more flexible, intuitive and versatile but just as easy to calculate, requiring only statistics that will typically be available: e.g. a p value or test statistic and the dimension of the alternative hypothesis.
This article develops and explores a robust Bayes factor derived from a calibration technique that makes it particularly compatible with elicited prior knowledge. Building on previous explorations, the particular robust Bayes factor, dubbed a neutral-data comparison, is adapted for broad comparisons with existing robust Bayes factors, such as the fractional and intrinsic Bayes factors, in configurations defined by informative priors. The calibration technique is furthermore developed for use with flexible parametric priors—that is, mixture prior distributions with components that may be symmetric or skewed—, and demonstrated in an example context from forensic science. Throughout the exploration, the neutral-data comparison is shown to exhibit desirable sensitivity properties, and to show promise for adaptation to elaborate data-analysis scenarios.
Interpretability and uncertainty quantification in machine learning can provide justification for decisions, promote scientific discovery and lead to a better understanding of model behavior. Symbolic regression provides inherently interpretable machine learning, but relatively little work has focused on the use of symbolic regression on noisy data and the accompanying necessity to quantify uncertainty. A new Bayesian framework for genetic-programming-based symbolic regression (GPSR) is introduced that uses model evidence (i.e., marginal likelihood) to formulate replacement probability during the selection phase of evolution. Model parameter uncertainty is automatically quantified, enabling probabilistic predictions with each equation produced by the GPSR algorithm. Model evidence is also quantified in this process, and its use is shown to increase interpretability, improve robustness to noise, and reduce overfitting when compared to a conventional GPSR implementation on both numerical and physical experiments.
Drawing on entrepreneurship policy literature and resource orchestration theory, we ask whether asset management capabilities of countries explain differences in entrepreneurship outcomes. We define the term asset management capability as the orchestration of inputs and outputs to improve entrepreneurial activity. Using time-series data of 512 inputs and outputs in 219 countries (2000–2018), we model for non-linear dynamic latent variables, allow for inefficiencies and slack, deconfound the model to improve predictive inference, and use cross-validation. The results illustrate systematic variations in asset management capabilities across countries and have implications for entrepreneurship and policymakers.
Frequently, biomedical researchers need to choose between multiple candidate statistical models. Several techniques exist to facilitate statistical model selection including adjusted R2, hypothesis testing and p-values, and information criteria among others. One particularly useful approach that has been slow to permeate the biomedical literature is the notion of posterior model probabilities. A major advantage of posterior model probabilities is that they quantify uncertainty in model selection by providing a direct, probabilistic comparison among competing models as to which is the “true” model that generated the observed data. Additionally, posterior model probabilities can be used to compute posterior inclusion probabilities which quantify the probability that individual predictors in a model are associated with the outcome in the context of all models considered given the observed data. Posterior model probabilities are typically derived from Bayesian statistical approaches which require specialized training to implement, but in this paper we describe an easy-to-compute version of posterior model probabilities and inclusion probabilities that rely on the readily-available Bayesian information criterion. We illustrate the utility of posterior model probabilities and inclusion probabilities by re-analyzing data from a published gait study investigating factors that predict required coefficient of friction between the shoe sole and floor while walking.
Identifying sublethal pesticide effects on aquatic organisms is a challenge for environmental risk assessment. Long-term population experiments can help assessing chronic toxicity. However, population experiments are subject to stochasticity (demographic, environmental, and genetic). Therefore, identifying sublethal chronic effects from “noisy” data can be difficult. Model-based analysis can support this process.
We use stochastic, age-structured population models applied to data from long-term population experiments with Daphnia galeata in 1L aquaria with and without chronic pesticide treatments (diazinon and diuron) at sublethal concentrations. Posterior analysis following Bayesian inference of model parameters and states helped choosing an adequate description of life-history characteristics under the specific experimental conditions (a zero-inflated negative binomial distribution for reproduction and mortality without density dependence). For the insecticide treatments, the inferred marginal posterior parameter distributions indicated the need for a mortality rate that increases with time, indicating cumulative chronic toxic effects of diazinon on Daphnia populations. With this study, we demonstrate how stochastic models can be used to infer mechanisms from population data to help identifying sublethal pesticide effects.
ResearchGate has not been able to resolve any references for this publication.