ArticlePublisher preview available

Variable-fidelity probability of improvement method for efficient global optimization of expensive black-box problems

Authors:
Xiongfeng Ruan
Xiongfeng Ruan
Xiongfeng Ruan
  • This person is not on ResearchGate, or hasn't claimed this research yet.
Ping Jiang at Huazhong University of Science and Technology
Ping Jiang
Qi Zhou at Huazhong University of Science and Technology
Qi Zhou
Hu Jiexiang at Huazhong University of Science and Technology
Hu Jiexiang
Show all 5 authorsHide
Read publisher preview
Download citation
To read the full-text of this research, you can request a copy directly from the authors.

Abstract and Figures

Variable-fidelity (VF) surrogate models have attracted significant attention recently in simulation-based design because they can achieve a desirable accuracy at a reasonable cost by making use of the data from both low-fidelity (LF) and high-fidelity (HF) simulations. To facilitate the usage of VF surrogate models assisted efficient global optimization, there are still challenging issues on (1) how to construct the VF surrogate model for simulations with variable-fidelity levels under the non-nested sampling data, (2) how to determine the location and fidelity level of the samples simultaneously, and (3) how to handle constraints when VF surrogate models are also used for constraints. In this work, a variable-fidelity probability of improvement (VF-PI) method is proposed for computationally expensive black-box problems. First, a multi-level generalized Co-Kriging (GCK) model, which is extended from the two-level GCK model, is developed for VF surrogate modeling of simulations with three or more levels of fidelities under non-nested sampling data. Second, to determine the location and fidelity level of the sequential samples, an extended probability of improvement (EPI) function is developed. In EPI function, the model correlation and cost ratio between the LF and HF models, together with the sample density, are considered. Third, the probability of satisfying the constraints is introduced and combined with the EPI function, enabling the proposed approach to handle VF optimization problems with constraints. The comparison results illustrate that the proposed VF-PI method is more efficient and robust than the four compared methods on the illustrated cases.
Figures - available from: Structural and Multidisciplinary Optimization
This content is subject to copyright. Terms and conditions apply.
A preview of this full-text is provided by Springer Nature.
Learn more
Content available from Structural and Multidisciplinary Optimization
This content is subject to copyright. Terms and conditions apply.
RESEARCH PAPER
Variable-fidelity probability of improvement method for efficient
global optimization of expensive black-box problems
Xiongfeng Ruan
1,2
&Ping Jiang
2
&Qi Zhou
1
&Jiexiang Hu
2
&Leshi Shu
2
Received: 19 December 2019 /Revised: 16 May 2020 /Accepted: 4 June 2020
#Springer-Verlag GmbH Germany, part of Springer Nature 2020
Abstract
Variable-fidelity (VF) surrogate models have attracted significant attention recently in simulation-based design because they can
achieve a desirable accuracy at a reasonable cost by making use of the data from both low-fidelity (LF) and high-fidelity (HF)
simulations. To facilitate the usage ofVF surrogate models assisted efficient globaloptimization, there are still challenging issues
on (1) how to construct the VF surrogate model for simulations with variable-fidelity levels under the non-nested sampling data,
(2) how to determine the location and fidelity level of the samples simultaneously, and (3) how to handle constraints when VF
surrogate models are also used for constraints. In this work, a variable-fidelity probability of improvement (VF-PI) method is
proposed for computationally expensive black-box problems. First, a multi-level generalized Co-Kriging (GCK) model, which is
extended from the two-level GCK model, is developed for VF surrogate modeling of simulations with three or more levels of
fidelities under non-nested sampling data. Second, to determine the location and fidelity level of the sequential samples, an
extended probability of improvement (EPI) function is developed. In EPI function, the model correlation and cost ratio between
the LF and HF models, together with the sample density, are considered. Third, the probability of satisfying the constraints is
introduced and combined with the EPI function, enabling the proposed approach to handle VF optimization problems with
constraints. The comparison results illustrate that the proposed VF-PI method is more efficient and robust than the four compared
methods on the illustrated cases.
Keywords Variable-fidelity surrogate model .Sequential optimization .Probability of improvement .Co-Kriging
1 Introduction
Surrogate models have been widely used in engineering opti-
mization problems to replace the computationally expensive
simulations (Booker et al. 1999; Jiang et al. 2019a;Zhouetal.
2017). They are constructed based on available input param-
eter values and the corresponding quantity of interests (QoIs).
Generally, the computational burden to construct an accurate
surrogate in the whole design domain is unaffordable in the
surrogate model–baseddesignandoptimization(SBDO)
(Jiang et al. 2020; Zhang et al. 2019;Zhouetal.2019).
Therefore, efficient global optimization (EGO) approaches,
which sequentially allocate samples to balance the exploration
and exploitation, were put forward and have reached a great
success (Forrester et al. 2008;Queipoetal.2005;Wangand
Shan 2007). The development of different criteria for allocat-
ing the sequential samples results in different EGO ap-
proaches. Commonly used EGO approaches are expected im-
provement (EI) (Jones et al. 1998), lower confidence
bounding (LCB) (Cox and John 1992), and probability of
improvement (PI) (Jones 2001). The variations and extensions
of these three EGO approaches have also attracted significant
attention nowadays (Cheng et al. 2019;Dongetal.2018;Liu
et al. 2018b; Qian et al. 2019; Shi et al. 2019; Wang and
Ierapetritou 2018).
Although previous works demonstrated the apparent merits
of EGO approaches, for the computationally expensive high-
fidelity (HF) models, even performing the number of simula-
tions required for constructing a surrogate model could be too
Responsible Editor: Michael Kokkolaras
*Qi Zhou
qizhou@hust.edu.cn
1
School of Aerospace Engineering, Huazhong University of Science
& Technology, Wuhan 430074, Hubei, People’s Republic of China
2
The State Key Laboratory of Digital Manufacturing Equipment and
Technology, School of Mechanical Science and Engineering,
Huazhong University of Science & Technology, Wuhan 430074,
People’s Republic of China
https://doi.org/10.1007/s00158-020-02646-9
Structural and Multidisciplinary Optimization (2020) 62:3021–3052
/Published online: 20 August 2020
Content courtesy of Springer Nature, terms of use apply. Rights reserved.
... This is obtained through a learner represented by the multifidelity acquisition function U(x, l) that extends the infill criteria of Bayesian optimization, and selects the pair of sample and the associated level of fidelity to query (x new , l new ) ∈ max x∈χ,l∈L U(x, l) that is likely to provide higher gains with a regard for the computational expenditure. Among different formulations, well known multifidelity acquisition functions to address optimization problems are the Multifidelity Probability of Improvement (MFPI) [113], Multifidelity Expected Improvement (MFEI) [114], Multifidelity Predictive Entropy Search (MFPES) [115], Multifidelity Max-Value Entropy Search (MFMES) [116], and non-myopic multifidelity expected improvement [21]. These formulations of the acquisition function define adaptive learning schemes that retain the infill principles characterizing the single-fidelity counterpart, and account for the dual decision task balancing the gains achieved through accurate queries with the associated cost during the optimization procedure. ...
... The Multifidelity Probability of Improvement (MFPI) acquisition function provides an extended formulation of the probability of improvement suitable for the multifidelity scenario as follows [113]: ...
Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal
Preprint
  • Mar 2023
Science and Engineering applications are typically associated with expensive optimization problems to identify optimal design solutions and states of the system of interest. Bayesian optimization and active learning compute surrogate models through efficient adaptive sampling schemes to assist and accelerate this search task toward a given optimization goal. Both those methodologies are driven by specific infill/learning criteria which quantify the utility with respect to the set goal of evaluating the objective function for unknown combinations of optimization variables. While the two fields have seen an exponential growth in popularity in the past decades, their dualism and synergy have received relatively little attention to date. This paper discusses and formalizes the synergy between Bayesian optimization and active learning as symbiotic adaptive sampling methodologies driven by common principles. In particular, we demonstrate this unified perspective through the formalization of the analogy between the Bayesian infill criteria and active learning criteria as driving principles of both the goal-driven procedures. To support our original perspective, we propose a general classification of adaptive sampling techniques to highlight similarities and differences between the vast families of adaptive sampling, active learning, and Bayesian optimization. Accordingly, the synergy is demonstrated mapping the Bayesian infill criteria with the active learning criteria, and is formalized for searches informed by both a single information source and multiple levels of fidelity. In addition, we provide guidelines to apply those learning criteria investigating the performance of different Bayesian schemes for a variety of benchmark problems to highlight benefits and limitations over mathematical properties that characterize real-world applications.
View
Multifidelity Bayesian Optimization: A Review
Article
Full-text available
  • Feb 2025
  • AIAA J
Resided at the intersection of multifidelity optimization (MFO) and Bayesian optimization (BO), MF BO has found a niche in solving expensive engineering design optimization problems, thanks to its advantages in incorporating physical and mathematical understandings of the problems, saving resources, addressing exploitation–exploration trade-off, considering uncertainty, and processing parallel computing. The increasing number of works dedicated to MF BO suggests the need for a comprehensive review of this advanced optimization technique. This paper surveys recent developments of two essential ingredients of MF BO: Gaussian process (GP) based MF surrogates and acquisition functions. First the existing MF modeling methods and MFO strategies are categorized to locate MF BO in a large family of surrogate-based optimization and MFO algorithms. Then, the common properties shared between the methods from each ingredient of MF BO are exploited to describe important GP-based MF surrogate models and to review various acquisition functions. This presentation aims to provide a structured understanding of MF BO. Finally, important aspects are examined that require further research for applications of MF BO in solving intricate yet important design optimization problems, including constrained optimization, high-dimensional optimization, optimization under uncertainty, and multiobjective optimization.
View
Inaccurate search based sequential approximate constrained optimization method for Solid Rocket Motor design
Article
  • Oct 2024
Surrogate based constrained optimization methods are widely utilized in practical solid rocket motor (SRM) design problems. To improve the design performance, this paper proposes an inaccurate search based sequential approximate constrained optimization (ISSACO) method consisting of an efficient recursive evolution Latin hypercube design (RELHD) method and a three-stage constrained sampling method. The RELHD method divides large-sample experiment design problem into several small-sample design problems to ensure the computational efficiency and the design uniformity at the same time. The three-stage constrained sampling method employs the elite archives mechanism to balance the feasibility and the optimality in optimization. Results of the numerical cases indicate that the proposed ISSACO algorithm is competitive with other state-of-art constrained optimization method in efficiency and effectiveness. The finocyl grain design and the impulse mass ratio optimization are also conducted to make further verifications, and the ISSACO method can efficiently obtain feasible optimum and proves to be a potential method for computationally intensive solid rocket motor design problems.
View
Efficient Impedance Controller Tuning of Robot–Environment Interaction With Improved Transient and Steady-State Performance
Article
  • Jan 2024
This paper proposes an efficient tuning strategy for impedance controllers based on multi-objective Bayesian optimization to achieve accurate force tracking in robot-environment interaction tasks. Contrasting with the single-objective optimization-based tuning methods, we fully exploit the force feedback information to construct objective functions that characterize both the transient and steady-state behaviors, thereby enabling a joint optimization of the response performance of the interaction force. The proposed method employs Gaussian processes as surrogate models for the objective functions. Following the approximation of the Pareto frontier using the NSGA-II algorithm, impedance parameters are adjusted based on the Hypervolume indicator. We compare our proposed method with three mainstream impedance controller tuning strategies and three multi-objective optimization-based tuning methods, demonstrating its superiority in tuning efficiency and interaction performance. Furthermore, the proposed method is experimentally verified using a self-developed robotic manipulator across various interaction environments. The method realizes efficient and high-performance controller tuning via only a handful of interaction attempts. Finally, the robustness and convergence of the proposed method are discussed in detail to promote its applicability in real-world interaction control.
View
View
View
Diagnosing Incipient Faults for a Faster Adoption of Sustainable Aerospace Technologies
Article
  • Jul 2024
Next-generation aircraft require the development and integration of a deal of innovative green technologies to meet the ambitious sustainability goals set for aviation. Those transformational efforts are associated with a tremendous increase in the complexity of the onboard systems and their multiphysics-coupled behaviors and dynamics. A critical aspect relates to the identification of the coupled faults resulting from the integration of those green technologies, which introduce damage identifiability issues and demand new approaches for the efficient and accurate identification of non-nominal fault conditions. Model-based fault detection and identification (FDI) methodologies have been proven essential to identify onboard the damages affecting the systems from physical signal acquisitions, but existing methods are typically computationally expensive and fail to capture incipient coupled faults, which prevents their adoption and scaling with the increasing complexity of novel multiphysics systems. This work introduces a multifidelity framework to accelerate the identification of fault modes affecting complex systems. An original two-stage compression computes an optimally informative and highly reduced representation of the monitoring signals for the minimum demand of onboard resources. A multifidelity scheme for Bayesian inversion is developed to infer multidomain fault parameters from the compressed signals: variable cost and fidelity models are optimally queried for a major reduction of the overall computational expense. The framework is demonstrated and validated for aerospace electromechanical actuators affected by incipient multimodal faults. Remarkable accelerations of the FDI procedure are observed, and the exact identification of the incipient fault condition was achieved one order of magnitude faster than with standard algorithms.
View
View
View
View
A two-stage adaptive multi-fidelity surrogate model-assisted multi-objective genetic algorithm for computationally expensive problems
Article
Full-text available
  • Jan 2021
  • ENG COMPUT-GERMANY
Surrogate model-assisted multi-objective genetic algorithms (MOGA) show great potential in solving engineering design problems since they can save computational cost by reducing the calls of expensive simulations. In this paper, a two-stage adaptive multi-fidelity surrogate (MFS) model-assisted MOGA (AMFS-MOGA) is developed to further relieve their computational burden. In the warm-up stage, a preliminary Pareto frontier is obtained relying only on the data from the low-fidelity (LF) model. In the second stage, an initial MFS model is constructed based on the data from both LF and high-fidelity (HF) models at the samples, which are selected from the preliminary Pareto set according to the crowding distance in the objective space. Then the fitness values of individuals are evaluated using the MFS model, which is adaptively updated according to two developed strategies, an individual-based updating strategy and a generation-based updating strategy. The former considers the prediction uncertainty from the MFS model, while the latter takes the discrete degree of the population into consideration. The effectiveness and merits of the proposed AMFS-MOGA approach are illustrated using three benchmark tests and the design optimization of a stiffened cylindrical shell. The comparisons between the proposed AMFS-MOGA approach and some existing approaches considering the quality of the obtained Pareto frontiers and computational efficiency are made. The results show that the proposed AMFS-MOGA method can obtain Pareto frontiers comparable to that obtained by the MOGA with HF model, while significantly reducing the number of evaluations of the expensive HF model.
View
Real-time estimation error-guided active learning Kriging method for time-dependent reliability analysis
Article
Full-text available
  • Jan 2020
  • APPL MATH MODEL
Time-dependent reliability analysis using surrogate model has drawn much attention for avoiding the high computational burden. But the surrogate training strategies of existing methods do not directly consider the estimation error of failure probability, leading to the limitations that some computationally expensive samples are wasted or some algorithms tend to terminate prematurely. To address the challenges, this work proposes a real-time estimation error-guided sampling method. As the classification of random points may be not completely accurate, the wrong-classification probability is calculated by using the mean and variance of Kriging prediction. With this probability, the total numbers of wrongly classified points are obtained. Furthermore, the confidence intervals of the numbers are computed based on the probability theory, and the estimation error of failure probability is calculated through the confidence intervals. Subsequently, the point maximizing the probability is identified as the new sample for decreasing the estimation error, and the maximum error is used to guide when to stop the refinement of Kriging model. Results of three cases demonstrate the performance of the proposed method.
View
Efficient Aerodynamic Shape Optimization Using Variable-Fidelity Surrogate Models and Multilevel Computational Grids
Article
Full-text available
  • May 2019
  • CHINESE J AERONAUT
A variable-fidelity method can remarkably improve the efficiency of a design optimization based on a high-fidelity and expensive numerical simulation, with assistance of lower-fidelity and cheaper simulation(s). However, most existing works only incorporate “two” levels of fidelity, and thus efficiency improvement is very limited. In order to reduce the number of high-fidelity simulations as many as possible, there is a strong need to extend it to three or more fidelities. This article proposes a novel variable-fidelity optimization approach with application to aerodynamic design. Its key ingredient is the theory and algorithm of a Multi-level Hierarchical Kriging (MHK), which is referred to as a surrogate model that can incorporate simulation data with arbitrary levels of fidelity. The high-fidelity model is defined as a CFD simulation using a fine grid and the lower-fidelity models are defined as the same CFD model but with coarser grids, which are determined through a grid convergence study. First, sampling shapes are selected for each level of fidelity via technique of Design of Experiments (DoE). Then, CFD simulations are conducted and the output data of varying fidelity is used to build initial MHK models for objective (e.g. CD) and constraint (e.g. CL, CM) functions. Next, new samples are selected through infill sampling criteria and the surrogate models are repetitively updated until a global optimum is found. The proposed method is validated by analytical test cases and applied to aerodynamic shape optimization of a NACA0012 airfoil and an ONERA M6 wing in transonic flows. The results confirm that the proposed method can significantly improve the optimization efficiency and apparently outperforms the existing single-fidelity or two-level-fidelity method.
View
A sequential constraints updating approach for Kriging surrogate model-assisted engineering optimization design problem
Article
Full-text available
  • Jul 2020
  • ENG COMPUT-GERMANY
Kriging surrogate model has been widely used in engineering design optimization problems to replace computational cost simulations. To facilitate the usage of the Kriging surrogate model-assisted engineering optimization design, there are still challenging issues on the updating of Kriging surrogate model for the constraints, since there exists prediction error between the Kriging surrogate model and the real constraints. Ignoring the interpolation uncertainties from the Kriging surrogate model of constraints may lead to infeasible optimal solutions. In this paper, general sequential constraints updating approach based on the confidence intervals from the Kriging surrogate model (SCU-CI) are proposed. In the proposed SCU-CI approach, an objective switching and sequential updating strategy is introduced based on whether the feasibility status of the design alternatives would be changed because of the interpolation uncertainty from the Kriging surrogate model or not. To demonstrate the effectiveness of the proposed SCU-CI approach, nine numerical examples and two practical engineering cases are used. The comparisons between the proposed approach and five existing approaches considering the quality of the obtained optimum and computational efficiency are made. Results illustrate that the proposed SCU-CI approach can generally ensure the feasibility of the optimal solution under a reasonable computational cost.
View
A Python surrogate modeling framework with derivatives
Article
Full-text available
  • Mar 2019
  • ADV ENG SOFTW
The surrogate modeling toolbox (SMT) is an open-source Python package that contains a collection of surrogate modeling methods, sampling techniques, and benchmarking functions. This package provides a library of surrogate models that is simple to use and facilitates the implementation of additional methods. SMT is different from existing surrogate modeling libraries because of its emphasis on derivatives, including training derivatives used for gradient-enhanced modeling, prediction derivatives, and derivatives with respect to training data. It also includes unique surrogate models: kriging by partial least-squares reduction, which scales well with the number of inputs; and energy-minimizing spline interpolation, which scales well with the number of training points. The efficiency and effectiveness of SMT are demonstrated through a series of examples. SMT is documented using custom tools for embedding automatically tested code and dynamically generated plots to produce high-quality user guides with minimal effort from contributors. SMT is maintained in a public version control repository.
View
View
A lower confidence bounding approach based on the coefficient of variation for expensive global design optimization
Article
  • Apr 2019
  • ENG COMPUTATION
Purpose Engineering design optimization involving computational simulations is usually a time-consuming, even computationally prohibitive process. To relieve the computational burden, the adaptive metamodel-based design optimization (AMBDO) approaches have been widely used. This paper aims to develop an AMBDO approach, a lower confidence bounding approach based on the coefficient of variation (CV-LCB) approach, to balance the exploration and exploitation objectively for obtaining a global optimum under limited computational budget. Design/methodology/approach In the proposed CV-LCB approach, the coefficient of variation (CV) of predicted values is introduced to indicate the degree of dispersion of objective function values, while the CV of predicting errors is introduced to represent the accuracy of the established metamodel. Then, a weighted formula, which takes the degree of dispersion and the prediction accuracy into consideration, is defined based on the already-acquired CV information to adaptively update the metamodel during the optimization process. Findings Ten numerical examples with different degrees of complexity and an AIAA aerodynamic design optimization problem are used to demonstrate the effectiveness of the proposed CV-LCB approach. The comparisons between the proposed approach and four existing approaches regarding the computational efficiency and robustness are made. Results illustrate the merits of the proposed CV-LCB approach in computational efficiency and robustness. Practical implications The proposed approach exhibits high efficiency and robustness in engineering design optimization involving computational simulations. Originality/value CV-LCB approach can balance the exploration and exploitation objectively.
View
View
Variable-Fidelity Lower Confidence Bounding Approach for Engineering Optimization Problems with Expensive Simulations
Article
  • Aug 2019
Engineering design optimization with expensive simulations is usually a computationally prohibitive process. As one of the most famous efficient global optimization approaches, the lower confidence bounding (LCB) approach has been widely applied to relieve this computational burden. However, the LCB approach can be used only for design optimization problems with the single-fidelity level. In this paper, a variable-fidelity lower confidence bounding (VF-LCB) approach is developed to extend the LCB approach to engineering problems with multifidelity levels. First, a VF-LCB function is analytically derived to adaptively select LF or HF samples according to the predicted values and uncertainty from the VF metamodel. Then the coefficient of variations (CoV) of the predicted values and uncertainty of the VF model, which can reflect the degree of dispersion of the prediction values and uncertainty, respectively, are introduced to balance the global exploration and local exploitation objectively. To illustrate the effectiveness and merits of the proposed VF-LCB approach, eight numerical examples, and the design of micro-aerial vehicle (MAV) fuselage are tested. Comparative results between the proposed approach and the other five existing methods show that the average cost savings are about 25% for eight numerical examples and about 45% for the MAV problem compared with the other five existing methods.
View
Filter-Based Sequential Radial Basis Function Method for Spacecraft Multidisciplinary Design Optimization
Article
  • Dec 2018
Spacecraft system design is practically a complex multidisciplinary design optimization (MDO) problem. Because of the application of high-fidelity simulation models, the massive computational cost of spacecraft MDO problems becomes a bottleneck and challenging problem in engineering practices. To address the issue, this paper proposes a novel filter-based sequential radial basis function (FSRBF) method for effectively and efficiently solving spacecraft MDO problems. In FSRBF, to handle expensive constraints, a filter is constructed, augmented, and refined based on the concept of Pareto nondomination, which is then combined with a support vector machine (SVM) to construct the filter-based region of interest (FROI) for sequentially bias sampling. During the optimization process, the expensive multidisciplinary analysis process is approximated by RBF metamodels to reduce the computational cost. The RBF metamodels are gradually updated via consecutively sampling within the FROI, which leads the search to rapidly reach the feasible optimum. A number of numerical benchmark problems are used to demonstrate the desirable performance of the proposed FSRBF compared with several alternative methods. In the end, FSRBF is applied to the design of an all-electric GEO satellite and a small Earth observation satellite to illustrate its capability for real-world spacecraft MDO problems. The results show that FSRBF can successfully obtain feasible solutions to improve the design quality of satellite systems. Moreover, the required computational cost of FSRBF is much lower than that of competitive methods, which illustrates the effectiveness and practicality of the proposed FSRBF for solving spacecraft MDO problems.
View