Efficient Estimation and Optimization of Expensive to Evaluate Black-box Functions

In many complex engineering problems, efficient estimation and global optimization of black-box functions is a major concern. It requires an extensive number of evaluations which is often not possible if the function is expensive to evaluate. For efficient estimation, we propose an innovative active learning methodology based on the integration of the Laplacian regularization and Active Learning - Cohn (ALC) criterion for identification of the most informative points for learning expensive noisy black-box functions with a minimum number of points. We propose two simple greedy search algorithms for sequential optimization of the tuning parameters and determination of subsequent points based on the information from the previously evaluated points. The proposed methodology is particularly suited for problems involving the estimation of expensive black-box functions with a high level of noise and plenty of unevaluated points. Using a case study for analysis of the kinematics of pitching in baseball as well as simulation experiments, we demonstrate the performance of the proposed methodology against existing methods in the literature in terms of estimation error. For global optimization, we propose two novel acquisition functions to help find the global minimum of expensive black-box functions with different levels of noise. The first acquisition function is a multi-armed bandit regularized expected improvement (BREI) method to adaptively adjust the balance between exploration and exploitation for efficient global optimization of long-running computer experiments with low noise. We also develop a multi-armed bandit strategy based on Thompson sampling for adaptive optimization of the tuning parameter of the regularization term based on the preexisting and newly tested points. Using a case study on optimization of the collision avoidance algorithm in mobile robot motion planning as well as extensive simulation studies, we validate the proposed algorithm against some of the existing methods in the literature under different levels of noise. For second acquisition function, we apply the concept of multi-armed bandit regularization to the knowledge gradient acquisition function to optimize the functions with high noise. Using simulation studies, we validate the performance of BRKG against other existing methods in the literature under different levels of noise. Finally, we discuss a case study in which we use orthogonal array tuning method to optimize the hyperparameters of deep learning models to predict the total wind power of a 130 MW utility-scale wind farm. The case study in this thesis provides a baseline and can be extended to develop a toolbox to incorporate proposed global optimization acquisition functions to efficiently optimize the hyperparameters with few evaluations.