What family of Radial Basis Functions to use in Direct Policy Search? A comparative analysis

Direct policy search (DPS) is increasingly being used to design adaptive policies for multi-objective state-based control policies. DPS is a promising approach that can easily find policies for many heterogenous objective functions, particularly when coupling global approximators (i.e., radial basis functions) with evolutionary algorithms. Nonetheless, specifying the topology and the family of radial basis functions is usually done by trial and error for practical applications and is often not reported in the literature. How does the selected family of radial basis functions affect the quality of the resulting control policies? Does the chosen family influence the search behavior of the evolutionary algorithms? Can we formulate recommendations for which families are more or less suitable in general, or given the characteristics of the control problem? We test a suite of radial basis functions to address these questions for finding Pareto optimal reservoir control policies using an established reference case. This reference case is the Conowingo reservoir, a transboundary water body in the Susquehanna River Basin in North-East US. The reservoir needs to meet multiple competing water demands for hydropower production, environmental flows, recreation, cooling water for the Peach Bottom atomic power plant, and urban water supply for Baltimore, MD, and Chester, PA. To optimize the Pareto optimal reservoir control policies, we use the epsilon-NSGA2 algorithm. Our study provides guidance on the effect of using different families of radial basis functions, particularly their impact on the recommended reservoir operations, the resulting tradeoffs across the different sectors, and the search behavior of the evolutionary algorithm.