Abstract
In this talk, I will introduce a method for model selection and averaging in Bayesian generalized nonlinear regression [1]. Our approach generates nonlinear features hierarchically—similar to deep learning but with even greater flexibility—encompassing most existing statistical learning models as special cases while allowing them to be combined within a single Bayesian model averaging (BMA) framework. Model priors are designed to penalize the complexity of nonlinear features and encourage parsimony. As a result, we often obtain highly interpretable nonlinear models that recover data-generating processes for physical laws [1], logical [2] and fractional [3] polynomials, or governing equations in Sparse Identification of Nonlinear Dynamical Systems (SINDy). When no true data-generating model is feasible, BMA still accounts for model uncertainty, ensuring robust predictive performance.
For Bayesian inference and posterior model probability estimation, we employ a genetically modified mode-jumping Markov chain Monte Carlo (GMJMCMC) algorithm [1,2]. To scale to larger datasets, we also provide subsampling-based MCMC [4] and an Evolutionary Variational Bayes method [5].
I will demonstrate this approach through various applications, accompanied by hands-on examples using the FBMS R package (CRAN link). Examples will include recovering physical laws and SINDy solutions, logic and fractional polynomial regressions, extensions to latent Gaussian nonlinear models, and predictive modeling tasks.
References:
[1] Hubin, Aliaksandr; Storvik, Geir & Frommlet, Florian (2021). Flexible Bayesian Nonlinear Model Configuration. The Journal of Artificial Intelligence Research. ISSN 1076-9757. 72, p. 901–942.
[2] Hubin, Aliaksandr; Storvik, Geir Olve & Frommlet, Florian (2018). A Novel Algorithmic Approach to Bayesian Logic Regression (with Discussion). Bayesian Analysis. ISSN 1936-0975. 15(1), p. 263–311. doi: 10.1214/18-BA1141
[3] Hubin, Aliaksandr; Heinze, Georg & De Bin, Riccardo (2023). Fractional Polynomial Models as Special Cases of Bayesian Generalized Nonlinear Models. Fractal and Fractional. 7(9), p. 1–23. doi: 10.3390/fractalfract7090641
[4] Lachmann, Jon; Storvik, Geir Olve; Frommlet, Florian & Hubin, Aliaksandr (2022). A subsampling approach for Bayesian model selection. International Journal of Approximate Reasoning. ISSN 0888-613X. 151, p. 33–63. doi: 10.1016/j.ijar.2022.08.018
[5] Hauglie Sommerfelt, Philip Sebastian & Hubin, Aliaksandr (2024). Evolutionary variational inference for Bayesian generalized nonlinear models. Neural Computing & Applications. ISSN 0941-0643. p. 1–18. doi: 10.1007/s00521-024-10349-1