How do we push the boundaries of Bayesian Optimization to handle complex, irregular, or non-numerical search spaces? In Embed-then-Regress, we use LLM embeddings as a universal representation for string inputs, unlocking BO's potential across new frontiers where traditional methods fall short. Embed-then-Regress achieves state-of-the-art results across synthetic, combinatorial, and hyperparameter optimization tasks.

Paper: arxiv.org/abs/2410.10190
Code: github.com/google-researc…

🧵A thread:

#BayesianOptimization #LLMs #GoogleDeepMind (1/7) The key idea of our method is to represent all candidates in the search space with a unified JSON string format, embed these strings using a pre-trained LLM, and train a regression model on top of this embedding space. This approach enables generalization to variable-length, combinatorial, or even non-numerical inputs.

Introducing ClimaX, the first foundation model for weather and climate. A fast and accurate one-stop AI solution for a range of atmospheric science tasks.

Paper: arxiv.org/abs/2301.10343

Blog: microsoft.com/en-us/research…

Thread🧵
#ML #Climate #Weather #FoundationModel The current approach to numerical weather and climate modeling is to simulate a system of differential equations relating the flow of energy and matter in different Earth systems. The science is good, but often computationally expensive and imperfect at longer time scales.