Thread by @tungnd_13

Introducing ClimaX, the first foundation model for weather and climate. A fast and accurate one-stop AI solution for a range of atmospheric science tasks.

Paper: arxiv.org/abs/2301.10343

Blog: microsoft.com/en-us/research…

Thread🧵
#ML #Climate #Weather #FoundationModel

The current approach to numerical weather and climate modeling is to simulate a system of differential equations relating the flow of energy and matter in different Earth systems. The science is good, but often computationally expensive and imperfect at longer time scales.

Fortunately, weather and climate is also a data-rich field courtesy satellites, radar, & other sensors. While numerical models do not scale with data, ML models benefit from both data & compute. What if we could distill knowledge of the Earth’s atmosphere into a large neural net?

Unlike recent ML attempts aimed at specific tasks like weather forecasting, CliMax is a foundation model that allows quick and easy adaptation to any spatiotemporal predictive task in the atmospheric sciences.

To enable this adaptation, we follow a pretrain-finetune regime. We propose to use climate simulation datasets (CMIP6) for pretraining. Enables finetuning on reanalysis datasets!

Architecture: ClimaX extends ViT with novel tokenization and aggregation modules that allow learning from heterogeneous data sources while remaining computationally efficient.

Forecasting: ClimaX can produce global and regional forecasts all the way from a few hours to days and weeks into the future Better/competitive with IFS as lead time grows!

Climate Projections: ClimaX can also be used to project future climates under different greenhouse forcings. State-of-the-art performance on ClimateBench!

Downscaling: ClimaX can downscale low-resolution outputs of climate models and fix biases. Outperforms all other CNN-style architectures.

This paper was the result of a fruitful year-long collaboration between UCLA & Microsoft. Very excited to see both the ML & climate community build on these results for next-generation climate science!

w/ @jo_brandstetter, @akapoor_av8r, @rejuvyesh and @adityagrover_