Today is a big day @InfluxDB. We're announcing the release of InfluxDB IOx, the new core of the database written in #rustlang and built with @ApacheArrow. It delivers infinite cardinality, tiered data storage, SQL compatibility and so much more. Here's some of the background...
When we released #InfluxDB 1.0 in 2016, it was written in #golang and had a custom storage engine we built from scratch that was specifically optimized for metrics workloads. We called it the Time Structured Merge Tree (or TSM for short).
The design of TSM paired a time series storage database with an inverted index to map metadata to underlying time series. This delivered excellent performance for low to medium cardinality workloads. But as the precision of information in the metadata increased, the DB suffered.
However, we always had a vision that #InfluxDB would be not just a pure time series database, but a database that could store raw events at scale and yield time series on the fly at query time. Achieving this required a complete rearchitecture of the underlying DB.
In addition to supporting events at scale, we wanted to separate compute from storage, using object storage to achieve much greater scale while reducing costs for large scale workloads. We also wanted to add support for SQL and bulk data imports and exports.
We started researching what tools to use for this project in the spring of 2020. By the summer we landed on writing in #rustlang (influxdata.com/blog/rust-can-…) and @ApacheArrow based on our confidence in their promise and their already existing strengths (influxdata.com/blog/apache-ar…)
Those bets have paid off as #rustlang and @ApacheArrow have advanced significantly over the last two years. Importantly, we've also contributed significantly to DataFusion, the Rust based SQL parser, planner, optimizer and execution engine and the Rust implementation of Parquet
Along the way we've released a #rustlang @apachekafka library, an object storage library, and a library that translates protobuf to json. github.com/influxdata/rsk… github.com/apache/arrow-r… github.com/influxdata/pbj…
We have so much more work to do, but as of today, we're taking signups for customers running in production on #InfluxDB IOx in our Cloud offering. I wrote a bit more about it here: influxdata.com/blog/influxdb-… and signup is here: influxdata.com/influxdb-engin….
• • •
Missing some Tweet in this thread? You can try to
force a refresh
