If you are using Postgres for embeddings, JSON or large text columns, or all of the above... then you are using TOAST, whether you know it or no.
TOAST can have massive impact on performance. So lets talk about it a bit: 🧵 Postgres has 8K pages, and you can't spread a row across pages. In fact, Postgres prefers to leave a bit of room for additional versions of rows or additional rows.

So if a row exceeds 2K (modifiable TOAST_TUPLE_THRESHOLD), Postgres is going to try and do something about it.

I've ran into quite a few social posts that conflate relational databases, SQL, transactions and ACID. This is wrong in the sense that these are different concepts and you can mix and match them in different ways.

But turns out that historically, RDBMS predates SQL and ACID by… twitter.com/i/web/status/1… The intersection between transactions and ACID is especially interesting.

You can have ACID without transactions (BerkleyDB had ACID guarantees on single statements).

But once you had transactions, they interact with all letters.

Atomic: fully committed or fully rolled back… twitter.com/i/web/status/1…

I didn't expect to attend in-person conferences this year. It is hard to justify taking time away from a very new company and product.

But @QConSF is convincing enough that I'll make an exception.

The talks I'm looking forward to the most: 🧵 Microservices: The whole track looks great, but Orchestration vs Choreography by @anatomic and Enabling Data Mesh by @zhamakd are topics that I've been interested in for years and would love to discover new perspectives.

Future of Data prediction from @sriramsubram: "One DB will never happen. One platform will".
My thoughts: When I started my career, the DB world was pretty clear - You had OLTP database for running transactional workloads, you had a DWH for analytical workloads and your had ETL/ELT in between.

One of the ways developers get in trouble is when they try to outsmart the framework they are using.

The framework has some limitations, and they work hard to get around these limitations without understanding what they are trading off when they do this. Example! You have a Kafka producer, writing events about a large set of entities to a topic. You want events about each entity to be processed in order. And you also want some degree of parallelism between the entities.

Some advice on the enterprise SKU where you need to contact sales. But only very little, because I know very little about enterprise sales.

https://twitter.com/awakecoding/status/1447001677794627586

First, if your target persona is sales-averse (all developers), you'll win by minimizing the situations they have to talk to a salesperson. Do you really need them to contact sales, or maybe just provide an option? "For discounts, contact sales" may be better.

I am not sure SaaS companies understand how challenging and important SKUs are. A lot of what I see on pricing pages seems completely random. A thread. 1/ Why are they important? First, they are critical for landing customers. Each target user needs to look at your pricing page and go, "Yes. This SKU has the features I need, at a price that I am willing to pay".

I noticed that my communication style tends to drift toward the style of the leader I work most closely with. Is it a weird trait, or someone everyone does? When I worked with someone who went "mmm.... not exactly, this is a bit subtle", I also softened my disagreements. Now I work with someone comfortable saying "No" as a complete sentence, and I'm doing this more. A year ago, I'd have told you that this isn't my style.

Trying to square the whole "developers are kingmakers" and developer-marketing/evangelism trends with the observed reality in which developers are terrible at getting budget for things that make their own lives better. It is surprisingly common to hear "I'm stuck managing my own Kafka, even though it is a waste of my time, and even though I'm pretty bad at it because there is no budget for a managed service".

I'm thinking a lot recently about building kickass Kafka applications. I don't think we've spent enough time as a community discussing this and sharing best practices. Running @confluentcloud means that we are seeing lots of clients. So, what makes Kafka apps awesome? 1. Resilience: Kafka can be slow to respond, it can return errors, requests can time out, requests can be retried infinite times, responses can be garbage, etc. You need to think through the desired behavior under error conditions and write apps that will survive.

Live tweeting @AnnaPovzner talk on Quotas in @apachekafka Proud to have her in my team ❤️

Not getting results you want from data lake? You may think of trying data lake in the cloud - but it's the same paradigm with same issues. @zhamakd at #QConSF The problem is that the layers in data lake design map to technical components. This doesn't work - change is orthogonal.

#QConSF - @criccomini shares that WePay data infrastructure is based on Airflow, Kafka and BigQuery. The roadmap to data maturity

At @apachekafka meetup in Singapore and Agoda, an online travel agency is talking about their Kafka architecture. Should be good! 5bn messages a day, 190TB a day. 9000+ services. 5 data centers. 230 Kafka brokers.

LinkedIn sharing their streaming architecture at #strangeloop Explaining the importance of my favorite pattern: stream-table join.