Thread by @LukeYoungblood

Web3 data is broken...

I didn't quite realize how badly until just recently, but we hit some hard scaling challenges at @MoonwellDeFi as we crossed 40,000 wallets on @Base. 🌜🔵🌛

In the last 3 months, our community on Base has quadrupled. This is a great problem to have... 🧵

Before our launch on Base last year, we designed a new data architecture that enabled us to show beautiful market and portfolio graphs in our app. Using GraphQL, we were able to capture daily snapshots of your positions, and show your portfolio history over time.

https://twitter.com/86364019/status/1688320044659552256

This worked beautifully, and was well received by our community, but we knew it was a somewhat inefficient way to do this. As we recently reached over 40,000 wallets, it was now taking several hours per day to index all of this onchain data and generate these daily snapshots.

https://twitter.com/86364019/status/1688320165975584768

For context, it takes almost a month now for us to index our GraphQL data from the beginning of our protocol launch in 2022. We planned on refactoring our data architecture this summer so we could scale to the millions of wallets that will come onchain. 🌜🌞🌛

https://twitter.com/86364019/status/1688320158740398080

This is not unique to us. @Uniswap's subgraph takes a month to sync as well, and they only have to index swap and LP events. Imagine being a data engineer at Uniswap or Moonwell and having to wait a month to test a new feature... 😅

I woke up April 18th to this ominous message:

We use both @Goldskyio and @AlchemyPlatform for subgraph hosting. They are some of the best web3 hosts in the space, and our subgraph had failed on both, with this ominous message in the logs.

We had hit an unknown limit or bug in The Graph node itself: github.com/graphprotocol/…

One of the core maintainers of The Graph node was very interested in fixing our problem, however, when I asked our friends at @Goldsky how soon we might get a fix, they indicated The Graph doesn't typically cut a release for months... We struggled to come up with a workaround.

Coincidentally, @x0s0l and I had already been researching an alternative backend, @ponder_sh, as an eventual replacement for The Graph. One of the challenges we were aiming to solve was multi-chain support. We support 5 networks now, and have 2 subgraphs on each.

One of the challenges with having so many subgraphs is that our app needs to pull data from all of them, so we're running identical queries across 10 different backend APIs at once instead of just one. And making a change to add a new column of data requires super long reindexing

With @ponder_sh we could index all of the networks we support in parallel from a single indexer, and we could query data on all the networks with a single API endpoint. We could also index on our laptops locally, which was too slow using the open source node from The Graph.

Here we were a week ago, with a subgraph that crashed due to a previously unknown scalability limit, with no fix in sight. We had two options:
- Refactor our subgraph to make it more efficient by only snapshotting N accounts per event
- Migrate to @ponder_sh, a refactor in itself

We decided to try @ponder_sh. There were two major factors leading to our decision:
1) With some early testing we determined we could capture the daily snapshots in ~5 minutes per day instead of ~3 hours, and we could index all our data in just a few hours, vs. ~30 days

2) Even if we could work around the issue in The Graph, hitting this limit at only ~30,000 wallets indicated that it would be extremely unlikely that we could ever scale to 100,000, much less the millions of users that will soon be using @Base as @Coinbase onboards retail users.

The only challenge with @ponder_sh is hosting: you're kind of on your own, and we prefer to use cloud services. We found this really awesome service called @Railway that makes hosting super easy. I used to work at @AWScloud and this is 100x easier. Here's how I setup our backend:

First, I setup a staging environment, so we could test upgrades and improvements, as well as west and east coast production environments, so you could access our data in the region closest to you, as measured by latency.

For a robust @ponder_sh environment suitable for production you'll want something like this:
- A Postgres database
- An indexer deployment: this system indexes all the data from each network's RPCs
- A serve-only deployment: this contains 2 or more replica ponder serve containers

The east coast region looks exactly the same. This gives you more resilience in case one of the regions is having a bad day, and also enables you to make changes to one at a time to perform zero downtime upgrades.

I recommend pointing each region at a different branch in Github.

To pull it all together and provide a more reliable and performant service, we front it with a @Cloudflare load balancer. Cloudflare can do "traffic steering," which enables it to send traffic to the closest region by latency, and it performs health checks from 8 regions globally

With a Cloudflare load balancer and traffic steering in place, now we have a consistent URL for Moonwell data:

Anyone can query Moonwell's open data using a very familiar interface: GraphQL. This is also what the app uses to render those beautiful graphs. ponder.moonwell.fi

We still have a lot more optimization to do in order to prepare to scale to the millions of new wallets that will be created during onchain Summer. While we've overcome the current scaling challenge, I'm sure we'll encounter more and need to optimize our data backend further.

@base Going forward, I'm confident that @ponder_sh can meet our current and future data needs. In this multi-chain world we're in now, where there are dozens of networks, the single API per network model for The Graph just doesn't seem to scale, and indexing speed is a pain point.

I want to thank @typedarray and the @ponder_sh team for their help and being super responsive in Telegram. They are really shipping improvements fast. Ponder is still very new and a little rough around the edges, but it's rapidly becoming a data powerhouse.

I also think Ponder might be the future of web3 data, and if so, I'm sure the fine people at @Goldsky and @AlchemyPlatform will start offering higher level hosting services for Ponder so that you don't have to build your own infra like we did.

What's next? We still need to migrate our governance subgraphs over to the new data backend, and there are a number of ancillary services that need to be updated, things like Discord bots, etc. We're staying Based and Optimistic, in this new world of multi-chain web3 data! 🌜🤔🌛

I hope you've found this thread about our Based data journey helpful.

Follow @MoonwellDeFi and me, @LukeYoungblood for more.

Like/Repost the quote below if you can:

https://twitter.com/86364019/status/1784244530071605612