Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Adarsh Profile picture
Adarsh
@adarsh_mamgain
23, developer | AI writing assistant: https://t.co/UI8c0KOtJu | Track your brand keywords on all social platforms: https://t.co/oFyw1ukjJI
Feb 9 • 9 tweets • 2 min read
tldr

spent the weekend debugging why our scraping workers kept crashing 😅
turns out processing everything in parallel was eating up memory and causing OOM errors
switched to sequential processing, added some GC triggers, and bumped up memory limits.
back to 95%+ uptime now 🎉 Image 1. how it was working before:
we were scraping 7 platforms (X, reddit, youtube, tiktok, instagram, threads, pinterest) all at once using Promise.all()
worked fine initially. but as we scaled and sessions got bigger, memory started spiking...