Thread by @erickwendel_

How to consume any Database as a Stream for heavy data processing using JavaScript 🤯🔥

🧵/1

#javascript #nodejs #streams #sql #nosql #tutorial #dataprocessing #database #business #data #strategy #pipeline @nodejs

The secret for processing anything using JavaScript is to handle data on demand.

Imagine data you wanna migrate data from a SQL database to a NoSQL DB. You would need to apply some business rules, clean up fields, filter data and then output them to the final output.

/2

You might know that you can block the Node.js (and the data source you're consuming) if you handle too much data at once in memory

The best practice then is to limit results, send individual data to a stream pipeline, and then ask for more data until you've consumed it all.

/3

In Node.js you can now use async generators as a data source. This is game-changing for those who really need to use JavaScript to process terabytes of data if needed.

/4

I've seen people installing packages and struggling to consume data as Streams - and I'm not speaking only about databases

This strategy applies to anything. Is it a file? Database? External API? You can handle them using Node.js if you keep in mind to process them on demand

/5

I have a good video on my Youtube channel where I show how you can throttle data to respect 3rd-party-API limits and this might also help you a lot on heavy tasks using Node.js.



/6

How about you, how would you migrate terabytes of data from one source to another using JavaScript?

/7