Thread by @lymanstoneky

I have a dataset with something like 65 million rows in it, in a tab-delimited .txt file, with some extraneous quotation marks. I have been able to import a related dataset with 51 million rows and 2 columns without any issue.

But even when I suppress quote binding, Stata seems to be crapping out on importing it. It's supposed to be able to handle 20 billion observations. There's only like 50 columns too.

Now, it is eating up a ton of my memory while it imports. But what I'm wondering is: do I just need to leave it overnight and let it burn? Or do I need to fix a setting? Because doing the 51 million rows with 2 variables only took like 30 seconds to import.

It just doesn't make sense to me that a 2*51,000,000 matrix would take 30 seconds but a 50*51,000,000 would take >2 hours, since if processing time is stable by matrix size (which idk if it is) it should be about 10-15 minutes import time.

In terms of filesize, the one I handled successfully was 897 mb of .txt, whereas the troublesome one is 15.4 gb.

For the curious, I have 16 GB installed RAM.

So after it runs the import for about 3 minutes this is what happens, EVEN IF I tell it to only import *the first column*, which is numeric.

So, my question is: should I just let it run and it will eventually pound through and get the job done? Or is this a situaion where I need to actually do something differently?

Also, the real crisis here: it uses up enough memory that my Imperator savegame as Cappadocia doesn't run smoothly, and if I don't knock of Mithridates, he'll soon become a real threat.