Efficient way to storage large lists in a file(s)
Using ByteCount
will tell you to expect more than 1 GB:
ByteCount @ { alist, blist, clist }
(* 1 136 880 448 *)
Compress
I would thus try Compress
:
cexpr = Compress @ { alist, blist, clist };
(* Export["pathname\\data.m", cexpr ]; *)
ByteCount @ cexpr
(* 437 359 280 *)
I am getting a 433 MB file (which roughly matches ByteCount
). You can Uncompress
the expression after loading.
BinarySerialize
Another possibility as of Version 11.1 or later is BinarySerialize
:
bexpr = BinarySerialize @ { alist, blist, clist };
ByteCount @ bexpr
(* 378 170 128 *)
So we are down to about 378 MB (the file is 312 MB on my computer). You can use BinaryDeserialize
to get the original expression again (see below for explicit instructions for writing/reading binary data).
If we give the option PerformanceGoal -> "Size"
bexpr = BinarySerialize[ {alist,blist,clist}, PerformanceGoal -> "Size" ];
ByteCount @ bexpr
(* 327 494 245 *)
we are down to about 327 MB.
Writing and Reading Binary Data
The documentation tells you how to write/read binary data:
stream = OpenWrite[ "pathname\\data.mx", BinaryFormat -> True ];
BinaryWrite[ stream, bexpr ];
Close @ stream;
Reading the data:
data = BinaryDeserialize @ ByteArray @ BinaryReadList[ "pathname\\data.mx", "Byte" ];