Speed up Flatten[] of a large nested list
The difference between using Flatten
and using Join
as in @kglr's answer is that Flatten
unpacks. Here is a smaller example:
SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]
{{0.269558, 0.445678, 0.158104, 0.751213, 0.965444}, {0.0518202, 0.675946, 0.698472}, {0.344389, 0.830322, 0.556863}}
Turn on packing messages:
On["Packing"]
Then, using Flatten
:
Flatten[list]
Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.
Developer`FromPackedArray::punpack: Unpacking array with dimensions {5} in call to Flatten.
Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.
Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.
Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.
General::stop: Further output of Developer`FromPackedArray::unpack will be suppressed during this calculation.
Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.
General::stop: Further output of Developer`FromPackedArray::punpack will be suppressed during this calculation.
{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}
and using Join
:
Join @@ list
{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}
As you can see, using Join
generates no unpacking messages, which is why it is much faster.
Apply
ing Join
is much faster than Flatten
:
SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], {n, 100}];
fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First
8.2375848
fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First
0.29729
fullFlatten2 == fullFlatten
True
ByteCount /@ {fullFlatten, fullFlatten2, jaggedList}
{1462957016, 487652456, 487667608}