Speed up Flatten[] of a large nested list

The difference between using Flatten and using Join as in @kglr's answer is that Flatten unpacks. Here is a smaller example:

SeedRandom[1]
list = Table[RandomReal[1, RandomSample[2;;5, 1]], 3]

{{0.269558, 0.445678, 0.158104, 0.751213, 0.965444}, {0.0518202, 0.675946, 0.698472}, {0.344389, 0.830322, 0.556863}}

Turn on packing messages:

On["Packing"]

Then, using Flatten:

Flatten[list]

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {5} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

Developer`FromPackedArray::unpack: Unpacking array in call to HoldForm.

General::stop: Further output of Developer`FromPackedArray::unpack will be suppressed during this calculation.

Developer`FromPackedArray::punpack: Unpacking array with dimensions {3} in call to Flatten.

General::stop: Further output of Developer`FromPackedArray::punpack will be suppressed during this calculation.

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

and using Join:

Join @@ list

{0.269558, 0.445678, 0.158104, 0.751213, 0.965444, 0.0518202, 0.675946, 0.698472, 0.344389, 0.830322, 0.556863}

As you can see, using Join generates no unpacking messages, which is why it is much faster.


Applying Join is much faster than Flatten:

SeedRandom[1]
jaggedList = Table[RandomReal[1, RandomSample[Range[400000, 800000], 1]], {n, 100}];

fullFlatten = Flatten@jaggedList; // AbsoluteTiming // First

8.2375848

fullFlatten2 = Join @@ jaggedList; // AbsoluteTiming // First

0.29729

fullFlatten2 == fullFlatten

True

ByteCount /@ {fullFlatten, fullFlatten2, jaggedList}

{1462957016, 487652456, 487667608}