Memory optimization in PHP array

A little bit late to the party, but if you have a multidimensional array you can save a lot of RAM when you store the complete array as json.

$array = [];

$data = [];
$data["a"] = "hello";
$data["b"] = "world";

To store this array just use:

$array[] = json_encode($data);

instead of

$array[] = $data;

If you want to get the arrry back, just use something like:

$myData = json_decode($array[0], true);

I had a big array with 275.000 sets and saved about 36% RAM consumption.

EDIT: I found a more better way, when you zip the json string:

$array[] = gzencode(json_encode($data));

and unzip it when you need it:

$myData = json_decode(gzdecode($array[0], true));

This saved me nearly 75% of RAM peak usage.


If you want a real indexed array, use SplFixedArray. It uses less memory. Also, PHP 5.3 has a much better garbage collector.

Other than that, well, PHP will use more memory than a more carefully written C/C++ equivalent.

Memory Usage for 1024x1024 integer array:

  • Standard array: 218,756,848
  • SplFixedArray: 92,914,208

as measured by memory_get_peak_usage()

$array = new SplFixedArray(1024 * 1024); // array();
for ($i = 0; $i < 1024 * 1024; ++$i)
  $array[$i] = 0;

echo memory_get_peak_usage();

Note that the same array in C using 64-bit integers would be 8M.

As others have suggested, you could pack the data into a string. This is slower but much more memory efficient. If using 8 bit values it's super easy:

$x = str_repeat(chr(0), 1024*1024);
$x[$i] = chr($v & 0xff); // store value $v into $x[$i]
$v = ord($x[$i]);        // get value $v from $x[$i]

Here the memory will only be about 1.5MB (that is, when considering the entire overhead of PHP with just this integer string array).

For the fun of it, I created a simple benchmark of creating 1024x1024 8-bit integers and then looping through them once. The packed versions all used ArrayAccess so that the user code looked the same.

                   mem    write   read
array              218M   0.589s  0.176s
packed array       32.7M  1.85s   1.13s
packed spl array   13.8M  1.91s   1.18s
packed string      1.72M  1.11s   1.08s

The packed arrays used native 64-bit integers (only packing 7 bytes to avoid dealing with signed data) and the packed string used ord and chr. Obviously implementation details and computer specs will affect things a bit, but I would expect you to get similar results.

So while the array was 6x faster it also used 125x the memory as the next best alternative: packed strings. Obviously the speed is irrelevant if you are running out of memory. (When I used packed strings directly without an ArrayAccess class they were only 3x slower than native arrays.)

In short, to summarize, I would use something other than pure PHP to process this data if speed is of any concern.


In addition to the accepted answer and suggestions in the comments, I'd like to suggest PHP Judy array implementation.

Quick tests showed interesting results. An array with 1 million entries using regular PHP array data structure takes ~200 MB. SplFixedArray uses around 90 megabytes. Judy uses 8 megs. Tradeoff is in performance, Judy takes about double the time of regular php array implementation.

Tags:

Php

Arrays

Memory Management

Recent Posts