Array of Matrices in MATLAB

Use cell arrays. This has an advantage over 3D arrays in that it does not require a contiguous memory space to store all the matrices. In fact, each matrix can be stored in a different space in memory, which will save you from Out-of-Memory errors if your free memory is fragmented. Here is a sample function to create your matrices in a cell array:

function result = createArrays(nArrays, arraySize)
    result = cell(1, nArrays);
    for i = 1 : nArrays
        result{i} = zeros(arraySize);
    end
end

To use it:

myArray = createArrays(requiredNumberOfArrays, [500 800]);

And to access your elements:

myArray{1}(2,3) = 10;

If you can't know the number of matrices in advance, you could simply use MATLAB's dynamic indexing to make the array as large as you need. The performance overhead will be proportional to the size of the cell array, and is not affected by the size of the matrices themselves. For example:

myArray{1} = zeros(500, 800);
if twoRequired, myArray{2} = zeros(500, 800); end

If all of the matrices are going to be the same size (i.e. 500x800), then you can just make a 3D array:

nUnknown;  % The number of unknown arrays
myArray = zeros(500,800,nUnknown);

To access one array, you would use the following syntax:

subMatrix = myArray(:,:,3);  % Gets the third matrix

You can add more matrices to myArray in a couple of ways:

myArray = cat(3,myArray,zeros(500,800));
% OR
myArray(:,:,nUnknown+1) = zeros(500,800);

If each matrix is not going to be the same size, you would need to use cell arrays like Hosam suggested.

EDIT: I missed the part about running out of memory. I'm guessing your nUnknown is fairly large. You may have to switch the data type of the matrices (single or even a uintXX type if you are using integers). You can do this in the call to zeros:

myArray = zeros(500,800,nUnknown,'single');