Resizing N # of squares to be as big as possible while still fitting into box of X by Y dimensions. (Thumbnails!)

Probably not optimal (if it works which I haven't tried), but I think better than you current approach :

w: width of rectangle

h: height of rectangle

n: number of images

a = w*h : area of the rectangle.

ia = a/n max area of an image in the ideal case.

il = sqrt(ia) max length of an image in the ideal case.

nw = round_up(w/il): number of images you need to stack on top of each other.

nh = round_up(h/il): number of images you need to stack next to each other.

l = min(w/nw, w/nh) : length of the images to use.

The solution on https://math.stackexchange.com/a/466248 works perfectly.

An unoptimized javascript implementation:

var getMaxSizeOfSquaresInRect = function(n,w,h) 
{
    var sw, sh;
    var pw = Math.ceil(Math.sqrt(n*w/h));
    if (Math.floor(pw*h/w)*pw < n) sw = h/Math.ceil(pw*h/w);
    else sw = w/pw;
    var ph = Math.ceil(Math.sqrt(n*h/w));
    if (Math.floor(ph*w/h)*ph < n) sh = w/Math.ceil(w*ph/h);
    else sh = h/ph;
    return Math.max(sw,sh);
}