Stacking rectangles to take as little space as possible

I'd recommend starting with a simple greedy approach, and seeing if that is good enough for your needs. If your input is well-behaved or small, that may be all you need--and the complexity will go up quickly when you try to do something more sophisticated.

For example: sort the rectangles by size, largest first. Add the rectangles one at a time, trying each possible position for the new rectangle. Pick the position that results in the smallest bounding box.

Another greedy approach would be to pick a starting rectangle, then repeatedly add the rectangle which results in the most dense arrangement (where density is defined as the percentage of the bounding box's area that is filled).

http://www-rcf.usc.edu/~skoenig/icaps/icaps04/icapspapers/ICAPS04KorfR.pdf

Apparently this problem is harder than it looks at first. It's an interesting algorithm, since first it guesses a solution and then improves on it, so if you don't want to wait for the optimal solution, you can just run it for a set number of iterations to get an approximate solution (the longer you run it, the better the approximation).