Converting a list of colors into an image

colors = RandomColor[{5, 5}]

Image[colors, ImageSize -> 1 -> 32]   

If you a list of list of numbers instead of colors as input, you can use

list = colors /. RGBColor -> List;
Image[list, ImageSize -> 1 -> 32]

same picture

Your idea of using Rasterization can be made to work. You must take window magnification into account to get to proper size in pixels.

swatchToRaster[color_?ColorQ, size_ /; size > 0, mag_Real: 1.25] :=
  Rasterize[color , ImageSize -> size/mag]

Row[Prepend[swatchToRaster[#, 32, 1.5] & /@ {Red, Green, Blue}, Spacer[50]]]

I used the ruler tool from the xTools app to prove that the size of rasterized swatches are indeed 32 x 32 pixels.

Note: when defining swatchToRaster, I gave the mag argument the default of 1.25 because I normally work with notebooks having magnification set to 125%. You should change this default if you normally use a different window magnification.

Each color is represented by a triplet of values {r,g,b} where each element is drawn from 0 to 1. You can get a matrix of these, turn it into an image with Image, and then resize:

Image[RandomReal[{0, 1}, {4, 5, 3}], ImageSize -> 1 -> 32]