How to set transparency per polygon in a shapefile in QGIS?

You can use data defined properties for this. Use the style tab on the layer properties and click the expression button to the right of the color.

In there you can use the function color_rgba( red, green, blue, alpha ) to create the color. All values need to be between 0 and 255.

Example:

color_rgba( 255, 0, 0, ( 1 - "transparency" ) * 255 )

This will give you a totally red style with the alpha defined from a field "transparency" as per your specification in the question.

In the expression editor you will find a number of other color related functions that may better suit your needs, just explore them and read their documentation directly in the expression editor.

Another example for nominal (qualitative) values which you would normally classify is to work in the hsv space:

There is the function color_hsva( hue, saturation, value, alpha ) to create the color. The values need to be between

• hue: 0-360 (meaning see below)
• value and saturation: 0-100
• alpha: 0-255.

Example:

color_hsva( 
  CASE 
    WHEN "classification" = 'red' THEN 0
    WHEN "classification" = 'blue' THEN 240
  END CASE, -- hue
  80,  -- saturation
  80,  -- value
  ( 1 - "transparency" ) * 255  -- alpha
)

For QGIS >= 2.12 also consider @ndawsons answer.

As a follow up to Matthias' answer, in QGIS >= 2.12 you can use a data defined fill color with the expression:

 set_color_part(@value, 'alpha', ( 1 - "transparency" ) * 255 )

What this expression does is takes the original polygon color (@value) , and replaces the alpha channel (opacity) with the value calculated from the field). This means you can set the fill color expression and then happily go ahead and use categorised or graduated renderers to automatically set the base fill color and the opacity rule will still be respected. No need for hard coding the rgb values this way!

You can also set this values in the Layer Properties of the layer (right click -> Properties).
Use Style -> Categorized then select the column you want to use, for example 'value', choose the colors in the color ramp and click 'Clasify'.
Then you can define the color of each value and its transparency.

If you add new values to the used layer click on Clasify to add them to your existing style.