How to convert a hex color string to RGBColor?

Using IntegerDigits to convert directly to base 256:

hexToRGB = RGBColor @@ (IntegerDigits[
  ToExpression@StringReplace[#, "#" -> "16^^"], 256, 3]/255.) &

hexToRGB["#FF8000"]
(*   RGBColor[1., 0.501961, 0.]  *)

Edit

Shorter version, since somebody mentioned golfing...

hexToRGB = RGBColor @@ (IntegerDigits[# ~StringDrop~ 1 ~FromDigits~ 16, 256, 3]/255.) &

Starting from version 10.1 you can use RGBColor directly:

RGBColor["#FF8000"]
(* RGBColor[1., 0.5019607843137255, 0.] *)

RGBColor["#45A"]
(* RGBColor[0.26666666666666666`, 0.3333333333333333, 0.6666666666666666] *)

ToColor[RGBColor["#FF8000"], Hue]
(* Hue[0.08366013071895424, 1., 1.] *)

Function that converts string to a list of 3 numbers: for R, G and B component:

toRGBSequence[i_] := 
  Composition[FromDigits[#, 16] &, StringJoin] /@ 
   Partition[Characters[StringDrop[i, 1]], 2] /. List -> Sequence;

• First, dropping the # sign.
• Converting to a list, using the Characters function.
• Partitioning in groups of 2.
• Composition of function to first join the two letters and than convert it to a base-10 format.
• Map over a list.
• Converting everything to Sequence to use with RGBColor function.

Usage:

RGBColor[toRGBSequence["#FF5500"]]

PS: This may, or may not be the most accurate and fast solution.