How to create a window which will close in a certain amount of time?

This is what I understood OP needs:

SetAttributes[foo, HoldFirst];

foo[proc_] := Module[{dialog},
  dialog = MessageDialog["whatever"];
  RunScheduledTask[NotebookClose @ dialog, {3}];
  proc]

a more advanced example can be found in

Palette button with progress bar

Oh, sorry everyone. I've got an answer here right after I posted my answer......

But I think this solution may be helpful for other users annoyed with Pause:

new = CreateDialog[Plot[x^2, {x, 0, 1}]];
PrintTemporary@DynamicModule[{t = Now, check = True},
   Dynamic[
    If[check && Now - t >= Quantity[3, "Seconds"], NotebookClose[new];
      check = False], UpdateInterval -> .2]];
Print@1
Prime /@ Range@1000000;

Much smoother.

The key of this answer is to use dynamic link between Kernel and FrontEnd instead of the normal link used for calculations. In this way, the dynamic will keep running while normal evaluations running too!

But there still is a problem: Here I used PrintTemporary and Print to deal with the Dynamic to make it disappear and stop wasting energy after it finished its duty. How to solve this problem better? This method is so rude and not that elegant......

A self-closing dialog window using NotebookDynamicExpression

selfclose1[content_, closetime_] := CreateDialog[
  Column[{
    content,
    CancelButton[ImageSize -> Full]}],
  NotebookDynamicExpression -> 
   Dynamic[Refresh[
     If[Clock[closetime, closetime, 1] == closetime, NotebookClose[]],
      UpdateInterval -> closetime, TrackedSymbols :> {}], None, 
    UpdateInterval -> Infinity, TrackedSymbols :> {}]]

or a DynamicWrapper

selfclose2[content_, closetime_] := CreateDialog[
  Column[{
    DynamicWrapper[
     content,
     Refresh[
      If[Clock[closetime, closetime, 1] == closetime, 
       NotebookClose[]], UpdateInterval -> closetime, 
      TrackedSymbols :> {}], UpdateInterval -> Infinity, 
     TrackedSymbols :> {}, Evaluator -> None],
    CancelButton[ImageSize -> Full]}]]

Test 1:

selfclose2[Plot[x^2, {x, 0, 1}], 3]

Test 2:

AbsoluteTiming[
 First@AbsoluteTiming[
    With[{t = 3}, 
     selfclose1[ProgressIndicator[Dynamic[Clock[t, t, 1]], {0, t}], t]]; 
    Prime /@ Range@500000] - 
  First@AbsoluteTiming[Prime /@ Range@500000]
 ]

---

Instead of a Clock, one could also use

AbsoluteTime[TimeZone -> 0] - ("MemoryModificationTime" /. NotebookInformation[])

or increment a local variable.