Serre duality in derived category

I think you can prove this statement from (a weaker form of) the usual Serre duality statement using some compatibilities between derived functors, namely:

• For all $\mathcal{F}^{\bullet}\in D^{b}(X)$ we have a natural isomorphism $$ R\Gamma(X,-)\circ R\mathcal{Hom}(\mathcal{F}^{\bullet},-)\cong R\operatorname{Hom}(\mathcal{F}^{\bullet},-). $$
• For all $n\in \mathbb{N}$ we have a natural isomorphisms $$ \operatorname{Hom}(\mathcal{O}[-n],-)\cong H^{n}(R\Gamma(X,-)). $$
• For all $\mathcal{F}^{\bullet},\mathcal{G}^{\bullet},\mathcal{E}^{\bullet}\in D^{b}(X)$ we have an isomorphism $$ \operatorname{Hom}(\mathcal{F}^{\bullet},\mathcal{E}^{\bullet}\otimes^{L}\mathcal{G}^{\bullet})\cong \operatorname{Hom}(R\mathcal{Hom}(\mathcal{E}^{\bullet},\mathcal{F}^{\bullet}),\mathcal{G}^{\bullet}). $$ This follows from formula (3.15) in Huybrechts' book after applying derived global sections and taking cohomology on degree zero. The formula (3.15) is functorial, because it is based on a sequence of canonical isomorphisms coming from exercise II.5.1 in Hartshorne's book. Hence the previous isomorphism is functorial as well.

As a starting point consider the following statement, which is part of the usual Serre duality statement:

For all line bundles $\mathcal{L}$ on $X$ we have a natural perfect pairing $$ \operatorname{Hom}(\mathcal{L},\omega_{X})\times H^{n}(X,\mathcal{L})\to H^{n}(X,\omega_{X})\cong k $$

We generalize this statement to bounded complexes of coherent sheaves:

For all $\mathcal{F}^{\bullet}\in D^{b}(X)$ we have a natural perfect pairing $$ \operatorname{Hom}(\mathcal{F}^{\bullet},\omega_{X})\times \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F}^{\bullet})\to {\operatorname{Hom}(\mathcal{O}_{X}[-n],\omega_{X})}\cong k $$

Proof:

We have to check that the induced morphism $$ \operatorname{Hom}(\mathcal{F}^{\bullet},\omega_{X})\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F}^{\bullet})^{\vee} $$ is an isomorphism. We divide the proof into smaller steps:

1. The result is true for finite direct sums of line bundles, because all functors are additive and we know the result for line bundles already.
2. The result is true for all coherent sheaves (complexes concentrated on degree zero). To see this we use that every coherent sheaf $\mathcal{F}$ on a smooth projective variety admits a surjection from a finite direct sum of line bundles $\mathcal{E}=\oplus_{i=1}^{l}\mathcal{L}_{i}$. So we have a distinguished triangle $$ \mathcal{K} \to \mathcal{E} \to \mathcal{F} \to \mathcal{K}[1] $$ Apply the cohomological functors $\operatorname{Hom}(-,\omega_{X})$ and $\operatorname{Hom}(\mathcal{O}_{X}[-n],-)^{\vee}$ to obtain a ladder diagram with first row the exact sequence $$ 0\to \operatorname{Hom}(\mathcal{F},\omega_{X})\to \operatorname{Hom}(\mathcal{E},\omega_{X})\to \operatorname{Hom}(\mathcal{K},\omega_{X}) $$ and second row the exact sequence $$ 0\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{F})^{\vee}\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{E})^{\vee}\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{K})^{\vee}. $$ The zero on the first row corresponds to a negative ext group between coherent sheaves and the one on the second row follows from Grothendieck vanishing. By exactness of the rows and the result for finite direct sums of line bundles, the composition $\operatorname{Hom}(\mathcal{F},\omega_{X})\to \operatorname{Hom}(\mathcal{O}_{X}[-n],\mathcal{E})^{\vee}$ is injective. By commutativity of the diagram the map $\operatorname{Hom}(\mathcal{F},\omega_{X})\to\operatorname{Hom}(\mathcal{O}_{X},\mathcal{F})^{\vee}$ is injective as well. Since $\mathcal{F}$ was an arbitrary coherent sheaf on $X$, this is also true for the corresponding map for $\mathcal{K}$. Hence we can apply the 5-lemma (the version which is sometimes refered to as 4-lemma) to get the isomorphism that we wanted.
3. The result is true for all bounded complexes of coherent sheaves. This can be shown by induction on the length of the complex. The result is already known for complexes of length zero, so let $\mathcal{F}^{\bullet}$ be a complex of length $n+1$ for some integer $n\geqslant 0$. For simplicity assume that this complex is concentrated between degrees $-n$ and $0$. Let $\mathcal{F}^{\bullet}_{\leqslant -1}$ be the stupid truncation (not preserving the cohomology of the complex), so that we get a distinguished triangle $$ \mathcal{F}^{\bullet}_{\leqslant -1}\to \mathcal{F}^{\bullet}\to \mathcal{F}^{0} \to \mathcal{F}^{\bullet}_{\leqslant -1}[1] $$ By induction hypothesis and arguing with the 5-lemma as before we obtain the desired isomorphism.

[End of proof]

Now we can use the compatibilities mentioned at the beginning to deduce the result in the form stated in Huybrechts' book. The conclusion is the following:

The category $D^{b}(X)$ has a Serre functor given by $$ \mathcal{F}^{\bullet} \mapsto \mathcal{F}^{\bullet}\otimes \omega_{X}[n] $$

Proof:

For all $\mathcal{F}^{\bullet}$ and $\mathcal{G}^{\bullet}$ in $D^{b}(X)$ we have the following sequence of functorial isomorphisms: $$ \operatorname{Hom}(\mathcal{F}^{\bullet},\mathcal{G}^{\bullet}) $$ $$ \cong \operatorname{Hom}(\mathcal{F}^{\bullet}\otimes \omega_{X},\mathcal{G}^{\bullet}\otimes \omega_{X}) $$ $$ \cong \operatorname{Hom}(R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}),\omega_{X}) $$ $$ \cong \operatorname{Hom}(\mathcal{O}_{X}[-n],R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}))^{\vee} $$ $$ \cong \operatorname{Hom}(\mathcal{O}_{X},R\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n]))^{\vee} $$ $$ \cong H^{0}(R\Gamma(\mathcal{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n])))^{\vee}$$ $$ \cong H^{0}(R\operatorname{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n]))^{\vee}$$ $$ \cong \operatorname{Hom}(\mathcal{G}^{\bullet},\mathcal{F}^{\bullet}\otimes \omega_{X}[n])^{\vee}.$$ [End of proof]