Functors in Isbell duality exchange $f^*a$ and $f_*a$
Let's give names to these two separate conditions: \begin{align} Nat(B(b,f-),hom(a,-)) &\cong B(fa,b) \tag{1} \\ Nat(B(f-,b),hom(-,a)) &\cong B(b,fa) \tag{2} \end{align} Neither of these must be true in general, though they hold under some natural conditions. I will treat (2) first, since (1) is completely dual.
As a counterexample, take $A = 1$ the terminal category, then $f$ corresponds to an object of $B$, and (2) reduces to a natural isomorphism $$ Set(B(f,b),1) \cong 1 \cong B(b,f) $$ which is false unless $f$ is a terminal object in $B$.
A sufficient condition for (2) is that $f : A \to B$ is both fully faithful and dense, meaning that there are natural isomorphisms \begin{align} A(a,a') &\cong B(fa,fa') \tag{$f$ fully faithful}\\ B(b,b') &\cong Nat(B(f-,b),B(f-,b')) \tag{$f$ dense} \end{align} We then derive (2) in two steps: $$ Nat(B(f-,b),hom(-,a)) \cong Nat(B(f-,b),B(f-,fa)) \cong B(b,fa) $$ (As an aside, it is perhaps worth mentioning that fully faithful and dense functors play a role in the theory of monads with arities.)
Essentially the same counterexample works with (1) (now taking $f : 1 \to B$ to be any non-initial object of $B$), and for a sufficient condition we can replace density with the codensity-like assumption that the functor $f_* : B \to [A,Set]^\circ$ is fully faithful, i.e., that there is an isomorphism $B(b,b') \cong Nat(B(b',f-),B(b,f-))$.
Update (14 June): Restoring original answer (with expanded commentary) after realizing the flaw in the OP's counterargument. The isomorphisms ${\mathcal O} \circ f^* \cong f_*$ and $Spec \circ f_* \cong f^*$ cannot follow by preservation of (left/right) Kan extensions along (left/right) adjoints, since $f^*$ is a left Kan extension while $f_*$ is a right Kan extension.