Epimorphisms preserved by pullback functor with right adjoint

The important thing to note here is that an arrow $g: X \to B$ is an epimorphism if and only if the square $\require{AMScd}$ $$ \begin{CD} X @>g>> B\\ @VgVV @VV Id_B V\\ B @>>Id_B> B \end{CD} $$ is a pushout. In a slice category, colimits are computed just as in the underlying category (see e.g. proposition 3.4 on this nLab page for the dual statement). That means that if $g$ is an epi in $\mathcal C$, then the above pushout will also be a pushout in $\mathcal C / B$. Then since the pullback functor $f^*: \mathcal C / B \to \mathcal C / A$ has a right adjoint, it preserves colimits and thus in particular it preserves pushouts. Therefore, we have a pushout $$ \begin{CD} f^*(X) @>f^*(g)>> A\\ @V f^*(g) VV @VV Id_A V\\ A @>>Id_A> A \end{CD} $$ in $\mathcal{C} / A$, which by the same result as before means that this is a pushout in $\mathcal C$. Then we can conclude that $f^*(g)$ is indeed an epimorphism in $\mathcal C$.

Edit: as was pointed out in the comments below, the last step is not as trivial as I first thought. The problem is that the second square may not be a pushout in $\mathcal C$. So far we have only shown that it is a pushout in $\mathcal{C}/A$. From that we can conclude that $f^*(g)$ is an epimorphism in $\mathcal{C}/A$.

An answer by Max to this question shows that when $\mathcal{C}$ has products, then an epimorphism in $\mathcal{C}/A$ is also an epimorphism in $\mathcal{C}$. See the little 'proposition' (yellow block) towards the end of his answer. Whether or not $\mathcal{C}$ is supposed to have products seems to be more of an issue with what exact definition is used, as is apparent from the comments on this answer.

Another way is to note that the forgetful functor $\mathcal{C} / A \to \mathcal{C}$ has $(-) \times A: \mathcal{C} \to \mathcal{C} / A$ as right adjoint. So in particular, the forgetful functor will preserve epimorphisms.

As mentioned in the comments to Mark's answer, this is a proof of the claim in the question with the assumption that $C$ has products and pullbacks (and no assumptions of (local) cartesian closedness). I don't know if that helps, because as they stand the hypotheses of the question do not (seem to - at least to me - EDIT : apparently Balaji Krishna assumes products in their definition of local cartesian closedness, so in this case it is sufficient) imply the existence of products.

So let $f:A\to B$ be such that $f^* : C/B\to C/A$ has a right adjoint, in particular it preserves pushouts and thus epis (epis in $C/B$, of course)

Now let $g:D\to B$ be an epi, we wish to show that $f^*g : D\to A$ is epi in $C$. To avoid confusion, I will write objects of the comma categories as couples.

Note that $g$ can also be seen as a morphism $(D,g)\to (B,id_B)$, and obviously this morphism is epi in $C/B$. Therefore, applying $f^*$ to it is still epi. Now you can check that $f^*$ of the morphism $g$ is actually still $f^*g$ (for the simple reason that $f^*(B,id_B) = (A,id_A)$ and that $f^*$(morphism $g$) is a morphism from $f^*(D,g)$ to $f^*(B,id_B)=(A,id_A)$.

Therefore $f^*g$ is epi in $C/A$, when it's seen as a morphism to $(A,id_A)$. Now we are reduced to the following claim :

Let $C$ be a category with products, and assume $k : (D,g) \to (E,h)$ is epi in $C/A$. Then $k:D\to E$ is epi.

(I am too lazy to draw the diagrams on here, but it's much better if you do that)

Proof : Let $l_1,l_2 : E\to T$ be arrows in $C$ such that $l_1\circ k = l_2\circ k$; and consider the object $(T\times E, h\circ \pi_E)$ of $C/A$. Then we have, for $i\in\{1,2\}$, a map $\tilde{l_i} : (E,h)\to (T\times E, h\circ \pi_E)$ : on $T$ it's $l_i$ and on $E$ it's $id_E$. It is very easy to see that it's a map of $C/A$, moreover one easily checks that $\tilde{l_1}\circ k = \tilde{l_2}\circ k$, so that by epi-ness of $k$ in $C/A$, $\tilde{l_1}=\tilde{l_2}$.

Now look at the $T$ coordinate, you get $l_1=l_2$, so $k$ is epi in $C$.

It follows that $f^*g$ is epi in $C$ and we are done.