Prove that $1 \leq A \leq \frac{5}{4}$ and $0 \leq B < \frac{81}{16}$

Let $a,b,c,d$ be the roots of $P$.

First we show the bound of $A$. By Rolle's theorem, we know that $P'(x)=4x^3-9x^2+6x-A$ has at least $3$ roots.

At the same time, $P''(x)=12x^2-18x+6=6(x-1)(2x-1)$

We can easily see that $P'(x)$ is increasing in $]- \infty,\frac1 2]$ decreasing in $[\frac1 2,1]$ and increasing in $[1, +\infty[$.

However, these variations imply that $P'$ has a root in $]- \infty,\frac1 2]$, we call it $\alpha$ (because $\lim_{x \to -\infty} P'(x)=-\infty$,$\lim_{x \to +\infty} P'(x)=+\infty$ and $P'$ must cross at least $3$ times the $x$-axis)

Then $P'(\frac1 2)\geqslant P'(\alpha)=0 \implies \frac 5 4-A \geqslant 0$, hence $A \leqslant \frac 5 4$ as $P'$ increasing here.

Secondly, we prove the other bound in a different way,using Vieta's formula as $B$ can easily be expressed this way.

We have $a+b+c+d=3$ and $ab+ac+ad+bc+bd+cd=3$ and $B=abcd$.

The bound we want to show is $\left(\frac 3 2 \right)^4$, so if we can show that all roots are positive roots less than $\frac 3 2$, this will conclude.

Thus we now see that $(a+b+c+d)^2-2(ab+ac+ad+bc+bd+cd)=a^2+b^2+c^2+d^2=3$

To deal with this sum, we use Cauchy-Schwarz inequality to bound the roots:

$3-a^2=b^2+c^2+d^2 \ge \frac{(b+c+d)^2}3=\frac{(3-a)^2}3$.

But, if $f(x)=3-x^2-\frac{(3-x)^2}3$ is a quadratic function with is positive if $x\in [0,\frac 3 2]$ This leads to the bound we wanted.

(I didn't see your edit while writing...)

Denote the roots by $r_1,r_2,r_3,r_4$, and let $e_i$ denote the $i^{\text{th}}$ elementary symmetric polynomial on the roots. By Vieta's formula, we have $$e_1=3,\quad e_2=3,\quad e_3=A,\quad e_4=B.$$ We may then maximize and minimize $e_3$ subject to the above conditions on $e_1$ and $e_2$ using Lagrange multipliers, for example, to find $$1\leq A\leq \frac{5}{4}\quad \textrm{and}\quad 0\leq B\leq \frac{3}{16}.$$ (Note my upper bound for $B$ is significantly tighter than the one you propose.) The minimum values are obtained when $$P(x)=x(x-1)^3=x^4-3x^3+3x^2-x,$$ and the maximum values when $$P(x)=\left(x-\frac{3}{2}\right)\left(x-\frac{1}{2}\right)^3=x^4-3x^3+3x^2-\frac{5}{4}x+\frac{3}{16}.$$

Another way to get estimations for $B$.

Let $a$, $b$, $c$ and $d$ are roots.

Thus, $$3=ab+ac+bc+ad+bd+cd\leq\frac{(a+b+c)^2}{3}+d(3-d)=\frac{(3-d)^2}{3}+d(3-d),$$ which gives $$2d^2-3d\leq0$$ or $$0\leq d\leq\frac{3}{2},$$ which gives $$0\leq abcd\leq\frac{81}{16}$$ and $$0\leq B\leq\frac{81}{16}.$$ The right equality does not occur because if $a=b=c=d=\frac{3}{2}$, so $a+b+c+d=6,$ which is a contradiction,

which says $B<\frac{81}{16}.$