Prolog - simplify derivative

I would rather see this as two separate problems:

First, get derivation right (you're probably getting close, depending on your concrete requirements).

Then, work on simplifying expressions on an algebraic level. Exploit algebraic identities, see if applying the laws of commutativity / associativity / distributivity on some subexpressions enable their rewriting into something equivalent (but simpler / more compact).

As a starting point, you may want to look at the somewhat related question "Replacing parts of expression in prolog".

---

Here's a simplistic sketch how you could do the simplification—using iwhen/2 to safeguard against insufficient instantiation:

expr_simplified(A, B) :-
   iwhen(ground(A), xpr_simplr(A,B)).

xpr_simplr(A, B) :-
   (  atomic(A)
   -> A = B
   ;  ( A = X+0 ; A = 0+X ; A = 1*X ; A = X*1 )
   -> xpr_simplr(X, B)
   ;  ( A = 0*_ ; A = _*0 )
   -> B = 0
   ;  A = X+X
   -> B = X*2
   ;  A = X*X
   -> B = X**2
   ;  A = X**1
   -> B = X
   ;  A =.. [F|Xs0],                          % defaulty catch-all
      maplist(xpr_simplr, Xs0, Xs),
      B =.. [F|Xs]
   ).

Let's see what it does with the expression you gave. We apply expr_simplified/2 until we reach a fixed point:

?- A = ((1*x+x*1+(0*x+2*1)+0)*(3*x)-(x*x+2*x+3)*(0*x+3*1))/(3*x*(3*x)),
   expr_simplified(A,B),
   expr_simplified(B,C),
   expr_simplified(C,D).
A = ((1*x+x*1+(0*x+2*1)+0)*(3*x)-(x*x+2*x+3)*(0*x+3*1))/(3*x*(3*x)),
B = ((x+x+(0+2))*(3*x)-(x**2+2*x+3)*(0+3))/(3*x)**2,
C = ((x*2+2)*(3*x)-(x**2+2*x+3)*3)/(3*x)**2,
D = C.                                        % fixed point reached

As imperfect as the simplifier is, the expression got a lot more readable.