Jacobian for a Cartesian to Polar-Coordinate Transformation

The problem is the wrong usage of things like $dx$ and $dy$. People once worked with them as "infinitesimals", but the problem is just that, you can get into confusion pretty quickly. The true rigorous $dx$ and $dy$ are differential forms. They are functions that assign to each point of space one object called alternating tensor. For simplicity, one can consider a tensor to be a multilinear function of vectors, i.e. a function that takes various vectors as parameters, returns numbers and is linear in each parameter with the others held fixed.

The alternating character has to do also with the product of such objects, called the wedge product. This product is such that $dx\wedge dy = -dy\wedge dx$ for example. In your case this is sufficent to establish the fact.

Indeed, the first part of computations is correct:

$$dx = \cos \theta dr-r\sin\theta d\theta,$$

$$dy=\sin\theta dr+r\cos\theta d\theta,$$

now we have

$$dx\wedge dy=(\cos\theta dr-r\sin\theta d\theta)\wedge(\sin\theta dr+r\cos\theta d\theta),$$

but this product is distributive, so that we have

$$dx\wedge dy=(\cos\theta dr)\wedge(\sin\theta dr)+(\cos\theta dr)\wedge(r\cos\theta d\theta)+(-r\sin\theta d\theta)\wedge(\sin\theta dr)+(-r\sin\theta d\theta)\wedge(r\cos\theta d\theta),$$

also scalars can be put outside, so that

$$dx\wedge dy = (\cos\theta\sin\theta)dr\wedge dr+(r\cos^2\theta)(dr\wedge d\theta)-(r\sin^2\theta)d\theta\wedge dr-(r^2\sin\theta\cos\theta)d\theta\wedge d\theta$$

Now, any 1-form $\omega$ satisfies $\omega\wedge \omega = 0$, this is because the alternating property grants that $\omega\wedge\omega=-\omega\wedge\omega$ and so this follows. Because of that, $dr\wedge dr = 0$ and $d\theta\wedge d\theta = 0$. Finally we have

$$dx\wedge dy =r\cos^2\theta dr\wedge d\theta - r\sin^2\theta d\theta\wedge dr,$$

And finally using again the alternating property $-d\theta\wedge dr = dr\wedge d\theta$ and so

$$dx\wedge dy = r\cos^2\theta dr\wedge d\theta + r\sin^2\theta dr\wedge d\theta = r dr\wedge d\theta.$$

Of course, it's not possible to explain everything of differential forms in this single answer, just to show a little of how this fits in your problem. To see more on this, look at Spivak's Calculus on Manifolds (this one is a heavy book), or take a look at "Elementary Differential Geometry" by O'neill, this one has a good introduction to differential forms.