Why Mathematicians and Physicists Approach Integration on Manifolds Differently?

I'd say volume forms are tensors, though I'm a mathematician so I'm not sure what a tensor means:)

The two chunks of integration you are describing are related in the following way (and note that this is just an outline): In general, you have the differential forms of degree $k$ (which you might call $(k,1)$ tensor fields or something similar), which is a relatively abstract object, and may be defined in a coordinate-free way, and we also have their integrals. If your manifold is orientable, there are also volume forms, which are simply defined as nonzero differentiable form of top degree. For instance, the manifold $\mathbb{R}^n$ has the form $dx_{1}\wedge\dots\wedge dx_{n}$. (Their existance is equivalent to orientability). Given a volume form, its integral over your manifold is called the volume of the manifold. Try to convince yourself of this in the case where your manifold is some open bounded set in $\mathbb{R}^n$

Next, consider a metric $g$ on your orientable manifold. then one can construct a distinguished volume form using this metric. The idea is to use the metric to specify orthonormal bases, which have nice properties. For instance the form on $\mathbb{R}^n$ from above is precisely the form obtained from the standard metric on $\mathbb{R}^n$.

Finally, in a chart of your manifold with coordinates $x_1,...,x_n$, you can calculate the volume form, and it turns out to be $\sqrt{|g|}dx_{1}\wedge\dots\wedge dx_{n}$.

I don't think that this is an issue between mathematics and physics. Both concepts you refer to are available in some places in the mathematics literature, but dsicussing them is not so common. The main notion needed here it the one of a density. This is a type of geometric objects (like functions,vector fields and $k$-forms) that is availabla on any manifold. In a physics approach you can define densities as being described by a single function in coordinates but with a transformation under changes of coordinates which is different from standard functions. This transformation law is basicallz given the absolute value of the Jacobian matrix of the coordinate change. There also is an approach to define them as sections of an appropraite associate bundle to the frame bundle of the manifold. However, the behavior under coordinate changes is the main issue for what follows.

Together with the transformation law for multiple integrals, this behavior under coordinate change implies a basic statement on coordinate invariant integration. Technically, this applies to densities which are identically zero outside of a compact subset contained in a chart. For those, you can integrate the cooridnate representations and the result is independent of the choice of coordinates. From there, it is just a technical game (using partitions of unity, etc.) to get a well defined integral for general densities. The problem is that this is not (yet) related to other geometric objects, so you don't know how this concept of integration interacts with the rest of differential geometry.

One way how it does interact is via pseudo-Riemannian geometry. Given a pseudo-Reimannian metric, described by an invertible symmetric matrix $g$ in coordinates, the expression $\sqrt{\det(g)}$ turns out to define a density, the volume density of $g$. Multiplying a density by a function again gives a density, so now you can define an integral of smooth functions by actually intgrating their product with the volume density. Now if you apply this to classical integration theory on $\mathbb R^n$, you see that you can reasonably interpret the symbol $dx$ or $dx^1\dots dx^n$ that occurs there as denoting the volume density on $\mathbb R^n$ and this seems to be the approach taken in Carrol's book. It is important to observe that the resulting integral is NOT invariant under diffeomorphisms of $\mathbb R^n$ but only under isometries of the metric you use to define the volume form (so these are rigid motions or Lorentz transformations).

The other interaction with differential geometry is via forms, but this also needs a restriction. The main observation here is that $n$-forms on manifolds of dimension $n$ behave in almost exactly the same way as densities: In a chart, they are described by a single function, whose transformation law involves the determinant of the Jacobian matrix of the coordinate change. So the only difference to densities is that the absolute value is missing. This can be by-passed by putting an additional structure on your manifold, called an orientation. This is not aways possible, but only on manifolds which are orientable (but mostly this is not too big a problem). This leads to a stronger compatibility condition between charts, which implies that all chart changes have Jacobians with positive determinant. Restricting to this setting $n$-forms can be identified with densities and thus be intergrated as above. The resulting integral is diffeomorphism invariant and also connects to the calculus of differntial forms, say via Stoke's theorem. Also you can then integrate forms of smaller degrees over (oriented) submanifolds. Since this is much more flexible and general, this is the version of integration that is usually taught. In this version any $n$-form on $\mathbb R^n$ can be written as $fdx^1\wedge \dots\wedge dx^n$, and this should be the approach discussed in Fortney's book. Inspired by this approach to integration, it has become common to also use the notation $dx^1\wedge \dots\wedge dx^n$ in integration theory on $\mathbb R^n$, which explains Caroll's notion. Probably $|dx^1\wedge \dots\wedge dx^n|$ would be more conceptual here ...