Help evaluating triple integral over tetrahedron

To get an idea what to do in 3D, try to understand the 2D case first and try to generalize. When integrating over the triangle with vertices $(0,0)$, $(0,1)$ and $(1,0)$, it is often a good idea to first let $x$ go from zero to $1-y$ and then let $y$ go from zero to 1.

In your case, you can proceed analogously: let $x$ range in $[0,1-y-z]$, then $y$ in $[0,1-z]$ and finally $z$ in $[0,1]$. That is, $$ \int_{\text{tetrahedron}}f(x,y,z)dxdydz = \int_0^1\int_0^{1-z}\int_0^{1-y-z}f(x,y,z)dxdydz. $$ In your case $f(x,y,z)=xyz$. To make the iterated integral structure clear, you can write it as $$ \int_0^1\left(\int_0^{1-z}\left(\int_0^{1-y-z}xyzdx\right)dy\right)dz $$ and start by doing the innermost integral. The first integral is $\int_0^{1-y-z}xyzdx=\frac12yz(1-y-z)^2$. If you get this right, you are on the right track.

I agree entirely with the accepted answers; that is how the integral should be attempted analytically. However, if you wish to compute integrals of this form over the unit tetrahedron efficiently, or even just check your answers, it is worth noting that there is an explicit formula for the result as a function of the powers.

$$\int_0^1 \int_0^{1-z} \int_0^{1-y-z} x^a y^b z^c \, dx \, dy \, dz = \frac{1}{3!} \frac{a! \, b! \, c!}{(a + b + c)!} \frac{1}{3 + a + b + c \choose 3} $$

See This mathoverflow question for proof and an extension to other dimensions (triangles, 4-D tetrahedra, etc...).

Using the facts that the projection of the solid in the xy-plane is the triangle with vertices (0,0), (0,1), and (1,0), and that the top of the solid is the plane $x+y+z=1$, we can set up the integral as

$\displaystyle\int_{T} f(x,y,z) \;dV =\int_{0}^{1}\int_{0}^{1-x}\int_{0}^{1-x-y} f(x,y,z) \;dz dy dx$.