Double Integral $\int\limits_0^b\int\limits_0^x\sqrt{a^2-x^2-y^2}\,dy\,dx$

Of course this is an exercise in integration. But nevertheless it can be done using elementary geometry alone. We are told to compute the volume of a body $K$ which is bounded by several planar faces and a peace of a spherical surface. The following figure shows the situation as seen from the tip of the $x$-axis. I have put $a=1$ and written $p$ instead of $b$.

The body $K$ contains (a) the pyramid $P$ with base the triangle with vertices $(0,0,0)$,$(p,0,0)$, $(p,p,0)$, and of height $\sqrt{1-2p^2}$. The volume of this pyramid is $${\rm vol}(P)={1\over6}p^2\sqrt{1-2p^2}\ .$$ Furthermore $K$ contains (b) part of a sector $S$ of central angle $$\alpha:=\arctan{p\over\sqrt{1-2p^2}}$$ of a spherical segment. The two radii of this segment are $1$ and $\sqrt{1-p^2}$, and its thickness is $p$. The volume of the full sector $S$ is therefore given by $${\rm vol}(S)={\alpha\over 2\pi}{\pi\over6} p\bigl(3+3(1-p^2) +p^2\bigr)={\alpha\over6}p(3-p^2)\ .$$ From the volume of $S$ we (c) have to deduct the volume of a triangular spherical sector $T$, whereby the angles of the spherical triangle in question (shaded in the figure) are ${\pi\over2}$, ${\pi\over4}$ and a certain $\beta$. One leg of this triangle is $\alpha$, and a standard formula for right spherical triangles then tells us that $$\cos\beta=\sin{\pi\over4}\cos\alpha={1\over\sqrt{2}}\cos\alpha\ .$$ The spherical area of the triangle is then $\beta-{\pi\over 4}$, so that $${\rm vol}(T)={1\over3}\bigl(\beta-{\pi\over 4}\bigr)\ .$$ Finally $${\rm vol}(K)={\rm vol}(P)+{\rm vol}(S)-{\rm vol}(T)\ ,$$ which maybe can be simplified somewhat.

Suppose $$I=\int_0^b\int_0^x\sqrt{a^2-x^2-y^2}dy\,dx$$ Let $y=\sqrt{a^2-x^2}\sin \theta$ then $dy=\sqrt{a^2-x^2}\cos \theta\,d\theta$, so $$I=\int_0^b\int_0^{\arcsin\frac{x}{\sqrt{a^2-x^2}}}(a^2-x^2)\cos^2 \theta\,d\theta\,dx$$ $$=\frac{1}{2}\int_0^b(a^2-x^2)\left[\theta+\frac{1}{2}\sin 2\theta\right]_0^{\arcsin\frac{x}{\sqrt{a^2-x^2}}}\,dx$$ and now note that $\sin \theta=\frac{x}{\sqrt{a^2-x^2}}$, so $\cos \theta=\sqrt{\frac{a^2-2x^2}{a^2-x^2}}$ and $\frac{1}{2}\sin 2\theta=\sin \theta\cos \theta=\frac{x\sqrt{a^2-2x^2}}{a^2-x^2}$. therefore $$I=\frac{1}{2}\left(\int_0^b(a^2-x^2)\arcsin\frac{x}{\sqrt{a^2-x^2}}\,dx+\int_0^bx\sqrt{a^2-2x^2}\,dx\right)=\frac{1}{2}(I_1+I_2).$$ for evaluating $I_1$ use integrating by parts. If you let $u=\arcsin\frac{x}{\sqrt{a^2-x^2}}$ and $dv=(a^2-x^2)dx$, then $$du=\frac{a^2}{(a^2-x^2)\sqrt{a^2-2x^2}}dx,v=a^2x-\frac{x^3}{3}$$ therefore $$I_1=\frac{3a^2b-b^3}{3}\arcsin\frac{b}{\sqrt{a^2-b^2}}+I_3$$ and $$I_3=-\int_0^b\frac{a^2(a^2-\frac{x^2}{3})x}{(a^2-x^2)\sqrt{a^2-2x^2}}\,dx$$ now let $u=\sqrt{a^2-2x^2}$, then $du=\frac{-2x}{\sqrt{a^2-2x^2}}dx$ and $x^2=\frac{a^2-u^2}{2}$. so $$I_3=\frac{a^2}{2}\int_a^{\sqrt{a^2-2b^2}}\frac{a^2-\frac{a^2-u^2}{6}}{a^2-\frac{a^2-u^2}{2}}\,du=\frac{a^2}{6}\int_a^{\sqrt{a^2-2b^2}}\frac{u^2+5a^2}{u^2+a^2}\,du$$ $$=\frac{a^2}{6}\int_a^{\sqrt{a^2-2b^2}}\left(1+\frac{a^2}{u^2+a^2}\right)\,du=\frac{a^2}{6}(\sqrt{a^2-2b^2}-a)+\frac{2a^3}{3}\left(\arctan\frac{\sqrt{a^2-2b^2}}{a}-\frac{\pi}{4}\right)$$ for $I_2$ we have $$I_2=\int_0^bx\sqrt{a^2-2x^2}\,dx=\frac{-1}{6}\left[(a^2-2x^2)\sqrt{a^2-2x^2}\right]_0^b=\frac{1}{6}(a^3-(a^2-2b^2)\sqrt{a^2-2b^2})$$ Hence,

$$I=\frac{3a^2b-b^3}{6}\arcsin\frac{b}{\sqrt{a^2-b^2}}+\frac{a^3}{3}\arctan\frac{\sqrt{a^2-2b^2}-a}{\sqrt{a^2-2b^2}+a}+\frac{b^2}{6}\sqrt{a^2-2b^2}.$$

$\int_0^b\int_0^x\sqrt{a^2-x^2-y^2}~dy~dx$

$=\int_0^b\left[\dfrac{y\sqrt{a^2-x^2-y^2}}{2}+\dfrac{a^2-x^2}{2}\sin^{-1}\dfrac{y}{\sqrt{a^2-x^2}}\right]_0^x~dx$ (according to http://en.wikipedia.org/wiki/List_of_integrals_of_irrational_functions)

$=\int_0^b\dfrac{x\sqrt{a^2-2x^2}}{2}dx+\int_0^b\dfrac{a^2-x^2}{2}\sin^{-1}\dfrac{x}{\sqrt{a^2-x^2}}dx$

Can you take it from here?