Why is law of conservation of angular momentum (seemingly) being violated over here?

I think that this is similar to something we learn in gravitation quite commonly called a double star system.

In that 2 bodies of some mass rotate about their stationary COM. This is just the same case except one of the masses is extremely large.

It is known that $L_{p} = L_{com,p} + L_{system,com}$. Using this we can first prove that the angular momentum is conserved about COM, since the velocities of both the masses are constant. Also, we can say that $L_{com,p} = 0$ since the velocity of the centre of mass is zero in this case. Thus $L_{p} = L_{system,com}=$ contant.

Thus the angular momentum about P should be conserved.

I think the mistake made by you was that you did not consider the mass at the centre. Even though it has a small velocity, since it has a large mass it can make a difference.

One more thing I would like to add is that, since the force of gravity acts in a pair, the torque is $0$ about any axis. This further confirms conservation of angular momentum.

Hope this helps!

What am I doing wrong here that I am getting this conclusion?

The problem is that even if the inner mass is quite large then too it would have angular velocity and hence angular momentum.

Is it possible to show mathematically that angular momentum is conserved?

Yes it is. The following picture says most of it. (A description is coming ahead):

Description

The outer mass is at a distance $R_{out}$ from the com and has a linear momentum as $\mathbf p_{out}$. The inner mass has is a distance $R_{in}$ from the center and has linear momentum $\mathbf p_{in}$. Two things to note here:

1. The momentum of each particle isn't going to change because

   • the distance between the two objects is always the same (i.e., $R+R'$).

   • the force (here gravitational) is centripetal.

2. the centre of mass is always inbetween these two on the line joining these. Therefore the angular velocity, $\omega$, of both the particles is the same.

Now in the centre of mass reference frame:

$$\mathbf L_{total} = \mathbf R_{in} \times \mathbf p_{in} +\mathbf R_{out} \ \mathbf p_{out}$$

$$\mathbf L_{total} = (p_{in}R_{in}+ p_{out}R_{out}) \hat k$$

Here you can see that angular momentum of the system is constant with time so it is conserved. Note that $\hat k$ is vector pointing out of your screen perpendicular to the plane. Also using vectors would make things easier.

Also since position of centre of mass in this reference frame is at origin:

$$\Rightarrow R_{in}m_{in} + R_{out}m_{out}= 0$$

[This equation is going to be quite useful in the last part of derivation]

Now let's shift the axis to a point on the orbit of outer particle. The labels diagram is as follows:

Inner Particle

The position vector $\mathbf r_{in}(t)$ of the inner particle with shifted axis is:

$$\begin {align} \mathbf r _{in}(t) & = \mathbf {OB} - \mathbf {OA} \\ & = R _{in} \hat r - R_{out} \hat i \end {align}$$

And $\mathbf p = p_{in} \hat {\theta}$

$$\begin {align} \boldsymbol {\ell} _{in} & = \mathbf r _{in}(t) \times \mathbf p \\ & = (R_{in} \hat r - R_{out} \hat i) \times p_{in} \hat {\theta} \\ & = R_{in}p_{in} \hat k- R_{out}p_{in} (\hat i \times \hat {\theta}) \end {align}$$

$$\mathbf {\ell}_{in} = R_{in}p_{in} \hat k- R_{out}p_{in} (\hat i \times \hat {\theta}) \tag 1$$

Now carrying this process similarly for outer particle we get:

$$\boldsymbol {\ell}_{out} = R_{out}p_{out} \hat k- R_{out}p_{out} (\hat i \times \hat {\theta} ) \tag 2$$

Total Angular momentum $L_{total}'$ is given by:

$$\mathbf L_{total}' = \boldsymbol {\ell}_{out} + \boldsymbol {\ell}_{in}$$

$$\mathbf L_{total}' = (R_{out}p_{out} + R_{in}p_{in}) \hat k - R_out (p_{out} +p_{in} )(\hat i \times \hat {\theta})$$

Now

$$p_{out} +p_{in} = \omega R_{in}m_{in} + \omega R_{out}m_{out}$$ And using equation $(1)$ we get:

$$p_{out} +p_{in}=0$$

Therefore

$$\mathbf L_{total}' = (R_{out}p_{out} + R_{in}p_{in}) \hat k$$

Clearly this equation is constant therefore this implies that the angular momentum is conserved.

This problem is seriously over-contrived. Gravity is not necessary, the COM stuff is not required to get at the main concept.

The main concept is that angular momentum is not a vector, is a pseudo-vector. True vectors are independent of choice of coordinate origin, pseudo vectors: not so much.

Consider a bead on a hoop moving around at constant angular velocity: it at has an oscillating angular momentum about any fixed point on the hoop. There is a "central" (for the hoop) force at all times keeping the bead on the hoop, but that force is not central in the wonky coordinate system about the fixed point on the hoop: it applies a torque (also a pseudo-vector) that satisfies:

$$ {\bf{\tau}}=\frac{d{\bf{L}}}{dt}$$

Note that angular momentum and torque are also axial-vectors, but that has to do with their positive parity and the fact that, while they rotate as vectors, they are really antisymmetric rank-2 tensors.