The center of the circumcircle lies on a side of a triangle

Observe that $\angle OPA=\angle PAO=\frac\alpha2=\angle BAP\implies OP\parallel AB$. Thus $$\frac{CO}{OA}=\frac{CP}{PB}\iff \frac dR=\frac{20}{16}=\frac54$$ Also, Power of a point yields $$\begin{align*}\text{Pow}(C)_{(APB)}=\lvert d^2-R^2\rvert&=20\cdot 36\\\iff \left\lvert\left(\frac54R\right)^2-R^2\right\rvert&=720\\\iff \frac9{16}R^2&=720\\\iff R&= 16\sqrt{5} \end{align*}$$ Use $OP\parallel AB$ again in order to infer

$$AB=R\cdot \frac{36}{20}=16\sqrt5\cdot \frac95=\frac{144\sqrt5}{5}$$

$\angle POC = \angle A$, so $\triangle COP \sim \triangle CAB$

Hence $\frac{R}{AB} = \frac{20}{36} = \frac{5}{9}$ ...(i)

$\angle APB = \frac{1}{2} \angle AOB = 90^0 - \angle A$

$AB = 2R \sin \angle APB = 2R \cos A$

So from (i), $\frac{R}{2R \cos A} = \frac{5}{9}$

$\cos A = \frac{9}{10} = 1 - 2 \sin^2{\frac{A}{2}} \implies \sin \frac{A}{2} = \frac{1}{2\sqrt5} $

$16 = 2R \sin \frac{A}{2} \implies R = 16\sqrt5 $

$AB = 2R \cos A = 32\sqrt5 \times \frac{9}{10} = \frac{144}{\sqrt5}$

Alternative solution using neither trigonometry nor power of a point:

You already figured out that $OP \parallel AB$, so draw two perpendicular lines $OQ$ and $BR$.

Let the radii $OA = OB = OP = 10x$.

Since $\triangle ABC \sim \triangle OPC$, we have $AB = \frac{20+16}{20} \cdot OP = 18x$, so $AQ = BQ = 9x$.

Therefore, $OQ = \sqrt{OA^2 - AQ^2} = \sqrt{19} x$, so $BR = \sqrt{19} x$.

$RP = OP - OR = 10x - 9x = x$.

$BP = \sqrt{BR^2 + RP^2} = 2 \sqrt{5} x$.

Given that $BP = 16$, we conclude that $x = \frac{16}{2\sqrt{5}} = \frac85 \sqrt{5}$.

Finally, $AB = 18 \cdot \frac85 \sqrt{5} = \frac{144}{5} \sqrt{5}$.