How to find center of an arc given start point, end point, radius, and arc direction?

Don't write down too many equations to be solved, but produce the desired center ${\bf c}=(a,b)$ in a forward movement instead. Let ${\bf z}_i=(x_i,y_i)$ $\ (i=0,1)$ be the two given points, put $\epsilon:=1$ if the arc should go from ${\bf z}_0$ to ${\bf z}_1$ counterclockwise, and put $\epsilon:=-1$ otherwise.

Next, let $d:=|{\bf z_1}-{\bf z_0}|=\sqrt{(x_1-x_0)^2+(y_1-y_0)^2}$ be the distance and ${\bf m}:=\bigl({x_0+x_1 \over2}, {y_0+y_1\over2}\bigr)$ be the midpoint of ${\bf z_0}$ and ${\bf z_1}$. Then

$${\bf n}:=(u,v):=\Bigl({x_1-x_0\over d},{y_1-y_0\over d}\Bigr)$$

is the unit normal in the direction ${\bf z_1}-{\bf z}_0$, and ${\bf n}^*:=(-v,u)$ is the unit vector you get by rotating ${\bf n}$ counterclockwise by $90^\circ$.

Given $r>0$ the center ${\bf c}$ has a distance $h:=\sqrt{r^2 -d^2/4}$ from ${\bf m}$, and the given $\epsilon$ together with ${\bf n}^*$ tell us in which direction we should go. In vectorial notation the center is given by

$${\bf c}\ =\ {\bf m}+\epsilon\ h\ {\bf n}^*\ ,$$

so that coordinate-wise we get

$$a={x_0+x_1 \over2}-\epsilon\ h\ v, \qquad b={y_0+y_1 \over2}+\epsilon\ h\ u\ .$$

To be explicit, your equations are,

$$r^2=(x-x_0)^2+(y-y_0)^2 \\ r^2=(x-x_1)^2+(y-y_1)^2$$

Subtracting them gives the bisector line:

$$0=2x(x_1-x_0)+x_0^2-x_1^2+2y(y_1-y_0)+y_0^2-y_1^2$$

You can solve any two of these simultaneously to get the two candidate centers.

Let's try a geometric approach:

We are given the points $A$ and $B$ on the circle and the radius $r$ of the circle.

I'll do it for a specific case:

Suppose $A$ and $B$ are in the first quadrant with $A$ "to the left and above" $B$.

Let $D$ be the midpoint of the line segment $\overline{AB}$ and set $\gamma$ equal to the acute angle formed by $\overline{AB}$ and the horizontal line $l$ through $A$.

Let $V=\overline{DC}$ where $C$ is the center of the circle.

Then:

1) The length of $V$ is $\sqrt{r^2-{{|\overline{AB}|^2\over 4 }}}$.

2) The acute angle formed by the horizontal line through $D$ and $\overline{DC}$ is ${\pi\over 2}-\gamma$.

Since you can compute the coordinates of $D$ and $\gamma$ from the coordinates of $A$ and $B$, you can find the coordinates of $C$ using the information in 1) and 2).