The Three-Cornered Duel

You and Mosteller are calculating slightly different quantities.

Let's look at what you are calculating for case 1:

You write $P(\text{A survives in case A shots B})$ which means $P\text{(A survives given A kills B}) \equiv P(\text{A survives|A kills B})$

But in the right hand side of your equation you are calculating $P(\text{A survives}\ \cap\ \text{A kills B})$ which is equal to $P(\text{A survives|A kills B})\cdot P(\text{A kills B}) = \frac{3}{13}\cdot 0.3$

So you are calculating the intersection of two events (event 1 and event 2) instead of the conditional (event 1 given event 2). Note that if you use the right name/description for the probability you are calculating then your calculations are agreeing with Mosteller.

The same goes for case 2. You are calculating $P(\text{A survives}\ \cap \ \text{A shoots at B and misses})$

Now that we have named the probabilities more accurately, you can ask yourself: How do they help me in answering the question? Isn't it more useful if I know the conditional probabilities instead? Yes it is. This is what will help you decide on the optimal action for A.

So as Monteller says: If A has to pick a target, he has to go for B. If A misses B, then B kills C on the next round, and on the round after that A has a single chance at B. So A survives with probability $0.3$. If A kills B then we find that the probability of surviving a duel with C (where C goes first) is $\frac{3}{13}$. So we see it's better to miss. And indeed A can choose to deliberately miss, and this is what he should do.

Finally here's another way to find A's survival probability against a duel with C. Let's define two probabilities, $P_A$ and $P_C$: $$ P_A \equiv P(\text{A shoots first and A survives at the end}) \\ P_C \equiv P(\text{C shoots first and C survives at the end}) $$

Now note the relationship between the two, that creates a $2\times 2$ system that can be easily solved: $$ \left. \begin{array}{} P_A = 0.3 + 0.7\cdot (1-P_C)\\ P_C = 0.5 + 0.5\cdot (1-P_A) \end{array} \right\} \iff \begin{array}{} P_A = \frac{6}{13}\\ P_C = \frac{10}{13} \end{array} $$

What we care about in our scenario is person A surviving given that person C goes first, which is equal to $1-P_C = \frac{3}{13}$