Probability sum of 5 before sum of 7
A simple approach would be to "split the world" into two events:
- The event in which a sum of $5$ appears before a sum of $7$
- The event in which a sum of $7$ appears before a sum of $5$
Since these are complementary events, the probability that either one of them will occur is $1$.
Now, the probability of a sum of $5$ is $\color\red{1/9}$, and the probability of a sum of $7$ is $\color\green{1/6}$.
Therefore:
- The probability that a sum of $5$ appears before a sum of $7$ is $\frac{\color\red{1/9}}{\color\red{1/9}+\color\green{1/6}}=\frac25$
- The probability that a sum of $7$ appears before a sum of $5$ is $\frac{\color\green{1/6}}{\color\red{1/9}+\color\green{1/6}}=\frac35$
I do not give an answer to your question(s) (yet), but hand over a different approach.
Probability corresponding with sum $5$ is $\frac{4}{36}$ and probability corresponding with sum $7$ is $\frac{6}{36}$.
Neglecting the other outcomes we find a probability $\frac{4}{10}$ for sum $5$ and $\frac{6}{10}$ for sum $7$.
If on base of these probabilities a choice is made then the probability that sum $5$ is chosen (i.e. comes before sum $7$) is: $$\frac25$$
edit:
Your first approach is wrong and your second is correct (but tedious). See the answer of Micapps.
Let $E_{k}$ denotes the event that at the $k$-roll for the first time a sum of $5$ or $7$ appears.
Then more formally we have: $$P\left(\text{sum }5\text{ appears first}\right)=\sum_{k=1}^{\infty}P\left(\text{sum }5\text{ appears first}\mid E_{k}\right)P\left(E_{k}\right)=$$$$\sum_{k=1}^{\infty}P\left(\text{sum }5\text{ appears at roll }k\mid E_{k}\right)P\left(E_{k}\right)=\sum_{k=1}^{\infty}\frac{2}{5}P\left(E_{k}\right)=\frac{2}{5}$$
In your first approach the events are not disjoint: For example the event where the first two rolls are $5$ is contained in both of the first two events.
In the second approach the events are disjoint and their union is the event that $5$ appeared before $7$, so this is the way to go.
Indidentally, if we can assume the throws are all independent, a neat trick in solving this sort of problem is as follows: Let $p$ be the probability that $5$ appears before $7$. By the law of complete probability we have:
$p = \Pr[5\text{ appears on first roll}] + \Pr[\text{first roll is not }5\text{ or }7]\cdot\Pr[5\text{ appears before }7\ |\text{ first roll is not }5\text{ or }7]$
Now, because of the self-similar nature of the problem we get:
$\Pr[5\text{ appears before }7\ |\text{ first roll is not }5\text{ or }7]=p$
Plugging numbers in the first equation we get:
$$p=\frac{4}{36}+\frac{26}{36}p\implies\frac{5}{18}p=\frac{1}{9}\implies p=\frac{2}{5}$$
Regarding "mathematical formulation" of the second approach: The way you've presented the problem by defining the events $A_n,B_n$ is correct as far as I can tell, in the sense that it encodes the intuition given at the start of the question.
Regarding independence: In order to move from the probability of an intersection of events to the product of their probabilities in general you need independence. You may be able to get away with something "weaker" (which you would have to specify), but I have no clue what the "weakest" assumption would be.