Factoring quadratics - how to?
When expanding any factored expression of the form $(x+a)(x+b)$ it is useful to remember the FOIL rule, meaning, first-outer-inner-last, specifying all the terms that need to be multiplied to get the expansion. That is,
First: $x\cdot x = x^2$
Outer: $x\cdot b = bx$
Inner: $a\cdot x = ax$
Last: $a\cdot b = ab$
Now sum, to obtain $(x+a)(x+b) = x^2 + bx+ax+ab = x^2 + (a+b)x + ab$. This is the rationale behind factoring an expression of the form $x^2 + cx + d$, we want to find the $a$ and $b$ values in $(x+a)(x+b)$ such that $a+b=c$ and $ab = d$.
In your example, we are just doing that independently for the numerator and the denominator to get both of them in factored form. Then we can cancel the common terms and obtain the desired answer.
Response to update:
The quadratic formula is a fool-proof way of obtaining the roots of an expression of the form $ax^2+bx+c$ (roots meaning the values of $x$ where $ax^2+bx+c=0$). It's a useful equation that is worth committing to memory. With practice, you will be able to see what the roots of an equation are without applying the formula, but if the roots aren't nice numbers, the quadratic equation is a good way of finding them.
In finding the roots of the numerator, most of it is right. One small error is you wrote $3^2-4a10$, when in fact, it should be $3^2-(4)(-10) = 3^2+40 = 49$. You've written it down correctly two lines down though!
For the roots of the denominator, you are right, except for the last line, the roots are $x=(6+4)/2=5$ and $x=(6-4)/2=1$. Remember, roots are the points that make the equation equal to zero, so $(x-5)(x-1)=0$ only when $x=5$ or $x=1$.
With that you have arrived at your answer since the root $x=5$ is common between the numerator and denominator and thus the $(x-5)$ terms cancel.
Let me know if you want me to explain any steps further!