how to factorize $a^3 - ab^3 + a^2 + b^2 + 1$

The part they don't show in olympiad solutions is the fifteen minutes of pure trial and error, testing different terms to add and subtract to see if anything works. They all demonstrate it as if it's something one ought to see right away.

Maybe, with lots of experience, one can spot it right away, or at least know which ones are more likely to work, but I don't think there is a general rule one can apply to find the relevant term.

Since this is a degree-$4$ polynomial, it's natural to try writing it as a product of two degree-$2$ polynomials, so the $-ab^3$ term needs to come from a $-ab/k$ multiplied with a $kb^2$, and we may as well take $k=1$. Applying the same logic to the $a^3b$ term (my earlier comment implies $a^3$ is meant to read $a^3b$, or we'd never get this factorization), we try $(ca^2-ab+d)(b^2+ab/c+1/d)$. The $a^2$ coefficient forces $c=d$; the $b^2$ coefficient forces $c=1$, determining the entire factorisation.

In reality, there is no actual method to always determine what terms to add. The best thing to do is to just try different combinations and memorize certain patterns that come up more often.

For more factorization like this, best to look through olympiad papers and olympiad lecture notes.