Compute the expression $(a^2 + 4b - 1)(b^2 + 4a - 1)$ without calculating the roots of $x^2 - x - 5 = 0$

Make repeated use of $a^2=a+5$, $b^2=b+5$, $a+b=1$, $ab=-5$.

This is an exercise in elementary symmetric polynomials. Write $s_1=a+b$ and $s_2=ab$. From the equation $$ x^2-x-5=(x-a)(x-b)=x^2-(a+b)x+ab=x^2-s_1x+s_2 $$ we can read that $s_1=1$ and $s_2=-5$.

The number that you wanted to know is $$ (a^2+4b-1)(b^2+4a-1)=a^2b^2+4(a^3+b^3)+16ab-4(a+b)-(a^2+b^2)+1. $$ We need to express the quantities in parenthesis in terms of $s_1$ and $s_2$. This is not too difficult, because from $$ s_1^2=a^2+2ab+b^2=(a^2+b^2)+2s_2 $$ we get $a^2+b^2=s_1^2-2s_2=11$. Similarly from $$ s_1^3=a^3+3a^2b+3ab^2+b^3=(a^3+b^3)+3ab(a+b) $$ we get that $a^3+b^3=s_1^3-3s_1s_2=16.$

Putting all this together gives $$ (a^2+4b-1)(b^2+4a-1)=25+64-80-4-11+1=-5. $$

What makes this tick is that $(a^2+4b-1)(b^2+4a-1)$ is symmetric in the unknowns $a$ and $b$. IOW if you swap the values of $a$ and $b$ nothing will change. Such polynomial functions can always be written in terms of the elementary symmetric polynomials $s_1$ and $s_2$. This result can be generalized to several unknowns. The buzzword "elementary/basic symmetric polynomials" should give you enough material. The power sums such as $a^3+b^3$ are a well-known special case. The buzzword "Newton's identities" helps you there.

Hint:

$$ a^2 + 4b - 1 = a^2 - a - 5 + 4b + a + 4 = 4b + a + 4 $$