Gregory-Laflamme Instability of Black Strings and $p$-Branes
They are looking at s-wave solutions, ie zero angular momentum. They are also looking at modes that are pure tensor with respect to the Schwarzchild geometry (ie they set the vector and scalar parts to zero). So the angular parts of the perturbation are just proportional to the angular parts of the background metric (in a sense the angular parts are proportional to the identity, because the angular momentum is zero). The interesting parts of the perturbation then are only allowed to involve $r$ and $t$, and this is the most general form for the perturbation that mixes $r$ and $t$ but leaves the angular parts alone.
The instability comes from the fact that there are solutions that are regular at infinity that blow up at the horizon. See the discussion after equation 11.
There are many tricks to find instabilities. The methods used in the Gregory-Laflamme paper are certainly valid, but their methods are by no means the only nor the most common ones. Generally speaking, the question is whether the split into background + perturbations is a good one in the sense that the perturbations remain small. The perturbations might fail to remain small for many reasons--you might have a runaway potential that becomes infinitely negative (hamiltonian unbounded below) or at least pushes you away from your chosen background (tachyon instability), you might have negative kinetic energy (ghost instability), you might have negative gradient energy (gradient instability)--there are lots of different ways for instabilities to manifest themselves. Here, the method is essentially brute force: numerically construct solutions and show explicitly that there are valid solutions to the perturbation equations where the perturbations become infinitely large.
The answer accepted here is, in fact, incorrect in its comments in part 2. This is not at all how the instability arises.
The instability comes from the fact that the solution is of the form $e^{\Omega t}e^{i\omega z}H_{\mu\nu}(r)$ with $\Omega\in \mathbb{R}_+$, i.e. exponentially growing. They numerically solve the decoupled ODEs for $H_{\mu\nu}(r)$ to find a solution that is regular at the horizon and regular at infinity. This is a subtle point: Schwarzschild coordinates do NOT cover the horizon, hence one needs to examine the perturbation at the horizon in coordinates that extend across it, for example, Ingoing Eddington Finkelstein. Now, this allows you to identify the right branch of asymptotic solution at the horizon in the singular Schwarzschild coordinates.