What is the correct definition of localisation of a category?

Actually, both of these definitions look weird to me.

I would say there are two natural ways to define the localization $C[S^{-1}]$ by a universal property, as follows. For any category $D$, let ${\rm Fun}_S(C,D)$ denote the full subcategory of the functor category ${\rm Fun}(C,D)$ spanned by those functors that send the morphisms in $S$ to isomorphisms. Let $Q:C\to C[S^{-1}]$ be a functor that sends the morphisms in $S$ to isomorphisms; then there is an induced functor $(-\circ Q): {\rm Fun}(C[S^{-1}],D) \to {\rm Fun}_S(C,D)$. I would say that $Q$ is a strict localization if this functor $(-\circ Q)$ is an isomorphism of categories, and a weak localization if $(-\circ Q)$ is an equivalence of categories.

If we unravel that, then being a strict localization means that

1. For any functor $F:C\to D$ sending $S$-morphisms to isomorphisms, there exists a literally unique functor $G:C[S^{-1}]\to D$ such that $F = G\circ Q$, and
2. condition KS3.

while being a weak localization means that

1. For any functor $F:C\to D$ sending $S$-morphisms to isomorphisms, there exists some functor $G:C[S^{-1}]\to D$ such that $F \cong G\circ Q$ (isomorphic, not equal!), and
2. condition KS3.

The definition you quoted from KS lies somewhere in between these two: it asks only that $G$ exists rather than being unique, but it asks this $G$ to factor $F$ strictly. This corresponds to asking that the functor $(-\circ Q)$ be a surjective-on-objects equivalence, which is a rather odd condition. (The nlab gives the "weak localization" definition.

Note that being a strict localization is the same as the lemma you quoted from GZ, but that it is actually strictly stronger than their definition as you quoted it: saying that $(-\circ Q)$ is an isomorphism requires $G$ to be literally unique given $F$, not merely unique up to isomorphism. Moreover, the up-to-isomorphism GZ definition, as Simon says, is not "categorically correct", and should not be written as the definition of localization. (I believe that GZ were writing before questions such as the difference between isomorphism and equivalence of categories was widely appreciated, though, which may somewhat excuse this sloppiness.)

However, I believe that in this special case, it happens to be true that if a strict localization exists (which, in $\rm Cat$, it always does), then any GZ-localization is equivalent to this strict localization. For applying both universal properties yields functors back and forth comparing the two localizations, which commute strictly with the localization functors from $C$. Then the second clauses of the two universal properties imply that both composites of these functors are isomorphic to identities, hence form an equivalence of categories. So if you "only care about determining categories up to equivalence", then the two definitions are in fact equivalent — but if you only care about determining categories up to equivalence, then you should really be using the definition of weak localization, since it is the only one that's invariant under replacing everything by something equivalent.

Regarding your problem with showing uniqueness of the isomorphism, note that "unique up to unique isomorphism" doesn't mean that there is a unique isomorphism $G_1\cong G_2$ period, it means that there is a unique such isomorphism that commutes with all the other data. The latter condition removes any dependency on automorphisms of the input. For instance, a product object $A\times B$ is unique up to unique isomorphism, even though any automorphism of $A$ or $B$ induces an automorphism of $A\times B$: such nontrivial automorphisms don't commute with the projection maps. In the case of localization, the weak localization condition should be enough to show that there is a unique isomorphism that commutes with the other data.

See Chapter 6 of the book Derived Categories for various notions of localization, including the Ore conditions. (Also available at the arXiv at https://arxiv.org/abs/1610.09640.)

Here's a sample of the type of stuff in loc. cit. It captures the 2-categorical nature of localization