Why is this category invalid?

It not so clear what is they mean, but I guess what they mean is that if you consider the graph above (in which edges with different labels are different) then you cannot put on that graph a structure of a category.

To prove that you have to use reductio ab absurdum: if there where any category structure on that graph there should be a law of composition such that $g \circ h = \text{id}_B$ and also $f \circ g = \text{id}_A$ (that's follows for what is said in the link you posted above) and so it should also be the case that $$f = f \circ \text{id_B} = f \circ (g \circ h) = (f \circ g) \circ h = \text{id}_A \circ h = h \ .$$

This would implies that $f=h$ but by hypothesis $f \ne h$ hence you've arrived to an absurd, so you cannot find any composition law that give to the graph the structure of a category.

Hope this helps.

hint: What is $hgf$? Write it in two different ways.