How to define “self-citation”?

There is no exact, official, legally binding definition of self-citation. The term occurs usually in the context of research metrics and scientific evaluations, and every developer of a system to measure such things is free to use his own definition of self-citation (or even, to ignore it).

That said, if I were to implement such a system, I would treat self-citation as a property of a paper, not as a property of an author. That means, I would consider a citation of paper A within paper B as a self-citation, if A and B have at least one author in common. The reason is that some overlap between the author lists will already make it more likely that paper A is referenced in paper B.

Note that "the citation of A within B is a self-citation" is not a moral statement. It does not mean that the authors of A, or the authors of B, or the joint authors of A and B did anything wrong. It just means that there is some aspect that makes this particular citation statistically more likely than a citation of A within another paper C, where A and C have no authors in common. Research metrics consider events that are more likely (say, a publication in arXiv) as less valuable than events that are less likely (say, a publication in Nature). They are based on statistical observations, not on a legal process that decides for each individual citation whether it is justified or not. Don't take it personally.