Branch of math studying relations
This is probably too advanced, but there is Roland Fraïssé's book (which I'm surprised no one has mentioned yet):
Review of 1986 edition by Arnold William Miller, Bulletin of the American Mathematical Society (N.S.) 23 #1 (July 1990), pp. 206-209.
Review of 2000 edition by Peter van Emde Boas, Nieuw Archief voor Wiskunde (5) 5 #3 (September 2004), p. 251. [Boas's review is in English.]
(added the next day)
More useful, I think, would be to gather up a lot of undergraduate level set theory texts (Enderton, Schaum's outline, Dalen/Doets/De Swart, Devlin, Hrbacek/Jech, Monk, Roitman, Vaught, etc.) and compile a list of basic results about relations from the text material and the exercises (most will probably be in the exercises). I've often used this method to learn something new. In the U.S. you can find many such books in most any college library under the Library of Congress headings QA 9 and QA 248. As you compile and orgainize the results, you'll become better acquainted with subject, and sometimes you'll even come up with some new results on your own by extending ideas in the results you have. (In my case, I almost always later come across my "original result" published somewhere, usually as an exercise in a book or as an aside in a research paper.)
While it is not a systematic study of relations, there is a paper by Smullyan which may be of interest to you, "Equivalence Relations and Groups ". Abstract:
Our purpose is to show how the logic of relations can be uitilized in the study of group theory. There are some striking similarities in certain theorems in group theory and certain results about equivalence relations, and we show how the former can be derived as consequences of the latter. This transition is accomplished by means of certain ismorphism theorems, proved in considerable generality in section 2, and applied to groups in section 3. In section 1 we give several miscellaneous theorems on equivalence relations, which later turn out to have their analogues in the theory of groups.
The theory of binary relations over a set is in a sense the same as the theory of (non-weighted) directed graphs. The theory of symmetric relations is in a sense the same as the theory of (non-weighted) undirected graphs. Indeed, every graph can be seen as the set of vertices and the adjacency relation (which is symmetric in the case of undirected graphs).
Since relations can be composed, and the composition is associative, they are of interest in semigroup theory. If you google "semigroup of binary relations", you will find many hits. These semigroups were studied quite intensively in the 1960s.
There are connections between semigroup-theoretic properties of relations and standard notions such as transitivity or being an equivalence relation. One simple example is that every equivalence relation is idempotent with respect to composition. Indeed, every preorder is. A relation $R$ is transitive iff $R\circ R\subseteq R.$ A relation $R$ over a set $X$ is interpolative, that is $(\forall a,b\in x) ((a,b)\in R \Longrightarrow (\exists c\in X) ((a,c)\in R\wedge (c,b)\in R))$, iff $R\circ R\supseteq R.$