Why can't matter anihilate with itself?

One has to realize that words in physics have a definite meaning. This meaning, for words used in physics, is dependent on the specific mathematical model that uses it.

The mathematical model for particle physics is called the standard model

All matter and energy emerges from interactions and composites of these particle. The model encapsulates the large number of particle data measured over the last half century.

In all particle interactions , particles may transmute at the vertices changing quantum numbers and identity as defined in the table. Changing identity is not called annihilation. The basic definition of annihilation as used in particle physics is that after the interaction all the quantum numbers of the initiating particles add up to zero. The particles coming out of the vertex add up to zero, and thus new pairs can appear with different quantum numbers.

Thus we get, as an example, e+e- annihilating into a quark antiquark pair, which have completely different quantum numbers than the electron positron pair, but their addition adds up to zero.

In your example an electron_antineutrino scattering on an electron will make the lepton number zero but not the charge, so it is not within the definition of "annihilation". It is merely an interaction.

Of course an electron_neutrino on an electron gives a lepton number of 2, and it can only be a scattering.

Searching for Feynman diagrams I did find this:

A Feynman diagram representing the annihilation of an electron neutrino and a positron to a muon neutrino and a muon.

So the author does not include charge zero within the definition of annihilation so one might find such a usage of the word, but it is not mainstream and it should be noted that it is from an astrophysics course, not a particle physics :).