Difference between algebra and geometry

You are right, that the attempted distinction makes sense only from a rather naive viewpoint, popular though it may be. Among professional mathematicians, these inherited "legacy" labels have a surprising endurance, which I think is mostly/only due to their widespread recognition among amateurs and professionals alike, despite their inaccuracy. Traditions die hard, even in the face of supposedly rational considerations.

200+ years ago, anyone who could "really prove" things, as opposed to giving a physical/physics-y quasi-heuristic, was a "geometer"... but this didn't mean you were "doing geometry", it only meant something about conforming to the alleged standards of proof of the ancient Greek geometers. "Arithmetization" of analysis in the mid-to-late 19th century really meant "finding a way to make a rigorous foundation", as opposed to invocation of "physical intuition" about continuity and such. (This in contrast to both Newton's and Leibniz' explanations of how to do calculus, in terms that were difficult to "ground" until A. Robinson.)

So, no, these labels do not accurately refer to much, but they are popular, and people speak in terms of them. :)