Defining vertical tangent lines
Speaking as a geometer, I want "tangency" to be independent of the coordinate system. Particularly, if $f$ is a real-valued function of one variable defined in some neighborhood of $a$, and if $f$ is invertible in some neighborhood of $a$, then the line $x = a$ should be tangent to the graph $y = f(x)$ at $a$ if and only if the line $y = b = f(a)$ is tangent to the graph $y = f^{-1}(x)$ at $b$.
For an elementary calculus course I'd want:
$f$ continuous in some neighborhood of $a$;
$f$ invertible in some neighborhood of $a$;
$f'(a) = \pm\infty$, i.e., $(f^{-1})'(b) = 0$ (the graph $y = f^{-1}(x)$ has $y = a$ as horizontal tangent).
Condition 1 does not guarantee invertibility near $a$ (as the cusp shows), so in my book it's out.
Condition 2 implies all three items of my wish list. ($f$ is implicitly assumed differentiable in some neighborhood of $a$; the derivative condition guarantees the derivative doesn't change sign in some neighborhood of $a$, and that $f'(a) = \pm\infty$.)
Condition 3 does not imply continuity (as the step function shows), so it's out.