What is the definition of differentiability?
First of all, there is no such thing a correct definition. When we pick a definition we just want to be sure that it satisfies some properties that we like. In this way, definitions can be good, but not correct. For this discussion, it suffices to discuss functions $f:\mathbb{R}\to\mathbb{R}$. We want differentiable to have the following properties as a bare minimum.
- $f$ is differentiable implies $f$ is continuous.
- The converse is false, as we don't want "kinks" in our graph.
So, based on what you've said, it does seem like a bad definition of the derivative.
However, I think the mistake you have made is to confuse the "left hand derivative" and the "limit of the derivative from the left." So let's just clear that up right now. The "left hand derivative" means $$ \lim_{h\to 0^-} \frac{f(x+h)-f(x)}{h} $$
I believe you looked at $$ \lim_{h\to 0^-} f'(x+h) $$ which would be called the "limit of the derivative from the left". These are quite different, as you've discovered. Clearly the "limit of the derivative from the left" is not a good definition of derivative.
However, it turns out that the difference quotient makes for a decent definition. It's easy to show from this that $f$ will be continuous at $x$. So this makes us happy. Furthermore, $f$ won't have a kink at $x$. Otherwise, the left and right limit would disagree. So this makes us happy as well.
Then there's the tangent definition. This one is slightly more troublesome, as you'd have to define tangent without mentioning derivative. This is possible, but what does it buy you? What most people do is define tangent in terms of derivatives!
To address your last question, if a function is not defined at a point, how could it be anything at that point? Continuity is not defined outside the domain of a function.
A more general definition of differentiability is:
$$\text{Function $f:\mathbb R\to \mathbb R$ is said to be differentiable if $\exists\,a\in \mathbb R$ such that $\lim_{h\to 0}\frac{f(x+h)-f(x)-ah}{\lvert h\rvert}=0\;$}$$
It can be shown that this definition is equivalent to the conventional definition that relies on the existence of $\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$ (in fact, $a$ is just $\lim_{h\to 0}\frac{f(x+h)-f(x)}{h}$ here!).
This definition can be extended naturally to multivariate function. We can define a multivariate function $g:\mathbb R^n\to\mathbb R$ as differential if:
$$\text{$\exists\,\mathbf A(\mathbf X)\in\mathbb R^n$ such that $\lim_{\|\mathbf H\|\to 0}\frac{g(\mathbf X+\mathbf H)-g(\mathbf X)-\mathbf A(\mathbf X)\cdot\mathbf H}{\|\mathbf H\|}=0$}$$
Likely, $\mathbf A(\mathbf X)$ can be proved to be exactly $(\nabla g)(\mathbf X)$, i.e., if $g$ is differentiable at $\mathbf X$, then the first order partial derivatives exist at $\mathbf X$ under above definition.
A function is differentiable (has a derivative) at point x if the following limit exists:
$$ \lim_{h\to 0} \frac{f(x+h)-f(x)}{h} $$
The first definition is equivalent to this one (because for this limit to exist, the two limits from left and right should exist and should be equal). But I would say stick to this definition for now as it's simpler for beginners. The second definition is not rigorous, it is quite sloppy to say the least.
Also, there's a theorem stating that: if a function is differentiable at a point x, then it's also continuous at the point x.