Differentiation: Vectors or scalars

You seem to be regarding $\mathbb{R}$ as an affine space that includes both points and vectors. With this view of the world, points and vectors are two different things, and it doesn't make sense to add together two points, as you say.

The Wikipedia article on affine spaces says:
In an affine space, there is no distinguished point that serves as an origin. Hence, no vector has a fixed origin and no vector can be uniquely associated to a point. In an affine space, there are instead displacement vectors, also called translation vectors or simply translations, between two points of the space. Thus it makes sense to subtract two points of the space, giving a translation vector, but it does not make sense to add two points of the space. Likewise, it makes sense to add a vector to a point of an affine space, resulting in a new point translated from the starting point by that vector.

In your definition of the derivative, I think $x$ should be regarded as a point, and $h$ as a vector. The function $x \mapsto y(x)$ should be regarded as mapping points to points. Then

• $x+ h$ is a point (adding a vector to a point)
• $y(x+h)$ is a point (a value of the function $y$)
• $y(x+h) - y(x)$ is a vector (difference of two points)
• $y'(x)$ is a vector

This point of view makes sense (I think), and it's consistent with the concept of directional derivative in higher dimensions.

As mentioned in one of the comments, there is a very good discussion of the point-versus-vector issue in the answers to this question.

While others have pointed out that an addition of points can be defined (and it is of course defined to be the same as when you interpret them as vectors), I do think that your line is morally right. (BTW: points of more general spaces, topological spaces, can not always be added. So being able to add two entities is indeed something specific, though not unique, to vectors.)

Even when derivatives are generalized to functions between geometric objects, say from a circle to a donut shaped thing, this is done by first approximating the domain and co-domain by vector spaces near the point where the derivative is desired. However, here you already see that the domain and codomain need not really consist of vectors, but it suffices that small changes in the inputs or outputs can be expressed via (tangential) vectors.

In summary, derivatives answer the question: If I change the input a bit in direction X, in which direction do the function values go. And vectors are the mathematical formalization of direction. In your question, the $h$ should arguable be thought as a vector, as well as the value of the derivative. But not necessarily the input $x$ and the output $f(x)$.

In a nitpicking sense, the derivative $$ f'(x) = \lim_{h \to 0} \frac{f(x + h) - f(x)}{h} $$ of a function $f$ is indeed a scaling map that specifies how a (small) displacement (vector) $h$ at a point $x$ gets stretched into a displacement vector $f'(x)\, h$ at $y = f(x)$.

Commonly this relationship is expressed in terms of points as $f(x + h) \approx f(x) + f'(x)\, h$. The derivative itself, $f'(x)$, is neither a point nor a vector.

In more general settings (such as coordinate-free formulations of mechanics), "points" live in a space $M$ (a "manifold"). To formulate differential calculus, one introduces an entirely new space $TM$ (the "tangent bundle"), whose elements may be viewed as pairs $(x, h)$ consisting of a point $x$ of $M$ and a displacement $h$ at $x$.

In this setting, if $f:M \to N$ is a differentiable mapping, its "(total) derivative" is a mapping $f_{*}:TM \to TN$ defined by $$ f_{*}(x, h) = \bigl(f(x), f'(x)\, h\bigr). $$ The derivative $f'(x):T_{x} M \to T_{f(x)} N$ is a "linear transformation" between tangent spaces, a generalization of a scaling map.

The conceptual asymmetry between points and vectors becomes starkly visible in this setting. Adding points makes no sense at all. The sum "$x + h$" of a point and a vector cannot be viewed as a point of $M$. It can be viewed as a point of $TM$, but doing so doesn't have much use. Instead, one "probes the infinitesimal structure of $M$" with "paths through $x$ whose velocity is $h$".