What is "calculus" on a manifold?

Tangency revisited

Before talking about differential calculus, it may be best to conceptually understand what calculus is trying to tell us. One way to look at it is this: calculus is trying to tell us that fundamentally the functions $f$ with $f'(0) \neq 0$ is different from the functions $g$ with $g'(0) = 0$.

To emphasize this, let's go back to the notion of continuity.

Definition Given $(X,\tau)$ and $(Y,\sigma)$ two topological spaces, a function $f:X\to Y$ is said to be continuous at $x$ if for every open neighborhood $V$ of $f(x)$ there exists some open neighborhood $U$ of $x$ such that $f(U) \subseteq V$.

On the level of "continuity", we cannot tell apart $f$ and $g$ from above, since continuity lacks a quantitative expression to allow us to say that "as the neighborhood of $x$ shrinks, the corresponding neighborhood of $f(x)$ shrinks much faster". The basic idea of calculus is this:

Idea We say that $f:X\to Y$ is tangent to a constant at $x$ if for every open neighborhood $V$ of $f(x)$ there exists some open neighborhood $U$ of $x$, and a function $\gamma:(0,1] \to (0,1]$ satisfying $\lim_{s\to 0} \gamma(s)s^{-1} = 0$, such that for every $s$, $f(sU) \subseteq \gamma(s) V$.

Here, the undefined notation $sU$ means the intuitive notion of "shrinking $U$ by a factor of $s$", and similarly for $\gamma(s) V$. This notion can be made precise when $X$ is a topological vector space: the scaling $sU$ of $U$ around the point $x$ is $$ sU = \{ x + s(y-x): y\in U\}.$$ An indeed this gives rise to the standard definition of tangency in topological vector spaces (which may or may not have a norm).

Note that if you suitably define the scaling operations, you can generalize the notion of "tangency to a constant" to much more general settings.

In the case where $X$ and $Y$ are topological vector spaces, linearity allows us to add and subtract functions. This allows us to say that

Definition Two functions $f,g:X\to Y$ between topological vector spaces are said to be tangent at the point $x$ if $f(x) = g(x)$ and the function $f-g$ is tangent to 0 at $x$.

Definition A function $f:X\to Y$ is said to be differentiable at $x$ if there exists a continuous linear mapping $L:X\to Y$ such that $f$ is tangent to the function $x'\mapsto f(x) + L(x')$ at $x$. The linear mapping $L$ is said to be the derivative of $f$.

Exercise Prove the chain rule for topological vector spaces. That is to say, suppose $X,Y,Z$ are topological vector spaces, and suppose $f:X\to Y$ and $g:Y\to Z$ are function such that $f$ is differentiable at $x$ and $g$ is differentiable at $f(x)\in Y$, then $g\circ f$ is differentiable at $x$, and the derivative is the continuous linear map $M\circ L$ where $L$ is the derivative of $f$ at $x$ and $M$ is the derivative of $g$ at $f(x)$.

(... to be continued ...)

Willie Wong has already said most of what needs to be said (though if you'd like a reference, see Lang's Differential and Riemannian Manifolds). But for a nice summary of those ideas in the context of the most general setting in which one can do calculus, you might also like to read the introduction of the book Calculus in Vector Spaces without Norm, by Frölicher and Bucher.

Let me also say that you seem to have an impoverished view of the notion of manifold. You are thinking of smooth manifolds modeled on Euclidean spaces. But manifolds can be modeled on more general spaces, such as Banach spaces and even Fréchet spaces. The resulting manifolds are called -- drumroll -- Banach manifolds and Fréchet manifolds. Since differentiation (and, in particular, the chain rule) makes sense in Banach spaces and in Fréchet spaces, we can lift the idea of differentiation to manifolds modeled on such spaces. Of course, given the greater generality, this project encounters some difficulties: it's not true that all the nice features of high school differential calculus carry over. But some semblance of the core notion of derivative does.