Does local mean infinitesimally small?

Firstly there is the mathematical understanding of locality, i.e. https://en.wikipedia.org/wiki/Local_property. Roughly "local" means "in some (sufficiently small) open set". This is also very relevant for physics, especially in GR, since the definition of a manifold (e.g. space-time) is that it looks locally like $\mathbb{R}^n$. More precisely locally here means that for every point on the manifold there exists an open neighborhood of that point which is homeomorphic to an open set in $\mathbb{R}^n$. This has to be contrasted with the term global. Very roughly this can be explained by an example, e.g. the circle $\mathbb{S}^1$, which looks locally like an interval $(0,1) \subset \mathbb{R}$ by the homeomorphism $s \mapsto (\cos 2\pi s, \sin 2\pi s)$. Globally it is however different. If you go once around the circle you end up in the same place, which you can not do in $\mathbb{R}$.

Now I agree with Vadim that as you describe it in you question "local" means "infinitesimally", since just knowing a Hessian at some point (or Gradient etc.) tells you something about the function at that point only and not in a neighborhood of that point. It tells you something about infinitesimal variations of that point. On the other hand if you know all derivatives of a function at a point, under certain assumptions, you may be able to know the function everywhere (see Taylor expansion) and knowing some derivatives gives you an approximation, that gets arbitrarily good in a neighborhood of that point as you shrink it arbitrarily close to the point. So there is some connection between the previous definition and this one.

Note also that knowing some relation of derivatives locally (i.e. on an open subset) gives you a differential equation that in combination with some conditions may or may not give you the function locally (or globally), but this is another story.

Then there is of course also the concept of a local theory or local interaction, which is correctly characterized in Vadims answer. For example in particle physics this means that interaction terms in the Lagrangian density only depend on the same space-time point. Otherwise it would lead a violation of causality. This is again another story.

Yes, local here means infinitesimally small, although it is a less well-defined term than infinitesimal. One also speaks of local theories, meaning the description of physical phenomena in terms of differential equations involving derivatives up to a finite order. Obviously, taking a derivative also means taking an infinitesimal limit. In this context non-local is associated with interaction occurring via finite distances with no continuous physical entity to mediate the interaction, famously known as the spooky action at a distance.

What the existing answers kinda imply but do not point out exactly is that there are two notions of locality, and one needs to exercise judgement in telling them apart.

Local can mean "in an open neighborhood", which is always finite.

Example: If $A$ is a closed $k$-form on a manifold $M$, there is a theorem (Poincaré's lemma) which states that then $A$ is locally exact as well. What this means is that each point $x\in M$ has an open neighborhood $U$ such that there is a $k-1$-form $B$ on $U$ satisfying $A|_U=dB$. The domain $U$ in question is finite.

There is also a notion of locality that is infinitesimal, which can be stated more rigorously using derivatives/jets. Some examples:

Example 1: It is often stated that every metric tensor is "locally flat". What this means that each point $x\in M$ has a neighborhood $U$ that is a coordinate neighborhood with some coordinate system $x^\mu$ such that at $x$ we have $g_{\mu\nu}(x)=\eta_{\mu\nu}$ and $\partial_\kappa g_{\mu\nu}(x)=0$.

Note that the neighborhood $U$ is finite, but the result is essentially valid for the "first-order infinitesimal neighborhood" of the point only. Without using some other framework such as synthetic differential geometry there is no way of stating this rigorously, but one can imagine that the first-order infinitesimal neighborhood of $x$ is the (fictious) region $U_1$ which contains $x$ and has the property that for any point $x+dx$ which is also in $U_1$ (i.e. infinitesimaly close to $x$) we have $f(x+dx)=f(x)+\partial_\mu f(x)dx^\mu$ as an exact (rather than approximative) relation for any smooth function $f$.

Example 2: Differential operators. The exterior derivative $d$, for example is a local operator in both sense. It is a local operator in the finite neighborhood sense because if $A$ and $B$ are differential forms that agree on some open neighborhood of $x\in M$, then $dA=dB$ on that neighborhood, but it is also an "infinitesimally local" operator in the sense that if $A,B$ are differential forms on $M$ such that at $x\in M$ we have $j^1_xA=j^1_xB$ (this essentially means that $A(x)=B(x)$ and in any chart they have the same first derivatives at $x$), then $dA(x)=dB(x)$.

For OP's examples, the curvature tensor is an infinitesimal measure of curvature. If the curvature tensor vanishes at a point it means that any loop in the second-order infinitesimal neighborhood of that point has integrable parallel transport.

The vanishing of the curvature at a point has no finite bearings on the manifold's geometry.

To complicate things, I am also noting that if the curvature tensor vanishes in the entire manifold, its effect on parallel transport is also only local, but now finite-local. If the entire curvature tensor vanishes, then it guarantees that parallel transport is path-independent in some open neighborhood of each point, but the corresponding global statement is not necessarily true, due to purely topological obstructions, a notion captured in the so-called null-holonomy (cf. Aharonov-Bohm effect).