Intrinsic Curvature of a Cylinder

An observer embedded in a surface can tell if the surface is curved by walking around what should be a square. That is his path is $4$ segments of equal length with $90^{\circ}$ turns between them.

On earth, such a path might start on the equator. It would go north to the north pole, turn left, go south to the equator, turn left, go west $1/4$ of the way around the world (which is back to the start), turn left, and go north to the north pole.

Since he doesn't wind up back where he started, he can conclude the surface is curved.

Curvature in space time as measured in General Relativity involves parallel transport around such a path. As you walk along the path, carry an arrow and keep it pointing in the same direction. If the arrow points in a different direction when you are done, the surface is curved. The change in direction is a measure of the curvature.

For example, start with the arrow pointing north. As you walk the first leg, it points ahead of you. After you turn left, it point to your right, which is east. After you turn left and walk along the equator, it points behind you, which is east. After you turn left and walk north again, it points to your left, which is east. When you reach the end at the pole, it is pointing to your left. If you turn left again, it is pointing ahead of you. But this is a different direction that when you started.

The example path was large because it is easy to visualize the difference in direction and position. And the differences are large for a large path. But there would still be a small difference if the sides of the square were $1$ mile long.

Also note that instead of walking around the entire square, you can walk halfway around, and see if you wind up at the same place if you walk the other half. In flat space, both halves wind up at the opposite diagonal. In curved space, they wind up in different places.

If you pick a square path on a cylinder, it should be easy to convince yourself that you would come back to the original point, and an arrow you carried would not change direction.

You can see that space time is curved from the fact that time runs slightly slower near the surface of the earth than it does far from the earth.

In space time, you can pick a square where two of the sides are time-like. So start away from earth. In one half path, wait $1$ sec and move $100$ miles closer. In the other half, move $100$ miles closer and wait $1$ sec. Since time moves slower when you wait near the earth, you wind up at the same point, but at different times. These are different points in space time.

There are two different understandings of curvature conflicting here, namely intrinsic and extrinsic curvature. Intrinsic curvature is the actual curvature in that is detectable to someone moving inside the space. On the other hand extrinsic curvature can only be defined if the space is embedded in another higher dimensional space, for example the cylinder embedded in $\mathbb{R}^3$. It has an extrinsic curvature but it intrinsic curvature is that of the plane, i.e. it is not curved intrinsically. Note that in the context of general relativity space-time has an intrinsic curvature but it is not assumed that space-time is embedded in any higher dimensional space.
For your question about the observer returning to the start-point. This is not really related to curvature but more about the topology of the space, namely that a cylinder can be viewed an infinite 2-dimensional tape but with its two sides identified.

Look also at https://math.stackexchange.com/questions/2002965/intrinsic-and-extrinsic-curvature.