What does "atomic" mean in programming?

Here's an example: Suppose foo is a variable of type long, then the following operation is not an atomic operation (in Java):

foo = 65465498L;

Indeed, the variable is written using two separate operations: one that writes the first 32 bits, and a second one which writes the last 32 bits. That means that another thread might read the value of foo, and see the intermediate state.

Making the operation atomic consists in using synchronization mechanisms in order to make sure that the operation is seen, from any other thread, as a single, atomic (i.e. not splittable in parts), operation. That means that any other thread, once the operation is made atomic, will either see the value of foo before the assignment, or after the assignment. But never the intermediate value.

A simple way of doing this is to make the variable volatile:

private volatile long foo;

Or to synchronize every access to the variable:

public synchronized void setFoo(long value) {
    this.foo = value;
}

public synchronized long getFoo() {
    return this.foo;
}
// no other use of foo outside of these two methods, unless also synchronized

Or to replace it with an AtomicLong:

private AtomicLong foo;

"Atomic operation" means an operation that appears to be instantaneous from the perspective of all other threads. You don't need to worry about a partly complete operation when the guarantee applies.

It's something that "appears to the rest of the system to occur instantaneously", and falls under categorisation of Linearizability in computing processes. To quote that linked article further:

Atomicity is a guarantee of isolation from concurrent processes. Additionally, atomic operations commonly have a succeed-or-fail definition — they either successfully change the state of the system, or have no apparent effect.

So, for instance, in the context of a database system, one can have 'atomic commits', meaning that you can push a changeset of updates to a relational database and those changes will either all be submitted, or none of them at all in the event of failure, in this way data does not become corrupt, and consequential of locks and/or queues, the next operation will be a different write or a read, but only after the fact. In the context of variables and threading this is much the same, applied to memory.

Your quote highlights that this need not be expected behaviour in all instances.