Under what circumstances can malloc return NULL?

Yes.

Just try to malloc more memory than your system can provide (either by exhausting your address space, or virtual memory - whichever is smaller).

malloc(SIZE_MAX)

will probably do it. If not, repeat a few times until you run out.

You need to do some work in embedded systems, you'll frequently get NULL returned there :-)

It's much harder to run out of memory in modern massive-address-space-and-backing-store systems but still quite possible in applcations where you process large amounts of data, such as GIS or in-memory databases, or in places where your buggy code results in a memory leak.

But it really doesn't matter whether you've never experienced it before - the standard says it can happen so you should cater for it. I haven't been hit by a car in the last few decades either but that doesn't mean I wander across roads without looking first.

And re your edit:

I'm not talking about memory exhaustion, ...

the very definition of memory exhaustion is malloc not giving you the desired space. It's irrelevant whether that's caused by allocating all available memory, or heap fragmentation meaning you cannot get a contiguous block even though the aggregate of all free blocks in the memory arena is higher, or artificially limiting your address space usage such using the standards-compliant function:

void *malloc (size_t sz) { return NULL; }

The C standard doesn't distinguish between modes of failure, only that it succeeds or fails.