What prevents the usage of a function argument as hidden pointer?

A function is allowed to assume its return-value object (pointed-to by a hidden pointer) is not the same object as anything else. i.e. that its output pointer (passed as a hidden first arg) doesn't alias anything.

You could think of this as the hidden first arg output pointer having an implicit restrict on it. (Because in the C abstract machine, the return value is a separate object, and the x86-64 System V specifies that the caller provides space. x86-64 SysV doesn't give the caller license to introduce aliasing.)

Using an otherwise-private local as the destination (instead of separate dedicated space and then copying to a real local) is fine, but pointers that may point to something reachable another way must not be used. This requires escape analysis to make sure that a pointer to such a local hasn't been passed outside of the function.

I think the x86-64 SysV calling convention models the C abstract machine here by having the caller provide a real return-value object, not forcing the callee to invent that temporary if needed to make sure all the writes to the retval happened after any other writes. That's not what "the caller provides space for the return value" means, IMO.

That's definitely how GCC and other compilers interpret it in practice, which is a big part of what matters in a calling convention that's been around this long (since a year or two before the first AMD64 silicon, so very early 2000s).

---

Here's a case where your optimization would break if it were done:

struct Vec3{
    double x, y, z;
};
struct Vec3 glob3;

__attribute__((noinline))
struct Vec3 do_something(void) {  // copy glob3 to retval in some order
    return (struct Vec3){glob3.y, glob3.z, glob3.x};
}

__attribute__((noinline))
void use(struct Vec3 * out){   // copy do_something() result to *out
    *out = do_something();
}


void caller(void) {
    use(&glob3);
}

With the optimization you're suggesting, do_something's output object would be glob3. But it also reads glob3.

A valid implementation for do_something would be to copy elements from glob3 to (%rdi) in source order, which would do glob3.x = glob3.y before reading glob3.x as the 3rd element of the return value.

That is in fact exactly what gcc -O1 does (Godbolt compiler explorer)

do_something:
    movq    %rdi, %rax               # tmp90, .result_ptr
    movsd   glob3+8(%rip), %xmm0      # glob3.y, glob3.y
    movsd   %xmm0, (%rdi)             # glob3.y, <retval>.x
    movsd   glob3+16(%rip), %xmm0     # glob3.z, _2
    movsd   %xmm0, 8(%rdi)            # _2, <retval>.y
    movsd   glob3(%rip), %xmm0        # glob3.x, _3
    movsd   %xmm0, 16(%rdi)           # _3, <retval>.z
    ret     

Notice the glob3.y, <retval>.x store before the load of glob3.x.

So without restrict anywhere in the source, GCC already emits asm for do_something that assumes no aliasing between the retval and glob3.

---

I don't think using struct Vec3 *restrict out wouldn't help at all: that only tells the compiler that inside use() you won't access the *out object through any other name. Since use() doesn't reference glob3, it's not UB to pass &glob3 as an arg to a restrict version of use.

I may be wrong here; @M.M argues in comments that *restrict out might make this optimization safe because the execution of do_something() happens during out(). (Compilers still don't actually do it, but maybe they would be allowed to for restrict pointers.)

Update: Richard Biener said in the GCC missed-optimization bug-report that M.M is correct, and if the compiler can prove that the function returns normally (not exception or longjmp), the optimization is legal in theory (but still not something GCC is likely to look for):

If so, restrict would make this optimization safe if we can prove that do_something is "noexcept" and doesn't longjmp.

Yes.

There's a noexecpt declaration, but there isn't (AFAIK) a nolongjmp declaration you can put on a prototype.

So that means it's only possible (even in theory) as an inter-procedural optimization when we can see the other function's body. Unless noexcept also means no longjmp.

Substantially rewritten:

I understand, that an alias of pointer out (e.g. as global variable) could be used in do_something and thus [out] cannot be passed as hidden pointer to do_something: if it would, out would be changed inside of do_something and not when do_something returns, thus some calculations might become faulty.

Except with respect to aliasing considerations inside do_something(), the difference in timing with respect to when *out is modified is irrelevant in the sense that use()'s caller cannot tell the difference. Such issues arise only with respect to accesses from other threads, and if that's a possibility then they arise anyway unless appropriate synchronization is applied.

No, the issue is primarily that the ABI defines how passing arguments to functions and receiving their return values works. It specifies that

If the type has class MEMORY, then the caller provides space for the return value and passes the address of this storage in %rdi

(emphasis added).

I grant that there's room for interpretation, but I take that as a stronger statement than just that the caller specifies where to store the return value. That it "provides" space means to me that the space in question belongs to the caller (which your *out does not). By analogy with argument passing, there's good reason to interpret that more specifically as saying that the caller provides space on the stack (and therefore in its own stack frame) for the return value, which in fact is exactly what you observe, though that detail doesn't really matter.

With that interpretation, the called function is free to assume that the return-value space is disjoint from any space it can access via any pointer other than one of its arguments. That this is supplemented by a more general requirement that the return space not be aliased (i.e. not through the function arguments either) does not contradict that interpretation. It may therefore perform operations that would be incorrect if in fact the space were aliased to something else accessible to the function.

The compiler is not at liberty to depart from the ABI specifications if the function call is to work correctly with a separately-compiled do_something() function. In particular, with separate compilation, the compiler cannot make decisions based on characteristics of the function's caller, such as aliasing information known there. If do_something() and use() were in the same translation unit, then the compiler might choose to inline so_something() into use(), or it might choose to perform the optimization you're looking for without inlining, but it cannot safely do so in the general case.

It was suggested to me, that using restrict should solve the problem,

restrict gives the compiler greater leeway to optimize, but that in itself does not give you any reason to expect specific optimizations that might then be possible. In fact, the language standard explicitly specifies that

A translator is free to ignore any or all aliasing implications of uses of restrict.

(C2011, 6.7.3.1/6)

restrict-qualifying out expresses that the compiler doesn't need to worry about it being aliased to any other pointer accessed within the scope of a call to use(), including during the execution of functions other functions it calls. In principle, then, I could see a compiler taking advantage of that to shortcut the ABI by offering somebody else's space for the return value instead of providing space itself, but just because it could do does not mean that it will do.

What prevents the usage of out as hidden pointer?

ABI compliance. The caller is expected to provide space that belongs to it, not to someone else, for storage of the return value. As a practical matter, however, I don't see anything in the restrict-qualified case that would invalidate shortcutting the ABI, so I take it that that's just not an optimization that has been implemented by the compiler in question.

NB: The desired (or very similar) behavior would be achieved for a slightly different function-signature: [...]

That case looks like a tail-call optimization to me. I don't see anything inherently inconsistent in the compiler performing that optimization, but not the one you're asking about, even though it is, to be sure, a different example of shortcutting the ABI.