What does a return statement do inside a setter?

Lets look at the simple assignment:

11.13.1 Simple Assignment ( = )

The production AssignmentExpression : LeftHandSideExpression = AssignmentExpression is evaluated as follows:

1. Let lref be the result of evaluating LeftHandSideExpression.
2. Let rref be the result of evaluating AssignmentExpression.
3. Let rval be GetValue(rref).

So rval is assigned the value that is going to be assigned to left hand side (o.a). In your case 5.

4. Throw a SyntaxError exception if the following conditions are all true: [...]
5. Call PutValue(lref, rval).

This is where the value is assigned to the left hand side (PutValue(o.a, 5)). As you can see, nothing is done with whathever PutValue returns (it doesn't return anything).

6. Return rval.

It simple returns the value that was assigned, in your case 5.

---

The assignment expression always returns the value that was assigned.

The value of an assignment expression is always equal to the value being assigned:

var a;
console.log(a = 7); // logs 7

This is true no matter whether the thing you are assigning to is a variable, plain property, setter, read-only property, or whatever:

var a = {};
Object.defineProperty(a, "b", { value: 8, writable: false });
console.log(a.b = 9);  // logs 9
console.log(a.b);      // logs 8, because a.b is read-only

As you said yourself, a return value in a setter is meaningless, so when you used a return value in your setter, why did you expect it to actually do anything special? To answer your question "What does a return statement do inside a setter?" - Basically nothing. It evaluates the expression to the right of the return and then throws away the result.

You cannot return anything from a property setter, and are just getting the value of the assignment (which is 5 because you assigned 5).

what does return in a setter do?

Nothing. Better remove that return statement. At best it's ignored.