What values can a constructor return to avoid returning this?

The exact condition is described on the [[Construct]] internal property, which is used by the new operator:

From the ECMA-262 3rd. Edition Specification:

13.2.2 [[Construct]]

When the [[Construct]] property for a Function object F is called, the following steps are taken:

1. Create a new native ECMAScript object.
2. Set the [[Class]] property of Result(1) to "Object".
3. Get the value of the prototype property of F.
4. If Result(3) is an object, set the [[Prototype]] property of Result(1) to Result(3).
5. If Result(3) is not an object, set the [[Prototype]] property of Result(1) to the original Object prototype object as described in 15.2.3.1.
6. Invoke the [[Call]] property of F, providing Result(1) as the this value and providing the argument list passed into [[Construct]] as the argument values.
7. If Type(Result(6)) is Object then return Result(6).
8. Return Result(1).

Look at steps 7 and 8, the new object will be returned only if the type of Result(6) (the value returned from the F constructor function) is not an Object.

As a side note, the return value or this is just part of the equation.

For example, consider this:

function Two() { return new Number(2); }
var two = new Two;
two + 2; // 4
two.valueOf = function() { return 3; }
two + 2; // 5
two.valueOf = function() { return '2'; }
two + 2; // '22'

As you can see, .valueOf() is internally used and can be exploited for fun and profit. You can even create side effects, for example:

function AutoIncrementingNumber(start) {
    var n = new Number, val = start || 0;
    n.valueOf = function() { return val++; };
    return n;
}
var auto = new AutoIncrementingNumber(42);
auto + 1; // 43
auto + 1; // 44
auto + 1; // 45

I can imagine this must have some sort of practical application. And it doesn't have to be explicitly a Number either, if you add .valueOf to any object it can behave as a number:

({valueOf: function() { return Math.random(); }}) + 1; // 1.6451723610516638

You can exploit this to make an object that always returns a new GUID, for instance.

Concrete examples http://jsbin.com/zivivucahi/1/edit?html,js,console,output

/*
ECMA 262 v 5
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-262.pdf
"4.3.2
primitive value
member of one of the types Undefined, Null, Boolean, Number, Symbol, or String as defined in clause 6"
*/

var Person = function(x){
  return x;

};


console.log(Person.constructor);
console.log(Person.prototype.constructor);
console.log(typeof(Person));
console.log(typeof(Person.prototype));

function log(x){
  console.log(x instanceof Person);
  console.log(typeof x);
  console.log(typeof x.prototype);
}

log(new Person(undefined));
log(new Person(null));
log(new Person(true));
log(new Person(2));
log(new Person(""));

//returns a function not an object
log(new Person(function(){}));


//implementation?
//log(new Person(Symbol('%')));

I couldn't find any documentation on the matter, but I think you're correct. For example, you can return new Number(5) from a constructor, but not the literal 5 (which is ignored and this is returned instead).