How to detect if a function is called as constructor?

NOTE: This is now possible in ES2015 and later. See Daniel Weiner's answer.

I don't think what you want is possible [prior to ES2015]. There simply isn't enough information available within the function to make a reliable inference.

Looking at the ECMAScript 3rd edition spec, the steps taken when new x() is called are essentially:

• Create a new object
• Assign its internal [[Prototype]] property to the prototype property of x
• Call x as normal, passing it the new object as this
• If the call to x returned an object, return it, otherwise return the new object

Nothing useful about how the function was called is made available to the executing code, so the only thing it's possible to test inside x is the this value, which is what all the answers here are doing. As you've observed, a new instance of* x when calling x as a constructor is indistinguishable from a pre-existing instance of x passed as this when calling x as a function, unless you assign a property to every new object created by x as it is constructed:

function x(y) {
    var isConstructor = false;
    if (this instanceof x // <- You could use arguments.callee instead of x here,
                          // except in in EcmaScript 5 strict mode.
            && !this.__previouslyConstructedByX) {
        isConstructor = true;
        this.__previouslyConstructedByX = true;
    }
    alert(isConstructor);
}

Obviously this is not ideal, since you now have an extra useless property on every object constructed by x that could be overwritten, but I think it's the best you can do.

(*) "instance of" is an inaccurate term but is close enough, and more concise than "object that has been created by calling x as a constructor"

As of ECMAScript 6, this is possible with new.target. new.target will be set if the function is called with new (or with Reflect.construct, which acts like new), otherwise it's undefined.

function Foo() {
    if (new.target) {
       console.log('called with new');
    } else {
       console.log('not called with new');
    }
}

new Foo(); // "called with new"
Foo(); // "not called with new"
Foo.call({}); // "not called with new"