How does C++ free the memory when a constructor throws an exception and a custom new is used

Fundamentally, if there is no delete operator which corresponds to the new operator, then nothing is done. Nothing is done also in the case of placement new, because the corresponding placement delete operator is a no-op. The exception is not diverted: it continues its course, so the caller of the new has the opportunity (and responsibility) for freeing the memory allocated.

Placement new is called that because it is used to place the object in memory otherwise acquired; since the memory was not acquired by the new operator, it would be unlikely that it could be released by the delete operator. In practice, the question is moot because (since C++03, at least) it is not permitted to replace the placement new operator (which has prototype operator new(size_t, void*) or delete (operator delete(void*, void*)). The supplied placement new operator returns its second argument, and the supplied placement delete operator is a no-op.

Other new and delete operators may be replaced, either globally or for a specific class. If a custom new operator is called, and the constructor throws an exception, and there is a corresponding delete operator, then that delete operator will be called to clean up before the exception is propagated. However, it is not an error if there is no corresponding delete operator.

First, an example:

#include <cstddef>
#include <iostream>

struct S
{
    S(int i) { if(i > 42) throw "up"; }

    static void* operator new(std::size_t s, int i, double d, char c)
    {
        std::cout << "allocated with arguments: "
                  <<i<<", "<<d<<", "<<c<<std::endl;
        return new char[s];
    }

    static void operator delete(void* p, int i, double d, char c)
    {
        std::cout << "deallocated with arguments: "
                  <<i<<", "<<d<<", "<<c<<std::endl;
        delete[] (char*)p;
    }

    static void operator delete(void* p)
    {
        std::cout << "deallocated w/o arguments"<<std::endl;
        delete[] (char*)p;
    }
};

int main()
{
    auto p0 = new(1, 2.0, '3') S(42);

    S* p1 = nullptr;
    try
    {
        p1 = new(4, 5.0, '6') S(43);
    }catch(const char* msg)
    {
        std::cout << "exception: "<<msg<<std::endl;
    }

    delete p1;
    delete p0;
}

Output:

allocated with arguments: 1, 2, 3
allocated with arguments: 4, 5, 6
deallocated with arguments: 4, 5, 6
exception: up
deallocated w/o arguments

---

The canonical definition of an operator new overload is void *operator new(std::size_t, heap h)

I don't see how this is canonical, since it's not allowed: Ok, now it's a valid placement-form of new :)

[basic.stc.dynamic.allocation]/1

An allocation function shall be a class member function or a global function; a program is ill-formed if an allocation function is declared in a namespace scope other than global scope or declared static in global scope. The return type shall be void*. The first parameter shall have type std::size_t. The first parameter shall not have an associated default argument. The value of the first parameter shall be interpreted as the requested size of the allocation.

[emphasis mine]

You can overload the allocation function to be called for the placement-form of new, see [expr.new] (it's not explicitly allowed in [basic.stc.dynamic.allocation] for non-template functions, but also not forbidden). The placement given in new(placement) is generalized here to an expression-list. Each expression in the expression-list for a specific new-expression is passed as an additional arguments to the allocation function. If the deallocation function is called (e.g. because the called ctor throws an exception), the same arguments plus a leading void* (the return value of the allocation function) are passed to the deallocation function.

[expr.new]/18 states:

If any part of the object initialization described above terminates by throwing an exception, storage has been obtained for the object, and a suitable deallocation function can be found, the deallocation function is called to free the memory in which the object was being constructed, after which the exception continues to propagate in the context of the new-expression. If no unambiguous matching deallocation function can be found, propagating the exception does not cause the object’s memory to be freed. [Note: This is appropriate when the called allocation function does not allocate memory; otherwise, it is likely to result in a memory leak. — end note ]

and /21

If a new-expression calls a deallocation function, it passes the value returned from the allocation function call as the first argument of type void*. If a placement deallocation function is called, it is passed the same additional arguments as were passed to the placement allocation function, that is, the same arguments as those specified with the new-placement syntax.

and /20

A declaration of a placement deallocation function matches the declaration of a placement allocation function if it has the same number of parameters and, after parameter transformations, all parameter types except the first are identical. Any non-placement deallocation function matches a non-placement allocation function. If the lookup finds a single matching deallocation function, that function will be called; otherwise, no deallocation function will be called. If the lookup finds the two-parameter form of a usual deallocation function and that function, considered as a placement deallocation function, would have been selected as a match for the allocation function, the program is ill-formed. [Example:

struct S {
    // Placement allocation function:
    static void* operator new(std::size_t, std::size_t);
    // Usual (non-placement) deallocation function:
    static void operator delete(void*, std::size_t);
};

S* p = new (0) S; // ill-formed: non-placement deallocation function matches
                  // placement allocation function

— end example ]

Going back to [basic.stc.dynamic.deallocation]:

1 Deallocation functions shall be class member functions or global functions; a program is ill-formed if deallocation functions are declared in a namespace scope other than global scope or declared static in global scope.

2 Each deallocation function shall return void and its first parameter shall be void*. A deallocation function can have more than one parameter.