How does this implementation of std::is_class work?

What you are looking at is some programming technologie called "SFINAE" which stands for "Substitution failure is not an error". The basic idea is this:

namespace detail {
  template <class T> char test(int T::*);   //this line
  struct two{
    char c[2];
  };
  template <class T> two test(...);         //this line
}

This namespace provides 2 overloads for test(). Both are templates, resolved at compile time. The first one takes a int T::* as argument. It is called a Member-Pointer and is a pointer to an int, but to an int thats a member of the class T. This is only a valid expression, if T is a class. The second one is taking any number of arguments, which is valid in any case.

So how is it used?

sizeof(detail::test<T>(0))==1

Ok, we pass the function a 0 - this can be a pointer and especially a member-pointer - no information gained which overload to use from this. So if T is a class, then we could use both the T::* and the ... overload here - and since the T::* overload is the more specific one here, it is used. But if T is not a class, then we cant have something like T::* and the overload is ill-formed. But its a failure that happened during template-parameter substitution. And since "substitution failures are not an error" the compiler will silently ignore this overload.

Afterwards is the sizeof() applied. Noticed the different return types? So depending on T the compiler chooses the right overload and therefore the right return type, resulting in a size of either sizeof(char) or sizeof(char[2]).

And finally, since we only use the size of this function and never actually call it, we dont need an implementation.

Part of what is confusing you, which isn't explained by the other answers so far, is that the test functions are never actually called. The fact they have no definitions doesn't matter if you don't call them. As you realised, the whole thing happens at compile time, without running any code.

The expression sizeof(detail::test<T>(0)) uses the sizeof operator on a function call expression. The operand of sizeof is an unevaluated context, which means that the compiler doesn't actually execute that code (i.e. evaluate it to determine the result). It isn't necessary to call that function in order to know the sizeof what the result would be if you called it. To know the size of the result the compiler only needs to see the declarations of the various test functions (to know their return types) and then to perform overload resolution to see which one would be called, and so to find what the sizeof the result would be.

The rest of the puzzle is that the unevaluated function call detail::test<T>(0) determines whether T can be used to form a pointer-to-member type int T::*, which is only possible if T is a class type (because non-classes can't have members, and so can't have pointers to their members). If T is a class then the first test overload can be called, otherwise the second overload gets called. The second overload uses a printf-style ... parameter list, meaning it accepts anything, but is also considered a worse match than any other viable function (otherwise functions using ... would be too "greedy" and get called all the time, even if there's a more specific function t hat matches the arguments exactly). In this code the ... function is a fallback for "if nothing else matches, call this function", so if T isn't a class type the fallback is used.

It doesn't matter if the class type really has a member variable of type int, it is valid to form the type int T::* anyway for any class (you just couldn't make that pointer-to-member refer to any member if the type doesn't have an int member).

The std::is_class type trait is expressed through a compiler intrinsic (called __is_class on most popular compilers), and it cannot be implemented in "normal" C++.

Those manual C++ implementations of std::is_class can be used in educational purposes, but not in a real production code. Otherwise bad things might happen with forward-declared types (for which std::is_class should work correctly as well).

Here's an example that can be reproduced on any msvc x64 compiler.

Suppose I have written my own implementation of is_class:

namespace detail
{
    template<typename T>
    constexpr char test_my_bad_is_class_call(int T::*) { return {}; }

    struct two { char _[2]; };

    template<typename T>
    constexpr two test_my_bad_is_class_call(...) { return {}; }
}

template<typename T>
struct my_bad_is_class
    : std::bool_constant<sizeof(detail::test_my_bad_is_class_call<T>(nullptr)) == 1>
{
};

Let's try it:

class Test
{
};

static_assert(my_bad_is_class<Test>::value == true);
static_assert(my_bad_is_class<const Test>::value == true);

static_assert(my_bad_is_class<Test&>::value == false);
static_assert(my_bad_is_class<Test*>::value == false);
static_assert(my_bad_is_class<int>::value == false);
static_assert(my_bad_is_class<void>::value == false);

As long as the type T is fully defined by the moment my_bad_is_class is applied to it for the first time, everything will be okay. And the size of its member function pointer will remain what it should be:

// 8 is the default for such simple classes on msvc x64
static_assert(sizeof(void(Test::*)()) == 8);

However, things become quite "interesting" if we use our custom type trait with a forward-declared (and not yet defined) type:

class ProblemTest;

The following line implicitly requests the type int ProblemTest::* for a forward-declared class, definition of which cannot be seen by the compiler right now.

static_assert(my_bad_is_class<ProblemTest>::value == true);

This compiles, but, unexpectedly, breaks the size of a member function pointer.

It seems like the compiler attempts to "instantiate" (similarly to how templates are instantiated) the size of a pointer to ProblemTest's member function in the same moment that we request the type int ProblemTest::* within our my_bad_is_class implementation. And, currently, the compiler cannot know what it should be, thus it has no choice but to assume the largest possible size.

class ProblemTest // definition
{
};

// 24 BYTES INSTEAD OF 8, CARL!
static_assert(sizeof(void(ProblemTest::*)()) == 24);

The size of a member function pointer was trippled! And it cannot be shrunk back even after the definition of class ProblemTest has been seen by the compiler.

If you work with some third party libraries that rely on particular sizes of member function pointers on your compiler (e.g., the famous FastDelegate by Don Clugston), such unexpected size changes caused by some call to a type trait might be a real pain. Primarily because type trait invocations are not supposed to modify anything, yet, in this particular case, they do -- and this is extremely unexpected even for an experienced developer.

On the other hand, had we implemented our is_class using the __is_class intrinsic, everything would have been OK:

template<typename T>
struct my_good_is_class
    : std::bool_constant<__is_class(T)>
{
};

class ProblemTest;

static_assert(my_good_is_class<ProblemTest>::value == true);

class ProblemTest
{
};

static_assert(sizeof(void(ProblemTest::*)()) == 8);

Invocation of my_good_is_class<ProblemTest> does not break any sizes in this case.

So, my advice is to rely on the compiler intrinsics when implementing your custom type traits like is_class wherever possible. That is, if you have a good reason to implement such type traits manually at all.