Overload resolution of template functions

For each of your function calls the compiler has 2 functions to choose from and chooses the best one. Unknown template parameters are deduced from the arguments apart from RT which must be explicitly specified and can't be deduced.

auto a = ::max(4, 7.2);

As RT is not specified and can't be deduced, the second overload is not usable so is ignored. The first is chosen and the types are deduced as int and double.

auto b = ::max<double>(4, 7.4);

RT is now specified so the compiler can choose to either use max<double,int,double> or max<double, double>, the argument types for the 3 template parameter version match the function arguments exactly whereas the 2 template parameter version would require a cast from int to double so the 3 parameter overload is chosen.

auto c = ::max<int>(7, 4.);

RT is now specified so the compiler can choose to either use max<int,int,double> or max<int, double>, the argument types both functions are now the same so the compiler can't choose between them.

the template argument deduction does not take the return type into account,

Yes. Template argument deduction is performed based on function arguments.

so why max<int> is ambiguous and not max<double>?

Given ::max<int>(7, 4.), for the 1st overload, the 1st template parameter T1 is specified as int, and T2 is deduced as double from the 2nd function argument 4., then the instantiation would be double max(int, double). For the 2nd overload, the 1st template parameter RT is specified as int, T1 is deduced as int from 7, T2 is deduced as double from 4., then the instantiation would be int max(int, double). Overload resolution doesn't consider return type too, the two overloads are both exact match and then ambiguous.

Given ::max<double>(7, 4.), for the 1st overload, the 1st template parameter T1 is specified as double, and T2 is deduced as double from 4., so the instantiation would be double max(double, double). For the 2nd overload, the 1st template parameter RT is specified as double, T1 is deduced as int from 7, T2 is deduced as double from 4., then the instantiation would be double max(int, double). Then the 2nd overload wins in overload resolution because it's an exact match, the 1st one requires the implicit conversion from int to double for the 1st argument 7.

Let's look at what specifying double as an argument does for the compiler during overload resolution.

For the "Number1" max template, it specifies that the first argument must be of type double. When attempting to do the template match, the compiler deduces that the second argument is of type double. So the resultant signature is auto max(double, double). That's a match, though it involves casting the first argument from int to double.

For the "Number2" max template, it specifies that the return type is double. The argument types are deduced. So the resultant signature is double max(int, double). That's an exact match, removing any ambiguity.

Now let's look at specifying int. Now the two signatures are auto max(int, double) and double max(int, double). As you can see, there's no difference which is relevant to overload resolution, resulting in the ambiguity.

Essentially, by passing in double, you've poisoned one of the overloads by forcing an unnecessary conversion; the other overload thereby gets to dominate. Passing in int, in contrast, does not further constrain either overload's ability to be a perfect match.