template argument type deduction from std::function return type with lambda

Don't use std::function in tandem with template argument deduction. In fact, there's very likely no reason to use std::function in a function or function template argument list. More often than not, you should not use std::function; it is a very specialized tool that is very good at solving one particular problem. The rest of the time, you can dispense with it altogether.

In your case you don't need template argument deduction if you use a polymorphic functor to order things:

struct less {
    template<typename T, typename U>
    auto operator()(T&& t, U&& u) const
    -> decltype( std::declval<T>() < std::declval<U>() )
    { return std::forward<T>(t) < std::forward<U>(u); }

    // operator< is not appropriate for pointers however
    // the Standard defines a 'composite pointer type' that
    // would be very helpful here, left as an exercise to implement
    template<typename T, typename U>
    bool operator()(T* t, U* u) const
    { return std::less<typename std::common_type<T*, U*>::type> {}(t, u); }
};

You can then declare:

template<typename Iter, typename Criterion, typename Comparator = less>
void sort_by(Iter first, Iter last, Criterion crit, Comparator comp = less {});

and comp(*ita, *itb) will do the right thing, as well as comp(crit(*ita), crit(*itb)) or anything else as long as it makes sense.

How about something like this:

template<  typename Titer
         , typename Tmaker
         , typename Tcompare
         >
void
sort_by(Titer first, Titer last,
        Tmaker criterion_maker,
        Tcompare comparator)
{
  typedef decltype(criterion_maker(*first)) Tcriterion;
  /*
    Now that you know the actual type of your criterion,
    you can do the real work here
  */
}

The problem is that you can obviously not use a default for the comparator with this, but you can easily overcome that by providing an overload that doesn't take a comparator and fills in std::less internally.

To do it like you originally suggested, the compiler would have to be able to "invert" the template instantiation process. I.e. for a given std::function<> instantiation, what parameter do I have to supply as the result to get it. This "looks" easy, but it is not!