When passing a class by-value, does the caller or callee call the destructor?

The standard answers this question in [expr.call]/4, with a surprising amount of elaboration:

... The initialization and destruction of each parameter occurs within the context of the calling function. [ Example: The access of the constructor, conversion functions or destructor is checked at the point of call in the calling function. If a constructor or destructor for a function parameter throws an exception, the search for a handler starts in the scope of the calling function; in particular, if the function called has a function-try-block (Clause 18) with a handler that could handle the exception, this handler is not considered. —end example ]

In other words, the destructor is invoked by the calling function.

The caller destroys it. See https://en.cppreference.com/w/cpp/language/lifetime. Quoting:

All temporary objects are destroyed as the last step in evaluating the full-expression that (lexically) contains the point where they were created, and if multiple temporary objects were created, they are destroyed in the order opposite to the order of creation.

Also keep this as general rule - one, who creates, destroys. Usually in reversed order.