how to implement overlap-checking memcpy in C

The only portable way to determine if two memory ranges overlap is:

int overlap_p(void *a, void *b, size_t n)
{
    char *x = a, *y =  b;
    for (i=0; i<n; i++) if (x+i==y || y+i==x) return 1;
    return 0;
}

This is because comparison of pointers with the relational operators is undefined unless they point into the same array. In reality, the comparison does work on most real-world implementations, so you could do something like:

int overlap_p(void *a, void *b, size_t n)
{
    char *x = a, *y =  b;
    return (x<=y && x+n>y) || (y<=x && y+n>x);
}

I hope I got that logic right; you should check it. You can simplify it even more if you want to assume you can take differences of arbitrary pointers.

What you want to check is the position in memory of the source relatively to the destination:

If the source is ahead of the destination (ie. source < destination), then you should start from the end. If the source is after, you start from the beginning. If they are equal, you don't have to do anything (trivial case).

Here are some crude ASCII drawings to visualize the problem.

|_;_;_;_;_;_|          (source)
      |_;_;_;_;_;_|    (destination)
            >-----^    start from the end to shift the values to the right

      |_;_;_;_;_;_|    (source)
|_;_;_;_;_;_|          (destination)
^-----<                 start from the beginning to shift the values to the left

Following a very accurate comment below, I should add that you can use the difference of the pointers (destination - source), but to be on the safe side cast those pointers to char * beforehand.

In your current setting, I don't think that you can check if the operation will fail. Your memcpy prototype prevents you from doing any form of checking for that, and with the rule given above for deciding how to copy, the operation will succeed (outside of any other considerations, like prior memory corruption or invalid pointers).