Old Cordless Phone

Python 3, 195 185 164 bytes

-4 bytes thanks to @notjagan
-27 bytes thanks to @FelipeNardiBatista

lambda n,l:min(g([*x],l,n)for x in permutations(range(n)))
def g(x,l,n,r=0):
 for p in l:a=x.index(p);x=[x.pop(a)]+x;r-=~min(a,n-a)
 return r
from itertools import*

Try it online!

L is taken as a list of integers from [0, N)