How can I get to use the sideset command on unusual symbols without an error message

\sideset wants as its third argument an operator such as \sum; you get what you want by

\mathbin{_\sigma\times_f}