How to represent the floor function using mathematical notation?
For a real number $x$, $$\lfloor x\rfloor=\max\{n\in\mathbb{Z}\mid n\leq x\}.$$ I'd like to add though, that "$\lfloor x\rfloor$" is mathematical notation, just as much as the right side of the above equation is; the right side might use more "basic" constructions, but you can then ask about $$\max,\qquad\in,\qquad \mathbb{Z},\qquad {}\mathbin{\mid}{},\qquad \leq$$ and so on. At some point you just have to start writing notation and explaining it in words and hope your readers understand. So I disagree with your phrasing of the question.
$\lfloor x \rfloor = x - \arctan(\tan(\pi x))/\pi$ ?...
I am an engineer, perhaps this helps if you do not expect too much.
$x-(x$ mod $1$)