How I can define this operator in Mathematica?

g[f_][x_?NumericQ] := If[EvenQ@Floor[x], f@FractionalPart[x], 1/f@FractionalPart[x]]

The ?NumericQ part is important because EvenQ returns False immediately for anything that is not a number.

In this case, f needs to be an actual function. Sin[x] and Sin[x]+2 are not functions. They are expressions in terms of x. Sin and Sin[#]+2& are functions. Look up Function to see what # and & mean.

Example:

g[Sin][5/2]
(* Sin[1/2] *)

As an alternative to Szabolcs' answer I would sugest a different approach, that quite literally follows the mathematical definition of $G(f)$. Note that we may avoid the premature evaluation of any argument that is not numeric by using Divisible instead of EvenQ or OddQ:

g[ f_Function ] := Function[ x, 
    Piecewise[
        {
          {     f[ FractionalPart @ x ],       Divisible[ Floor[x], 2 ] },
          { 1 / f[ FractionalPart @ x ], Not @ Divisible[ Floor[x], 2 ] }
        },
        Indeterminate (* in all other cases *)
    ]
]

We may then use this for numeric arguments:

f = Function[ x, 2 + Sin[x] ];
h = g[f]; (* or directly g[f] @ x *)
h[ 5/2 ]

$2 + \text{Sin}[\frac{1}{2}]$

In the given form we can now also work symbolically:

h[x] // Head

Piecewise