Maximum of $\int_a^b \frac{f(x)}{x}\,\mathrm dx$

$\def\d{\mathrm{d}}\def\peq{\mathrel{\phantom{=}}{}}$For any fixed $f \in F$, define$$ g(x) = \int_a^x f(u) \,\d u, \quad \forall x \in [a, b] $$ then $g' = f$ almost everywhere and $g(a) = g(b) = 0$. Since $|f(x)| \leqslant M$, then for any $a \leqslant x \leqslant \dfrac{a + b}{2}$,$$ g(x) = \int_a^x f(u) \,\d u \leqslant M (x - a), $$ and for any $\dfrac{a + b}{2} < x \leqslant b$,$$ g(x) = g(b) - \int_x^b f(u) \,\d u \leqslant M(b - x). $$ Therefore by integration by parts,$$ \int_a^b \frac{f(x)}{x} \,\d x = \int_a^b \frac{g'(x)}{x} \,\d x = \left. \frac{g(x)}{x} \right|_a^b + \int_a^b \frac{g(x)}{x^2} \,\d x = \int_a^b \frac{g(x)}{x^2} \,\d x\\ = \int_a^{\tfrac{a + b}{2}} \frac{g(x)}{x^2} \,\d x + \int_{\tfrac{a + b}{2}}^b \frac{g(x)}{x^2} \,\d x \leqslant M \int_a^{\tfrac{a + b}{2}} \frac{x - a}{x^2} \,\d x + M \int_{\tfrac{a + b}{2}}^b \frac{b - x}{x^2} \,\d x = M \ln\frac{(a + b)^2}{4ab}. $$

Note that the above inequality becomes equality for$$ f(x) = \begin{cases}M;& a \leqslant x \leqslant \dfrac{a + b}{2}\\-M;& \dfrac{a + b}{2} < x \leqslant b\end{cases}, $$ thus the maximum is $M \ln\dfrac{(a + b)^2}{4ab}$.

While this answer I cant claim is complete proof, I will share it, maybe others can complete it or find the ideas useful.

I would approach this using calculus of variations. However, if $f(x)$ must be continuous, I think the calculus of variations would dictate $f(x) \equiv 0$ (and the functional won't be a maximum). So, let us assume that $f$ is discontinuous at a point $ a < c < b$, such that: $$ f(x) = \begin{cases} f_1(x), & a \leq x \leq c \\ f_2(x), & c < x \leq b\end{cases}. $$

Our functional reads: $$ I_{f_1,f_2} = \int_a^b \dfrac{f(x)}{x} dx = \int_a^c \frac{f_1(x)}{x} dx + \int_c^b\frac{f_2(x)}{x} dx$$ and when we write the stationary (extremum) condition of $I$ with respect to independent variations of $f_1, f_2$ we get $\delta f_1 = \delta f_2 = 0$ (so they are constant functions). Let us assume now:

$$ f(x)/M = \begin{cases} k_1 , & a \leq x \leq c \\ -k_2, & c < x \leq b\end{cases}. $$

From the constraint $\int_a^b f(x) dx=0$, we get: $c = \dfrac{k_1 a + k_2 b}{k_1+k_2}$. We ask that $a < c < b$, so $0 < k_{1,2} \leq 1$ (because $\vert f(x) \vert \leq M$). Not all pairs $(k_1,k_2)$ need to be studied (see the scaling relation in the following, to keep in mind).

All that is left to do is to compute the functional and maximize it with respect to $k_1, k_2$. We have: $$I(k_1,k_2)/M = k_1 \ln \frac{c}{a} + k_2 \ln \frac{c}{b} = \ln \left[ \left(\frac{c}{a}\right)^{k_1} \left(\frac{c}{b}\right)^{k_2} \right],$$ with $c=c(k_1,k_2)$ given above.

Note that $I(\lambda k_1, \lambda k_2) = I^\lambda(k_1,k_2)$. This allows us to simplify the search for $k_1, k_2$ to obtain maximum value. Namely, we only could look for $0 < t < 1$ at $I(1,t)$, $I(t,1)$, and $I(1,1)$ to understand the behavior for all values $(k_1,k_2)$ (using the scaling relation above, we can scale the arguments down by the one that is the biggest). Turns out (I did plotting only) that the maximum is attained for $k_1=k_2=1$. This is a bit of a hand-waving proof, maybe somebody else can complete this point.

With the above in mind, the maximum value of the functional is the same as one obtained by Saad above: $$ \text{max } I_{f_1,f_2} = M \ln\dfrac{(a + b)^2}{4ab}.$$