Improper Integral $\int_0^1\left(\left\{\frac1x\right\}-\frac12\right)\frac{\log(x)}xdx$
This integral is not equal to zero.
We may obtain the following closed form.
$$ \begin{align} \int_0^1 \left(\left\{\frac{1}{x}\right\}-\frac{1}{2}\right)\frac{\log(x)}{x} \mathrm{d}x & = \dfrac{\ln^2(2\pi)}{4}-\dfrac{\gamma^2}{4}+\dfrac{\pi^2}{48}-\dfrac{\gamma_1}{2}-1\tag1 \\\\ \end{align} $$
where $\left\{x\right\}$ denotes the fractional part of $x$, $\gamma$ denotes the Euler–Mascheroni constant and where $\gamma_{1}$ denotes the Stieltjes constant defined by $$ \gamma_{1} = \lim_{N \rightarrow \infty}\left(\sum_{k=1}^{N}\frac{\ln k}{k}-\frac{\ln^{2}N}{2} \right). $$ Consequently, we have the numerical evaluation:
$$ \begin{align} \int_0^1 \left(\left\{\frac{1}{x}\right\}-\frac{1}{2}\right)\frac{\log(x)}{x} \mathrm{d}x = \color{red}{0.00}31782279542924256050500... . \tag2 \end{align} $$
Here is an approach.
Step 1. Let $s$ be a complex number such that $0<\Re{s}<1$. Then $$ \int_{0}^{1} x^{s-1}\left\{\frac{1}{x}\right\}\mathrm{d}x = -\frac{1}{1-s} -\frac{\zeta(s)}{s}\tag3 $$ where $\left\{x\right\}$ denotes the fractional part of $x$ and where $\zeta$ denotes the Riemann zeta function.
Proof. Let us assume that $0<\Re{s}<1$. We may write $$ \begin{align} \int_{0}^{1} x^{s-1}\left\{\frac{1}{x}\right\}\mathrm{d}x & = \sum_{k=1}^{\infty} \int_{1/(k+1)}^{1/k} x^{s-1}\left\{\frac{1}{x}\right\}\mathrm{d}x \\ & = \sum_{k=1}^{\infty} \int_{k}^{k+1} \left\{x\right\} \frac{\mathrm{d}x}{x^{s+1}} \\ & = \sum_{k=1}^{\infty} \int_{k}^{k+1} (x-k) \frac{\mathrm{d}x}{x^{s+1}} \\ & = \sum_{k=1}^{\infty} \int_{0}^{1}\frac{v}{(v+k)^{s+1}}\mathrm{d}v \\ & = \sum_{k=1}^{\infty} \int_{0}^{1}\left(\frac{1}{(v+k)^{s}}-\frac{k}{(v+k)^{s+1}}\right)\mathrm{d}v \\ & = \sum_{k=1}^{\infty} \left.\left(\frac{1}{(-s+1)(v+k)^{s-1}} +\frac{k}{s(v+k)^s}\right) \right|_{0}^{1} \\ & = -\frac{1}{1-s}-\frac{\zeta(s)}{s}. \end{align} $$
Step 2. We have $$ \int_{0}^{1} x^{s-1}\left(\left\{\frac{1}{x}\right\}-\frac{1}{2}\right)\log(x)\mathrm{d}x = -\frac{1}{(1-s)^2} +\frac{1}{2s^2} +\frac{\zeta(s)}{s^2} -\frac{\zeta'(s)}{s}. \tag4 $$ Using $(3)$, we readily get $$ \int_{0}^{1} x^{s-1}\left(\left\{\frac{1}{x}\right\}-\frac{1}{2}\right)\mathrm{d}x = -\frac{1}{1-s}-\frac{1}{2s} -\frac{\zeta(s)}{s} $$ which we differentiate with respect to $s$ to obtain $(4)$.
Step 3. For $s$ near $0$, we take into account the Taylor series expansion of the Riemann $\zeta$ function: $$ \begin{align} & \zeta(s) =-\frac12-\dfrac{\ln(2\pi)}{2} s +\left(\dfrac{\gamma^2}{4}-\dfrac{\pi^2}{48}+\ln(2\pi)-\dfrac{\ln^2(2\pi)}{4}+\dfrac{\gamma_1}{2}\right)s^2+\mathcal{O}(s^3) \\& \zeta'(s) =-\dfrac{\ln(2\pi)}{2} +\left(\dfrac{\gamma^2}{2}-\dfrac{\pi^2}{24}+2\ln(2\pi)-\dfrac{\ln^2(2\pi)}{2}+\gamma_1\right)s+\mathcal{O}(s^2) \end{align} $$ and upon letting $s$ tend to $0^+$ in $(4)$ we obtain $(1)$.
Remark: A related result to $(3)$.
Here is an alternative approach similar to my solution of Closed form of integral over fractional part $\int_0^1 \left\{\frac{1}{2}\left(x+\frac{1}{x}\right)\right\}\,dx$. The closed form expression of the integral is traced back to the asymptotic behaviour of $g(n) = \sum_{k=1}^n \log(k)^2$.
1. Calculation
Letting $x=1/y$, and splitting the resulting integration range into equidistant pieces from $k$ to $k+1$ ($k=1,2,3,...$) gives
$$i:=\int_{0}^1 \frac{\log(x)}{x} (\{\frac{1}{x}\}-\frac{1}{2})\,dy=-\int_{1}^\infty \frac{\log(y)}{y} (\{y\}-\frac{1}{2})\,dy=\sum_{k=1}^\infty a(k)$$
Where
$$a_k = -\int_0^1 \frac{\left(\xi -\frac{1}{2}\right) \log (k+\xi )}{k+\xi } \, d\xi \\ =-\frac{1}{2} \log ^2(k+1)+\frac{1}{4} \left(\log ^2(k+1)-\log ^2(k)\right)+\frac{1}{2} \left((k+1) \log ^2(k+1)-k \log ^2(k)\right)-((k+1) \log (k+1)-k \log (k))+1$$
Forming the partial sum of $a_k$ most of the terms telescope witht the result
$$i_s(n) := \sum_{k=1}^n a_k = i_{s1}(n) + i_{s2}(n)$$ $$i_{s1}(n) = -(-n+(n+1) \log (n+1)-\frac{1}{4} \log ^2(n+1)-\frac{1}{2} (n+1) \log ^2(n+1))$$ $$i_{s2}(n) = -\frac{1}{2}g(n+1)$$
With
$$g(n) = \sum_{k=1}^{n}(\log(k))^2$$
In order to find the asymptotic behaviour of $g(n)$, we notice first that
$$\nu(n,x) := \sum_{k=1}^n k^x =H(n,-x)$$
where $H$ is the generalized harmonic number, is a generating function for our finite sum.
Its asymptotic expression is provided by Mathematica:
$$\nu_a(n,x) = \left(\frac{-x^3+3 x^2-2 x}{720 n^3}+\frac{n}{x+1}+\frac{x}{12 n}+\frac{1}{2}\right) n^x+\zeta (-x)$$
Hence we have $$g_{a}(n) = \frac{\partial ^2 \nu_a(n,x)}{\partial x^2}| x\to 0 \\ = \frac{1}{120 n^3}+2 \left(-\frac{1}{360 n^3}-n+\frac{1}{12 n}\right) \log (n)+2 n+\left(n+\frac{1}{2}\right) \log ^2(n)+\gamma _1+\frac{\gamma ^2}{2}-\frac{\pi ^2}{24}-\frac{1}{2} (\log (2)+\log (\pi ))^2$$
The asymptotics of $i_{s1}(n)$ is easily calculated
$$i_{s1}(n)\to \frac{5}{12 n^2}-\frac{5 \log (n)}{12 n^2}+n+\frac{1}{2} n \log ^2(n)+\frac{3 \log ^2(n)}{4}-n \log (n)+\frac{\log (n)}{n}-1$$
Finally,
$$i_{s} = \lim_{n\to\infty} (i_{s1}(n)+i_{s2}(n)) \\=-\frac{\gamma_1}{2}-\frac{\gamma^2}{4}+\frac{\pi^2}{48}-1+\frac{1}{4} (\log(2 \pi )^2= 0.0031782279542924256051 $$
in agreement with the previously obtained result of Olivier Oloa.
2. Discussion
In the OP it was stated that Mathematica returns a strange result when the integral is calculated directly (I assume NIntegrate was used). I confirm that. So the representation as an infinite sum is better suited for numerical purposes.
The following graph shows how the partial sums approoach the limiting value with increasing number of terms
The changing sign of the partial sums indicates that a false value $0$ can be understood on the basis of missing accuracy.