Solve for $(5x-1)(x-3)<0$

When you get $x<\frac15$ and $x>3$, the conclusion that you should get is not that is wrong. What happens here is that this condition is impossible, that it, it never holds. Still another way of seeing this is to insert that it holds if and only if $x\in\emptyset$. And the other condition holds if and only if $x\in\left(\frac15,3\right)$. So the set of those numbers $x$ such that $(5x-1)(x-3)<0$ is $\emptyset\cup\left(\frac15,3\right)$, which is equal to $\left(\frac15,3\right)$.

There's no contradiction.

You simply showed that there is no value of $x$ for which $5x-1<0$ and $x-3>0$. This is not "wrong".

To see the solution in more simpler terms sketch the sign scheme.

$(5x-1)(x-3)<0$

set each of these factors to zero, you will get two numbers $\frac{1}{5}$ and $3$. Locate these numbers in the number line.

If you put any number greater than $3$ you get the expression to be $>0$. One more thing this graph is always alternating its sign. If $>3$ is positive then $<3$ will be negative and so on.

Hence as per your question you will find your required solution in $(\frac{1}{5},3)$. Its open interval because putting either of the two values will make it zero and we are particularly interested in $<0$ negative solutions.