Why can't DSolve find a solution for my system of ODEs?

I'm not sure why this doesn't work. Until someone finds the reason, here is a workaround to solve the system of equations:

ode={x'[t] == x[t]/t - y[t], y'[t] == x[t] + y[t]/t}

Using the change of variables $x(t)=t X(t)$ and $y(t)=tY(t)$,

ode /. {x -> ((# X[#]) &), y -> ((# Y[#]) &)}
(* X[t] + t X'[t] == X[t] - t Y[t], Y[t] + t Y'[t] == t X[t] + Y[t] *)

which is now easily solved by DSolve:

DSolve[%, {X[t], Y[t]}, {t}]
(* X[t] -> C[1] Cos[t] - C[2] Sin[t], Y[t] -> C[2] Cos[t] + C[1] Sin[t] *)

Here is another way to solve in Mathematica.

\begin{align*} x^{\prime}\left( t\right) & =\frac{x\left( t\right) }{t}-y\left( t\right) \\ y^{\prime}\left( t\right) & =x\left( t\right) -\frac{y\left( t\right) }{t} \end{align*}

Write as

$$ \dot{X}=AX $$

Where $A= \begin{pmatrix} \frac{1}{t} & -1\\ 1 & \frac{-1}{t} \end{pmatrix} $. Let us assume the solution is $X\left( t\right) =X\left( 0\right) e^{\int_{0}^{t}A\left( \tau\right) d\tau}$. However, this only works (since this is matrix time varying system) if $A$ commutes with $e^{\int_{0}^{t}A\left( \tau\right) d\tau}$. i.e. if $$ A\left( t\right) e^{\int_{0}^{t}A\left( \tau\right) d\tau}=e^{\int_{0} ^{t}A\left( \tau\right) d\tau}A\left( t\right) $$

If it does not commute, things get little more complicated. We can check, using Mathematica, that it does indeed commute in this example, so we are lucky

(mat = {{1/t, -1}, {1, 1/t}}) // MatrixForm
(bmat = MatrixExp[Integrate[mat, t]]) // MatrixForm

 mat.bmat == bmat.mat

Therefore, we can ask Mathematica now to solve it, using MatrixExp

x0 = {c1, c2}; (*constant of integration*)
(sol = x0.bmat) // MatrixForm

The above is the solution. First row is $x(t)$ and second row is $y(t)$.

Why did Mathematica not solve it? I do not know.

Fixed in version 11.1 Now DSolve can solve it directly

 $Version

11.1.0 for Microsoft Windows (64-bit) (March 13, 2017)

 ClearAll[x,t,y];
 DSolve[{x'[t]==x[t]/t-y[t],y'[t]==x[t]+y[t]/t},{y[t],x[t]},t]

 {{x[t]->t C[1] Cos[t]-t C[2] Sin[t],y[t]->t C[2] Cos[t]+t C[1] Sin[t]}}