Where are the bandits at Nilheim?

Tom is right. A little ways off Nilheim there will be a man with a broken cart called Telrav. You escort him back to his hideout and all the friendly "guards" reveal their bandit nature and ambush you. The problem is that you can get the generic bounty quest before meeting Telrav and thus they'll all stay guards. If you kill them all while they're just guards it bugs the Telrav encounter and it cannot be completed. If you only kill the "bandit chief" it should be fine.

I... kind of shot the head guard off a cliff.

Quest completed!

I guess I missed something XD

I had the same issue just guards in and around the tower I killed the sleeping guard that the quest marker pointed to, he appears to be the only one in the tower he insisted that he was on my side and would not attack me, I then got the quest completed message when he died. Strangely non of the guards would attack me but if I rode my horse up to them they would attack it if I got off they totally ignored me and just attacked the horse even when I attacked them. I think the quest is a bit bugged hope it's the only one.