How to professionally quit a student job?

It sounds to me like you don't exactly need to quit. Remind him of the conditions under which you took the job, and ask him to stick to the agreement. If he can't do so because his requirements have changed, then that's his choice. You might say something like this:

As you recall, when I accepted this job, it was under the condition that I would only work 5 hours per week so it wouldn't interfere with working on my thesis. Because of your workshop, I have been working far more time than we agreed. I'm concerned that you need more support than the 5 hours a week I am able to provide. What do you think?

Note that I said "5 hours per week" twice for emphasis. Give him a chance to respond. At this point, if he promises to stick to 5 hours per week in future, and allows you to take the next few weeks off to catch up on your thesis, then perhaps you might give him one more chance.

However, if he indicates that he needs you to work more hours, or be available whenever he has deadlines, then you might say something like this:

Unfortunately, I can't work more than 5 hours per week, and it sounds like that just won't be enough. Perhaps the best way to solve this is turn the project over to someone else. I'll do what I can to ensure a smooth transition.

Instead of "quitting", you're working with him to solve his requirements. If it turns out that the only way to meet those requirements is for you to hand off the work to someone else, then that's his choice.

As for the time you worked for free at the start of the contract, I think it is going to be difficult to get paid for that without burning bridges. Of course, you are entitled to be paid, and you could pursue that legally, but you may not want to. In future, remember never to work without a contract.

German university students leave HiWi positions all the time; essentially all universities have forms that allow you to break a running contract (Auflösung) for essentially any reason that you so choose, given the requisite notice; typically this is one month before the new end date of the contract.

Your situation is not that unusual, in that you're doing a HiWi in the same group that you're doing a master's thesis (and intend to do your PhD in). Since it's clear you aren't really don't want to do the HiWi work right now, you can mention, as you suggested, that you're worried about finishing your master's thesis on time and doing a good job with it. However, leaving this position in this manner means you won't be able to take another HiWi position within the group (although you might be able to accept something else in another chair).

However, you shouldn't worry about quitting the position because of time constraints. This happens fairly frequently for lots of reasons. A good graduate student and research group will understand this and not be bothered by it. (If they do have a problem with it, then you should reconsider the wisdom of doing your PhD there!)

[On a side note, though, it seems your PhD student supervisor had unreasonable expectations of the position and what you were supposed to do. Most of the issues are on his side—particularly since he agreed that you could work just five hours per week in the first place!]