Saying no to student requests for excessive appointments and help solving homework
I would limit the amount of time you dedicate to a single student to a maximum of 20 minutes a week. This is a reasonable amount of time to discuss the main problems a student has. This does not mean that you cannot make exceptions, but it is good to have a general rule.
As a side remark: You will not make everybody happy. If somebody wants an appointment and you say "no", and he insists, you can still say "no" and walk away.
One of the harder things I've had to do is to say 'no'.
[...]
how do I tell them that their homework is not meant for me?
It seems that you are struggling with a far more general problem than the specific one you asked about (which other answers have given good responses to), so I will address the more general problem: by your own description, you are a person who finds it very difficult to say "no" to requests from others. This is not good, and is the heart of your problem; the issue with office hours is only a particular manifestation of it.
It's therefore important to emphasize that saying no is a very, very important life and career skill, and I strongly recommend that you take steps to master it as early as possible. If you do, you will be handsomely rewarded in increased quality of life and professional and personal success.
Here are some recommendations for things you can do to get better at saying no:
Spend some time reflecting on why you find it so difficult to say no. It may go back to something in your culture or upbringing, an insecurity you have that causes you to have an extreme reluctance to displease others, or something else. If you understand the causes, you may have better success fighting this tendency.
Spend some time thinking of all the times when this trait has caused you to do things you didn't want to do, or to not do things you did want to do, and to think how much better off you would be if you didn't have this problem. That should help with motivation to work on fixing the problem.
Practice saying no by doing role-playing thought experiments in your head (or in front of the mirror, or with a friend) in which you imagine situations where someone asks you to do something you don't want and you say no. Imagine their negative reaction, and practice doing it even knowing that the reaction will be negative.
Practice saying no in the real world. You can start with the office hours situation, but I'm sure plenty more opportunities will come along soon enough. Over time, it will become much easier.
These are my own ideas. This article has several more suggestions and an analysis of why some people find it hard to say "no". Good luck!
You mentioned that you feel like you are solving there homework for them. This is indeed something you should not encourage. In fact you should try to never answer questions about current exercises directly. Students will of course always try, but it is entirely fair to just tell them no, or to come back next week, after they handed them in and then you can discuss their solution.
If you do not want to cut things as short, for example during the actual office hours, what you can always do is try to deflect the question. When I am asked about some specific exercise problem, I usually try to explain the general underlying concept instead. I may even discuss an older different but related problem, but I will never give more than just a small hint.
In general, if you want to be helpful to your student, it is always good to remember that your job is not to get them good grades or to help them pass the exam, but to help them learn something instead.