What to answer an author asking me if I reviewed his/her paper?

You could say something like

"I am not in the habit of telling people whether I did or did not review their papers, sorry"

or

"I don't feel comfortable answering this question."

Or you could defuse the question with a humorous answer, e.g.

"I would tell you but then I'd have to kill you"

or

"I don't remember, I always take an amnesia pill immediately after reviewing a paper"

or some similar kind of obviously nonserious smart-assery. None of these answers provide any useful information to the asker, and all of them convey some level of disapproval on your part at being asked, making it unlikely that the asker would press the case any further.

I think it is inappropriate of someone to ask you if you reviewed their paper. As you point out, if you are in a position where you would only confirm if you were positive, and declining to respond implies a negative review, you are essentially forced to confirm unless you explicitly deny being a reviewer. In fact just having your jaw drop as the question is asked is probably confirmation enough. So I think the right answer to the question is an immediate Whether I reviewed that or not is confidential, and that's an inappropriate question.

That being said, I'm not aware of rules that would prevent you from acknowledging the review. As I mentioned in a comment, for conferences I've seen people explicitly unblind themselves in reviews by choice. If you wanted to acknowledge it, I think you could. I've occasionally admitted that to someone in the past, but not because the author asked.

There's a principle I go by now that took me a long time to learn, mainly because it involved a lot of unlearning things I'd internalized since childhood:

It's always okay to lie when someone asks you an inappropriate question. And usually it's the best answer, especially if you have it prepared, since the whole point of the inappropriate question is usually to extract information from you involuntarily through your reaction.

As such the best answer is just "no", regardless of whether you did. If you feel comfortable with the power dynamics, you can follow up with something like "You know, really, you shouldn't go around asking people that. If they did happen to be the reviewer, it would put them in a really bad position."