Should I accept an invitation to give a talk from someone who might review my proposal?

It's the potential reviewer's responsibility to declare or avoid any conflict of interest, not yours.

If the potential reviewer isn't asked to review your proposal (or declines for other reasons) then there's no conflict of interest.

If the potential reviewer does end up reviewing your proposal and you've declined an invitation to speak at his institution that potential reviewer might hold it against you.

If you do accept the invitation, this wouldn't by itself be sufficient to create a conflict of interest that would prevent the potential reviewer from reviewing your proposal.

Thus I wouldn't hesitate to accept the invitation.

It's possible that this person was selected to review your proposal, has begun reading it, and thinks enough of your ideas to want to learn more. It's also possible that he wants to hear, in your talk, things that were already in your proposal --- the point here is that a reviewer is not allowed to use the content of the proposal for any purpose other than writing the review, but if he also gets the same content by another route, like your talk, then he can build on it (on his own or in collaboration with you).

In any case, I don't think your accepting the invitation and giving a talk would create a conflict of interest for the reviewer, even if your talk is on the same topic as your proposal.