Am I allowed to use personal conversation as a source?

I think it would be dangerous to source an unrecorded interview with a prominent person that involves opinion. Actually, it seems dangerous to do it if it is unrecorded and the rest of the first sentence just adds additional reasons.

If the person, at some point, objects to your characterization of their words you will have some troubles.

However, you can refer to personal communication in general if you have a record of it. One way to obtain the record is to send them a message (mail or email) in which you state your interpretation of their words and ask "(a) did I get it right, and (b) can I use that in a paper I'm writing?"

But a recorded and transcribed interview would also do the trick.

However, don't use things that could possibly be disputed without any agreed upon record, or without permission.