Changing field for PhD - How much do I need to know before I start talking to advisers/applying?

First of all, do you have enough time to change your topic? If so, I would strongly recommend to talk to your supervisor first. If he/she knows you and your abilities, he/she can guide you on choosing a topic; you can finish up and get a PhD. Second of all, with a good help of your supervisor and your commitment you can finish a PhD.

Note on "Scary Parts": Nothing is scary!. I myself had no idea about a subset of computing semantics, and it felt like jumping from an airplane when I started to learn about it. However, at the end of the learning period, I realized why I was so afraid? Don't be scared, it will take your focus away, on doing your job. Your new slogan should be: Keep calm and learn!

I changed fields between my master's degree and my PhD and I also found it scary. From my experience, there is no reason to be scared. Most students entering the graduate program will have little experience and you will have enough time to learn as you take classes or start your research.

I would say that with a background in Physics you would be ready to 'seriously apply' right away.

I will refer you to a somewhat similar question I answered a year ago, and the comments below it for some more info: https://academia.stackexchange.com/a/42326/31255