When being asked, what's the most harmless way to address my past record of leaving Ph.D. program in a different field?

Based on the information you provided in the question, I think you should focus on reason (b). You should probably avoid using words like "abuse" or "confrontation", at least in the beginning of the conversation, as they come off as really strong and thus tend to motivate people to question deeper. State that you have had problems with the previous environment (including the adviser and the department) and that it was in the mutual best interest that you continued your career somewhere else. Avoid blaming anyone. If the conversational partner seeks to question further, reveal details to what extent you deem best, but again it is good practice to avoid bad-mouthing your former environment. Stay as objective as possible.

Like you said, most people don't care that much. For those that do care, you can't really stop them from taking it negatively (much as you can't stop anyone from taking anything about you negatively), but you can stay true to yourself, by being honest and objective without bad-mouthing your previous adviser, department, etc.

Apart from the detailed answer provided by @user3209815, an alternative answer could be

"I felt that my previous environment was not suited for my research. Besides, after much progress I realised that I'd do better in field X within my current environment."

Switching fields isn't entirely uncommon, especially in PhD. The same also goes switching from one grad school to another. There's nothing to feel guilty about in this issue.

If it is a topic you just don't want to discuss, and your interlocutor shouldn't care about, you can just say that you discovered that topic wasn't your cup of tea. If they insist, say that [new topic] is much more interesting for [couple of reasons].

A more truthful answer is that you had disagreements with your advisor on the direction or purpose of the project. That doesn't mean that either of you was at fault, or that there was a culprit in the first place. Don't give much detail, and if they insist on asking, politely say you don't want to talk about it.

If they do have a reason for asking, you can elaborate saying that you found some flaws in the previous work that your professor and you disagreed on its importance and the necessary effort to fix it. But in any case, going from computational chemistry to pure mathematics is a big jump. A bad advisor would explain changing university, not field, and I think this is where you should focus. It is also what would strike people the most.