Intellij IDEA creates multiple process for each debug session?
You can't do it OS wide AFAIK, and you wouldn't want it to be that way; how many other Java processes may be running that aren't associated with your debug sessions?? (Intellij would be one of them)
What you can do is, in your run configuration settings, set one or more of your configurations to be "Single instance only"; it's a checkbox in the upper right-hand corner of the dialogue. That will force you to kill any running with that config before starting another.
IntelliJ Idea 2017
Mark this checkbox:
Run -> Edit configurations -> Single instance only
(top right corner)
It won't let you run another instance before running the next one.
from IntelliJ IDEA 2016.1 Help /Run/Debug Configuration:
"IntelliJ IDEA 2016.1 Help /Run/Debug Configuration: Remote
Common options
Single instance only
If this check box is selected, this run/debug configuration cannot be launched more than once.
Every time a new run/debug configuration is launched, IntelliJ IDEA checks the presence of the other instances of the same run/debug configuration, and displays a confirmation dialog box. If you click OK in the confirmation dialog box, the first instance of the runner will be stopped, and the next one will take its place.
This makes sense when the usage of certain resources can cause conflicts, or when launching two run/debug configurations of the same type consumes too much of the CPU and memory resources.
If this check box is not selected, it is possible to launch as many instances of the runner as required. So doing, each runner will start in its own tab of the Run tool window."