How to open VLC in a new window instance?

1. Go to Tools > Preferences.
2. Make sure that Allow only one instance under Instances is unchecked.
3. Choose All at the bottom-left corner, under "Show Settings".
4. Make sure that Allow only one running instance under "Performance options" is unchecked.
5. Make sure that One instance when started from file under "Performance options" is unchecked.

Use:

vlc.exe --no-one-instance

Reference: VLC command-line help