Can I demand re-submission of a manuscript that has an odd document layout?

No. It is not your job as a reviewer to demand things. You can inform the editor that given the format of the manuscript that you will not review it. The editor will then likely look at the manuscript and either decide that you are being a pain and find a new reviewer or that the manuscript format is ridiculous. In that case, they will apologize to you and tell you that they are requesting a reformatted version.

Unless the format is really awful (like less than 10 lines a page or greater than 100 characters per line), the editor will probably conclude you are a pain.