This Program Can not Be Run in DOS Mode

Background

That message is from the basic header that every Windows (PE format) executable has. The message (and the code that displays it) is technically editable, but all compilers seem to just emit code that displays that same string and then exits. It's 16-bit executable code (like a DOS .COM file), intended to be displayed when you try to run the program on MS-DOS or similar pre-Windows systems. Windows' program loader typically skips over it.

Explanation

However, there is one case where Windows might display that message: when you try to run a PE that is compiled for a more advanced version of Windows than the one you're using, especially if the main program code is for a different CPU architecture than the OS can support. (This is similar to, for example, trying to run Win32 code on 16-bit DOS.) From a GUI, you'd get a pop-up error, but in the command line, you may well just get that message.

The obvious candidate here is that you're trying to run code on the (hideously obsolete) Windows XP, and you say you're using "i686-w64-mingw32-gcc". If you're compiling 64-bit binaries for Windows XP, it's extremely likely they won't work; while a 64-bit build of XP technically exists (it's actually a different kernel version, 5.2 vs. 5.1, but it's branded as XP), it was never widely used; the extremely vast majority of XP boxes are 32-bit only. If you want a quick check for 32-bit vs. 64-bit from the command line, check for the presence of a \Windows\SysWOW64 directory (which holds 32-bit system binaries on 64-bit machines); if it doesn't exist, you're running on a 32-bit OS.

Recommendation

• Unless you A) need 64-bit for something, and B) know your target is 64-bit, only use 32-bit payloads.
• Make sure your code and compiler are targeting XP (Vista and later added a ton of new APIs that will prevent a program which uses them from running on XP).
• Consider spinning up an XP VM to test whether your payloads run locally before trying them on the target machine.

Make sure you have delivered the payload correctly, I had a similar issue when transferring accesschk.exe via ftp. FTP allows transfer in ascii and binary modes, if you transfer it in binary mode it should work.

Binary mode transfers the files, bit by bit, as they are on the FTP server. Ascii mode, however, will download the text directly. You can type ascii or binary to switch between the types.

To do this, connect to the ftp service and type "binary", you should get a response saying "200 Type set to I"

put the file again and run the executable.