Can one executable be both a console and GUI application?
Check out Raymond's blog on this topic:
https://devblogs.microsoft.com/oldnewthing/20090101-00/?p=19643
His first sentence: "You can't, but you can try to fake it."
Jdigital's answer points to Raymond Chen's blog, which explains why you can't have an application that's both a console program and a non-console* program: The OS needs to know before the program starts running which subsystem to use. Once the program has started running, it's too late to go back and request the other mode.
Cade's answer points to an article about running a .Net WinForms application with a console. It uses the technique of calling AttachConsole after the program starts running. This has the effect of allowing the program to write back to the console window of the command prompt that started the program. But the comments in that article point out what I consider to be a fatal flaw: The child process doesn't really control the console. The console continues accepting input on behalf of the parent process, and the parent process is not aware that it should wait for the child to finish running before using the console for other things.
Chen's article points to an article by Junfeng Zhang that explains a couple of other techniques.
The first is what devenv uses. It works by actually having two programs. One is devenv.exe, which is the main GUI program, and the other is devenv.com, which handles console-mode tasks, but if it's used in a non-console-like manner, it forwards its tasks to devenv.exe and exits. The technique relies on the Win32 rule that com files get chosen ahead of exe files when you type a command without the file extension.
There's a simpler variation on this that the Windows Script Host does. It provides two completely separate binaries, wscript.exe and cscript.exe. Likewise, Java provides java.exe for console programs and javaw.exe for non-console programs.
Junfeng's second technique is what ildasm uses. He quotes the process that ildasm's author went through when making it run in both modes. Ultimately, here's what it does:
- The program is marked as a console-mode binary, so it always starts out with a console. This allows input and output redirection to work as normal.
- If the program has no console-mode command-line parameters, it re-launches itself.
It's not enough to simply call FreeConsole to make the first instance cease to be a console program. That's because the process that started the program, cmd.exe, "knows" that it started a console-mode program and is waiting for the program to stop running. Calling FreeConsole would make ildasm stop using the console, but it wouldn't make the parent process start using the console.
So the first instance restarts itself (with an extra command-line parameter, I suppose). When you call CreateProcess, there are two different flags to try, DETACHED_PROCESS and CREATE_NEW_CONSOLE, either of which will ensure that the second instance will not be attached to the parent console. After that, the first instance can terminate and allow the command prompt to resume processing commands.
The side effect of this technique is that when you start the program from a GUI interface, there will still be a console. It will flash on the screen momentarily and then disappear.
The part in Junfeng's article about using editbin to change the program's console-mode flag is a red herring, I think. Your compiler or development environment should provide a setting or option to control which kind of binary it creates. There should be no need to modify anything afterward.
The bottom line, then, is that you can either have two binaries, or you can have a momentary flicker of a console window. Once you decide which is the lesser evil, you have your choice of implementations.
* I say non-console instead of GUI because otherwise it's a false dichotomy. Just because a program doesn't have a console doesn't mean it has a GUI. A service application is a prime example. Also, a program can have a console and windows.