Effects of switching between /SUBSYSTEM:CONSOLE to /SUBSYSTEM:WINDOWS in a DLL

This option only has an effect on applications (exe), not on libraries(dll).

Its documentation says:

The /SUBSYSTEM option specifies the environment for the executable.

The choice of subsystem affects the entry point symbol (or entry point function) that the linker will select.

This does not affect libraries, which have their own (optional) entry point.

Additional information following CyberShadow's comment: The subsystem field itself seems to be ignored when a DLL is loaded. This article about CSRSS says (emphasis mine):

Furthermore, each process is associated with one, certain subsystem; this property is being set by the linker (during the compilation process), and resides in the following PE structure field: [...]

My below reply is just some findings.

I tried to create below types of project within VS2015:

• Win32 console app project
• Win32 DLL project
• Win32 Windows app project

Below are their complete linking options:

Win32 console app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\ConsoleApplication3.pgd" /SUBSYSTEM:CONSOLE /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\ConsoleApplication3.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 DLL project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.dll" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /IMPLIB:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.lib" /DEBUG /DLL /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32DLLProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32DLLProject1.dll.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

Win32 Windows app project

/OUT:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.exe" /MANIFEST /NXCOMPAT /PDB:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pdb" /DYNAMICBASE "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "comdlg32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "odbc32.lib" "odbccp32.lib" /DEBUG /MACHINE:X86 /INCREMENTAL /PGD:"C:\Temp\ConsoleApplication3\Debug\Win32WindowsProject1.pgd" /SUBSYSTEM:WINDOWS /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /ManifestFile:"Debug\Win32WindowsProject1.exe.intermediate.manifest" /ERRORREPORT:PROMPT /NOLOGO /TLBID:1

So we can see, for a /DLL, /SUBSYSTEM:WINDOWS is defined.

Then I tried to build a DLL with 3 different SUBSYSTEM values:

• CONSOLE
• WINDOWS
• NOT SET

There are other values as well, but I can only build successfully with the above 3. Other values will lead to linking errors, which means some external symbols are needed.

The full list of possible SUBSYSTEM values is:

You can use the CFF Explorer to inspect the PE/COFF binaries.

The SUBSYSTEM header field locates at the file offset 0x14C of a PE/COFF file. It is part of the Optional Header.

/DLL /SUBSYESTEM:CONSOLE

/DLL /SUBSYESTEM:WINDOWS

/DLL and no /SUBSYESTEM option (by selecting NOT SET in VS2015 project properties page

Interestingly, the NOT SET and WINDOWS values lead to the same content in the binary header.

And I compared the whole binary of the 3 DLLs. It seems the rest of the binaries are all the same, except for some timestamp and debug info.

This is just some fact I found. How /SUBSYSTEM option affects the binary behavior depends on how the loader interprets this field.

In one of the projects I participated in, the /SUBSYSTEM:CONSOLE and /DLL are used together, which is different from the default combination for a DLL project. But it seems nothing bad happens. So I agree with @Frédéric Hamidi that this flag has no functional impact to a DLL.