How to check if an executable or DLL is build in Release or Debug mode (C++)
There is an old reversing tool called LordPE. It will allow you to open two files and diff the headers. I compiled a "hello world" program in VS2008 in Release and Debug mode and compared them. As in the other posters, I didn't see anything that would serve as an indicator.
But what I did find as an indicator was the padding in the .text section of the binary. There is over a hundred bytes with the value 0xCC after the last byte of code in the .text section in the Debug version. In the Release version, there were no 0xCC bytes. 0xCC bytes will show up as int3 or breakpoints in a debugger.
The question is very good, and as already stated, there are no real obvious (unique) indicators that flag of whether or not an image is debug or released.
As explained here and here, the presence of a Debug Directory is NOT an indicator about whether or not an image has been built in Release Mode. It is very common that released images are built with debug support . As a matter of fact, almost ALL Windows OS image files are built with debug support (otherwise, there would be NO possibility to link these Released images with the symbols files from the Microsoft Symbols Server). Even though these images are Release images!
Even the presence of the .debug section (actually, Sections names do NOT plays a role in the PE specification, the name of a section can be changed and set as you wish - the Loader don't care about it!) is NOT an indicator of Release vs. Debug image.
When you create a C++ project in Visual Studio it generates two configurations for you. The names of these configurations are Debug and Release. The Debug configuration includes generation of debug information, less optimization and support for Edit&Continue.
But this is only a starting point. You can create arbitrary configurations and even add all debug information to the Release configuration. So there is no clear Debug or Release build.
You can try to identify if the preprocessor symbol _DEBUG was defined. This is seldom changed and it is used in the version resource. The bit 0 of the FILEFLAGES field usually indicates that the symbols _DEBUG was defined when compiling the resource.
I need to find the mode exe/dll was build looking at its headers.
If by "headers" you mean PE sections or resources (headers won't tell you anything, and programs are not usually shipped with their development headers!), this is kind of possible, within limits, and unreliably. Otherwise, this is an entirely impossible endeavour unless you wrote the program yourself.
Generally, it is hard to do such a thing in a reliable way, even more so as "debug build" is a Microsoft Visual Studio simplification that does not exist as such under most compilers. For example, with GCC it is perfectly allowable to have an optimized build that nevertheless contains debug symbols. It is even possible to turn optimizations on and off with #pragma (and change the optimization level and even the target machine!) and thus have optimized functions (or groups of functions) in an unoptimized build, and vice versa.
The presence of debug symbols is your best guess for a program that you didn't write. It is not possible (not realistically, in a simple, automated way, anyway) to tell from a generated binary whether it has been optimized or not.
The sections .debug$S and .debug$T contain debug symbols and debug types, respectively. There are some other sections starting with .debug as well, but they're deprecated. A program that has been built in "debug mode" and that has not afterwards been stripped will contain some or all of these sections.
Using C++ with no external tools, you will want to skip over the DOS "MZ" stub and the PE header. After this come the section headers, which you can parse. Complete documenation of the file format can be downloaded here.
Most probably, reading the file in and doing a string match for .debug will be just as good.
Similarly, you can look at VERSIONINFO or the manifest file (they also allow to specify whether a program is a debug build), but these are not mandatory. You can write pretty much anything you want into these. Insofar, they're even less reliable than looking for debug symbols.
Another hint, unreliable again, would be to check what versions of system libraries a program was linked with. If it's the debug version, chances are it was a debug build. However, one could do a release build and still link with debug libraries, nothing can prevent you from doing that.
The next best guess would be the absence of calls to the CRT assert function (which you could do with a simple string match), since the assert macro (from which it's normally called) is entirely stripped out in a build with NDEBUG defined. No use of that symbol, no string present in the binary.
Unluckily, a program that doesn't have any asserts would be falsely identified as "release build" regardless of its actual build, and it is entirely possible to redefine the assert macro to do something completely different (such as printf a text and continue). And lastly, you don't know wheter some static 3rd party library that you link with (which obviously has already passed the preprocessor) contains calls to assert that you don't know about.
If you want to check a program that you wrote yourself, you can exploit the fact that the optimizer will completely remove things that are provably unreachable or not used. It may take 2-3 attempts to get it just right, but basically it should be as simple as defining a variable (or an exported function if your compiler/linker doesn't export symbols that aren't used) and writing two or three magic values to it from a program location that is not reachable. An optimizing compiler will at least collapse those several redundant moves into one, or more likely entirely eleminate them all.
You can then just do a binary string search for the magic values. If they're not present, it's an optimized build.