What SAFESEH:NO option actually do

Short answer: disabling SafeSEH will reduce your program security.

---

Details: SafeSEH is a compiler protection.

On a Windows environment SEH (Structured Exception Handler) records are laid out as follows

Stack data (pointed by TEB - thread environment block)
|
|  I) Pointer to next SEH record II
|  EH pointer
|
|  II) Pointer to next SEH record III
|  EH pointer
|
|  0xFFFFFF
|  default EH (MSVCRT)

Usually SEH-based attacks rely on overwriting one of the above records and having the application throw an exception: this will detour the control flow to your code (I'm not taking into account DEP/ASLR protection systems here so I'm assuming a known +X location). More precisely they often "simulate a EH return" and they fetch the next "evil-crafted" pointer to jump to the shellcode.

SafeSEH works by instructing the operating system to first check the handler pointers for validity (against a table of known valid EHs) before jumping to them. There are a few restrictions to this process and under special circumstances an application might still be vulnerable but a SEH-based attack is less likely to take place (or significantly harder to craft).

When linking against a non-safeSEH compiled module the linker won't be able to generate a "trusted table" of EH locations (it simply cannot tell where and if those are valid EHs) thus the error you're getting.

Some logistic restrictions on the Windows OS engineering, compatibility reasons and the problems bound to controlling addresses falling out of the range of loaded modules (and executable image) led to the choice of disabling this option by default and leaving the user the choice whether to enable it or not.

If your application desperately needs security and you repute the above scenario a potential threat, you should enable it and recompile your modules in order to use it.

/SAFESEH produces a "Safe Exception Handler Table":

>dumpbin safeseh_yes.dll /loadconfig | find "xcept"
            3001F4D0 Safe Exception Handler Table
                   1 Safe Exception Handler Count
    Safe Exception Handler Table
          30018FE0  __except_handler4

/SAFESEH:NO produces no table:

>dumpbin safeseh_no.dll /loadconfig | find "xcept"
            00000000 Safe Exception Handler Table
                   0 Safe Exception Handler Count

If the table is present the OS uses it to verify that a SEH handler is valid before calling it.