Porting VC++'s __try/__except EXCEPTION_STACK_OVERFLOW to MinGW

You would need to manually call the Windows API functions which register exception handling; namely, AddVectoredExceptionHandler. Note that by using MinGW which does not respect SEH exceptions, throwing any SEH exception or attempting to catch any such exception will result in undefined behavior, because the normal C++ stack unwinding semantic isn't done. (How does Windows know to nuke all those std::strings on the stack?)

You would also need to call RemoveVectoredExceptionHandler at the end of the time you want that SEH exception handler to be called.

Generally MinGW is lacking in support of Windows features like SEH and COM. Any reason you're trying to use that instead of MSVC++ (given that both compilers are free?)

This isn't well known, but the header file <excpt.h> in MinGW and MinGW-w64 provides macros __try1 and __except1 to produce gcc inline assembly for handling exceptions. These macros are not documented and are not easy to use. It gets worse. The x86_64 editions of __try1 and __except1 aren't compatible with the 32-bit editions. They use different callbacks with different arguments and different return values.

After a few hours, I almost had working code on x86_64. I needed to declare a callback with the same prototype as _gnu_exception_handler in MinGW's runtime. My callback was

long CALLBACK
ehandler(EXCEPTION_POINTERS *pointers)
{
    switch (pointers->ExceptionRecord->ExceptionCode) {
    case EXCEPTION_STACK_OVERFLOW:
        return EXCEPTION_EXECUTE_HANDLER;
    default:
        return EXCEPTION_CONTINUE_SEARCH;
    }
}

And my try-except code was

    __try1 (ehandler) {
        sum = sum1to(n);
        __asm__ goto ( "jmp %l[ok]\n" :::: ok);
    } __except1 {
        printf("Stack overflow!\n");
        return 1;
    }
ok:
    printf("The sum from 1 to %u is %u\n", n, sum);
    return 0;

It was working until I enabled optimization with gcc -O2. This caused assembler errors so my program no longer compiled. The __try1 and __except1 macros are broken by an optimization in gcc 5.0.2 that moves functions from .text to a different section.

When the macros did work, the control flow was stupid. If a stack overflow happened, the program jumped through __except1. If a stack overflow didn't happen, the program fell through __except1 to the same place. I needed my weird __asm__ goto to jump to ok: and prevent the fall-through. I can't use goto ok; because gcc would delete __except1 for being unreachable.

After a few more hours, I fixed my program. I copied and modified the assembly code to improve the control flow (no more jump to ok:) and to survive gcc -O2 optimization. This code is ugly, but it works for me:

/* gcc except-so.c -o except-so */
#include <windows.h>
#include <excpt.h>
#include <stdio.h>

#ifndef __x86_64__
#error This program requires x86_64
#endif

/* This function can overflow the call stack. */
unsigned int
sum1to(unsigned int n)
{
    if (n == 0)
        return 0;
    else {
        volatile unsigned int m = sum1to(n - 1);
        return m + n;
    }
}

long CALLBACK
ehandler(EXCEPTION_POINTERS *pointers)
{
    switch (pointers->ExceptionRecord->ExceptionCode) {
    case EXCEPTION_STACK_OVERFLOW:
        return EXCEPTION_EXECUTE_HANDLER;
    default:
        return EXCEPTION_CONTINUE_SEARCH;
    }
}

int main(int, char **) __attribute__ ((section (".text.startup")));

/*
 * Sum the numbers from 1 to the argument.
 */
int
main(int argc, char **argv) {
    unsigned int n, sum;
    char c;

    if (argc != 2 || sscanf(argv[1], "%u %c", &n, &c) != 1) {
        printf("Argument must be a number!\n");
        return 1;
    }

    __asm__ goto (
        ".seh_handler __C_specific_handler, @except\n\t"
        ".seh_handlerdata\n\t"
        ".long 1\n\t"
        ".rva .l_startw, .l_endw, ehandler, .l_exceptw\n\t"
        ".section .text.startup, \"x\"\n"
        ".l_startw:"
            :::: except );
    sum = sum1to(n);
    __asm__ (".l_endw:");
    printf("The sum from 1 to %u is %u\n", n, sum);
    return 0;

except:
    __asm__ (".l_exceptw:");
    printf("Stack overflow!\n");
    return 1;
}

You might wonder how Windows can call ehandler() on a full stack. All those recursive calls to sum1to() must remain on the stack until my handler decides what to do. There is some magic in the Windows kernel; when it reports a stack overflow, it also maps an extra page of stack so that ntdll.exe can call my handler. I can see this in gdb, if I put a breakpoint on my handler. The stack grows down to address 0x54000 on my machine. The stack frames from sum1to() stop at 0x54000, but the exception handler runs on an extra page of stack from 0x53000 to 0x54000. Unix signals have no such magic, which is why Unix programs need sigaltstack() to handle a stack overflow.

You might want to look into LibSEH for adding Structured Exception Handling compatibility for MinGW.