Forcing a Lua script to exit

You could use setjmp and longjump, just like the Lua library does internally. That will get you out of pcalls and stuff just fine without need to continuously error, preventing the script from attempting to handle your bogus errors and still getting you out of execution. (I have no idea how well this plays with threads though.)

#include <stdio.h>
#include <setjmp.h>
#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"

jmp_buf place;

void hook(lua_State* L, lua_Debug *ar)
{
    static int countdown = 10;
    if (countdown > 0)
    {
        --countdown;
        printf("countdown: %d!\n", countdown);
    }
    else
    {
        longjmp(place, 1);
    }
}

int main(int argc, const char *argv[])
{
    lua_State* L = luaL_newstate();
    luaL_openlibs(L);
    lua_sethook(L, hook, LUA_MASKCOUNT, 100);

    if (setjmp(place) == 0)
        luaL_dostring(L, "function test() pcall(test) print 'recursing' end pcall(test)");

    lua_close(L);
    printf("Done!");
    return 0;
}

The way to end a script is to raise an error by calling error. However, if the user has called the script via pcall then this error will be caught.

You could set a variable somewhere in your program and call it something like forceQuitLuaScript. Then, you use a hook, described here to run every n instructions. After n instructions, it'll run your hook which just checks if forceQuitLuaScript is set, and if it is do any clean up you need to do and kill the thread.

Edit: Here's a cheap example of how it could work, only this is single threaded. This is just to illustrate how you might handle pcall and such:

#include <stdlib.h>
#include "lauxlib.h"

void hook(lua_State* L, lua_Debug *ar)
{
    static int countdown = 10;
    if (countdown > 0)
    {
        --countdown;
        printf("countdown: %d!\n", countdown);
    }
    else
    {
        // From now on, as soon as a line is executed, error
        // keep erroring until you're script reaches the top
        lua_sethook(L, hook, LUA_MASKLINE, 0); 
        luaL_error(L, "");
    }
}

int main(int argc, const char *argv[])
{
    lua_State* L = luaL_newstate();
    luaL_openlibs(L);
    lua_sethook(L, hook, LUA_MASKCOUNT, 100);
    // Infinitely recurse into pcalls
    luaL_dostring(L, "function test() pcall(test) print 'recursing' end pcall(test)");
    lua_close(L);
    printf("Done!");
    return 0;
}

It seems like you could terminate the thread externally (from your main thread) since the lua script is user supplied and you can't signal it to exit.

If that isn't an option, you could try the debug API. You could use lua_sethook to enable you to regain control assuming you have a way to gracefully terminate your thread in the hook.