How can a shared library (.so) call a function that is implemented in its loading program?

You have two options, from which you can choose:

Option 1: export all symbols from your executable. This is simple option, just when building executable, add a flag -Wl,--export-dynamic. This would make all functions available to library calls.

Option 2: create an export symbol file with list of functions, and use -Wl,--dynamic-list=exported.txt. This requires some maintenance, but more accurate.

To demonstrate: simple executable and dynamically loaded library.

#include <stdio.h>
#include <dlfcn.h>

void exported_callback() /*< Function we want to export */
{
    printf("Hello from callback!\n");
}

void unexported_callback() /*< Function we don't want to export */
{
    printf("Hello from unexported callback!\n");
}

typedef void (*lib_func)();

int call_library()
{
   void     *handle  = NULL;
   lib_func  func    = NULL;
   handle = dlopen("./libprog.so", RTLD_NOW | RTLD_GLOBAL);
   if (handle == NULL)
   {
       fprintf(stderr, "Unable to open lib: %s\n", dlerror());
       return -1;
   }
   func = dlsym(handle, "library_function");

   if (func == NULL) {
       fprintf(stderr, "Unable to get symbol\n");
      return -1;
   }

   func();
   return 0;
}

int main(int argc, const char *argv[])
{
    printf("Hello from main!\n");
    call_library();
    return 0;
}

Library code (lib.c):

#include <stdio.h>
int exported_callback();

int library_function()
{
    printf("Hello from library!\n");
    exported_callback();
    /* unexported_callback(); */ /*< This one will not be exported in the second case */
    return 0;
}

So, first build the library (this step doesn't differ):

 gcc -shared -fPIC lib.c -o libprog.so

Now build executable with all symbols exported:

 gcc -Wl,--export-dynamic main.c -o prog.exe -ldl

Run example:

 $ ./prog.exe
 Hello from main!
 Hello from library!
 Hello from callback!

Symbols exported:

 $ objdump -e prog.exe -T | grep callback
 00000000004009f4 g    DF .text  0000000000000015  Base        exported_callback
 0000000000400a09 g    DF .text  0000000000000015  Base        unexported_callback

Now with the exported list (exported.txt):

{
    extern "C"
    {
       exported_callback;
    };
};

Build & check visible symbols:

$ gcc -Wl,--dynamic-list=./exported.txt main.c -o prog.exe -ldl
$ objdump -e prog.exe -T | grep callback
0000000000400774 g    DF .text  0000000000000015  Base        exported_callback

You'll need make a register function in your .so so that the executable can give a function pointer to your .so for it's later use.

Like this:

void in_main_func () {
// this is the function that need to be called from a .so
}

void (*register_function)(void(*)());
void *handle = dlopen("libmylib.so");

register_function = dlsym(handle, "register_function");

register_function(in_main_func);

the register_function needs to store the function pointer in a variable in the .so where the other function in the .so can find it.

Your mylib.c would the need to look something like this:

void (*callback)() = NULL;

void register_function( void (*in_main_func)())
{
    callback = in_main_func();
}

void function_needing_callback() 
{
     callback();
}

1. Put your main function's prototype in a .h file and include it in both your main and dynamic library code.

2. With GCC, simply compile your main program with the -rdynamic flag.

3. Once loaded, your library will be able to call the function from the main program.

A little further explanation is that once compiled, your dynamic library will have an undefined symbol in it for the function that is in the main code. Upon having your main app load the library, the symbol will be resolved by the main program's symbol table. I've used the above pattern numerous times and it works like a charm.