Segmentation fault when using a shared_ptr for private_key

Global variables, and especially singletons, are the scourge of multithreaded, complex applications. You'll be always running into such problems with this sort of design.

Here's what I usually do: everything global gets defined as a local variable in main or some sub-function, in proper order, so that it gets destroyed in an appropriate reverse order. Dependency-injection-like techniques can be used to pass those objects around in cases where "almost everything" depends on them. It took me some pain to realize that this was essentially the only way that was debuggable in large, complex applications (think 2M loc between the app itself and the dozens of libraries it uses outside of the C++ library). After the globals got eviscerated from the bespoke code, and then from a few problematic libraries, the specter of "death on closure" pretty much vanished. I don't guarantee that it'll fix everyone's problems - since people can be quite creative at coming up with new ones - but it's IMHO a step in the right direction.

This is an example of static de-initialization order ‘fiasco’.

There are technics to prevent this, but as you link libraries they might not work as you can't control their life time.

So the best solution might be to explicit clear the content of globalCreds before the program exits, at end of main or in an atexit function. Next best is to leak the structure if there is no need to clean up.

How to leak example taken from isocpp

TLS::credentials& x() {
  static TLS::credentials* creds = new TLS::credentials();
  return *creds;
}
TLS::credentials &globalCreds = x();

Yes that offends my feeling of neatness too.

But this is just work arounds, globalCreds should be created in main, passed to the classes (that are also created in main, after Creds) that needs it as a reference.

Author of Botan replied to me that

The problem is the globally defined object.

The problem is that the mlock pool is a singleton created on first use then destroyed sometime after main returns. First your object is created. It allocates memory. This results in the pool being created. Destruction happens LIFO. So first the pool is destroyed. Then your object is destroyed, and attempts to touch memory (to zero it) which has already been unmapped.

Workarounds,

• Create a Botan::Allocator_Initializer object to force initialization before your object is created (thus the pool lives until after your object has been destructed)
• Disable locking_allocator module
• Set env var BOTAN_MLOCK_POOL_SIZE to 0
• No global vars

In principle the locking allocator instead of munmaping the memory, just zeros it, and leave it to be unmapped by the OS on process exit. This might still break invariants, but not as badly. It also causes valgrind to reports leaks which is obnoxious.

I think because it was mmap'ed directly and not through malloc, valgrind doesn't track it.