Do (statically linked) DLLs use a different heap than the main program?

Let’s first understand heap allocation and stack on Windows OS wrt our applications/DLLs. Traditionally, the operating system and run-time libraries come with an implementation of the heap.

1. At the beginning of a process, the OS creates a default heap called Process heap. The Process heap is used for allocating blocks if no other heap is used.
2. Language run times also can create separate heaps within a process. (For example, C run time creates a heap of its own.)
3. Besides these dedicated heaps, the application program or one of the many loaded dynamic-link libraries (DLLs) may create and use separate heaps, called private heaps
4. These heap sits on top of the operating system's Virtual Memory Manager in all virtual memory systems.
5. Let’s discuss more about CRT and associated heaps:
   1. C/C++ Run-time (CRT) allocator: Provides malloc() and free() as well as new and delete operators.
   2. The CRT creates such an extra heap for all its allocations (the handle of this CRT heap is stored internally in the CRT library in a global variable called _crtheap) as part of its initialization.
   3. CRT creates its own private heap, which resides on top of the Windows heap.
   4. The Windows heap is a thin layer surrounding the Windows run-time allocator(NTDLL).
   5. Windows run-time allocator interacts with Virtual Memory Allocator, which reserves and commits pages used by the OS.

Your DLL and exe link to multithreaded static CRT libraries. Each DLL and exe you create has a its own heap, i.e. _crtheap. The allocations and de-allocations has to happen from respective heap. That a dynamically allocated from DLL, cannot be de-allocated from executable and vice-versa.

What you can do? Compile our code in DLL and exe’s using /MD or /MDd to use the multithread-specific and DLL-specific version of the run-time library. Hence both DLL and exe are linked to the same C run time library and hence one _crtheap. Allocations are always paired with de-allocations within a single module.

DLLs / exes will need to link to an implementation of C run time libraries.

In case of C Windows Runtime libraries, you have the option to specify, if you wish to link to the following:

1. Single-threaded C Run time library (Support for single threaded libraries have been discontinued now)
2. Multi-threaded DLL / Multi-threaded Debug DLL
3. Static Run time libraries.
4. Few More (You can check the link)

Each one of them will be referring to a different heap, so you are not allowed pass address obtained from heap of one runtime library to other.

Now, it depends on which C run time library the DLL which you are talking about has been linked to. Suppose let's say, the DLL which you are using has been linked to static C run time library and your application code (containing the main function) has linked to multi-threaded C Runtime DLL, then if you pass a pointer to memory allocated in the DLL to your main program and try to free it there or vice-versa, it can lead to undefined behaviour. So, the basic root cause are the C runtime libraries. Please choose them carefully.

Please find more info on the C run time libraries supported here & here

A quote from MSDN:

Caution Do not mix static and dynamic versions of the run-time libraries. Having more than one copy of the run-time libraries in a process can cause problems, because static data in one copy is not shared with the other copy. The linker prevents you from linking with both static and dynamic versions within one .exe file, but you can still end up with two (or more) copies of the run-time libraries. For example, a dynamic-link library linked with the static (non-DLL) versions of the run-time libraries can cause problems when used with an .exe file that was linked with the dynamic (DLL) version of the run-time libraries. (You should also avoid mixing the debug and non-debug versions of the libraries in one process.)