Reference a GNU C (POSIX) DLL built in GCC against Cygwin, from C#/NET

The main problem which you has is following. Before you can use your helloworld.dll a cygwin environment must be initialized (see http://cygwin.com/faq/faq.programming.html#faq.programming.msvs-mingw). So the following code in native C++ will works:

#include <windows.h>

typedef int (*PFN_HELLO)();
typedef void (*PFN_CYGWIN_DLL_INIT)();

int main()
{
    PFN_HELLO fnHello;
    HMODULE hLib, h = LoadLibrary(TEXT("cygwin1.dll")); 
    PFN_CYGWIN_DLL_INIT init = (PFN_CYGWIN_DLL_INIT) GetProcAddress(h,"cygwin_dll_init");
    init(); 

    hLib = LoadLibrary (TEXT("C:\\cygwin\\home\\Oleg\\mydll.dll"));
    fnHello = (PFN_HELLO) GetProcAddress (hLib, "hello");
    return fnHello();
}

Of cause the path to cygwin1.dll must be found. You can set C:\cygwin\bin as a current directory, use SetDllDirectory function or easy include C:\cygwin\bin in the global PATH environment variable (click on right mouse button on Computer, choose Properties then "Advanced System Settings", "Environment variables...", then choose system variable PATH and append it with ";C:\cygwin\bin").

Next if you compile you DLL, you should better to use DEF-file to define BASE address of DLL during compiling and makes all function names, which you exported more clear readable (see http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/gnu-linker/win32.html)

You can verify results with dumpbin.exe mydll.dll /exports, if you have Visual Studio installed. (don't forget start command promt from "Visual Studio Command Prompt (2010)" to have all Visual Studio set).

UPDATED: Because you don't write about the success I think there are exist some problems. In Win32/Win64 world (unmanaged world) it works. The code which I posted I have tested. Loading of CygWin DLLs in .NET can have some problem. In http://cygwin.com/faq/faq.programming.html#faq.programming.msvs-mingw one can read "Make sure you have 4K of scratch space at the bottom of your stack". This requirement can be wrong in .NET. A stack is a part of thread and not a process. So you can try to use CygWin DLLs in the new .NET Thread. Since .NET 2.0 one can define the maximum stack size for the thread. One other way is trying to understand http://cygwin.com/cgi-bin/cvsweb.cgi/~checkout~/src/winsup/cygwin/how-cygtls-works.txt?rev=1.1&content-type=text/plain&cvsroot=src and the code described in http://old.nabble.com/Cygwin-dll-from-C--Application-td18616035.html#a18616996. But the really interesting I find two ways without any tricks:

1. Compiling the DLL with respect of MinGW tools instead of CygWin tools. MinGW produce code which are much more Windows compatible. I am not using CygWin or MinGW myself, so I am not sure, that you will be able to compile all you existing code used POSIX function in MinGW. If it is do possible, this way can have more success. You can look at http://www.adp-gmbh.ch/csharp/call_dll.html for example, to see, that MinGW DLL can be called from C# exactly like a Windows DLL.
2. Usage of CygWin DLL inside of unmanaged process or unmanaged thread. This is a standard way described in CygWin documentation and it works (see example from my first post).

P.S. Please write short in the text of your question if you have success in one of this or in another way which you choose at the end. It's interesting for me independent on reputation and bounty.

You should first try to get your simple hello world sample running. There is not much sense in trying with a huge legacy C/C++ codebase written for POSIX depending on Cygwin if you don't even get the former right. And please note that if you are linking Cygwin in you have to GPL-license your library.

For that have a look at the documentation (e.g. you need to specify explicitly in your hello world example if you are using Cdecl (what you currently don't do)): Consuming Unmanaged DLL Functions

On the native side you have to initialize Cygwin (see winsup/cygwin/how-cygtls-works.txt)

Use P/Invoke to your library directly. There is no sense in PInvoking into Win32 libraries like LoadLibrary to then invoke your libraries. This just adds another layer for errors and is gaining you exactly nothing.

Make sure you are getting the architecture right (.Net applications run 64bit by default on 64bit machines). So make sure your dlls match/support that or limit the .Net to 32bits.

If that works try getting your other library to work. (And you will have to have some luck if you expect to use functions that mix two entirely different threading models)