Hooking DirectX EndScene from an injected DLL

I know this question is old, but this should work for any program using DirectX9, You are creating your own instance basically, and then getting the pointer to the VTable, then you just hook it. You will need detours 3.X btw:

//Just some typedefs:
typedef HRESULT (WINAPI* oEndScene) (LPDIRECT3DDEVICE9 D3DDevice);
static oEndScene EndScene;

//Do this in a function or whatever
HMODULE hDLL=GetModuleHandleA("d3d9");
LPDIRECT3D9(__stdcall*pDirect3DCreate9)(UINT) = (LPDIRECT3D9(__stdcall*)(UINT))GetProcAddress( hDLL, "Direct3DCreate9");

LPDIRECT3D9 pD3D = pDirect3DCreate9(D3D_SDK_VERSION);

D3DDISPLAYMODE d3ddm;
HRESULT hRes = pD3D->GetAdapterDisplayMode(D3DADAPTER_DEFAULT, &d3ddm );
D3DPRESENT_PARAMETERS d3dpp; 
ZeroMemory( &d3dpp, sizeof(d3dpp));
d3dpp.Windowed = true;
d3dpp.SwapEffect = D3DSWAPEFFECT_DISCARD;
d3dpp.BackBufferFormat = d3ddm.Format;

WNDCLASSEX wc = { sizeof(WNDCLASSEX),CS_CLASSDC,TempWndProc,0L,0L,GetModuleHandle(NULL),NULL,NULL,NULL,NULL,("1"),NULL};
RegisterClassEx(&wc);
HWND hWnd = CreateWindow(("1"),NULL,WS_OVERLAPPEDWINDOW,100,100,300,300,GetDesktopWindow(),NULL,wc.hInstance,NULL);

hRes = pD3D->CreateDevice( 
    D3DADAPTER_DEFAULT,
    D3DDEVTYPE_HAL,
    hWnd,
    D3DCREATE_SOFTWARE_VERTEXPROCESSING | D3DCREATE_DISABLE_DRIVER_MANAGEMENT,
    &d3dpp, &ppReturnedDeviceInterface);

pD3D->Release();
DestroyWindow(hWnd);

if(pD3D == NULL){
    //printf ("WARNING: D3D FAILED");
    return false;
}
pInterface = (unsigned long*)*((unsigned long*)ppReturnedDeviceInterface);


EndScene = (oEndScene) (DWORD) pInterface[42];
DetourTransactionBegin();
DetourUpdateThread(GetCurrentThread());
DetourAttach(&(PVOID&)EndScene, newEndScene);
DetourTransactionCommit();

And then your function:

HRESULT WINAPI D3D9Hook::newEndScene(LPDIRECT3DDEVICE9 pDevice)
{   
    //Do your stuff here

    //Call the original (if you want)
    return EndScene(pDevice);
}

You install a system wide hook. (SetWindowsHookEx) With this done, you get to be loaded into every process.

Now when the hook is called, you look for a loaded d3d9.dll.

If one is loaded, you create a temporary D3D9 object, and walk the vtable to get the address of the EndScene method.

Then you can patch the EndScene call, with your own method. (Replace the first instruction in EndScene by a call to your method.

When you are done, you have to patch the call back, to call the original EndScene method. And then reinstall your patch.

This is the way FRAPS does it. (Link)

---

You can find a function address from the vtable of an interface.

So you can do the following (Pseudo-Code):

IDirect3DDevice9* pTempDev = ...;
const int EndSceneIndex = 26 (?);

typedef HRESULT (IDirect3DDevice9::* EndSceneFunc)( void );

BYTE* pVtable = reinterpret_cast<void*>( pTempDev );
EndSceneFunc = pVtable + sizeof(void*) * EndSceneIndex;

EndSceneFunc does now contain a pointer to the function itself. We can now either patch all call-sites or we can patch the function itself.

Beware that this all depends on the knowledge of the implementation of COM-Interfaces in Windows. But this works on all windows versions (either 32 or 64, not both at the same time).

A slightly old question I know - but in case anyone is interested in doing this with C#, here is my example on hooking the Direct3D 9 API using C#. This utilizes EasyHook an open source .NET assembly that allows you to 'safely' install hooks from managed code into unmanaged functions. (Note: EasyHook takes care of all the issues surrounding DLL injection - e.g. CREATE_SUSPENDED, ACL's, 32 vs 64-bit and so on)

I use a similar VTable approach as mentioned by Christopher via a small C++ helper dll to dynamically determine the address of the IDirect3DDevice9 functions to hook. This is done by creating a temporary window handle, and creating a throw-away IDirect3Device9 within the injected assembly before then hooking the desired functions. This allows your application to hook a target that is already running (Update: note that this is possible entirely within C# also - see comments on the linked page).

Update: there is also an updated version for hooking Direct3D 9, 10 and 11 still using EasyHook and with SharpDX instead of SlimDX