Custom Dialog Box works in .exe but fails in .dll -> Why?

[Terry_Chappell]

@dale,

I have created a custom dialog box which is defined in a Resource.rc file and its dialog number is specified in a Resource.h file. Here are those files:

These files are used in a Visual Studio project that creates a stand alone Test Dialog.exe file. The Solution Explorer view of the .exe project is here:

They are also used in a separate project to create a mesh_and_cloud.dll as shown here:

The code in the 2 projects is very similar. The main difference is that the Test Dialog.exe project uses a WinMain procedure to call ShowCustomDialog while the mesh_and_cloud.dll project imbeds the call to ShowCustomDialog in a procedure called by a Python script. The .exe version works fine, creating the custom dialog box as shown here.

But when I put the same code in the mesh_and_cloud.dll project, no dialog box is shown. The rest of the mesh_and_cloud.dll works fine but the resource for the custom dialog box is not found as shown here:

The error says that the resource is not found in the image file. To check if the IDD_CUSTOM_MESSAGEBOX is in the mesh_and_cloud.dll file I used Resource Hacker to look at the .dll file and compare it to the Test Dialog.exe as shown here:

On the left is what the Resource Hacker found in the mesh_and_cloud.dll which is the same as in Test Dialog.exe shown on the right. Both the code and the image of the dialog box are the same.

My conclusion is that something in the flow for creating the custom dialog box in the mesh-and_cloud.dll is causing the failure. To investigate this, I compared some of the Properties of the 2 projects. The General Configuration Properties are very similar:

The Advanced Configuration Properties are also very similar:

The VC++ Directories are the same:

The C/C++ General settings appear to have small differences:

The Code Generation setting have 2 differences but do these matter?

The Precompiled header settings have expected differences:

The Linker General settings look the same:

while Linker Input setting show 1 difference in the Additional Dependencies:

I tried adding gdi32.lib;%(AdditionalDependencies) to the mesh_cloud.dll project but it made no difference.

Here is the C++ code for the Test Dialog.exe Project:
main.cpp (9.0 KB)

And here is the C++ code for the mesh_cloud.dll Project:
mesh_and_cloud.cpp (9.6 KB)

Somewhere there is a setting that is needed for the mesh_cloud.dll to find the custom dialog resource and display the dialog box. I have spent 3 days looking for it and feel very dumb at this point. Any help would be greatly appreciated.

By the way, a standard message box like MessageBox work fine in both projects. Only the custom dialog box (aka message box) is being nuked in the .dll project.

Regards,
Terry.

[stevebaer]

oh the joys of C++ resources… I believe you need to call GetModuleHandle with the name of your DLL. If you pass NULL, you are getting the handle for the top level executable (which is why it works for your exe).

[dale (Dale Fugier)]

This might be useful.

/// <summary>
/// Returns module handle where this function is running in: EXE or DLL.
/// </summary>
HMODULE MyGetModuleHandle()
{
  HMODULE hModule = NULL;
  ::GetModuleHandleEx(
    GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT,
    (LPCTSTR)MyGetModuleHandle, 
    &hModule
  );
  return hModule;
}

– Dale

[Terry_Chappell]

I tried a quick check using a manual setting of the dll name:

HINSTANCE hInstance = GetModuleHandleW((LPCWSTR)L"mesh_and_cloud");

which gave us our first success in displaying the custom dialog box in the DLL:

Strangely the message is now too small. Somehow in the DLL case a 0.5X scaling is being applied to both the window and the font of the message:

hRichEdit = CreateWindowEx(0, MSFTEDIT_CLASS, NULL,
    WS_CHILD | WS_VISIBLE | ES_MULTILINE | ES_AUTOVSCROLL | ES_AUTOHSCROLL | ES_READONLY,
    30, 20, 320, 70, hDlg, NULL, GetModuleHandle(NULL), NULL);

 // Set the font
 LOGFONT lf = { 0 };
 lf.lfHeight = 20; // Height of the font
 lf.lfWeight = FW_NORMAL; // Bold font
 wcscpy_s(lf.lfFaceName, L"Arial");

If I increase the window and font by 2X then the DLL dialog box matches the EXE’s:

I tried using the DLL’s Module Handle when calling CreateWindowEx but it made no difference.

Steve thanks for the boost!

Regards,
Terry.

[Terry_Chappell]

This works fine for both the DLL and EXE cases.

Thanks Dale!

Regards,
Terry.