Unit Test Runners, AppDomains, and Testing Plug-In Code

TLDR: If you’re trying to write an automated test for code that exists integrates between Rhino and Your plugin logic, run your tests via a command line interface or something like: https://github.com/MingboPeng/RhinoUnitTest.

This is somewhere between a plea for help and a cautionary tale for others trying to create integration tests for plugins. To be blunt AppDomains are your enemy. Both Visual Studio and Rider’s built in Unit Testing Frameworks have a continuously running process that scans for tests. When you then use them to run the tests, that process kicks off a child appdomain to run those tests. This means that your tests inside of the IDE’s are never running in the main appdomain. The main problem with this is that anything created in the mainAppdomain, singletons for example, will not share state between the loaded version of your *.rhp and the *.dll that your test is in.

For those curious, I did manage to get around this by leveraging: https://github.com/StephenCleary/CrossDomainSingleton/tree/master. And this was working great, except that custom userdata must inherit from Rhino.DocObjects.Custom.UserData, which means they can’t also inherit from MarshalByRefObject, so they can’t be passed across application domains. Not to mention the large amount of additional complexity taken on for sole purpose of unit testing.

After too many hours of digging through test frameworks, XUnit, NUnit, MSTest, and then finally finding this post: https://discourse.mcneel.com/t/how-does-rhinodoc-interact-with-the-appdomain-when-used-inside-autocad/162815/2, I’m reasonable certain that the only way (other that writing my own testing plugin for Rider or Visual Studio) to test of all this is either by running the tests via a CLI or another Rhino Plugin. The simple fact that Rhino is always created on the default application domain means that it doesn’t matter what settings you pass in to the different test frameworks, as long as you’re running them via the existing editor test runners you’re out of luck.

If someone else has run into this, or would like a sample project to fiddle with it themselves, please let me know.

Hey there @cullen.sarles,

I’ve done quite a lot of extensive testing for Rhino and there’s a few ways. I’ve done it via VSCode and/or Visual Studio and I think Rhider on Windows. And these tests have ranged from simple unit tests → Integration Tests → End to End tests.

Unit Tests

Unit tests use Rhino.Inside, which means Windows Only (for now). And these can be run from inside the IDEs Test Explorer.

Integration Tests

If you want to go beyond a RhinoDoc.Headless and interact with visual elements like views, UI elements etc. you’ll need to load Rhino up somehow and run it headed. These cannot currently be run from the IDE Test Explorer and must be run via Debug.

I have a few more questions, but I’m hoping the above might help you solve your issues.

@CallumSykes, thanks for the info.

I am using Rhino.Testing, it definitely makes it easier to setup up tests.

The first place I encounter problems is when programmatically loading my plugin.

My implementation of the RhinoSetupFixture is below. when I hook up to a debugger inside of an IDE, PlugIn.Find always returns null even after PlugIn.LoadPlugIn returns true. When watching the debug output, I can see that Rhino, it’s related assemblies, and my PlugIn are loaded in the root application domain, in my case this is “ReSharperTestRunner64.exe”, but all tests cases are executed in the “Tests” application domain.

If I use NUnitLite to run the test cases via the CLI, everything loads in the the same application domain and I don’t have these issues.

[SetUpFixture]
public class SetupFixture : Rhino.Testing.Fixtures.RhinoSetupFixture
{
    public override void OneTimeSetup()
    {
        AppDomain.CurrentDomain.AssemblyResolve += ResolveChimeraPlugin;
        
        base.OneTimeSetup();
        LoadChimera();
    }

    public static IChimeraPlugin LoadChimera()
    {
        Helpers.PrintAppDomain(nameof(SetupFixture));
        var plugin = PlugIn.Find(PluginConstants.RhinoPluginGuid);
        if (plugin == null)
        {
            if (PlugIn.LoadPlugIn(PluginConstants.RhinoPluginGuid))
            {
                plugin = PlugIn.Find(PluginConstants.RhinoPluginGuid);
            }
            else
            {
                var assemblyPath = Assembly.GetExecutingAssembly().Location;
                var assemblyDir = Path.GetDirectoryName(assemblyPath);
                var pluginPath = Path.Combine(assemblyDir,
                    $"{nameof(Chimera)}.PluginRhino.rhp");
                
                Console.WriteLine($"Did not find Rhino Plugin, attempting to explicitly load ${pluginPath}");
        
                PlugIn.LoadPlugIn(pluginPath, out var loadedGuid);
                if (loadedGuid != PluginConstants.RhinoPluginGuid)
                {
                    throw new Exception("Failed to load Rhino plugin");
                }
                Console.WriteLine($"Loaded ${pluginPath}");
                plugin = PlugIn.Find(PluginConstants.RhinoPluginGuid);
            }
        }
        ChimeraPlugin = plugin as IChimeraPlugin;
        return ChimeraPlugin;
    }

    public static IChimeraPlugin ChimeraPlugin { get; set; }
    
    /// <summary>
    /// Add Chimera libraries to the current Appdomain
    /// </summary>
    private static Assembly ResolveChimeraPlugin(object sender, ResolveEventArgs args)
    {
        var assemblyName = args.Name.Split(',').First();

        if (!assemblyName.StartsWith("Chimera")) return null;
        var assemblyPath = Assembly.GetExecutingAssembly().Location;
        var assemblyDir = Path.GetDirectoryName(assemblyPath);

        if (assemblyName.Contains("PluginRhino"))
        {
            assemblyName += ".rhp";
        }
        else if (assemblyName.Contains("PluginGrasshopper"))
        {
            assemblyName += ".gha";
        }
        else
        {
            assemblyName += ".dll";
        }

        var path = Path.Combine(assemblyDir ?? ".", assemblyName);
        
        return Assembly.LoadFrom(path);

    }
}

Do you have the exact same issue in Visual Studio? And you’re using NUnit to write these tests? If so, you have NuGet Gallery | NUnit3TestAdapter 4.5.0 installed correct?

Same behavior:

Sanitized Debug output, all the way into a test:

‘testhost.net48.exe’ (CLR v4.0.30319: domain-ce8773c8-Chimera.RhinoCore.Test.exe): Loaded ‘\Chimera\bin\DebugR7\net48\Chimera.Core.dll’. Symbols loaded.
Executed SetupFixture.LoadChimera on AppDomain: domain-ce8773c8-Chimera.RhinoCore.Test.exe
‘testhost.net48.exe’ (CLR v4.0.30319: testhost.net48.exe): Loaded ‘\Chimera\bin\DebugR7\net48\Chimera.PluginRhino.rhp’. Symbols loaded.
‘testhost.net48.exe’ (CLR v4.0.30319: testhost.net48.exe): Loaded ‘\Chimera\bin\DebugR7\net48\Chimera.RhinoCore.dll’. Symbols loaded.
‘testhost.net48.exe’ (CLR v4.0.30319: testhost.net48.exe): Loaded ‘\Chimera\bin\DebugR7\net48\Chimera.Core.dll’. Symbols loaded.
Executed PluginRhino…ctor on AppDomain: testhost.net48.exe
Executed RhinoDocumentHandler…ctor on AppDomain: testhost.net48.exe
‘testhost.net48.exe’ (CLR v4.0.30319: domain-ce8773c8-Chimera.RhinoCore.Test.exe): Loaded ‘C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\PrivateAssemblies\Runtime\Microsoft.VisualStudio.Debugger.Runtime.Desktop.dll’. Skipped loading symbols. Module is optimized and the debugger option ‘Just My Code’ is enabled.
Executed Setup.Setup on AppDomain: domain-ce8773c8-Chimera.RhinoCore.Test.exe
‘testhost.net48.exe’ (CLR v4.0.30319: testhost.net48.exe): Loaded ‘C:\Program Files\Rhino 7\System\websocket-sharp.dll’. Module was built without symbols.
‘testhost.net48.exe’ (CLR v4.0.30319: testhost.net48.exe): Loaded ‘\Chimera\bin\DebugR7\net48\nunit.framework.dll’. Skipped loading symbols. Module is optimized and the debugger option ‘Just My Code’ is enabled.
Executed CreateNewDocumentTest.CreateNewDocumentTest on AppDomain: domain-ce8773c8-Chimera.RhinoCore.Test.exe

You can see “testhost.net48.exe” loading in the dll’s and constructing the Plugin and that the Setup and Tests are being run in a subdomain.

My Package.Props file for Testing Packages:

<?xml version="1.0" encoding="utf-8"?>
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!-- 
    packages for testing dependencies 
  -->
  <ItemGroup>
    <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
    <PackageVersion Include="NUnit3TestAdapter" Version="4.5.0"/>
    <PackageVersion Include="NUnit" Version="4.1.0"/>
    <PackageVersion Include="NUnit.Analyzers" Version="4.2.0"/>
    <PackageVersion Include="NUnitLite" Version="4.1.0"/>
    <PackageVersion Include="Rhino.Testing" Version="8.0.16-beta"/>
    <PackageVersion Include="coverlet.collector" Version="6.0.2"/>
  </ItemGroup>
</Project>

I’m not sure if I got the problem correctly.

In all of my C# apps I use a so called ServiceProvider object as part of Dependency Injection, which provides me references to all singletons or provides me with new instances of all transient “service” objects. This is quite standard for modern .Net apps. You don’t need to use the Microsoft default implementation. It’s about the idea.

A single service provider object which acts as a single global variable, gives you all access within your app. During unit testing you init the same ServiceProvider during the global setup and replace untestable parts of your software, like an IOService, Logger or a DialogService with Mockups or empty implementations.

I never had issues with AppDomains because it will and should run isolated like this. Im not sure how feasible everything is inside the RhinoDomain, because plugin development is quite limiting in some regards. But I don’t see why it shouldn’t work.

It’s strange that you mix AppDomains. That alone sounds to be wrong. A singleton should init itself only once (threadsafe). But that should be it.

I’m completely onboard the not mixing AppDomains train. I only even started learning about them because of the behavior documented above. I would love to use DI for all of this. What are your thoughts on setting up unit tests for retrieving custom UserData from a RhinoObject?

For reference, my current approach follows this pattern. In a class library define an Interface as well as data retrieval callback delegates. Then implement that interface with custom userdata in the plugin and assign the callback functions for adding the data and retrieving the data during plugin startup. It all looks something like this:

IChimeraUserData.cs

namespace Chimera.Shared;

interface IChimeraUserData
{
    string Data {get; set;}
}

static class UserDataHelpers
{
    public delegate IChimeraRhinoData CreateUserObjectDelegate(RhinoObject ro);
    public delegate bool RemoveUserObjectDelegate(RhinoObject ro);
    public delegate bool TryGetUserObjectDelegate(ObjectAttributes objectAttributes, out IChimeraRhinoData rhinoData);

    public static CreateUserObjectDelegate CreateObjectFunc { get; set; }
    public static RemoveUserObjectDelegate RemoveObjectFunc { get; set; }
    public static TryGetUserObjectDelegate TryGetObjectFunc { get; set; }   
}

PluginRhino.cs

namespace Chimera.PluginRhino;

public class PluginRhino : Rhino.PlugIns.PlugIn
{
    protected override LoadReturnCode OnLoad(ref string errorMessage)
    {           
        UserDataHelpers.CreateObjectFunc = ChimeraData.CreateObject;
        UserDataHelpers.RemoveObjectFunc = ChimeraData.RemoveObject;
        UserDataHelpers.TryGetObjectFunc = ChimeraData.TryGetObject;
        return base.OnLoad(ref errorMessage);
    }
}

ChimeraData.cs

namespace Chimera.RhinoPlugin;

[Guid("00000000-0000-0000-0000-000000000000")] //blanked in this example
public class ChimeraData : Rhino.DocObjects.Custom.UserData, IChimeraUserData
{
    public ChimeraData()
    {
    }

    public override string Description => "Custom User Data for Chimera";

    public string Data {get; set;}

    protected override void OnDuplicate(UserData source)
    {
        base.OnDuplicate(source);
        if(source is not ChimeraData sourceData) return;
        Data = sourceData.Data;
    }

    #region Serialization
    
    private const int _MAJOR_VERSION = 0;
    private const int _MINOR_VERSION = 1;

    public override bool ShouldWrite => true;

    protected override bool Read(BinaryArchiveReader archive)
    {
        archive.Read3dmChunkVersion(out int major, out int minor);

        Data = archive.ReadString();

        return true;
    }

    protected override bool Write(BinaryArchiveWriter archive)
    {
        archive.Write3dmChunkVersion(_MAJOR_VERSION,_MINOR_VERSION);
        
        archive.WriteString(_data);

        return true;
    }
    
    #endregion

    #region UserData Callback Implementations
    public static IChimeraRhinoData CreateObject(RhinoObject ro)
    {
        if (TryGetReferenceObject(ro.Attributes, out var existingRef)) return existingRef;
        
        var roAtt = ro.Attributes.Duplicate();
        var userObject = new ChimeraData{ Data = "Initial Data"};
        roAtt.UserData.Add(userObject);
        ro.Document.Objects.ModifyAttributes(ro, roAtt, true);
        return userObject;
    }

    public static bool TryGetObject(ObjectAttributes attributes, out IChimeraRhinoData rhinoData)
    {
        var cdata = attributes.UserData.Find(typeof(ChimeraData));
        rhinoData = (IChimeraRhinoData)cdata;
        return rhinoData is not null;
    }

    public static bool RemoveObject(RhinoObject ro)
    {
        var roAtt = ro.Attributes.Duplicate();
        var userObject = roAtt.UserData.Find(typeof(ChimeraData));
        if (userObject is null) return false;
        if (!roAtt.UserData.Remove(userObject)) return false;
        ro.Document.Objects.ModifyAttributes(ro, roAtt, true);
        return true;
    }

    #endregion
} 

This all works and makes it easy to get user data in context of the shared library, the rhino plugin, or the associated grasshopper plugin. I’ve tested via normal debugging without issue. But creating unit tests for it has proven difficult, and is where I ran into the issue with Rhino.Inside, and Rhino in general, always running on the default application domain. With the unit test runners running the tests in a subdomain, and the plugin in being loaded an run on the default domain, the userdata is null. When I attach a debugger, I can see the userdata being added to the object, see it there in the context of the plugin, and then when I inspect the object in the context of the test, the userData on the object attributes has a null in the place where the userdata should be. In fact, because the callbacks are set as part of the plugin load function, they’re null when I try and use them in the context of the tests. This is where I got to about a month ago, and where I went looking for a way to handle dealing with cross-appdomain things.

Also had issues with UserData, must be accessed from the defining Rhino Plugin assembly. There’s a thread for that somewhere in here.

1 Like

I will also add, i have never managed to get UserData working from the non-plugin assembly. I ended up using UserDictionaries instead

@CallumSykes @sonderskovmathias

I think you’re thinking about this thread: add-userdata-object-reference-not-set-to-the-instance-of-an-object. I’ve been using what I posted there in my internal plugin for 2 years, it works great. I’m currently trying to add the capability to read the user data stored on a document into a block definition when you reference a whole document in, and so am going back through and writing tests for the existing behavior to make sure I don’t mess things up as I go forward.

I’m happy to talk about that implementation in a separate thread, let me know and I’ll create a pull request for the RhinoCommon example repo. I don’t want to get too far from the behavior that inspired this thread. Stepping back from UserData, you can observe the behavior with just this code inside of a test case. Assuming that you’ve loaded and registered your plugin, this code loads the plugin, but PlugIn.Find returns null, causing the last line to fail. At least for me, when running directly from the test explorers in either Visual Studio or Rider. If I run the test via NUnitLite as a standalone executable it succeeds. As far as I can tell this is because regardless of the AppDomain that Rhino.Inside is run from, it always creates the Rhino Process on the Default Application Domain.

[Test]
void LoadPluginTest()
{
    var plugin = PlugIn.Find(PluginConstants.RhinoPluginGuid);
    if (plugin is null)
    {
        if (PlugIn.LoadPlugIn(PluginConstants.RhinoPluginGuid))
        {
            plugin = PlugIn.Find(PluginConstants.RhinoPluginGuid);
        }
        else
        {
           Assert.Fail();
        }
    }
    Assert.That(plugin, Is.Not.Null);
}