Mesh.CreateBooleanIntersection leaks unmanaged memory (Rhino 8 MX engine)

Rhino Developer
, , , ,
1

Affects: Rhino 8.34 (8.34.26223.11002), macOS 26.5.2 (Apple Silicon) — reproduced identically on Rhino 8 for Windows
Severity: unmanaged memory leak, gigabytes per session in plugins that call mesh booleans repeatedly

Summary

Every call to Rhino.Geometry.Mesh.CreateBooleanIntersection (both the legacy overload and the MeshBooleanOptions overload — they appear to route into the same native entry point,
RHC_RhinoMeshBooleanIntDiffUni) leaks unmanaged memory inside the new MX mesh boolean engine.
The .NET managed heap stays flat; the leaked memory is native and is not reclaimed by
GC.Collect() + GC.WaitForPendingFinalizers(), nor when all managed result meshes are disposed.

In our Grasshopper plugin (which ran one small mesh boolean per candidate connection between
structural elements), this leaked ~750 MB per recompute, reaching 9+ GB of process memory in a
few minutes of normal use. Managed memory (dotMemory snapshots) stayed at ~150 MB throughout.

Evidence: macOS leaks with malloc stack logging

Launching Rhino with MallocStackLogging=lite and running leaks after ~8 recomputes:

Process 38766: 6877371 nodes malloced for 13001373 KB
Process 38766: 2207151 leaks for 11303250656 total leaked bytes.

11.3 GB across 2.2 million leaked blocks. The leaked object types are the engine’s internals:
ON_Mesh, ON_SimpleArray<bool>, ON_SimpleArray<unsigned int>, plus raw onmalloc blocks
(reported both as ROOT LEAK and ROOT CYCLE groups). Every top leak group shares this allocation stack (managed JIT frames elided):

...(managed frames)...
com.mcneel.rhinoceros.8      RHC_RhinoMeshBooleanIntDiffUni + 208
com.mcneel.rhinoceros.RhCore RhinoMeshBooleanIntersection(ON_SimpleArray<ON_Mesh const*> const&, ON_SimpleArray<ON_Mesh const*> c...
com.mcneel.opennurbs         MX_MeshMeshBooleanIntersection(ON_SimpleArray<ON_Mesh const*> const&, ON_SimpleArray<ON_Mesh const*>...
com.mcneel.opennurbs         _MX_MeshMeshBoolean(ON_SimpleArray<ON_Mesh const*> const&, double, double, bool*, ON_SimpleArray<ON_...
com.mcneel.opennurbs         MX::PublicIntersectionOps::ComputeMeshIntersections(MX::TessellationArray const&, MX::TessellationAr...
com.mcneel.opennurbs         onmalloc + 44
libsystem_malloc.dylib       _malloc_zone_malloc_instrumented_or_legacy + 152

(Some groups end at _MX_MeshMeshBoolean without the ComputeMeshIntersections frame.)
Full leaks output available on request (157 MB; excerpt attached).

Minimal repro (Grasshopper C# script or any RhinoCommon context)

Run this and watch process memory (Task Manager / Activity Monitor). Managed memory stays flat;
process memory grows by hundreds of MB and never returns, including after forced GC:

var boxA = Mesh.CreateFromBox(new BoundingBox(new Point3d(0, 0, 0), new Point3d(1, 1, 1)), 2, 2, 2);
var boxB = Mesh.CreateFromBox(new BoundingBox(new Point3d(0.5, 0.5, 0.5), new Point3d(1.5, 1.5, 1.5)), 2, 2, 2);

for(int i = 0; i < 20000; i++)
{
    var result = Mesh.CreateBooleanIntersection(new[] { boxA }, new[] { boxB });
    if(result != null)
        foreach(var m in result)
            m.Dispose(); // disposing results makes no difference
}

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();
// managed heap: flat. process memory: grew by GBs and stays there.

Notes from our testing:

  • Occurs with both CreateBooleanIntersection(IEnumerable<Mesh>, IEnumerable<Mesh>) and the MeshBooleanOptions overload — switching between them changed nothing, consistent with both routing into RHC_RhinoMeshBooleanIntDiffUni.
  • Our production calls included open (planar patch) meshes against closed boxes as well as
    closed-vs-closed boxes; leak stacks were recorded for both.
  • Given the shared native entry name (...IntDiffUni), CreateBooleanDifference/Union are likely affected as well (not separately profiled).
  • Reproduced on macOS (Apple Silicon, 8.34) and Windows (Rhino 8). Cross-platform, so the leak is in the shared engine code, not platform glue.

Impact / workaround

We removed mesh booleans from our hot paths entirely (replaced with analytic clipping), but our
remaining legitimate uses (terrain/excavation booleans on large meshes) still leak per call, and
any plugin or scripted workflow calling mesh booleans in a loop will exhaust memory. It seems there is no SDK-level way to reach the pre-MX engine (the +old fallback exists only as UI commands). What do?

Happy to provide the full leaks report.

Cheers
Ben

2

Hi @Benjamin1,

I believe this was fixed in the Rhino Beta - https://mcneel.myjetbrains.com/youtrack/issue/RH-96589. You might give it a try.

– Dale

3

Hi @dale and thanks for the quick response. Just out of interest: will this be patched in Rhino 8 as well?

Cheers
Ben

6

Hi @Benjamin1 - we’re looking into what it takes to push this fix into V8, not an easy task.

– Dale

7

@dale sure. Thanks!

8

This looks like it will be possible. See RH-96589

9

This will be released also in Rhino 8 SR36.

10

RH-96589 is fixed in Rhino 8 Service Release 36 Release Candidate