Making clickable icons through the display pipeline

Ah dang, I had actually written an example file for the AECtech workshop that does that too (i.e. shooting rays along the frustum line at the mouse position onto a list of meshes and drawing the closest hit mesh):

""" Highlight closest mesh under mouse position. Author: Anders Holden Deleuran """

import Rhino as rc
import Grasshopper as gh
import System.Drawing as sd
import GhPython.Assemblies.ExecutingComponent as component

class MyMouseCallback(rc.UI.MouseCallback):
    
    def __init__(self):
        self.move = None
        
    def OnMouseMove(self,arg):
        self.move = arg
        ghenv.Component.ExpireSolution(True)

class MyComponent(component):
    
    def RunScript(self,Meshes):
        self.meshes = Meshes
        self.material = rc.Display.DisplayMaterial()
        
    def DrawViewportMeshes(self,arg):
        
        # Get mouse viewport position
        if self.mouse.move:
            mpt = self.mouse.move.ViewportPoint
            
            # Make ray along frustum line at mouse
            r,fl = arg.Viewport.GetFrustumLine(mpt.X,mpt.Y)
            ray = rc.Geometry.Ray3d(fl.From,fl.Direction)
            
            # Shot ray onto meshes
            hitMeshes = []
            hitParams = []
            for m in self.meshes:
                t = rc.Geometry.Intersect.Intersection.MeshRay(m,ray)
                if t >= 0.0:
                    hitMeshes.append(m)
                    hitParams.append(t)
                    
            # Get closest hit mesh and draw it
            if hitParams:
                hm = hitMeshes[hitParams.index(max(hitParams))]
                arg.Display.DrawMeshShaded(hm,self.material)
                
    def __enter__(self):
        self.mouse = MyMouseCallback()
        self.mouse.Enabled = True
        
    def __exit__(self):
        self.mouse.Enabled = False
        del self.mouse


240705_HighlightMeshUnderMouse_00.gh (5.8 KB)

Which is close to what @mrhe is suggesting, I think. But also pretty off topic, which I think was clickable HUD elements. But who cares, this is a great topic now. Thanks guys and happy Friday :partying_face:

Edit: Just added a check to make sure we get a mouse position, which was causing an error report in the Rhino command line upon initialisation.

2 Likes

Thanks @AndersDeleuran this seems to be much more stable than my solution!

Silly question though, how do I simply output the index of the hit mesh?

1 Like

Iā€™ve never actually done this, but one can get/output data from inside DrawViewportMeshes like so:


240705_HighlightMeshUnderMouse_01.gh (6.3 KB)

If the goal is to output things and draw them downstream, it might be easier to do all the raycasting business inside the RunScript scope though.

Edit: Forgot to add self.hitMesh = None on line 49:
240705_HighlightMeshUnderMouse_02.gh (6.6 KB)

1 Like

Before optimizing prematurely, benchmark it. Ray casting is fast, really fast. Granted, Rhinoā€™s implementation is not as fast as embree or optix, but you should be easily able to cast a few thousand rays and maintain interactive performance. What tends to be the bottleneck, though, are simple view redraw calls with denser scenes.

Setup a test scene with 1.000 objects and redraw your view on mouse move. No other logic is executed there. Now, make it 10.000, then 100.000 objects. At some point your model will slow down to a crawl, and there is very little you can do about it.

As for ray casting itself. There are many strategies to speed this up:

  1. Multithread it
  2. Introduce a broad collision phase where you first check a bounding box and only if it is hit, you proceed to the narrow and more expensive collision test
  3. Use some sort of space partitioning structure. Rhino has the Rtree class, embree uses its own BVH (bounding volume hierarchy), but it could also be a screen space grid
  4. Join all meshes into one and only ever shoot one ray (youā€™d get the original mesh id based on a lookup table keeping track of original face indices). Joining meshes is slow but if you only do it once, it might be worth it
  5. Any combination of the above methods.

All depends on your use case.
Good luck exploring the options!

2 Likes

Thanks @AndersDeleuran ,

I still need to pull the raytrace logic out of the Draw function but I modified to be able to get a selected index as well so that it now supports selection and hover:

""" Highlight closest mesh under mouse position. Author: Anders Holden Deleuran """
import Rhino as rc
import scriptcontext as sc
import ghpythonlib.component as component  # Changed to this for Python 3


class MyMouseCallback(rc.UI.MouseCallback):

    def __init__(self):
        self.move = None
        self.click = None

    def OnMouseDown(self, e):
        self.click = True
        self.button = e.Button
        ghenv.Component.ExpireSolution(True)

    def OnMouseUp(self, arg):
        self.click = False
        ghenv.Component.ExpireSolution(True)

    def OnMouseMove(self, arg):
        self.move = arg
        ghenv.Component.ExpireSolution(True)


class MyComponent(component):

    def RunScript(self, Meshes: list[object]):
        self.meshes = Meshes
        self.material = rc.Display.DisplayMaterial()
        return self.hitMesh, self.hitIndex, self.SelectedIndex

    def DrawViewportMeshes(self, arg):
        # Get mouse viewport position
        if self.mouse.move:
            mpt = self.mouse.move.ViewportPoint

            # Make ray along frustum line at mouse
            r, fl = arg.Viewport.GetFrustumLine(mpt.X, mpt.Y)
            ray = rc.Geometry.Ray3d(fl.From, fl.Direction)

            # Shot ray onto meshes
            hitMeshes = []
            hitParams = []
            hitIndices = []
            for i, m in enumerate(self.meshes):
                t = rc.Geometry.Intersect.Intersection.MeshRay(m, ray)
                if t >= 0.0:
                    hitMeshes.append(m)
                    hitParams.append(t)
                    hitIndices.append(i)

            # Get closest hit mesh and draw it
            if hitParams:
                hm = hitMeshes[hitParams.index(min(hitParams))]
                hi = hitIndices[hitParams.index(min(hitParams))]
                # self.hitMesh = hm
                self.hitMesh = None
                self.hitIndex = hi
                if self.mouse.click and str(self.mouse.button) == "Left":
                    sc.sticky["sel_index"] = hi
                    self.SelectedIndex = sc.sticky["sel_index"]
                else:
                    self.SelectedIndex = sc.sticky["sel_index"]

            else:
                self.hitMesh = None
                self.hitIndex = None

    def __enter__(self):
        self.hitMesh = None
        self.hitIndex = None
        self.SelectedIndex = None
        self.mouse = MyMouseCallback()
        self.mouse.Enabled = True

    def __exit__(self):
        self.mouse.Enabled = False
        del self.mouse

Pretty sure thereā€™s some stuff in here I donā€™t need anymore so Iā€™ll comeback and clean it up and reupload that version but just sharing for now.

Thanks for the help!

1 Like

I really appreciate the advice and ideas, thank you for sharing as always!

1 Like

Hey @michaelvollrath,

Etoā€™s co-ordinates are in logical pixels which is not quite compatible when you have different DPIs, or even multiple screens (due to how Windows treats coordinates in system-dpi aware mode vs Etoā€™s multi-monitor dpi aware mode). To convert a point in physical pixels (e.g. what you get from the viewport, or windows forms APIs, etc), you can use Rhino.UI.EtoExtensions.ToEtoScreen(myPointOrRect). This API was added in Rhino 8.7, and directly calls Etoā€™s Eto.Forms.WpfHelpers.ToEtoScreen() in Eto.Wpf.dll.

Thereā€™s also a Rhino.UI.EtoExtensions.ToSystemDrawingScreen() to convert in the opposite direction, if needed.

Hope this helps!

4 Likes

Your unstoppable determination is sensational. I hope you know Iā€™m taking my sweet time with this example because itā€™s fun to see what happens when youā€™re left unattended with an exciting problem.

1 Like

Thank you @curtisw this is perfect! I went on vacation recently (laptop) and couldnā€™t work on my UI developments because the DPI thing had me completely stumped. Iā€™m looking forward to updating with the methods you mentioned, thank you for the information!

:laughing: Donā€™t take too long though, Iā€™m about to stress test on some massive files and quite confident these solutions wonā€™t stand the abuse and will likely need a life raft, but hopefully Iā€™ll be pleasantly surprised!

1 Like

Hereā€™s a quick update that demonstrates how to move all the business logic into RunScript instead (i.e. how to get the mouse location and frustrum line without the arg variable). One should probably just wrap it into a function actually. I also cranked up the mesh sphere count to 1000 and made them more dense to benchmark, this does not appear to affect performance notably on my system:

"""
Get closest mesh under mouse location.
Author: Anders Holden Deleuran
Version: 240706
"""

import Rhino as rc
import GhPython.Assemblies.ExecutingComponent as component

class MyMouseCallback(rc.UI.MouseCallback):
    
    def __init__(self):
        self.move = None
        
    def OnMouseMove(self,arg):
        self.move = arg
        ghenv.Component.ExpireSolution(True)

class MyComponent(component):
    
    def RunScript(self,Meshes):
        
        # Get mouse viewport position
        mpt = rc.RhinoDoc.ActiveDoc.Views.ActiveView.ScreenToClient(rc.UI.MouseCursor.Location)
        
        # Make ray along frustum line at mouse
        r,fl = rc.RhinoDoc.ActiveDoc.Views.ActiveView.ActiveViewport.GetFrustumLine(mpt.X,mpt.Y)
        ray = rc.Geometry.Ray3d(fl.From,fl.Direction)
        
        # Shot ray onto meshes
        hitMeshes = []
        hitParams = []
        for m in Meshes:
            t = rc.Geometry.Intersect.Intersection.MeshRay(m,ray)
            if t >= 0.0:
                hitMeshes.append(m)
                hitParams.append(t)
                
        # Get closest hit mesh and return it
        if hitParams:
            return hitMeshes[hitParams.index(max(hitParams))]
        else:
            return []
            
    def __enter__(self):
        self.mouse = MyMouseCallback()
        self.mouse.Enabled = True
        
    def __exit__(self):
        self.mouse.Enabled = False
        del self.mouse


240706_GetMeshUnderMouse_00.gh (6.7 KB)

Edit: When I went to 5000 spheres with 2500 faces each, the profiler started kicking in. It still doesnā€™t feel too laggy though, quite snappy actually:

2 Likes

Forgot about returning the index. Hereā€™s a nice terse method for dealing with that:


240706_GetMeshUnderMouse_01.gh (9.0 KB)

1 Like

Love it! Thanks @AndersDeleuran , works great on my end as well! Iā€™m about to test on some of the ā€œbloatedā€ models from the public ICON Initiative 99 competition repository here since a lot of these have no garbage management in them.

Also, happy anniversary! :partying_face:

1 Like

Hehe, likewise. Funny coincidence :slightly_smiling_face:

1 Like

Had some fun augmenting the script to allow multi-selection, un-selection, and selection clearing logic:

I think my conditional statements on index handling are just a little bit goofy in certain cases so Iā€™m trying to track down whatā€™s going on there but otherwise itā€™s a pretty fluid interaction I am very happy with so far.

EDIT:

Forgot to attach the code:

"""
Get closest mesh under mouse location.
Author: Anders Holden Deleuran
Version: 240706

Modified By: Michael Vollrath
Features Added: Python 3, Clickable Selection, Multi-Selection, De-Selection, Selection Clearing
"""

import Rhino as rc
import ghpythonlib.component as component  # Changed to this for Python 3
import scriptcontext as sc
from Rhino.UI import ModifierKey

class MyMouseCallback(rc.UI.MouseCallback):
    
    def __init__(self):
        self.move = None
        self.click = None
        self.button = None
        self.modifier = None
        sc.sticky["sel_indices"] = []  # Initialize as an empty list
        
    def OnMouseMove(self, arg):
        self.move = arg
        ghenv.Component.ExpireSolution(True)

    def OnMouseDown(self, e):
        self.button = e.Button
        self.ctrl = e.CtrlKeyDown
        self.shift = e.ShiftKeyDown
        self.click = True
        ghenv.Component.ExpireSolution(True)
    
    def OnMouseUp(self, e):
        self.click = False
        ghenv.Component.ExpireSolution(True)

class MyComponent(component):
    
    def RunScript(self, Meshes: list[object]):
        
        # Get mouse viewport position
        mpt = rc.RhinoDoc.ActiveDoc.Views.ActiveView.ScreenToClient(rc.UI.MouseCursor.Location)
        
        # Make ray along frustum line at mouse
        r, fl = rc.RhinoDoc.ActiveDoc.Views.ActiveView.ActiveViewport.GetFrustumLine(mpt.X, mpt.Y)
        ray = rc.Geometry.Ray3d(fl.From, fl.Direction)
        
        # Shot ray onto meshes
        hits = []
        for i, m in enumerate(Meshes):
            t = rc.Geometry.Intersect.Intersection.MeshRay(m, ray)
            if t >= 0.0:
                hits.append((t, i, m))
                
        # Get closest hit index/mesh and return those
        if hits:
            hits.sort(reverse=True)
            t, i, m = hits[0]
            # rc.RhinoApp.WriteLine("Get Hovered Sticky Value")
            sel_indices = sc.sticky["sel_indices"] if "sel_indices" in sc.sticky else []
            if self.mouse.click and str(self.mouse.button) == "Left":
                if self.mouse.shift:  # Multi-select with Shift key
                    if i not in sel_indices:
                        sel_indices.append(i)
                elif self.mouse.ctrl:  # Remove selection with Ctrl key
                    if i in sel_indices:
                        sel_indices.remove(i)
                else:  # Single select
                    sel_indices = [i]
                sc.sticky["sel_indices"] = sel_indices
                return i, sel_indices

            return i, sel_indices

        elif self.mouse.click and str(self.mouse.button) == "Left":
            # rc.RhinoApp.WriteLine("Clear Sticky")
            sc.sticky["sel_indices"] = []  # Clear Sticky If No Valid Hit On Mouse Click
            sel_indices = sc.sticky["sel_indices"]
            return [], sel_indices
        else:
            # rc.RhinoApp.WriteLine("Get Selected Sticky Value")
            sel_indices = sc.sticky["sel_indices"]
        
            return [], sel_indices
            
    def __enter__(self):
        self.mouse = MyMouseCallback()
        self.mouse.Enabled = True
        
    def __exit__(self):
        self.mouse.Enabled = False
        del self.mouse

2 Likes

Thatā€™s awesome. This has been a very fun AND useful one. And just for completeness, should OP ever come back, hereā€™s a quick go at drawing a rectangular 2D swatch/icon in the viewport foreground with a left mouse clickable state (purple dots in the video are right clicks, yellow are left clicks, using PowerToys):


240707_MouseOverClickableState_00.gh (5.6 KB)

1 Like

Very cool! This will definitely come in handy! Thanks for sharing!

(Also I forgot to add the code for the multi-select, editing the post above with that code now)

Oddly, the code ā€œworksā€ on Mac OS except that there is a big vertical offset in terms of where the ā€œhit locationā€ is and where the mouse cursor is. My guess is itā€™s happening in the RayCasting function or .ScreenToClient

Have you ever come across that? I imagine you mostly use Windows?

Afraid not, Iā€™m indeed mostly/only on Windows.

Just to rule things out, do the GhPython components Iā€™ve posted also not work as expected on Mac? Or is it just the new script editor CPython 3 implementations? Either way, itā€™s maybe something for @CallumSykes or @curtisw to have a look at.

Hmm Iā€™m not sure Iā€™ve only tested the Py3 versions. Good Question, Iā€™ll report back with findings

You are wizards! I love this community and what you share. Thank you so much!!! Amazing discussion in here.

2 Likes

@curtisw

Thanks for the answer! This is something we have been trying to do as well and we ended up using a lot of workaround to ensure we got the right screen point.

This might be a rather dumb question, but here goesā€¦
I tried downloaded the nuget Eto.Platform.Wpf.

When I tried to call this line, it returned with a CS0012 error.

I am not sure why that is because Eto.dll and Eto.Wpf.dll were both referenced in the packagesā€¦ What am I missing?

1 Like