Yes, you can assign macros (commands) to extra mouse buttons on the Mac. See Rhinoceros > Preferences > Mouse > Buttons.
However, in your case, you have changed the behavior of your mouse buttons so they no longer appear to be mouse actions but instead are keyboard actions. So you really want to assign keyboard shortcuts.
To do this:
- Select Rhinoceros > Commands > Customize in the menu.
- Create a new command set if you have not already done so.
- Select the Keyboard Shortcuts tab.
- Find the Group command button by typing in the search field in the lower left part of the window.
- Select the Group button. The Group details show up in the lower right panel. Note that Group already has a keyboard shortcut, and you are about to replace it.
- Click in the Keyboard shortcut field. The current shortcut text is selected.
- Press the mouse button you want to assign to Group. The shortcut is replaced with something like the screenshot below.
- Click Save NOW to save your work and dismiss the dialog. I just discovered that there is a bug in OS X 10.9.1 that will crash Rhino if you try to do much else in this same Commands dialog session. This crash doesn’t happen in either 10.8.5 or 10.7.5.
To change or add another keyboard shortcut, start again at step 1.