Short answer: Rhino for Mac supports only the SpaceNavigator, not other 3DConnexion devices. This is stated in the System Requirements section of the Rhino for Mac home page, Rhino for Mac wiki page and also in Frequently Asked Questions reachable from the Rhino for Mac Help menu. To be exact, these notes do not explicitly name the SpaceMouse (perhaps this is a new device), but the notes do point out only the SpaceNavigator is supported.
Longer answer: When 3DConnexion started introducing devices after the SpaceNavigator, they decided to take over the device interface development in each 3D CAD program. 3DConnexion now writes the interface software for Rhino for Windows and other CAD programs for all their devices. Rhino for Mac is not a released product and does not have a published SDK that companies like 3DConnexion can use to write device plug-ins. 3DConnexion does not publish enough information for us to support their newer devices, and we do not publish enough information for Rhino for Mac for 3DConnexion to take over the device interface development, so only the SpaceNavigator can be supported in Rhino for Mac at this time.
As far as the number of buttons supported, in the past some enterprising 3DConnexion device owners found that they could fiddle with with System Preferences for their device and get more buttons working. I don’t know what they did. Perhaps that unintentional capability has been removed by 3DConnexion and no longer works.