Rhino is 64-bit on both Windows and Mac.
Native C++ libraries can work, but they won’t straight from Windows compiled. You’ll have to make an effort in ensuring your code compiles cross-platform. Depending on what you are doing it is either straightforward, or less so.
After that you’ll have to do P/Invoke just like on Windows - it will just be differently set up.
RhinoCycles / Raytraced works like this. The Cycles engine and its wrapper are compiled as a single shared library (DLL on Windows, .so on Mac), and they are accessed through P/Invoke on the Mac just the same.
For instance the camera functions from the shared library are P/Invoked like this: https://github.com/mcneel/CCSycles/blob/rhino-6.x/csycles/CSycles.Camera.cs
For your C/C++ code just take care you write it cross-platform.
And before you ask - yes, the Cycles library is also used in Grasshopper through the GhShaderNodes plug-in.