Hey, I had the same problem but I managed to fix it!
Context: I was having issues with Rhino 6 itself (for other reasons that I won’t go into here), uninstalled it, installed Rhino 5, and then had both Rhino 5 and 6 for a little while (whereupon I was unable to install Octane for Rhino 6, which had previously been working fine). So I uninstalled Rhino 5 and re-attempted the Octane for Rhino 6 installation, and was still not having success.
Then I found this link: https://wiki.mcneel.com/rhino/5/rhi-problems
And running RhiFix.exe solved the problem.
Not sure how to solve the problem without uninstalling Rhino 5 though.
Uninstall Rhino 5, then run RhiFix.exe