My recollection, perhaps incorrect, is this error can occur with a previous release of Rhino 5. Which release of Rhino 5 did you install. If it is not the current one then you might try removing your current installation, download and install the latest release from http://www.rhino3d.com/download/rhino/5.0/latest
The problem started when you attempted to run the installer, stopped it for some reason, then the Windows Compatibility Assistant asked if it installed properly or not. You answered No so Windows tagged the executable as requiring Windows XP SP2.
The next time you ran the Rhino installer, Windows put you in WinXP SP2 mode. This. Makes the Rhino installer itself report you’re running in an unsupported OS.
Windows led you into a dark alley.
That support page describes how to use Windows to remove the compatibility tagging from the Rhino installer. Then it will run.
NEVER tell Windows something went wrong EVER. It will always screw up something else, and you’ll blame us.
We can’t fix Micrisoft’s heavy handed “helping”.