If the ray supplied to IVRay starts on a surface that falls in the list of obstacles, the component registers a hit on that surface. If you are following a ray as it bounces off, or refracts through, a set of surfaces this is unwanted - you want the surface the ray reaches, not the one it is coming from, which you already know. You therefore have to mess with your set of surfaces to exclude the starting surface each time.
It would simplify matters if IVRay excluded the starting surface as a matter of course or had a boolean input to switch such exclusion on.
reading this when you posted it i researched a bit and i am a bit confused, is IVRay now part of grasshopper?, i also happened to have it and was messing around with the Isovist plugin, but i might have had it from a different plugin that was already installed.
and that is part of vanilla grasshopper? i am asking dumb because i checked with 2 independent AI and both claimed it must be part of some different plugin. i am afraid that AI stuff can not be trusted around the corner.. sorry off topic.