Rhino Render in v6 is the off-line renderer (also sometimes called production renderer). This is pretty much the same renderer as in v5, with some improvements.
Raytraced viewport mode is indeed using a very different engine called Cycles. The engine is an open source production raytrace engine from Blender 3D. The settings you can find in
Tools > Options | Cycles are for that
Raytraced viewport mode. This gives you an interactive viewport integration of that Cycles engine. If you have a fairly recent GPU (Kepler and newer architectures for Nvidia GPUs, and at least GCN 2nd generation for AMD GPUs) Cycles, and thus
Raytraced can take advantage of that GPU to accellerate the raytracing.
In the future (v7) the Cycles engine will replace the old Rhino Render engine. This isn’t done yet, but will be in the near future. Once that is done the Rhino Render will be exactly the same as the
If you have any specific questions feel free to ask them.