What is the best system for a windows PC if you want to have real fast renders with Rhino in Raytraced mode?
Rendering is done via GPU and the most efficient way is to do it via the raytracing engine called OPTIX from nvidia. To use this you would need a GPU from nvidia, the latest series is the 5000 series. Of those, the fastest ist the 5090, though those are hard to get and if they are very overpriced at the moment. the next step down in terms of performance would be the 5080 - this isn’t as overpriced and more broadly available.
Sure you would want to pare the expensive GPU with a good CPU, RAM and Motherboard but those don’t really make the rendertime go down, that’s really only dependant on the GPU, as mentioned above.
AMD currently offers the fastest CPUs (for most workloads) and the AM5 plattform has a long lifetime ahead with two more generations you can upgrade to in the future (new CPU generation roughly every two years). The next Zen6 CPUs coming early next year are looking to deliver a pretty big IPC gain, compared to usual generational gains.
RAM you only need a lot if your models are big. Big for example because of huge and complex meshes, and or a lot of high res textures. If that is true then you would also want the GPU to have more VRAM. In my experience, even big and complex scenes are fine with 32GB main memory and 16GB VRAM. If that is enough depends entirely on the files you are working with.
The render engine that Rhino Cycles is using comes from blender. By looking at the blender benchmarks you can compare GPU performance numbers. Though I don’t know what blender/cycles version is in the current Rhino 8 or the current Rhino 9 Beta. Would be great to know.
You can also use AMD GPUs for rendering. Their equivalent to OPTIX is HIP-RT, though this isn’t yet mature and it isn’t even available in the blender benchmark, also not in the version that Rhino uses, afaik.