since i was asked by picajol in the octane forum to give some first hand info (since I use both), i will try to summarize some of my recent experiences with both render engines.
I is a bit hard, since it is quite a complex task, both packages are absolutely great imho, they excel in different areas.
First of all it is important to note that both engines come from different directions, although vray4Rhino has been receiving gpu rendering support (as part of the vray RT engine) in recent times, these capabilities are not 100% developed until now imho. So vrays origin is CPU rendering which can be done one one single computer or on a huge network. Octane4Rhino on the other hand is strictly gpu based rendering and until now limited to one machine, so it is by far not as scalable a v4r. At this point I have to say that concerning to Otoy there are current developments to change this and even to enable a cloud based commercial gpu power support.
Vray is a program which is used by many professionals around the world for many years now. It is very reliable and has tons of possibilities to archive every imaginable outcome. It has been used for several hollywood productions. I have been rendering resolutions with this package up to 150 megapixel! It has many ways to optimize render performance by doing e.g. undersampling passes and lets you control every fraction of the render process. While this is great for experienced professionals because it lets you fine tune every aspect of the process to cut down your frametime (especially important when doing animations…) it might become an obstacle for new users. Generally speaking the big range of options is often somewhat frightening to noobs and it takes you some time to find your way around. Vray was originally created for 3D Studio, and some of the 3D Studio features are not fully present in the rhino version, especially when looking at animation features ( incremental IR map passes, motion Blur…)
Octane is a relatively young engine. A great feature is the ability to see changes made to materials an lightning in real time. It offers different rendering pipelines, one is the ambient occlusion pipeline which can deliver very very fast but biased (no diffuse reflections!) visual feedback. So you really can save a whole lot of time when tweaking and experimenting with light and material. This is very nice for experienced users but even more useful for the new ones.
Like vray octane knows proxies, which is important for vegetation, it can use obj. files for that purpose which is nice.
The Octane render process is iterative which means a picture is permanently gaining quality which means it get less and less noisy.
While it is pretty straightforward in octane to create a decent interior picture of a simple room with the AO pipeline within 30 min (depending on your GPUs), there are situations when you can run into serious problems right now, when doing big interiors with many artificial light sources and the “unbiased” engines DL, PCM or pathtracing. Getting “noise free” hires pictures here can be quite time consuming and here the time gained in the “developing phase” of the rendering can often be lost by the render process itself. Another factor is the fact that octane is quite limited, again depending on your GPU, how much textures can be used in the scene. While this is not that much of an issue when you have one or more GTX Titans with 6GB Ram it can easily become a problem when you have e.g. a gtx 590 with 3GB which is further divided by 2 for each GPU resulting in 1,5 BG for texture usage!
Concerning the price it is a bit hard to tell, the initial price for the software is 279 Euro for the standalone version + the plugin which is a bargain imho. Vray is around 700 Euro which is also great for this software.
however to get optimal performance you also need to invest in your hardware, which means CPU power for vray or gpu for octane. At this point it gets quite complicated because you cannot compare both programs in an objective way concerning the given performance / Euro.
Vray can use a grid of computers e.g. in a larger office this is very nice, octane right now is limited to one machine which should be equipped with a few good GPUs and a good power supply and cooling.
That it from my side at the moment when i have time i might come back and tell a bit more about certain aspects of both programs…