Okay, after some further testing I can confirm that it works well with .exr files. In our software we are now saving the height maps as .exr files with either 16 or 32 bit depth per channel, depending on the input texture.
The quality of the mesh is now much better. It’s doesn’t look quite as smooth, but at least the accuracy is there now. We are slightly blurring the normal map for that super smooth look:
And in Rhino with the same height and color texture:
Of course speed is still an issue. We calculate everything on the GPU at 60fps. In Rhino it takes a good minute for extremely high quality.
Thanks for looking into this and creating the YT items!