Custom Rhino View Capture for ControlNet / Stable Diffusion Rendering

Grasshopper
,
1

Hi everyone,

I want to develop a workflow where I use ControlNet with Stable Diffusion to generate AI-enhanced renders that match the geometry of my GH scene. The idea is to train or guide the AI using structured views captured from Rhino that include o.a.:

Object edges - Normals - Z-depth - Custom color-per-object passes - etc

Basically, these captured viewports act, in unisson / parallel, like input “control images” that inform the AI model how to conform to the actual scene geometry.

To generate these views, I currently rely on:

  • A custom display mode, tweaked from “Monochrome,” to render Canny Edge with no shade and one for Ambient Occlusion (tweaked Artic view).
  • Some Material effects on the geometry to get Silhouette Mask effect to get Alpha mask render on the scene geometry, and some toggle to display grid and horizon (ground plane) to capture the camera orientation relative to XY plane
  • Use of Rhino commands like ShowZBuffer and _TestShowNormalMap, and using **C# scripted components to capture these.
  • Before I used GLSL shaders with same visual effects and a node to capture the generated view

But sharing this workflow others is cumbersome:

  • Users must install GLSL shader plugin (which i think is no longer maintained and i am not sure if R9 or GH2 will natively support shaders @stevebaer )
  • Or manually install custom .ini display mode file.
  • The workflow depends on internal or undocumented Rhino commands (_TestShowNormalMap) .

What I’m Looking For

  • Has anyone built a repeatable, shareable pipeline for this kind of view capture ? ideally just a Grasshopper definition with native C# nodes, sliders, / view capture — that produces ControlNet-ready render passes.

Would love to hear how others are doing this ( @Ambrosinus How do you do it in your Toolkit ?) — or if McNeel has plans to expose more of these visualization features natively (especially with the rise of ML-based visualization workflows).

Thanks in advance!

Edit:
Writing this I am thinking instead of trying to wrestle Gh into producing these views using internal workarounds and unsupported commands — I could maybe export my geometry to Three.js and generate those views there, as they provides well-documented, GPU-accelerated tools for: Depth rendering, Normal passes, Object-wise color encoding, or Wireframe or Edge display. — just clean, standard WebGL code. Plus, I get full control over render passes, camera setup, and resolution. Maybe this is way too optimitic think, any way would love to hear about other pipeline.