it looks like this is not yet fully exposed in plain rhino-python. In RhinoScript you can return and change the material index of a layer using:
Call Rhino.LayerMaterialIndex(strLayerName, intMaterialIndex)
which assigns the V-Ray material if it has been loaded before, while in Python i´ve found only the method to return the material index of a layer:
intIndex = rs.LayerMaterialIndex(strLayerName)
Speaking of the RhinoScript example, the preloaded V-Ray material gets assigned to the layer and it even renders regardless of the layers material source. (Which is not set to “plugin” yet). To set it to plugin i´ve used this in RhinoScript as a Rhino.Command after setting the layer`s material index:
_-Layer _Material "LayerName" _Plugin "V-Ray for Rhino" _EnterEnd
The result is identical as using the UI in Rhino to assign a V-Ray material to a layer. So if one clicks in the Layer dialog on the layer’s material name, the proper dialog offering the “browse”, “edit” and “match” buttons comes up having the “plugin” toggle checked.
@pfahy are you aware that V-Ray can be accessed as object via scripting too ?