Trim Brep does not work in Rhino6 but it work in Rhino5.
import rhinoscriptsyntax as rs
obj = rs.GetObjects("Select surface or polysurface to trim")
cutters = rs.GetObjects("Select cutting surface or polysurface")
trim = rs.TrimBrep(obj[0],cutters[0])
print trim
Your file tolerance is too large for the size of the objects. If you use normal Rhino Trim or Split, the front solid cylinder won’t trim. If you make the file tolerance 0.001, both the normal Rhino commands as well as the script work.
TrimBrep call the RhinoCommon Brep.Trim method to do the grunt work. The documentation for that gives a better explanation of the scope and limitations of the trim. Maybe that will shed some light? Brep.Trim Method (Brep, Double) (rhino3d.com)
Are you aware the open surface has its normal pointing inwards instead of outwards? Furthermore your closed surface is grouped for no obvious reason. Once the open surface normals point outwards, at least the commands Trim and Split work like expected.