I am developing a C++ Rhino plugin, what is the best way to check if a point at (x1,y1,z1) is inside/on_edge/outside a solid? and if two points (x1,y1,z1) and (x2,y2,z2) are both inside the solid, how can I check if the line connecting them does not go outside the solid?
The first thing to do is to get the bounding box of the Brep using ON_Brep::BoundingBox() and test to see if the point falls inside of it or not using ON_BoundingBox::IsPointIn(). This way you can quickly eliminate some points.
You can use ON_Brep::IsPointIn() to see if a point falls inside of a closed Brep. See opennurbs_brep.h for details.
You can use RhinoBrepClosestPoint() to see if the point is on an edge, or a face. See rhinoSdkUtilities.h for details.
To see if a line is completely inside of a closed Brep, create an ON_LineCurve from the points and then use RhinoCurveBrepIntersect() to see if they intersect or not. See rhinoSdkUtilities.h for details.