How does the Offset command know on which side of the curve the mouse is located?

You could use the built-in Curve.Contains method to check whether a point is inside, outside or on a curve. Or you could use the winding number approach as described here: