You could maybe check for self intersection and evaluate the start and end intersect parameters…

import rhinoscriptsyntax as rs
curve = rs.GetObject("Get Curve", rs.filter.curve)
# if only one curve is supplied as an argument, it checks for self intersections...
insec=rs.CurveCurveIntersection(curve)
if insec and len(insec)==2:
dom_start=min(insec[0][5],insec[1][5])
dom_end=max(insec[0][5],insec[1][5])
print "Olap domain start={} | Olap domain end={}".format(dom_start,dom_end)
#check
rs.AddPoint(rs.EvaluateCurve(curve,dom_start))
rs.AddPoint(rs.EvaluateCurve(curve,dom_end))
else:
print "Curve does not self-intersect"

Is that like a 2D CAM toolpath with an approach path and a radius entry/exit?

I can see the problem using the typical fillet tools, which require getting the trim parameters. I wonder if this - which doesn’t depend on parameters - might work better:

As a radical solution could be to split the rectangle away from the overlap area - say at the upper right and lower right corners - then you will have two separate curves. Do your filleting on each and join all back together afterward.

At this point I solved with a radical solution, exploded the curve, making fillet and then putting everything join together, a little more work but it works.