Create curves along on the four edges using CurveThroughPt though the ends of the contours. Smooth and simplify the edge curves as desired for the edges of the final surface. Use EdgeSrf to create a surface using the four edge curves. Then use Patch with the surface previously created as the starting surface for patch with Preserve Edges selected, and the points from the contours as input to Patch.