Finding the difference between two sets of referenced geometries

That isn’t true. Sets can consist of other data types like points and integers. Posting geometry and/or code is highly recommended.