This is also an interesting question to me, too.
Just want to contribute some algorithms I knew for collision detections.
AABB(Axis Aligned Bounding Box)
SAT(Separated Axis Theorem)
I found this page illustrate it well, so it might helps.
Although in the article it only mention simple collision of simple stuffs.
Such as:
Sphere
Box
According to this QA from StackOverflow:
work
I think the algorithm for more sophisticated meshes collision, might be a in a recursive structure, traversing the faces/ boundingBox in that mesh with kdTree, QuadTree or RTree helps.
These are just some of my ideas, a really interesting question.
I also think that McNeel uses something complex method consisting of multiple procedures including inspection for collision candidates based on AABB, OBB, and some tree structures.
I wish someday McNeel held conferences or sessions for developers.