Antialiasing in Rhino on a ATI Radeon is a know problem. ATI apparently doesn’t want to update their drivers for these cards to support OpenGL calls correctly. The ATI FirePro is geared towards OpenGL but also has problems with antialiasing.
Further reading (one among many such posts):
http://discourse.mcneel.com/t/please-help-choosing-video-card-for-rhino-3d-v5/26862