Just to make sure… because inputs are a bit confusing about number of points :
1 000 000 Points
// 171ms PointCloud (Color) camera very fluid.
// 4,9s Point3d (no Color) impossible to move camera.
// 123 ms GH_Point (no Color) impossible to move camera but a bit better.
// 5.0s GH_Point (Color) impossible to move camera.
Cheers 
//Adrian