When I increase the number of possible rotations for nesting what I expect is that the number of nested items and the total area of them should increase or at least stay the same as with the case of 4. However, if I increase it, fewer elements are getting nested. Why’s it so?

The rotations would rotate element incrementally by degree of 360/Rotations.
It depends, sometimes the orthogonal rotations can be nested better if your geometry is rectangular. Also if you need good nest it would take many iterations to check every position.

Run - runs the solver multiple times, and output the best solution by defined number of iterations. By default it runs only once. For running solver do not forget to make Interations number to 0, elsewise you will get the same result every iteration.