@runnie I’m not sure about win8, but here is what I would do to determine the amount of memory your system is using. Open Windows Resource Monitor in the memory tab and shut all the programs you won’t be using during a render and Rhino. Look at the memory bar (the green part of it) and see how much is being used. Add a small margin and I believe you’ll be safe.
Note that if you set the default geometry to static, vray will use up all the memory it can regardless of what you set the dynamic limit to. The dynamic memory limit is only important for auto our dynamic default geometries.