This bothers me too! The problem is the layer visibility attribute which updates everything. Super annoying on heavy definitions.
One thing that helped me is to be more particular with layer filtering.
@Czaja did you see the other topic:
@Japhy mentioned other possibilities for filters.