The controls are only enabled when the multiple text annotations that are selected are identical (same text, same rich text formatting). That's because there's no way to select a substring of the text if they are different: represented by (varies). When the text is the same but the rich text formatting is different that's also confusing: the text is displayed w/o the formatting since there's no way to show the different formatting of each selected item. That's why we opted to disable the controls in those cases.
I think what adds to the confusion is that it's not obvious that the controls do not change the style at all but only the rich text string. Ex: if you change the font in the middle of a string the font drop down will change as your cursor moves in and out of that substring.
Trying to have the same set of controls handle both the rich text and the style settings might add to the confusion. For example when a control overrides a style property the control label or border is blue to indicate that.
The top part of the dialog has controls that apply to the style so adding another set of font, bold, italics, underline controls there might be clearer: those new controls change the style whereas the original set just above the text box is for the rich text. The dialog may look to crowded though.
I don't see a way to represent all these different ways to edit text that is not completely intuitive so we appreciate your feedback and will make changes to make it work.